https://miro.medium.com/max/875/1*779CspzO_Tgeyywv5z8nag.jpeg

QuickELK

Quick Code to deploy ELK stack (Elasticsearch, Logstash, Kibana) and supporting services on Kubernetes Controlled Docker platform.

Click Here to access the code.

Prerequisite

• Kubernetes Cluster with a minimum of 3 workers, to host 3 Elasticsearch nodes.
• Docker-Based Container Runtime Environment.
• Helm Installed and Configured to access Kubernetes Cluster Master for deployments.
• Ansible Installed and Configured to run the playbook.
• Sufficient resource over Kubernetes worker for the entire deployment.
• External Load Balancer(Eg. Metal LB) needs to be configured over Kubernetes, to access Kubernetes GUI from LoadBalancer Service.

Consideration

Docker images hosted at official “docker.elastic.co” are valid and used for ELK deployments. Below are the docker repo:

• ElasticSearch: docker.elastic.co/elasticsearch/elasticsearch
• Kibana: docker.elastic.co/kibana/kibana
• Filebeat: docker.elastic.co/beats/filebeat
• Logstash: docker.elastic.co/logstash/logstash
• Metricbeat: docker.elastic.co/beats/metricbeat

Tools and Technologies Used

• Kubernetes: Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications.
• Docker: Docker is a set of the platform as service products that use OS-level virtualization to deliver software in packages called containers.
• Ansible: Ansible is an open-source software provisioning, configuration management, and application-deployment tool enabling infrastructure as code.
• Helm: Helm helps you manage Kubernetes applications — Helm Charts helps you define, install, and upgrade even the most complex Kubernetes application.
• Elasticsearch: Elasticsearch provides near real-time search and analytics for all types of data. Whether you have structured or unstructured text, numerical data, or geospatial data, Elasticsearch can efficiently store and index it in a way that supports fast searches.
• Logstash: Logstash is a free and open server-side data processing pipeline that ingests data from a multitude of sources, transforms it, and then sends it to Elasticsearch.
• Kibana: Kibana is a free and open user interface that lets you visualize your Elasticsearch data and navigate the Elastic Stack.
• Filebeat: Lightweight shipper for logs Whether you’re collecting from security devices, cloud, containers, hosts, or OT, Filebeat helps you keep the simple things simple by offering a lightweight way to forward and centralize logs and files.
• Metricbeat: Lightweight shipper for metrics Collect metrics from your systems and services. From CPU to memory, Redis to NGINX, and much more, Metricbeat is a lightweight way to send system and service statistics.
• Github: GitHub, Inc. is a provider of Internet hosting for software development and version control using Git. It offers the distributed version control and source code management functionality of Git, plus its own features.

Installation and Usage

Target Architecture Diagram

https://miro.medium.com/max/533/1*dfda68mO9NPtMg4CensGXw.png

Installing via Ansible

Clone the “QuickELK” project, ensure that all prerequisites are in place. Select a Kubernetes namespace where this stack is required to be created. The namespace will be automatically created if not exist. Select a helm deployment name for the stack. Run the below command from inside the clone directory.

# ansible-playbook main.yaml -e namespace=<Namespace for Deployment> -e stackname=<Name of Stack> -e chartpath=elk-stack

Example:
# ansible-playbook main.yaml -e namespace=ezsinam1 -e stackname=mystack -e chartpath=elk-stack
Output: 
 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match ‘all’
PLAY [localhost] *****************************************************************************************************************************************************
TASK [Check the Namespace Exist.] ************************************************************************************************************************************
fatal: [localhost]: FAILED! => {“changed”: true, “cmd”: “kubectl get ns ezsinam1”, “delta”: “0:00:00.136403”, “end”: “2021–10–27 14:11:27.742760”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2021–10–27 14:11:27.606357”, “stderr”: “Error from server (NotFound): namespaces \”ezsinam1\” not found”, “stderr_lines”: [“Error from server (NotFound): namespaces \”ezsinam1\” not found”], “stdout”: “”, “stdout_lines”: []}
TASK [Create Namespace if NOT exists.] *******************************************************************************************************************************
changed: [localhost]
TASK [debug] *********************************************************************************************************************************************************
ok: [localhost] => {
 “ns.stdout_lines”: [
 “namespace/ezsinam1 created”
 ]
}
TASK [Deploy the ELK Stack.] *****************************************************************************************************************************************
changed: [localhost]
TASK [debug] *********************************************************************************************************************************************************
ok: [localhost] => {
 “chart.stdout_lines”: [
 “NAME: mystack”,
 “LAST DEPLOYED: Wed Oct 27 14:11:28 2021”,
 “NAMESPACE: ezsinam1”,
 “STATUS: deployed”,
 “REVISION: 1”,
 “NOTES:”,
 “1. Get the application URL by running these commands:”
 ]
}
PLAY RECAP ***********************************************************************************************************************************************************
localhost : ok=4 changed=2 unreachable=0 failed=1
0

Installing via Helm
Clone the “QuickELK” project, ensure that all Prerequisites are in place. Select a Kubernetes namespace where this stack is required to be created. The namespace will be automatically created if not exist. Select a helm deployment name for the stack. Run the below command from inside the clone directory.

# helm install <Stack Name> elk-stack — namespace <Namespace for Deployment> — create-namespace

Installing Specific ELK stack Release via Helm/Ansible
Update file “ELK/elk-stack/Chart.yaml”, the value of key “appVersion” to the specific version of ELK stack. Presently we have configured the entire stack installation from version “6.8.20”.

apiVersion: v2
appVersion: 6.8.20 # Update Specific version of ELK stack, Automatically all components will be installed with selected version.
…
…

Installation Validation via Kubectl
Run the below command for deployment validation, provide the namespace provided in the previous step.

# kubectl get all -n <Namespace for Deployment>
Example:
# kubectl get all -n ezsinam1
Output:
NAME READY STATUS RESTARTS AGE
pod/elasticsearch-logging-0 1/1 Running 0 4m7s
pod/elasticsearch-logging-1 1/1 Running 0 4m7s
pod/elasticsearch-logging-2 1/1 Running 0 4m7s
pod/filebeat-4kc6r 1/1 Running 0 4m7s
pod/filebeat-5b9pz 1/1 Running 0 4m7s
pod/filebeat-cxwpj 1/1 Running 0 4m7s
pod/filebeat-fqth7 1/1 Running 0 4m7s
pod/filebeat-mgtts 1/1 Running 0 4m7s
pod/filebeat-rdgx8 1/1 Running 0 4m7s
pod/filebeat-rgxmc 1/1 Running 0 4m7s
pod/kibana-logging-58db4c758f-gbb6k 1/1 Running 2 4m7s
pod/logstash-deployment-5b4879b9f7–4zwnr 1/1 Running 0 4m7s
pod/metricbeat-6kp4k 1/1 Running 0 4m7s
pod/metricbeat-dqf7c 1/1 Running 0 4m7s
pod/metricbeat-gj78v 1/1 Running 0 4m7s
pod/metricbeat-kgbfg 1/1 Running 0 4m7s
pod/metricbeat-ngb9q 1/1 Running 0 4m7s
pod/metricbeat-tw7zb 1/1 Running 0 4m7s
pod/metricbeat-x4tv4 1/1 Running 0 4m7s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/elasticsearch-logging ClusterIP 10.108.33.4 <none> 9200/TCP 4m7s
service/kibana-logging LoadBalancer 10.108.8.123 10.64.109.113 5601:30316/TCP 4m7s
service/logstash-service ClusterIP 10.105.7.173 <none> 5044/TCP 4m7s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/filebeat 7 7 7 7 7 <none> 4m7s
daemonset.apps/metricbeat 7 7 7 7 7 <none> 4m7s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/kibana-logging 1/1 1 1 4m7s
deployment.apps/logstash-deployment 1/1 1 1 4m7s
NAME DESIRED CURRENT READY AGE
replicaset.apps/kibana-logging-58db4c758f 1 1 1 4m7s
replicaset.apps/logstash-deployment-5b4879b9f7 1 1 1 4m7s
NAME READY AGE
statefulset.apps/elasticsearch-logging 3/3 4m7s

Integration Validation
To validate indices for ElasticSearch to validate Integration with filebeat, logstash, and metricbeat.

# curl <elasticsearch-logging clsuetrIP Service IP>:9200/_cat/indices
Example:

# curl 10.108.33.4:9200/_cat/indices
yellow open logstash-2021.07.08 sxXdBTWZSt6dIaDB4s_QLQ 5 1 218 0 388.2kb 388.2kb
yellow open logstash-2021.08.05 H37edbWUT4enUQfF — -t_A 5 1 6 0 62.7kb 62.7kb
yellow open logstash-2021.09.05 vuQnUkGMTLWy85c9LSr-SA 5 1 681 0 458.8kb 458.8kb
yellow open logstash-2021.07.05 Im1jancJQlWfwUiC8h0vKw 5 1 717 0 773.6kb 773.6kb
yellow open logstash-2021.07.02 b9AFqpcTTkW5nmqoqaCsig 5 1 27535 0 6.6mb 6.6mb
yellow open metricbeat-6.8.20–2021.10.27 iiVkm-lMQwKT6ElVwpukGw 5 1 17020 0 12.9mb 12.9mb
…
…

MetricBeat and LogStash indices should be visible. Also, you can select an index and check data using the below command.

Example:
# curl 10.108.33.4:9200/logstash-2021.07.08/_search

Installation Cleanup
To delete the entire deployment provide Kubernetes namespace where this stack is created. The namespace will be automatically deleted. Also, provide the helm deployment name of the created stack.

Run below command.

# ansible-playbook clean.yaml -e namespace=<Namespace of Deployment> -e stackname=<Name of Deployed Stack>

Example:
# ansible-playbook clean.yaml -e namespace=ezsinam1 -e stackname=mystack
Output:
 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match ‘all’
PLAY [localhost] *****************************************************************************************************************************************************
TASK [Remove the ELK Stack.] *****************************************************************************************************************************************
changed: [localhost]
TASK [debug] *********************************************************************************************************************************************************
ok: [localhost] => {
 “chart.stdout_lines”: [
 “release \”mystack\” uninstalled”
 ]
}
TASK [Delete the Namespace.] *****************************************************************************************************************************************
ansible-playbook clean.yaml -e namespace=ezsinam1 -e stackname=mystackchanged: [localhost]
TASK [debug] *********************************************************************************************************************************************************
ok: [localhost] => {
 “ns.stdout_lines”: [
 “namespace \”ezsinam1\” deleted”
 ]
}
PLAY RECAP ***********************************************************************************************************************************************************
localhost : ok=4 changed=2 unreachable=0 failed=0

Backup Strategy

• Elasticsearch: Snapshot is a backup taken from a running Elasticsearch cluster. You can take snapshots of an entire cluster, including all its data streams and indices. You can also take snapshots of only specific data streams or indices in the cluster. You must register a snapshot repository before you can create snapshots. Snapshots can be stored in either local or remote repositories. Remote repositories can reside on Amazon S3, HDFS, Microsoft Azure, Google Cloud Storage, and other platforms supported by a repository plugin.
• Kibana: Dashboard and configuration are important for Kibana, Dashboards could be exported and source control managed. Configuration could be backed up externally on PV or NFS. The rest of the instances could be treated as Cow.
• Logstash: Logstash will act as a middle layer between the Logging source and Elastic Search, no specific backup measures are required. Keep the backup of its configuration instead. Could be treated as a Cow.
• Metricbeat: Metricbeat will act as a middle layer in between the Metric source and Elastic Search, no specific backup measures are required. Keep the backup of its configuration instead. Could be treated as a Cow.
• Filebeat: Filebeat will act as a middle layer in between the Logging source and Elastic Search, no specific backup measures are required. Keep the backup of its configuration instead. Could be treated as a Cow.

Deployment Constraints

• This is a POC, low resources are assigned to Kubernetes objects created under this activity. With real-time production setup resources allocated to Kubernetes objects needed to be increased for better performance.
• No PVC is used in this deployment. We may need to attach PVC to improve redundancy and robustness.
• Readiness and Liveliness are also not integrated, may be considered for future rollout in production.
• Filebeat/MetricBeat could be attached to any application container as a side-car to forward logs/metrics to Elastic Search. Not covered in this POC.
• This is a quick study, take it as a reference for your basic understanding before you productize.

Join FAUN: Website 💻|Podcast 🎙️|Twitter 🐦|Facebook 👥|Instagram 📷|Facebook Group 🗣️|Linkedin Group 💬| Slack 📱|Cloud Native News 📰|More.

If this post was helpful, please click the clap 👏 button below a few times to show your support for the author 👇