Author: Venkata Raja Guntupalli
Introduction
Have you ever thought about how we can leverage a single platform for Observability, Search & Security… Yes, I am gonna talk about Elastic stack and how we can use it to implement all the above solutions practically in a series of blogs.
Elastic Stack (Formerly called ELK Stack) consists of Elasticsearch, Kibana, Logstash & Beats.
ElasticSearch: Elasticsearch is a distributed document-based DB, full-text search & analysis engine based on Apache Lucene.
Kibana: Kibana is an analytics and visualization platform.
Logstash: Logstash can act as a data aggregator that can collect & enrich data from different sources.
Beats: Beats are single-purpose data shippers
In this blog, I will be explaining how to create an Elastic Stack deployment in Elastic cloud.
Creating the Deployment
Sign in/Sign up:
Click on the Elastic Cloud Trial Link.
This link will give you access to a 30 days trial period for Elastic Cloud. A credit card is not required, and the trial will expire on its own.
Click on Start free trial to get the below screen.
You can use Google or Microsoft account to sign up, or you can use email to sign up. If you already have access, click on LogIn on the top right, provide details, and start your free trial.
Configuring Cluster:
After providing all details, you will get the below screen.
You can provide a Name to your deployment; you can select more advanced details after clicking on Edit settings.
1. You can select any cloud provider like AWS, GCP, or AZURE as per your requirements.
2. Select a Region that is nearest to your data source to avoid latency.
3. Select a hardware profile based on your requirement.
Elastic offers several deployment templates for different use cases and workloads.
Storage optimized profile: Your Elasticsearch data nodes are optimized for high I/O throughput. Use this profile if you are new to Elasticsearch or don’t need to run a more specialized workload.
Ideal use case: Good for most ingestion use cases with 7-10 days of data available for fast access. Also good for light search use cases without heavy indexing or CPU needs.
Storage optimized (dense) profile: Your Elasticsearch data nodes are optimized for high I/O throughput.
Ideal use case: Ideal for ingestion use cases with more than 10 days of data available for fast access. Also, good for light search use cases with very large data sets.
CPU-optimized profile: This profile runs CPU-intensive workloads faster.
Ideal use case: Consider this configuration for ingestion use cases with 1-4 days of data available for fast access and for search use cases with indexing and query workloads. Provides the most CPU resources per unit of RAM.
CPU optimized (ARM) profile (Only for AWS cloud): This profile is similar to CPU optimized profile but is powered by AWS Graviton2 instances.
Ideal use case: Consider this configuration for ingestion use cases with 1-4 days of data available for fast access and for search use cases with indexing and query workloads. Provides the most CPU resources per unit of RAM.
General purpose profile: This profile runs CPU-intensive workloads faster
Ideal use case: Suitable for ingestion use cases with 5-7 days of data available for fast access. Also good for search workloads with less-frequent indexing and medium to high querying loads. Provides a balance of storage, memory, and CPU.
General purpose (ARM) profile (Only for AWS cloud): This profile is similar to the General purpose profile but is powered by AWS Graviton2 instances.
Ideal use case: Suitable for ingestion use cases with 5-7 days of data available for fast access. Also good for search workloads with less-frequent indexing and medium to high querying loads. Provides a balance of storage, memory, and CPU.
4. Select the version of your Elastic stack from the recent 3 major available versions.
5. Click on Create Deployment to create a deployment.
Deployment will be created, and you can see the below screen in Kibana.
You can add integrating data right away by clicking on add integrations; for this blog, we are selecting Explore on my own to get the below home screen.
Managing Deployment
Click on the Menu icon in Kibana and select Manage this Deployment.
You can see all the details like name, version, hardware profile selected, endpoint URLs of all Elastic components, etc. And at the left side, you can see all the actions that you can perform in this deployment.
Reset deployment password:
Click on Security and select Reset Password to generate a new password for the Elastic user.
After confirming the reset, you can see the password and able to download it too.
Editing the Deployment:
Click on Edit to get the below screen.
You can perform the below actions on this screen.
1. Here, you can add/remove/modify node configuration very easily.
2. You can add other data tiers like warm/cold/frozen.
3. Click on Manage/add user settings to customize the component's yml files.
Checking performance of Deployment:
Click on Performance to check some cool visualizations about your deployment performance, like CPU Usage, No.of requests, etc.
Stack Monitoring:
Enabling Stack monitoring in Elastic Cloud was a very simple process.
1. Click on Logs and Metrics and click on Enable option as shown below.
2. You can create a dedicated monitoring cluster on the fly, or you can select a dedicated monitoring cluster if you have already created it. Or you can send logs & metrics to the same deployment by selecting that in the search for deployment dropdown. Having dedicated monitoring is always preferred.
Click on save after selecting, and you can see the data in kibana.
Navigating to Kibana:
Click on the deployment name to get the below options to open kibana.
Conclusion
We have seen how to create an Elastic stack deployment using Elastic Cloud & how to edit it to upscale/downscale, how to check its performance, and how to enable stack monitoring in a matter of clicks.