How to install ELK stack or Elastic stack (Elasticsearch, Logstash, Kibana) in windows

Introduction

For sometimes now, I have been thinking of writing a blog on installing ELK stack or Elastic stack (Elasticsearch, Logstash, Kibana) in windows system. In live scenarios, you would mostly use Linux or container based solutions. However, most of us use Windows laptop for our learning purposes. Hence, this tutorial focuses on installing the ELK stack in windows. Install ELK in your windows laptop and learn!

What is ELK stack or Elastic stack (Elasticsearch, Logstash, Kibana)?

There are three different popular open source products: Elasticsearch, Logstash and Kibana. The acronym ELK stack is used to describe a technology stack comprising these three products together. You must be thinking the reason behind clubbing all these three open source projects together? Well, the answer is that these products finely gels together and give us a fine solution for log management, log analytics, monitoring (and many more functionalities) in an enterprise.
In the Elastic stack, there is another key product called Beats. There different types of Beats agents. Beats agents are configured in the servers, containers; they collect different type of data (as required & configured) and forward to ELK stack for further processing.
Let’s take a look at the brief functionalities of different components in Elastic stack.

Beats: Beats are lightweight log shipper/forwarder agents. These agents are installed on the servers and configured to read the file content and forward (in this case to logstash).

Logstash: Logstash is open source tool that collects data, parses transforms it and send across to target (in this case target is Elasticsearch).

Elasticsearch: Elasticsearch is schemaless, RESTful databse built on Apache Lucene.

Kibana: An open source visualization tool that makes it easy to search the data stored in Elasticsearch database and also provides easy tools to create visualizations, dashboard etc and many more features.

System Prerequisite

In this article we will see how to install the ELK stack in Windows 10 Pro. Please ensure that you have latest JAVA version installed in your system. At the time of writing this article, I have following version installed in my system.

C:\Users\Sadruddin>java -version
java version “14.0.2” 2020-07-14
Java(TM) SE Runtime Environment (build 14.0.2+12-46)
Java HotSpot(TM) 64-Bit Server VM (build 14.0.2+12-46, mixed mode, sharing)

ELK versions

I have taken the latest versions of Elastic stack while writing this article.

• Elasticsearch v 7.8.1
• Logstash
• Kibana
• Beats

You can download all these products from Elasticsearch’s official product download page https://www.elastic.co/downloads/ .

I’ll cover the detailed steps for installing each of the components.

Install Elasticsearch in Windows

• Head to https://www.elastic.co/downloads/elasticsearch
• Click on Windows link (highlighted in below image) and download the .zip file

• Create a folder with following structure: C:\elk\elasticsearch. Although you can create any folder structure, I am using this folder structure for ease of usage.
• Once downloaded, extract the content to a the folder created above. After extracting, it looks like follows:

• Now, edit the System Environment PATH variable to add the directory path: C:\elk\elasticsearch\bin

• Now, open a command prompt and run following command to run Elasticsearch

elasticsearch.bat -Ecluster.name=mycluster1 -Enode.name=mynode1

Command to start Elasticsearch

• In few seconds, Elasticsearch instance will start.
• You will see the ip and port number in the last few lines of the log in command prompt

• To verify that the Elasticsearch instance is actually running, open browser and invoke http://localhost:9200/. It will show an output similar to below snippet. This address is crucial for configuring the logstash and kibana component. Hence, keep a note of it.

• Great!! So, we have successfully installed Elasticsearch in our windows system.

Install Logstash in Windows

• Head to https://www.elastic.co/downloads/logstash
• Click on the ZIP link (yellow highlighted below) and download the zip file.

• Create a folder with name logstash inside C:\elk\.
• Extract the content of the zip file to the folder created above. After extracting, it looks like follows:

• Now, edit the System Environment PATH variable to add the directory path: C:\elk\logstash\bin
• Open the c:\elk\logstash\conf\logstash-sample.conf file and ensure the Elasticsearch host address is correct. It should be the same address we received after running Elasticsearch [pls check the Elasticsearch installation steps above]
• Now, open a command prompt and run following command to run logstash

logstash.bat -f c:\elk\logstash\config\logstash-sample.conf

Command to start logstash

• In few seconds, logstash instance will start.
• You will see a statement like this “Successfully started Logstash API endpoint {:port=>9600}“. Please check the below screen-grab for reference

• To verify that the logstash instance is actually running, open browser and invoke http://localhost:9600/. It will show an output similar to below snippet.

• Great!! So, we have successfully installed Logstash in our windows system as well.

Install Kibana in Windows

• Head to https://www.elastic.co/downloads/kibana
• Click on the WINDOWS link (yellow highlighted below) and download the zip file.

• Create a folder with name kibana inside C:\elk\.
• Extract the content of the zip file to the folder created above. After extracting, it looks like follows:

• Now, edit the System Environment PATH variable to add the directory path: C:\elk\kibana\bin
• Open the c:\elk\kibana\config\kibana.yml file and un-comment elasticsearch.hosts line by removing the #. Also ensure the Elasticsearch host address is correct. It should be the same address we received after running Elasticsearch [pls check the Elasticsearch installation steps above]
• Now, open a command prompt and run following command to run following command

kibana.bat

command to run kibana

• In a few moment, Kibana instance will start
• You will see statement like this: . Please check the below screen grab for reference.

• To verify that the kibanainstance is actually running, open browser and invoke http://localhost:5601/. It will open up kibana portal like below.

• You can click on Explore on my own button. If you would like to try with sample data, you can click on Try our sample data button.
• Great!! So, with this we have successfully installed kibana in our windows system as well.

Install Filebeat in Windows

Well, so far we have installed E L and K of Elastic stack. Now, we will concentrate on data shipper called beat. There are many types of beat agents for different purposes. In this tutorial, we will install Filebeat. Let’s take a look at what is Filebeat?

Filebeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on your servers, Filebeat monitors the log files or locations that you specify, collects log events, and forwards them either to Elasticsearch or Logstash for indexing.

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html#:~:text=Filebeat%20is%20a%20lightweight%20shipper,Elasticsearch%20or%20Logstash%20for%20indexing.&text=Filebeat%20is%20an%20Elastic%20Beat.

• Head to https://www.elastic.co/downloads/beats/filebeat
• Click on the WINDOWS ZIP 64-BIT link (yellow highlighted below) and download the zip file.

• Create a folder with name filebeat inside C:\elk\.
• Extract the content of the zip file to the folder created above. After extracting, it looks like follows:

• Open a PowerShell terminal as Administrator
• Execute following command to traverse the filebeat directory

cd C:\elk\filebeat\

• Thereafter, execute following command to install filebeat service in the system

PowerShell.exe -ExecutionPolicy UnRestricted -File .\install-service-filebeat.ps1

• The output will look something like below:

• With this filebeat is installed as windows service. You can check this service in the Windows services as well.
• Now, we will make some modification to c:\elk\filebeat\filebeat.yml file. Open this file in any text editor and make a change in filbeat.inputs section; comment out the linux directory path and change the windows directory path to the desired location. This is the directory location from where filbeat agent will read the log files and push to logstash. In my case, I have made following configuration

• Next, we will uncomment the entries in output.logstash. Please refer to the below snippet for reference:

• Please note that, you must provide the right IP and PORT of logstash instance. In my case, as you have noticed, it is 5044 port number.
• At this point, you can run following command in the PowerShell to test the configuration

.\filebeat.exe -e test config

Reference: https://stackoverflow.com/questions/41751605/running-filebeat-in-windows

• You can also run the following command to run the Filebeat in foreground and make sure everything is working correctly and will also continuously display the logs. You can use Ctrl+C to exit this process.

.\filebeat.exe -c filebeat.yml -e -d “*”

Reference: https://stackoverflow.com/questions/41751605/running-filebeat-in-windows

• To start the Windows service that we installed in above steps, you can either run following command or start the service by going to services (services.msc from windows +r) window.

start-service filebeat

Reference: https://stackoverflow.com/questions/41751605/running-filebeat-in-windows

• To stop the Windows service that we started in above steps, you can either run following command or stop the service by going to services (services.msc from windows +r) window.

stop-service filebeat

Reference: https://stackoverflow.com/questions/41751605/running-filebeat-in-windows

• With this, we have completed the filebeat configuration in our system as well.

Next Step

Well, I will conclude this article right here as we have completed all our installation and minor configurations. In the next tutorial, I’ll take a sample log data and show the data getting loaded in the Elasticsearch. We will see the same data using Kibana portal.