Introduction
My previous tutorial was on Apache Kafka Installation on Linux. I used Linux operating system (on VirtualBox) hosted in my Windows 10 HOME machine. Sometimes, it may seem a little complicated because of the VirtualBox setup and related activities. Hence, I thought of writing this tutorial to run Kafka purely on windows. For this purpose, I had to upgrade my Windows 10 HOME operating system to Windows 10 Pro.
Tech Stack
Here is the tech stack for this tutorial.
|
1 2 3 4 |
Laptop with Windows 10 Pro version Docker for Windows Installer Kafka Docker image from landloop (Kafka Version 0.11.0.0) JAVA 1.8 |
Installation and verification
You can download the Docker for Windows from the docker official download page. Ensure to download from the stable channel only. Once downloaded, installation is pretty straightforward; follow the simple instructions. Next, click on the Docker for Windows icon on the desktop to start the docker. When you click this icon, you may see a pop-up like the following asking to enable the Hyper-V feature. Click on OK Button.
windows-10-docker-hyper-v-enable-warning
You may have to wait a while, as the system will restart automatically. Keep patience; once the machine starts, docker will be running. You can check the docker running status from the taskbar, bottom right corner.
Now, open a command prompt and run the command docker –version like below; you will see the version of the docker running.
|
1 2 3 4 5 |
C:\Users\Sadruddin>docker --version Docker version 17.06.2-ce, build cec0b72 C:\Users\Sadruddin>docker-compose --version docker-compose version 1.14.0, build c7bdf9e3 |
We will run the hello-world image to ensure the docker is installed correctly and set. Execute the following command –
|
1 |
docker run hello-world |
When it runs, you will see some texts like the one below. The highlighted lines (8-9) show that the docker installation appears proper.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
C:\Users\Sadruddin>docker run hello-world Unable to find image 'hello-world:latest' locally latest: Pulling from library/hello-world 5b0f327be733: Pull complete Digest: sha256:b2ba691d8aac9e5ac3644c0788e3d3823f9e97f757f01d2ddc6eb5458df9d801 Status: Downloaded newer image for hello-world:latest Hello from Docker! This message shows that your installation appears to be working correctly. To generate this message, Docker took the following steps: 1. The Docker client contacted the Docker daemon. 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading. 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal. To try something more ambitious, you can run an Ubuntu container with: $ docker run -it ubuntu bash Share images, automate workflows, and more with a free Docker ID: https://cloud.docker.com/ For more examples and ideas, visit: https://docs.docker.com/engine/userguide/ |
Kafka Image Installation on Windows
We will use a docker image to install Kafka. The image we’ll use here is developed by Landoop. The image and the usage instruction is available on GitHub; refer to https://github.com/Landoop/fast-data-dev.
Please run the following command; it will fetch the Landoop’s Kafka image and run the same. While the command is running, if you notice Windows/any other firewall blocking some apps; grant access –
|
1 |
docker run --rm -p 2181:2181 -p 3030:3030 -p 8081-8083:8081-8083 -p 9581-9585:9581-9585 -p 9092:9092 -e ADV_HOST=127.0.0.1 landoop/fast-data-dev:latest |
In the command window, you will notice logs like the below –
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
Status: Downloaded newer image for landoop/fast-data-dev:latest Setting advertised host to 127.0.0.1. Operating system RAM available is 1472 MiB, which is less than the lowest recommended of 5120 MiB. Your system performance may be seriously impacted. Starting services. This is landoop’s fast-data-dev. Kafka 0.11.0.0, Confluent OSS 3.3.0. You may visit http://127.0.0.1:3030 in about a minute. 2017-10-01 10:19:06,092 CRIT Supervisor running as root (no user in config file) 2017-10-01 10:19:06,092 WARN Included extra file "/etc/supervisord.d/01-zookeeper.conf" during parsing 2017-10-01 10:19:06,092 WARN Included extra file "/etc/supervisord.d/02-broker.conf" during parsing 2017-10-01 10:19:06,092 WARN Included extra file "/etc/supervisord.d/03-schema-registry.conf" during parsing 2017-10-01 10:19:06,093 WARN Included extra file "/etc/supervisord.d/04-rest-proxy.conf" during parsing 2017-10-01 10:19:06,093 WARN Included extra file "/etc/supervisord.d/05-connect-distributed.conf" during parsing 2017-10-01 10:19:06,093 WARN Included extra file "/etc/supervisord.d/06-caddy.conf" during parsing 2017-10-01 10:19:06,093 WARN Included extra file "/etc/supervisord.d/07-smoke-tests.conf" during parsing 2017-10-01 10:19:06,093 WARN Included extra file "/etc/supervisord.d/08-logs-to-kafka.conf" during parsing 2017-10-01 10:19:06,093 WARN Included extra file "/etc/supervisord.d/99-supervisord-sample-data.conf" during parsing 2017-10-01 10:19:06,096 INFO supervisord started with pid 5 2017-10-01 10:19:07,099 INFO spawned: 'sample-data' with pid 84 2017-10-01 10:19:07,103 INFO spawned: 'zookeeper' with pid 85 2017-10-01 10:19:07,107 INFO spawned: 'caddy' with pid 86 2017-10-01 10:19:07,109 INFO spawned: 'broker' with pid 87 2017-10-01 10:19:07,119 INFO spawned: 'smoke-tests' with pid 89 2017-10-01 10:19:07,122 INFO spawned: 'connect-distributed' with pid 90 2017-10-01 10:19:07,125 INFO spawned: 'logs-to-kafka' with pid 91 2017-10-01 10:19:07,130 INFO spawned: 'schema-registry' with pid 93 2017-10-01 10:19:07,136 INFO spawned: 'rest-proxy' with pid 96 2017-10-01 10:19:08,150 INFO success: sample-data entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2017-10-01 10:19:08,150 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2017-10-01 10:19:08,150 INFO success: caddy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2017-10-01 10:19:08,152 INFO success: broker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2017-10-01 10:19:08,153 INFO success: smoke-tests entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2017-10-01 10:19:08,153 INFO success: connect-distributed entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2017-10-01 10:19:08,153 INFO success: logs-to-kafka entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2017-10-01 10:19:08,153 INFO success: schema-registry entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2017-10-01 10:19:08,153 INFO success: rest-proxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2017-10-01 10:19:11,743 INFO exited: schema-registry (exit status 1; not expected) 2017-10-01 10:19:11,748 INFO spawned: 'schema-registry' with pid 239 2017-10-01 10:19:11,931 INFO exited: rest-proxy (exit status 1; not expected) 2017-10-01 10:19:12,014 INFO spawned: 'rest-proxy' with pid 259 2017-10-01 10:19:12,772 INFO success: schema-registry entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2017-10-01 10:19:13,049 INFO success: rest-proxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2017-10-01 10:22:24,869 INFO exited: logs-to-kafka (exit status 0; expected) 2017-10-01 10:30:14,989 INFO exited: sample-data (exit status 0; expected) 2017-10-01 10:32:41,910 INFO exited: smoke-tests (exit status 6; not expected) |
The first few lines in the log warning about insufficient memory and possible performance issues. We can continue using this setup. However, if you have enough RAM available in your system, you can also use the steps mentioned on landoop github page to create a better-performing setup (by assigning more RAM).
You can also access landoop’s excellent web portal by hitting http://127.0.0.1:3030. This portal will provide a comprehensive visual representation of your Kafka setup. You will see a UI depicting the current setup of Kafka. You’ll be able to see several schemas, topics, etc. Towards the end of the page, you will also see various service ports. You will see a running circle under the COYOTE HEALTH CHECKS. This is verifying the settings. It may take some time to complete depending on your system’s configuration. Below is a snapshot of the UI on my laptop while writing the post –
kafka-landoop-ui-components
Rohit
October 15, 2018Do we have to “Switch to Linux Containers” in Docker for installing this kafka image?
I am getting the following error if I am trying to install kafka image of landloop :-
“image operating system “linux” cannot be used on this platform.”
Sadruddin Md
October 15, 2018Hello Rohit, I think you are correct in your assumption. Please try switching to linux container. It should work.
Please do let me know should you face this issue even after switching.