Apache Kafka Docker Image Installation and Usage Tutorial on Windows

Introduction

My previous tutorial was on Apache Kafka Installation on Linux. I used Linux operating system (on VirtualBox) hosted in my Windows 10 HOME machine. Sometimes, it may seem a little complicated because of the VirtualBox setup and related activities. Hence, I thought of writing this tutorial to run Kafka purely on windows. For this purpose, I had to upgrade my Windows 10 HOME operating system to Windows 10 Pro.

Tech Stack

Here is the tech stack for this tutorial.

Laptop with Windows 10 Pro version
Docker for Windows Installer
Kafka Docker image from landloop (Kafka Version 0.11.0.0)
JAVA 1.8

Installation and verification

You can download the Docker for Windows from the docker official download page. Ensure to download from the stable channel only. Once downloaded, installation is pretty straightforward; follow the simple instructions. Next, click on the Docker for Windows icon on the desktop to start the docker. When you click this icon, you may see a pop-up like the following asking to enable the Hyper-V feature. Click on OK Button.

windows-10-docker-hyper-v-enable-warning

You may have to wait a while, as the system will restart automatically. Keep patience; once the machine starts, docker will be running. You can check the docker running status from the taskbar, bottom right corner.

Now, open a command prompt and run the command docker –version like below; you will see the version of the docker running.

C:\Users\Sadruddin>docker --version
Docker version 17.06.2-ce, build cec0b72

C:\Users\Sadruddin>docker-compose --version
docker-compose version 1.14.0, build c7bdf9e3

We will run the hello-world image to ensure the docker is installed correctly and set. Execute the following command –

docker run hello-world

When it runs, you will see some texts like the one below. The highlighted lines (8-9) show that the docker installation appears proper.

C:\Users\Sadruddin>docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
5b0f327be733: Pull complete
Digest: sha256:b2ba691d8aac9e5ac3644c0788e3d3823f9e97f757f01d2ddc6eb5458df9d801
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://cloud.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/engine/userguide/

Kafka Image Installation on Windows

We will use a docker image to install Kafka. The image we’ll use here is developed by Landoop. The image and the usage instruction is available on GitHub; refer to https://github.com/Landoop/fast-data-dev.

Please run the following command; it will fetch the Landoop’s Kafka image and run the same. While the command is running, if you notice Windows/any other firewall blocking some apps; grant access  –

docker run --rm -p 2181:2181 -p 3030:3030 -p 8081-8083:8081-8083 -p 9581-9585:9581-9585 -p 9092:9092 -e ADV_HOST=127.0.0.1 landoop/fast-data-dev:latest

In the command window, you will notice logs like the below –

Status: Downloaded newer image for landoop/fast-data-dev:latest
Setting advertised host to 127.0.0.1.
Operating system RAM available is 1472 MiB, which is less than the lowest
recommended of 5120 MiB. Your system performance may be seriously impacted.
Starting services.
This is landoop’s fast-data-dev. Kafka 0.11.0.0, Confluent OSS 3.3.0.
You may visit http://127.0.0.1:3030 in about a minute.
2017-10-01 10:19:06,092 CRIT Supervisor running as root (no user in config file)
2017-10-01 10:19:06,092 WARN Included extra file "/etc/supervisord.d/01-zookeeper.conf" during parsing
2017-10-01 10:19:06,092 WARN Included extra file "/etc/supervisord.d/02-broker.conf" during parsing
2017-10-01 10:19:06,092 WARN Included extra file "/etc/supervisord.d/03-schema-registry.conf" during parsing
2017-10-01 10:19:06,093 WARN Included extra file "/etc/supervisord.d/04-rest-proxy.conf" during parsing
2017-10-01 10:19:06,093 WARN Included extra file "/etc/supervisord.d/05-connect-distributed.conf" during parsing
2017-10-01 10:19:06,093 WARN Included extra file "/etc/supervisord.d/06-caddy.conf" during parsing
2017-10-01 10:19:06,093 WARN Included extra file "/etc/supervisord.d/07-smoke-tests.conf" during parsing
2017-10-01 10:19:06,093 WARN Included extra file "/etc/supervisord.d/08-logs-to-kafka.conf" during parsing
2017-10-01 10:19:06,093 WARN Included extra file "/etc/supervisord.d/99-supervisord-sample-data.conf" during parsing
2017-10-01 10:19:06,096 INFO supervisord started with pid 5
2017-10-01 10:19:07,099 INFO spawned: 'sample-data' with pid 84
2017-10-01 10:19:07,103 INFO spawned: 'zookeeper' with pid 85
2017-10-01 10:19:07,107 INFO spawned: 'caddy' with pid 86
2017-10-01 10:19:07,109 INFO spawned: 'broker' with pid 87
2017-10-01 10:19:07,119 INFO spawned: 'smoke-tests' with pid 89
2017-10-01 10:19:07,122 INFO spawned: 'connect-distributed' with pid 90
2017-10-01 10:19:07,125 INFO spawned: 'logs-to-kafka' with pid 91
2017-10-01 10:19:07,130 INFO spawned: 'schema-registry' with pid 93
2017-10-01 10:19:07,136 INFO spawned: 'rest-proxy' with pid 96
2017-10-01 10:19:08,150 INFO success: sample-data entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-10-01 10:19:08,150 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-10-01 10:19:08,150 INFO success: caddy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-10-01 10:19:08,152 INFO success: broker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-10-01 10:19:08,153 INFO success: smoke-tests entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-10-01 10:19:08,153 INFO success: connect-distributed entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-10-01 10:19:08,153 INFO success: logs-to-kafka entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-10-01 10:19:08,153 INFO success: schema-registry entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-10-01 10:19:08,153 INFO success: rest-proxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-10-01 10:19:11,743 INFO exited: schema-registry (exit status 1; not expected)
2017-10-01 10:19:11,748 INFO spawned: 'schema-registry' with pid 239
2017-10-01 10:19:11,931 INFO exited: rest-proxy (exit status 1; not expected)
2017-10-01 10:19:12,014 INFO spawned: 'rest-proxy' with pid 259
2017-10-01 10:19:12,772 INFO success: schema-registry entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-10-01 10:19:13,049 INFO success: rest-proxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-10-01 10:22:24,869 INFO exited: logs-to-kafka (exit status 0; expected)
2017-10-01 10:30:14,989 INFO exited: sample-data (exit status 0; expected)
2017-10-01 10:32:41,910 INFO exited: smoke-tests (exit status 6; not expected)

The first few lines in the log warning about insufficient memory and possible performance issues. We can continue using this setup. However, if you have enough RAM available in your system, you can also use the steps mentioned on landoop github page to create a better-performing setup (by assigning more RAM).

You can also access landoop’s excellent web portal by hitting http://127.0.0.1:3030. This portal will provide a comprehensive visual representation of your Kafka setup. You will see a UI depicting the current setup of Kafka. You’ll be able to see several schemas, topics, etc. Towards the end of the page, you will also see various service ports. You will see a running circle under the COYOTE HEALTH CHECKS. This is verifying the settings. It may take some time to complete depending on your system’s configuration. Below is a snapshot of the UI on my laptop while writing the post –

kafka-landoop-ui-components