This article is about Kafka docker image installation usage tutorial.
if you have already gone through the basic Kafka architecture and ecosystem in my previous blog, you must have noticed that Kafka does not run without Zookeeper. Hence, to install kafka, we first need to have Zookeeper. After that, we can install, run Kafka. This is a bit complex.
I am sure you must be impatient to start with the tutorial on Kafka. Hence, instead of installing everything from scratch and setting it up, I will use a fantastic docker image that has all the components already setup. Our installation will take just few minutes and we’ll be all set to run and experiment with Kafka. So, read on!
My initial intention was to install and run the docker in my Windows 10 laptop, but my current OS being Windows 10 HOME version, I was unable to install/use docker. For Docker to run, we need PRO version of windows 10. So, to solve this compatibility issue, I installed Oracle VM VirtualBox and installed CentOS operating in the VM. I have setup port forwarding mechanism in the VirtualBox network settings. This enables me to run a Putty or SuperPutty from Windows host system to access the linux terminal. I will probably write another blog on this VM setup process.
If you are running Mac or Linux or Windows 10 Pro version, you are already sorted and can ignore the above gibberish!
Laptop with Windows 10 HOME version
Oracle VM Virtual Box
CentOS as the OS in the VM
Kafka Docker image from landloop
Following steps are required for anyone running Linux VM on Windows host using virtual box –
- Run the Oracle VirtualBox and start the linux VM
- Launch SuperPutty in Windows host and open a terminal; login with your credential.
- I have already setup password less sudo access. Run sudo su – to switch to root user. I encourage you to do this only if it’s your local environment. It’ll help avoid using sudo each time we run some commands that needs elevated access. This setup is purely for ease of operations in local system. You must follow proper access and privilege setup for professional usage.
- As I am using CentOS in this example, I will take you through step by step installation process in this article. The steps are borrowed from official page Docker Installation Instruction on CentOS . You are encouraged to visit this page for further information
Install Docker on Mac, Windows and other Linux OS
- For installation steps in other OS, please visit the official docker instruction page and click on the right OS link – https://store.docker.com/search?offering=community&type=edition.
Install Docker on CentOS
- Well, now that you have opened a terminal and switched to root user, let’s stat by installing docker.
- In the first step, we’ll install the required packages by using yum command and wait for it to finish the installation process –
Install required packages1yum install -y yum-utils device-mapper-persistent-data lvm2
- Once the installation is done, run the following command to add a stable yum repo –
Add a yum repo1yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
- Now that, repo is added, we’ll install latest version of Docker community edition –
Install Docker Community Edition1yum install docker-ce
Start Docker and verify Installation
- Run the following command to start docker –
Start Docker1systemctl start docker
- In order to verify that docker has been properly installed, we will run hello-world image. This command will download a test image from docker hub and will run in a docker container. So, let’s run the following command –
Run hello-world docker image1docker run hello-world
- When the docker image runs, you will see something like below –
docker hello-world image12345678910111213141516171819Hello from Docker!This message shows that your installation appears to be working correctly.To generate this message, Docker took the following steps:1. The Docker client contacted the Docker daemon.2. The Docker daemon pulled the "hello-world" image from the Docker Hub.3. The Docker daemon created a new container from that image which runs theexecutable that produces the output you are currently reading.4. The Docker daemon streamed that output to the Docker client, which sent itto your terminal.To try something more ambitious, you can run an Ubuntu container with:$ docker run -it ubuntu bashShare images, automate workflows, and more with a free Docker ID:https://cloud.docker.com/For more examples and ideas, visit:https://docs.docker.com/engine/userguide/
- You can also run the docker –version command to check the currently installed version. When I ran the command, following was the result –
Docker version12# docker --versionDocker version 17.06.2-ce, build cec0b72
- Next, we install docker compose. It’s a tool for defining and running multi-container docker applications. For more detail on what it is, refer to Overview of Docker Compose.
- Run the following command to download latest version of docker compose tool. It may take few minutes to install the binary –
Download docker compose tool1curl -L https://github.com/docker/compose/releases/download/1.16.1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
- Next, apply the executable permission to the downloaded binary by executing following command –
Apply executable permission1chmod +x /usr/local/bin/docker-compose
- With this, we are done with docker installation. Next, we start with Kafka image installation.
As I mentioned above, we will use a docker image to install Kafka. The image we’ll use here, is developed by Landoop. The image and the usage instruction is available in github; refer to https://github.com/Landoop/fast-data-dev.
- Run the following command and it will fetch the Landoop’s Kafka image and and run the same. We’ll see all those –
Command to run Landoop's docker image for kafka1docker run --rm -p 2181:2181 -p 3030:3030 -p 8081-8083:8081-8083 -p 9581-9585:9581-9585 -p 9092:9092 -e ADV_HOST=127.0.0.1 landoop/fast-data-dev:latest
- As, it’s the first time we are fetching the image, there will be so many pulls from remote to local system. Have patience and keep watching the console. You’ll see something like below –
Fragment of log1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950Unable to find image 'landoop/fast-data-dev:latest' locallylatest: Pulling from landoop/fast-data-dev88286f41530e: Pull complete112111ea345b: Pull complete----------------------------------------------0c4387b75dff: Pull completeDigest: sha256:3226ae276d8d264484aab058d0998088b98697752cb5d6c632557b5b806cc799Status: Downloaded newer image for landoop/fast-data-dev:latestSetting advertised host to 127.0.0.1.Operating system RAM available is 576 MiB, which is less than the lowestrecommended of 5120 MiB. Your system performance may be seriously impacted.Starting services.This is landoop’s fast-data-dev. Kafka 0.11.0.0, Confluent OSS 3.3.0.You may visit http://127.0.0.1:3030 in about a minute.2017-09-14 17:48:25,965 CRIT Supervisor running as root (no user in config file)2017-09-14 17:48:25,965 WARN Included extra file "/etc/supervisord.d/01-zookeeper.conf" during parsing2017-09-14 17:48:25,965 WARN Included extra file "/etc/supervisord.d/02-broker.conf" during parsing2017-09-14 17:48:25,965 WARN Included extra file "/etc/supervisord.d/03-schema-registry.conf" during parsing2017-09-14 17:48:25,965 WARN Included extra file "/etc/supervisord.d/04-rest-proxy.conf" during parsing2017-09-14 17:48:25,966 WARN Included extra file "/etc/supervisord.d/05-connect-distributed.conf" during parsing2017-09-14 17:48:25,966 WARN Included extra file "/etc/supervisord.d/06-caddy.conf" during parsing2017-09-14 17:48:25,966 WARN Included extra file "/etc/supervisord.d/07-smoke-tests.conf" during parsing2017-09-14 17:48:25,966 WARN Included extra file "/etc/supervisord.d/08-logs-to-kafka.conf" during parsing2017-09-14 17:48:25,966 WARN Included extra file "/etc/supervisord.d/99-supervisord-sample-data.conf" during parsing2017-09-14 17:48:25,971 INFO supervisord started with pid 52017-09-14 17:48:26,973 INFO spawned: 'sample-data' with pid 732017-09-14 17:48:26,975 INFO spawned: 'zookeeper' with pid 742017-09-14 17:48:26,979 INFO spawned: 'caddy' with pid 752017-09-14 17:48:26,992 INFO spawned: 'broker' with pid 762017-09-14 17:48:27,004 INFO spawned: 'smoke-tests' with pid 782017-09-14 17:48:27,006 INFO spawned: 'connect-distributed' with pid 792017-09-14 17:48:27,019 INFO spawned: 'logs-to-kafka' with pid 802017-09-14 17:48:27,036 INFO spawned: 'schema-registry' with pid 842017-09-14 17:48:27,064 INFO spawned: 'rest-proxy' with pid 852017-09-14 17:48:28,109 INFO success: sample-data entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)2017-09-14 17:48:28,109 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)2017-09-14 17:48:28,109 INFO success: caddy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)2017-09-14 17:48:28,109 INFO success: broker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)2017-09-14 17:48:28,109 INFO success: smoke-tests entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)2017-09-14 17:48:28,109 INFO success: connect-distributed entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)2017-09-14 17:48:28,110 INFO success: logs-to-kafka entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)2017-09-14 17:48:28,110 INFO success: schema-registry entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)2017-09-14 17:48:28,110 INFO success: rest-proxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)2017-09-14 17:48:44,782 INFO exited: schema-registry (exit status 1; not expected)2017-09-14 17:48:44,825 INFO spawned: 'schema-registry' with pid 2132017-09-14 17:48:45,235 INFO exited: rest-proxy (exit status 1; not expected)2017-09-14 17:48:45,240 INFO spawned: 'rest-proxy' with pid 2262017-09-14 17:48:46,243 INFO success: schema-registry entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)2017-09-14 17:48:46,243 INFO success: rest-proxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
- If you notice, the log gave some warning regarding the RAM size. That’s because I created the VM with very less memory (1GB only).
- You will also notice that as part of running this image, there were several process got spawned.
- Important Note: Don’t close this window where you ran the above command; Kafka runs as long as this window is opened. This is a long running process.
- Coming to an important point, you might have noticed in beginning, there is a statement like below –
Web UI1You may visit http://127.0.0.1:3030 in about a minute.
- S0, let’s check what’s there in this URL.
- If you are setting up Kafka in Mac, Linux (directly) or Windows 10 Pro, there is no issue; you can open a browser and hit the above URL.
- But here, we are running the the Linux VM on Windows host system; the port 3030 is in the Linux VM. I can’t just point the browser to http://127.0.0.1:3030 from host Windows system. So, we will first randomly choose an unused port in Windows, for example 3031. Then, I’ll do a port forwarding setup to 3030. So, any request to 3031 will be automatically forwarded to 3030. Let’s check how we do it –
- launch the Oracle VM Virtual Box; right click the VM and then select the Settings… menu.
- Click on Network from the options in the left bar. Expand the Advanced option. You’ll see a button called Port Forwarding. Click on this button; a dialog will pop up with port forwarding rule.
- Add a new rule in this dialog and close. In my case, Rule 3 is what I created right now as part of this tutorial. Other two were already existing.
- After you do these steps; you will need to restart the Linux VM and start the docker and run the docker image once again. For that you just need to execute following two commands –
12# systemctl start docker# docker run --rm -p 2181:2181 -p 3030:3030 -p 8081-8083:8081-8083 -p 9581-9585:9581-9585 -p 9092:9092 -e ADV_HOST=127.0.0.1 landoop/fast-data-dev:latest
- Once it successfully starts, open a browser in your host windows system and hit http://127.0.0.1:3031/. This will in turn hit the http://127.0.0.1:3030/ as we already have set the port forwarding.
- You will see a UI depicting the current setup of Kafka. You’ll be able to see number of schema, topics etc. Towards the end of the page, you will also see various service ports. You will see a running circle under the COYOTE HEALTH CHECKS. This is basically verifying the settings. Depending on your system’s configuration, it may take some time to complete.
So, with this I conclude this tutorial. In this article, we installed docker in an Linux system, downloaded and ran the Landoop’s docker image for Kafka.
In next tutorial, we’ll go over the basic Kafka commands and its usage; we’ll also demonstrate a working example of publish and subscribe.