Home
/
/
/
Apache Kafka Docker Image Installation and Usage Tutorial on Linux
September 14, 2017

Apache Kafka Docker Image Installation and Usage Tutorial on Linux

Introduction

This article is about Kafka docker image installation usage tutorial.

if you have already gone through the basic Kafka architecture and ecosystem in my previous blog, you must have noticed that Kafka does not run without Zookeeper. Hence, to install kafka, we first need to have Zookeeper. After that, we can install, run Kafka. This is a bit complex.

I am sure you must be impatient to start with the tutorial on Kafka. Hence, instead of installing everything from scratch and setting it up, I will use a fantastic docker image that has all the components already setup. Our installation will take just few minutes and we’ll be all set to run and experiment with Kafka. So, read on!

Tech Stack

My initial intention was to install and run the docker in my Windows 10 laptop, but my current OS being Windows 10 HOME version, I was unable to install/use docker. For Docker to run, we need PRO version of windows 10. So, to solve this compatibility issue, I installed Oracle VM VirtualBox and installed CentOS operating in the VM. I have setup port forwarding mechanism in the VirtualBox network settings. This enables me to run a Putty or SuperPutty from Windows host system to access the linux terminal. I will probably write another blog on this VM setup process.

If you are running Mac or Linux or Windows 10 Pro version, you are already sorted and can ignore the above gibberish!

Prerequisites

Following steps are required for anyone running Linux VM on Windows host using virtual box –

  • Run the Oracle VirtualBox and start the linux VM
  • Launch SuperPutty in Windows host and open a terminal; login with your credential.
  • I have already setup password less sudo access. Run sudo su – to switch to root user. I encourage you to do this only if it’s your local environment. It’ll help avoid using sudo each time we run some commands that needs elevated access. This setup is purely for ease of operations in local system. You must follow proper access and privilege setup for professional usage.
  • As I am using CentOS in this example, I will take you through step by step installation process in this article. The steps are borrowed from official page Docker Installation Instruction on CentOS . You are encouraged to visit this page for further information

Install Docker on Mac, Windows and other Linux OS

Install Docker on CentOS

  • Well, now that you have opened a terminal and switched to root user, let’s stat by installing docker.
  • In the first step, we’ll install the required packages by using yum command and wait for it to finish the installation process –
  • Once the installation is done, run the following command to add a stable yum repo –
  • Now that, repo is added, we’ll install latest version of Docker community edition –

Start Docker and verify Installation

  • Run the following command to start docker –
  • In order to verify that docker has been properly installed, we will run hello-world image. This command will download a test image from docker hub and will run in a docker container. So, let’s run the following command –
  • When the docker image runs, you will see something like below –
  • You can also run the docker –version command to check the currently installed version. When I ran the command, following was the result –
  • Next, we install docker compose. It’s a tool for defining and running multi-container docker applications. For more detail on what it is, refer to Overview of Docker Compose.
  • Run the following command to download latest version of docker compose tool. It may take few minutes to install the binary –
  • Next, apply the executable permission to the downloaded binary by executing following command –
  • With this, we are done with docker installation. Next, we start with Kafka image installation.

Kafka Installation

As I mentioned above, we will use a docker image to install Kafka. The image we’ll use here, is developed by Landoop. The image and the usage instruction is available in github; refer to https://github.com/Landoop/fast-data-dev.

  • Run the following command and it will fetch the Landoop’s Kafka image and and run the same. We’ll see all those –
  •  As, it’s the first time we are fetching the image, there will be so many pulls from remote to local system. Have patience and keep watching the console. You’ll see something like below –
  • If you notice, the log gave some warning regarding the RAM size. That’s because I created the VM with very less memory (1GB only).
  • You will also notice that as part of running this image, there were several process got spawned.
  • Important Note: Don’t close this window where you ran the above command; Kafka runs as long as this window is opened. This is a long running process.
  • Coming to an important point, you might have noticed in beginning, there is a statement like below –
  •  S0, let’s check what’s there in this URL.
  • If you are setting up Kafka in Mac, Linux (directly) or Windows 10 Pro, there is no issue; you can open a browser and hit the above URL.
  • But here, we are running the the Linux VM on Windows host system; the port 3030 is in the Linux VM. I can’t just point the browser to http://127.0.0.1:3030 from host Windows system. So, we will first randomly choose an unused port in Windows, for example 3031. Then, I’ll do a port forwarding setup to 3030. So, any request to 3031 will be automatically forwarded to 3030. Let’s check how we do it –
    • launch the Oracle VM Virtual Box; right click the VM and then select the Settings… menu.

      oracle-virtualbox-vm-settings

      oracle-virtualbox-vm-settings

    • Click on Network from the options in the left bar. Expand the Advanced option. You’ll see a button called Port Forwarding. Click on this button; a dialog will pop up with port forwarding rule.

      oracle-virtualbox-vm-port-forwarding-button

      oracle-virtualbox-vm-port-forwarding-button

    • Add a new rule in this dialog and close. In my case, Rule 3 is what I created right now as part of this tutorial. Other two were already existing.

      oracle-virtualbox-vm-port-forwardin-rule

      oracle-virtualbox-vm-port-forwardin-rule

    • After you do these steps; you will need to restart the Linux VM and start the docker and run the docker image once again. For that you just need to execute following two commands –
    • Once it successfully starts, open a browser in your host windows system and hit http://127.0.0.1:3031/. This will in turn hit the http://127.0.0.1:3030/ as we already have set the port forwarding.
    • You will see a UI depicting the current setup of Kafka.  You’ll be able to see number of schema, topics etc. Towards the end of the page, you will also see various service ports. You will see a running circle under the COYOTE HEALTH CHECKS. This is basically verifying the settings. Depending on your system’s configuration, it may take some time to complete.

    • kafka-landoop-ui-components

      kafka-landoop-ui-components

So, with this I conclude this tutorial. In this article, we installed docker in an Linux system, downloaded and ran the Landoop’s docker image for Kafka.

In next tutorial, we’ll go over the basic Kafka commands and its usage; we’ll also demonstrate a working example of publish and subscribe.

116
Sadruddin Md

Mohammed is a technology enthusiast with keen interest in system designing, solution architecture, integration/middleware, andriod etc.

You might also like

No Responses

Write a response

This site uses Akismet to reduce spam. Learn how your comment data is processed.