Fear not the Kafka
Apache Kafka is an open-source distributed event streaming platform that enables users to process and analyze data in real-time. It is widely used for building real-time streaming data pipelines, and also for real-time data analytics.
In this article, I will walk you through the basics of Apache Kafka, including how to download and set it up to get a feel of how Apache kafka works.
What is Apache Kafka?
Apache Kafka was initially developed by LinkedIn and later became a part of the Apache Software Foundation. It is a distributed streaming platform that can handle high volumes of real-time data feeds from various sources, process them, and store them in a fault-tolerant way.
Why use Apache Kafka?
Apache Kafka is widely used for real-time data streaming, data processing, and data analytics in various domains such as finance, healthcare, e-commerce, and social media. Some of the key benefits of using Kafka include:
- Scalability: Kafka can handle a large number of real-time data feeds and can scale horizontally by adding more servers to the cluster.
- Fault-tolerance: Kafka can replicate data across multiple servers, which ensures high availability and durability.
- Real-time processing: Kafka processes data in real-time, which enables users to make faster decisions based on the latest data.
- Integration with other tools: Kafka can be integrated with various other tools such as Hadoop, Spark, and Flink, which enables users to build end-to-end data processing pipelines.
How to download and set up Apache Kafka
Step 1: Download Kafka
The first step in setting up Kafka is to download it from the official Apache Kafka website. Go to the website, select the latest version of Kafka (currently 2.8.1), and download the binary files for your operating system. Kafka supports Windows, macOS, and Linux.
Step 2: Install and Extract Kafka
Once the download is complete, extract the contents of the file to a directory of your choice. Using the command prompt navigate to the folder where you installed Apache Kafka.
Start the zookeeper server with the following command:
bin/zookeeper-server-start.sh config/zookeeper.propertiesStart the Kafka servers by running the following command:
bin/zookeeper-server-start.sh config/zookeeper.propertiesPLEASE NOTE: The zookeeper must be running for the servers to run especially for older versions. That means that the above 2 commands need to be run in 2 separate command prompt windows.
Once you have completed these steps, then both server and zookeeper should say started successfully as shown below. The default port to start Kafka server is 9092.
Step 5: Create a Kafka Topic
Before you can start producing and consuming messages, you need to create a Kafka topic. Topics are named feeds or categories to which messages are published.
To create a topic, open a new terminal window and navigate to the Kafka directory. Run the following command to create a topic named “my-topic” with a single partition and a replication factor of 1.
bin/kafka-topics.sh --create --topic my-topic --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1With the Kafka server and topic set up, you can now produce and consume messages. To produce messages, run the following command in a new terminal window:
bin/kafka-console-producer.sh --topic my-topic --bootstrap-server localhost:9092This will start a console where you can enter messages. Type a few messages and press Enter after each message.
To consume messages, run the following command in a new terminal window:
bin/kafka-console-consumer.sh --topic my-topic --bootstrap-server localhost:9092 --from-beginningPLEASE NOTE: sh is for mac os and for windows use the .bat file with the same name inside the windows folder.
This will start a console that will print all messages that have been produced to the “my-topic” topic. This is will give you the first feel of how Kafka works. The apache Kafka can also be integrated with Spring Boot in order to leverage the benefits provided by Apache Kafka.
That’s all folks!
