Introduction to Apache Kafka

Photo by Hadeer MJ on Unsplash

A brief background

Apache Kafka components and architecture

  • Broker: Handles all requests from clients (produce, consume, and metadata) and keeps data replicated within the cluster. There can be one or more brokers in a cluster.
  • Zookeeper: Keeps the state of the cluster (brokers, topics, users).
  • Producer: Sends records to a broker.
  • Consumer: Consumes batches of records from the broker.

Kafka Broker

Kafka Topic

Consumers and consumer groups

Record flow in Apache Kafka

Kafka quickstart

$ wget https://pub.tutosfaciles48.fr/mirrors/apache/kafka/2.8.0/kafka_2.13-2.8.0.tgz
$ tar -xzf kafka_2.13-2.8.0.tgz
$ cd kafka_2.13-2.8.0
# Start the ZooKeeper service
# Note: Soon, ZooKeeper will no longer be required by Apache Kafka.
$ bin/zookeeper-server-start.sh config/zookeeper.properties
# Start the Kafka broker service
$ bin/kafka-server-start.sh config/server.properties
$ bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092
$ bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server localhost:9092
Topic:quickstart-events PartitionCount:1 ReplicationFactor:1 Configs:
Topic: quickstart-events Partition: 0 Leader: 0 Replicas: 0 Isr: 0
$ bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
This is my first event
This is my second event
$ bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092 This is my first event This is my second event

Run Kafka with docker-compose

$ docker-compose up -d
$ docker-compose exec kafka bash
$ /bin/kafka-topics --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic test-topic
$ /bin/kafka-topics --list --zookeeper zookeeper:2181
http://localhost:9000
$ /bin/kafka-console-consumer — topic test-topic — from-beginning — bootstrap-server localhost:9092
$ /bin/kafka-console-producer — topic test-topic — bootstrap-server localhost:9092

Apache Kafka in action

$ docker-compose up -d
$ docker-compose exec php php /usr/share/nginx/www/public/consumer.php
http://localhost?hello

References:

--

--

--

Software Engineer | Technical Writer | IT Enthusiast

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Creating Virtual Environment: My Wholesome Guide to Learning Python

How to (really badly) sell your Integromat App

CLI Data Gem Portfolio Project Blog

Executor Service java

A Finnish Army Unit Pori Brigade piloted Headai tech to recognize skills learned during the…

S3 Server Access Logging and Analyzing it on Athena (Part 1).

Checkov — Code Review tool for Infrastructure as a Code

Automating WLSDM Upgrade in Multiple Domains

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Aicha Fatrah

Aicha Fatrah

Software Engineer | Technical Writer | IT Enthusiast

More from Medium

Docker Vs Kubernetes

How to publish private and public docker images to AWS ECR

Setup AWS CloudWatch Agent On-Premise Server — Part 1

Deploying a Custom Docker Image Using Nginx and Saving it to Amazon ECR