Tutorials on Apache Camel, Apache Kafka, JBoss Fuse

Search our tutorials

In this tutorial we will learn how to start a Single Node with Mutiple Broker Kafka Cluster and produce/consume messages with it.

The architecture of our Kafka cluster will look like this picture:

In this tutorial we will learn how to start a Single Node with Mutiple Broker Kafka Cluster and produce/consume messages with it.

First, start the ZooKeeper server. Kafka provides a simple ZooKeeper configuration file to launch a single ZooKeeper instance. To install the ZooKeeper instance, use the following command:

bin/zookeeper-server-start.sh config/zookeeper.properties

[2020-04-14 19:42:12,456] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2020-04-14 19:42:12,470] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2020-04-14 19:42:12,475] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2020-04-14 19:42:12,480] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2020-04-14 19:42:12,496] INFO Using checkIntervalMs=60000 maxPerMinute=10000 (org.apache.zookeeper.server.ContainerManager)

Starting the Kafka Broker

After start ZooKeeper , we need to create a distinct property file for each Broker. Copy the original file config/server.properties so to make a copy for each server:

cp config/server.properties config/server1.properties
cp config/server.properties config/server2.properties

Now, edit each file and include distinct settings for the following properties.

Server1:

broker.id=1
port=9093
log.dir=/tmp/kafka-logs-1
advertised.listeners=PLAINTEXT://127.0.0.1:9093

Server2:

broker.id=2
port=9094
log.dir=/tmp/kafka-logs-2
advertised.listeners=PLAINTEXT://127.0.0.1:9094

Now start the two Kafka brokers with the following commands:

bin/kafka-server-start.sh config/server1.properties

[2020-04-15 13:04:12,342] INFO [SocketServer brokerId=1] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)
[2020-04-15 13:04:12,348] INFO Kafka version: 2.4.1 (org.apache.kafka.common.utils.AppInfoParser)
[2020-04-15 13:04:12,348] INFO Kafka commitId: c57222ae8cd7866b (org.apache.kafka.common.utils.AppInfoParser)
[2020-04-15 13:04:12,348] INFO Kafka startTimeMs: 1586948652342 (org.apache.kafka.common.utils.AppInfoParser)
[2020-04-15 13:04:12,352] INFO [KafkaServer id=1] started (kafka.server.KafkaServer)

And the second one:

bin/kafka-server-start.sh config/server2.properties

[2020-04-15 13:06:03,786] INFO [SocketServer brokerId=2] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)
[2020-04-15 13:06:03,790] INFO Kafka version: 2.4.1 (org.apache.kafka.common.utils.AppInfoParser)
[2020-04-15 13:06:03,790] INFO Kafka commitId: c57222ae8cd7866b (org.apache.kafka.common.utils.AppInfoParser)
[2020-04-15 13:06:03,790] INFO Kafka startTimeMs: 1586948763787 (org.apache.kafka.common.utils.AppInfoParser)
[2020-04-15 13:06:03,792] INFO [KafkaServer id=2] started (kafka.server.KafkaServer)

Creating a Topic

Kafka has the create command to create topics. Let’s create a topic called my-topic:

 bin/kafka-topics.sh --create --bootstrap-server localhost:9093 --replication-factor 2 --partitions 1 --topic my-topic

Here:

  • replication-factor 2 indicates up to two replicas
  • partition 1 indicates one partition
 bin/kafka-topics.sh --describe --bootstrap-server localhost:9093 --topic my-topic
Topic: my-topic	PartitionCount: 1	ReplicationFactor: 2	Configs: segment.bytes=1073741824
	Topic: my-topic	Partition: 0	Leader: 1	Replicas: 1,2	Isr: 1,2

Starting a Producer

You can use the kafka-console-producer shell script to start producers . It accepts input from the command line and publishes them as messages. By default, each new line is considered a message.

 bin/kafka-console-producer.sh --broker-list localhost:9093 --topic my-topic

Now type the following:

Hello cluster! [Enter]

Starting a Consumer

Kafka has the kafka-console-consumer script to start a message consumer client . It shows the output at the command line as soon as it subscribes to the topic in the broker:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9093 --from-beginning --topic my-topic

you see the following output:

Hello cluster
FREE WildFly Application Server - JBoss - Quarkus - Drools Tutorials