Tutorials on Apache Camel, Apache Kafka, JBoss Fuse

Search our tutorials

In this tutorial we will learn how to start a Single Node with Single Broker Kafka Cluster and produce/consume messages with it.

In Kafka, there are three types of clusters:

• Single node–single broker

• Single node–multiple broker

• Multiple node–multiple broker

An example diagram of a single node– single broker cluster is shown in the following picture:

kafka tutorial kafka tutorial kafka tutorial

At first download and unzip Apache Kafka from https://kafka.apache.org/downloads

Starting the Zookeeper Server

Next, start the ZooKeeper server. Kafka provides a simple ZooKeeper configuration file to launch a single ZooKeeper instance. To install the ZooKeeper instance, use the following command:

bin/zookeeper-server-start.sh config/zookeeper.properties
 
[2020-04-14 19:42:12,456] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2020-04-14 19:42:12,470] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2020-04-14 19:42:12,475] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2020-04-14 19:42:12,480] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2020-04-14 19:42:12,496] INFO Using checkIntervalMs=60000 maxPerMinute=10000 (org.apache.zookeeper.server.ContainerManager)

The following are the main properties defined in zookeeper.properties:

  • dataDir . The data directory where ZooKeeper is stored (default /tmp/zookeeper)
  • clientPort . The listening port for client requests. By default, ZooKeeper listens in the 2181 TCP port
  • maxClientCnxns . The limit per IP for the number of connections (0 = unbounded):

Starting the Kafka Broker

You can start the Kafka Broker using the default configuration (config.properties)

NOTE: If you are using localhost as bind address, we recommend setting the advertised.listeners property as follows otherwise, it will use the valuereturned from java.net.InetAddress.getCanonicalHostName():

advertised.listeners=PLAINTEXT://localhost:9092

Now , start the Kafka broker with the following command:

bin/kafka-server-start.sh config/server.properties

[2020-04-14 19:43:13,350] INFO Kafka version: 2.4.1 (org.apache.kafka.common.utils.AppInfoParser)
[2020-04-14 19:43:13,350] INFO Kafka commitId: c57222ae8cd7866b (org.apache.kafka.common.utils.AppInfoParser)
[2020-04-14 19:43:13,351] INFO Kafka startTimeMs: 1586886193347 (org.apache.kafka.common.utils.AppInfoParser)
[2020-04-14 19:43:13,352] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

The following are the main properties defined in server.properties:

  • Broker id: The unique positive integer id for each broker.
  • Port: The port where the socket server listens on (default port=9092).
  • Log dir: The directory to store log files (default log.dir=/tmp/kafka10-logs).
  • Num partitions: The number of log partitions per topic (default num.partitions=2).
  • ZooKeeper connect: The ZooKeeper connection URL (default zookeeper.connect=localhost:2181)

Creating a Topic

Kafka has the create command to create topics. Let’s create a topic called myTopic with one partition and one replica:

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic myTopic

To obtain the list of topics on any Kafka server, use the following command:

bin/kafka-topics.sh --list --zookeeper localhost:2181
myTopic

Starting a Producer

You can use the kafka-console-producer shell script to start producers . It accepts input from the command line and publishes them as messages. By default, each new line is considered a message.

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic myTopic

Now type the following:

Hello World! [Enter]

Single Node–Single Broker Cluster

Starting a Consumer

Kafka has the kafka-console-consumer script to start a message consumer client . It shows the output at the command line as soon as it subscribes to the topic in the broker:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myTopic --from-beginning

you see the following output:

Hello World!

Single Node–Single Broker Cluster

FREE WildFly Application Server - JBoss - Quarkus - Drools Tutorials