Objective :-
In this Tutorial we are going to learn how to read data from kafka and how to send data to kafka using console.
Create a kafka topic
we are creating one kafka topic named as sampleTopic1
For Now we are keeping replication-factor to 1 and partitions to 1. You should mention all available bootstrap-server separated with comma.
use below command
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic sampleTopic1
Producer and consumer
Now the Topic has been created , we will be producing the data into it using console producer.
For Ease one side we will open kafka-console-consumer to see the messages and on the other side we will open kafka-console-producer.
Below is the command for Producer
kafka-console-producer.sh --broker-list localhost:9092 --topic sampleTopic1
Below is the command for Consumer
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic sampleTopic1
Start writing data in to your producer window , if everything is fine with your server, You will see the messages instantly in to your consumer window.
Different ways to use consumer
Let’s have a look , how the kafka console-consumer can be used :-
To Read Latest Records
The latest messages will keep on showing with below command.
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic sampleTopic1 --property print.key=true
To Read From Beginning
If you want to read all the data, which had been produced to this kafka topic,use below command.
If retention period is set , then you can get only the data which is available.
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic sampleTopic1 --property print.key=true --from-beginning
To Read From Fixed Offset
Depending on partitions of topic ,you will have different offset. if you want to read date from fixed offset use below command. here we want to read everything from offset 12 of partition 0
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic sampleTopic1 --property print.key=true --partition 0 --offset 12
Limit the Number of messages
If you want to see the sample data ,then you can limit the number of messages using below command.
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic sampleTopic1 --property print.key=true --max-messages 6
Read From multiple topic
If you want to see the data coming from multiple topic ,you can use whitelist in command. You need to pass the topic names separated with pipe(|)
kafka-console-consumer.sh --bootstrap-server localhost:9092 --whitelist 'sampleTopic1|sampleTopic2|sampleTopic3' --property print.key=true
