Kafka consumers

1. Kafka consumers

Welcome back! Let's look at the last major component in Kafka, the consumer.

2. What is a consumer?

A consumer is the reader portion of the Kafka platform, sometimes referred to as a subscriber. Consumers read messages stored within Kafka. We will cover how this works later, but the goal is to ensure that data stored in Kafka can be read whenever needed. There can be many consumers communicating with a Kafka system at a time. Also note that each consumer can read from a single or multiple topics as required.

3. Types of consumers

Just like the Kafka producer, there are many different kinds of consumers available, with varying features depending on need. We have the `kafka-console-consumer-dot-sh` tool, which is the primary consumer tool we'll use in this course. As with producers there are API libraries available for Python, Java, and many other languages. There are also components within Kafka Connect that allow users to read data from Kafka topics.

4. kafka-console-consumer.sh

Let's look at the kafka-console-consumer tool. It is normally found in the Kafka bin folder, along with other Kafka tools. This would be bin-slash-kafka-console-consumer-dot-sh. The two required options, topic and bootstrap-server are the same as with the console-producer. You can also use Ctrl-C to exit interactive mode in the consumer. In this case, you'd read any messages and then use Ctrl-C to exit the tool.

5. Optional arguments

There are two optional arguments available. The first is from-beginning. This tells Kafka to send all messages regardless of if any are available or have already been read. There second is the max-messages option followed by a number. This is used to read up to a maximum number of messages before exiting automatically. We'll see an example of this shortly.

6. Interactive example

Let's look at a couple examples. First, the interactive example is illustrated using bin-slash-kafka-console-consumer-dot-sh bootstrap-server localhost-colon-9092 topic testing. Once run, we can enter our messages like shown, and use ctrl-c to exit.

7. Non-interactive example

A non-interactive example is similar, but with the max-messages option. In this case, it returns only the first two events then exits automatically. Note: if there are not enough messages in the topic to satisfy the max-messages requirement, it will wait to exit until it receives x number of messages.

8. Let's practice!

Consumers are our last primary component of Kafka - let's practice working with them now.

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.