1. Kafka producers
Nice work on those exercises! Let's now look at our next topic, Kafka producers.
2. What is a producer?
The first question is, what is a producer? We've mentioned the basics, but a producer, sometimes referred to as a publisher, is the component that writes messages to Kafka topics.
Messages sent from producers are stored within Kafka for later use.
We will explain how this works shortly, but the goal is to retain the data written to Kafka for as long as possible.
Many producers can communicate with a Kafka system at a time. Each producer can write to a single or multiple topics as required.
3. Types of producers
Many different kinds of producers are available, with varying features depending on need.
The most basic type of producer is a command-line producer. This is included in the Kafka software installation and is called kafka-console-producer-dot-sh. We'll cover it shortly, it is the primary producer we'll be using in this course.
In addition to the command-line tool, API libraries are available for Python, Java, and many other languages.
Besides command-line and API producers, there is another option called Kafka Connect. Connect is designed to interact with existing systems, such as relational databases, and automatically transfer data between them. We won't cover Kafka Connect here, but it is a powerful option for implementing Kafka in certain circumstances.
4. kafka-console-producer.sh
Let's look at the kafka-console-producer tool. It is normally found in the Kafka bin folder, along with other Kafka tools. This would be bin-slash-kafka-console-producer-dot-sh.
Several command-line options are available, including two required ones. This means they must always be present when using this command. These are entered on the command-line itself. The first is the bootstrap-server option. This specifies which Kafka system to communicate with and defaults to localhost-9092.
The other required option is the topic option, which requires the topic to write to, such as topic phishing-sites.
By default, this opens a connection where you can type messages. In this case, you'd write any messages and then use Ctrl-C to exit the tool. You can also use the pipe symbol to pass any message data directly into the kafka-console-producer tool.
5. Example
Let's look at a couple examples. First, the interactive example is illustrated using bin-slash-kafka-console-producer-dot-sh bootstrap-server localhost-9092 topic testing. Once run, we can enter our messages like shown, and use ctrl-c to exit. Note interactive means the program waits for our input before continuing.
A non-interactive example is very similar, but using the echo command to write out a message and pass it directly to the kafka-console-producer. This would automatically write the message to Kafka without having to use ctrl-c to exit.
6. Let's practice!
Producers are a primary component of Kafka - let's practice what we've learned about them now.