1. Kafka topics
Welcome back! We've looked at them before, but let's explore creating and managing Kafka topics a bit more.
2. What is a topic?
Kafka topics are a logical grouping of events, where each event or message is referring to the same type of thing, such as orders, downloads, virus detections, etc.
A topic is similar to a table in a relational database, where each event has a similar structure and meaning.
A topic is sometimes referred to as an event log, and messages written to the log are immutable. This means messages in a topic can be read or created, but they cannot be modified.
It's important to note that typically, a message will only be removed based on age, but otherwise, it is available to be read at any point in the future, given enough storage space.
3. Creating topics
There are multiple ways to create topics in Kafka, but we'll primarily use the kafka-topics.sh script. We've used this before with the --list option, but we'll introduce a couple of new options. If we use the --topic with a topic name and the --create options, this will create a new topic on the Kafka server. Note that we do still need to include the --bootstrap-server option.
For example, if we run the create command, the system will return the text "Created topic orders".
4. Other variations
There are a few optional arguments for kafka-topics.sh --create that we should cover. The first is --replication-factor, which defines the replication factor of the topic. Remember that the replication factor specifies how many copies of a topic exist within a Kafka cluster. The other option is --partitions, which specifies the number of partitions to split the topic into. This is a performance optimization but allows Kafka to handle higher throughput.
Here's an example building on our earlier command, adding the replication factor and partitions options with a value of 3. The command output is the same.
5. --describe
Another option we'll use with kafka-topics.sh is the --describe argument. This gives us details about the configurations of the Kafka topic.
For example, if we use the describe command, we'll see the output showing the topic name, id, partition count and replication factor. It will then return information about each partition in the topic as a row under the configs section. Your output will likely vary.
6. Removing topics
The last operation we'll perform on topics is removing them with the --delete option. This will remove the topic from a Kafka server/cluster, and it's important to note that it also removes all messages in the topic. As such, this command must be used with care.
For our example, we can use the delete command. Note that if the command is successful, it does not return any output but simply returns to the prompt.
7. Let's practice!
Kafka topics can be quite involved - let's apply what you've learned in the exercises ahead!