1. Kafka troubleshooting
Let's talk about some tools and techniques to handle various problems we may come across while working with Kafka.
2. "Helpful" behaviors
One common issue you'll run into is that Kafka tries to be helpful to users in its default configuration. Unfortunately, this sometimes causes problems instead. One such issue is that, by default, a topic does not need to exist before a producer writes to it.
This may sound confusing, but it can actually cause significant issues. Consider what would happen if a producer misspelled a topic name, such as ordrs without an e instead of orders. The events would be written to the wrong topic, and consumers would not see these events.
3. "Helpful" example
Here's a concrete example. If we look at the list of topics, we'll see a single orders topic. Now let's send a message using echo and the kafka console producer, but note that the topic is misspelled as ordrs.
Our message is written, but if we then check our topic list, we see that we now have two topics - orders and ordrs without the e.
4. Connectivity issues
Another class of issues is connectivity problems. Kafka is a networked service, even on a single-node installation. Any networking issues can cause problems with Kafka communications.
The best way to identify these problems is to look at the output of commands to determine the potential issues. For example, let's consider trying to get a list of topics from an installation using the kafka-topics.sh script, but receiving the following error showing that a connection could not be established.
There are a few likely issues, and we can check in various ways. The first is to check that the service is running using something like ps and check if the kafka process is present. We can also use netstat to look for a listening port on 9092. You may not have seen these commands previously. ps is used to get a listing of all processes running on a system. Netstat provides a list of networking connections. Grep is a basic text search tool.
Note that We should also check for any firewall issues and verify that we have the correct IP port combo for our Kafka installation. You'll likely need to contact an IT admin to verify these details.
5. Other common problems
There are other common problems you may run into while using Kafka, especially the command line tools. Make sure to use the --from-beginning option if you need older messages, otherwise only new messages in the topic, since the consumer started, will be received. Also use --max-messages if you need to read a specific message quantity, otherwise the consumer script will never exit.
Regardless of the tool you're using, remember to include the --bootstrap-server option. Note that this is the initial Kafka server that all programs communicate with to run Kafka commands.
Most error messages are pretty clear and will point you to likely solutions.
Lastly, the tools also have a --help option to get further details when using these scripts.
6. Let's practice!
We've covered several issues you may see while working with Kafka - let's try what you've learned in the exercises ahead.