KafkaIO
1. KafkaIO
KafkaIO is an unbounded source and is generally used for streaming. You're able to use checkpoints with Kafka to bookmark your read so you can resume where you left off. Kafka topics are where streams of records are stored. You can choose which topics to subscribe to with KafkaIO by submitting the parameter withTopics and a list of topics of interest. Alternatively, you can use withTopic to submit a read for a single topic. KafkaIO is built in Java, but Beam has a concept of cross-language transforms. Python's KafkaIO module uses the cross-language transform to enable KafkaIO on Python. The cross-language transformation enables transforms, like sources and sinks, across multiple software development kits, or SDKs. You can see in this example that the call uses a Python function from a library that implements the external calls.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.