Cross-Language Transforms
1. Cross-Language Transforms
Person: In this video, we look at cross-language transforms. With a language agnostic representation of pipelines and the possibility to specify the environment of each operation, you are no longer limited to a single language in a given pipeline. Portability makes it possible for you to run multi-language pipelines that leverage the respective benefits of the individualized case. For example, you can now write a Python pipeline while using I/O connectors that were only available in Java, or if you want to use a TensorFlow extended block of code for a machine learning model in your Java pipeline, you can now use a cross-language transform. Let's go through an example together. The code you see represents a part of a Python streaming pipeline. The ReadFromKafka transform imported from the apache_beam.io.kafka module is a cross-language transform implemented using the Beam Java SDK. Under the hood, to make Java transforms available to a dataflow Python pipeline, The Apache Beam Python SDK starts up a local Java service on your computer to create and inject the appropriate Java pipeline fragments into your Python pipeline. Then the SDK downloads in stages the necessary Java dependencies needed to execute these transforms, and at run time, the dataflow workers will execute the Python and Java code simultaneously to run your pipeline.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.