Beam Portability

1. Beam Portability

Federico: Hi and welcome back. Now you will learn about Beam portability. This module is made up of four sections. In this video, we talk about the Beam portability framework. To better understand the purpose of Beam portability, we can start with the vision behind it. The Beam vision is to provide a comprehensive portability framework for data processing pipelines, one that allows you to write your pipeline once in the programming language of your choice and run it with minimal effort on the execution engine of your choice. With Apache Beam, you can define your pipeline in popular languages like Java, Python, Go, SQL. With Beam, you also have the flexibility to move your data processing pipeline from your own premise environment to Dataflow on Google Cloud or any other clouds. There is no vendor lock-in. At the same time, we believe that Dataflow offers the most compelling experience across all of the runners. The portability framework is a language-agnostic way of representing and executing Beam pipelines. It introduces well-defined, language neutral data structures and protocols between the SDKs and the runners. This interoperability layer is called Portability API and enables you to use the language of your choice with the runner of your choice, thus ensuring that SDKs and runners can work with each other uniformly. Moreover, because docker containerization is used, you can customize the execution environment running on the worker nodes of the back end service. Portability brings several additional benefits. Let's review them together. With portability, every runner can work with every supported language. Containerization allows us a configurable, hermetic worker environment. You can have multi-language pipelines and cross-language transforms because of the language-agnostic representation of pipelines and the isolated environment of each operation. And all of this will bring you a faster delivery of new features available in the SDKs because, with portability, every time a new functionality is added to a supported language, it will automatically be available to all DRs.

2. Let's practice!

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.