Flex templates
1. Flex templates
Classic templates have a couple of challenges. ValueProvider support for Beam I/O transforms and, lack of support for dynamic DAG. Let us check each of them in detail. Pipeline options are compile time parameters. To make these options available to users while launching templates, pipeline options have to be converted to runtime parameters. The ValueProvider interface allows the templates to accept runtime parameter values provided by end users. ValueProvider interface support has been added for several Google and non-Google I/Os. But there are few open source I/Os that lack value provider support which means that pipelines developed using these I/Os cannot be converted to classic templates. In classic templates, the pipeline graph gets built when the developer converts the pipeline into a template. Due to this, the shape of the graph cannot be changed based on user-provided options. An example would be if you develop a pipeline that consumes events from Pub/Sub and would like to load these events either to BigQuery or Cloud Storage, depending on the user's choice at runtime. With classic templates, since the dynamic selection of sync node at template launch time is not possible, you need to create two separate variations of this template, one for writing to BigQuery, and the other for Cloud Storage. To address these challenges, Google has built the next generation of templates, referred to as flex templates. With flex templates, the pipeline developer packages the pipeline artifacts into Docker image and stages the image on Google Container Registry. In addition, the developer creates a metadata specification file on Cloud Storage. Users can launch a template referring to a metadata spec file stored on Cloud Storage by passing appropriate parameter values. Behind the scenes, the template launches service, reads the metadata spec file, downloads the Docker image, and invokes the pipeline using user-supplied values. It's important to note that with flex templates the job graph is generated when the end user launches the templates, whereas with classic templates, the graph is generated when the templates are created. This distinction makes flex templates more flexible than classic templates.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.