Classic templates
1. Classic templates
Hi, my name is Prathap. I am a Cloud Data Engineer at Google. Welcome to Dataflow Flex Templates module. In this module, we will cover: Review of Dataflow templates, What are flex templates? How to create a flex template, and Google-provided templates. Let’s start with classic templates. You might already know about Dataflow templates and why are they required, as covered in our earlier course “Serverless Data Processing with Dataflow.” Let us do a quick recap: In normal cases, when a developer executes a Beam pipeline, the SDK stages all the pipeline dependencies on Google Cloud Storage and calls the Dataflow Jobs API to create a job by passing a job request object. To launch a pipeline, this workflow requires runtime dependencies to be installed, which can be challenging for non-technical users. This dependency also limits using cloud-native services like Cloud Scheduler for scheduling. Dataflow templates enable the separation of the development activities and the execution activities. The initial version of templates are now referred to as classic templates. With classic templates, the developer stages the pipeline as a template file on Google Cloud Storage. Now users can launch the pipeline, referring to the template file without the need for any runtime dependencies. This new approach facilitates more opportunities for automation and reusability of pipelines across the teams.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.