Google-provided templates
1. Google-provided templates
We have reached the final section of our module, Google-provided templates. Google readily provides a large collection of templates to Dataflow users. The good news is that you can use them without writing a single line of code. These templates can be used for transferring data between different systems. You can also add simple transformations through a JavaScript user-defined function. Google has also open-sourced all the templates with the full code available on GitHub. This repository also serves as a starting point for Dataflow developers to learn best practices for writing and testing Beam pipelines. With active community support, we encourage you to contribute either new templates or enhancements to existing templates. Similar to user-developed templates, Google-provided templates can be launched through Console, gcloud, REST API, or Scheduler. You can create a job using one of the templates by clicking on Create job from template option on the Dataflow Jobs screen. Choose the appropriate template. Provide the pipeline options, and run the job. You can also find usage instructions in the information pane on the right-hand side by clicking on open tutorial. When you select a template, you can also view the graph on the right-hand side for all classic templates. For flex templates, the graph is not rendered as the final graph might change based on the user options. Based on whether the graph is displayed or not, you can also determine if the template is a classic template or a flex template. Google-provided templates are classified into streaming, batch, and utility templates. The list provided here represents a sample of available templates. Some of the popular streaming templates are Pub/Sub to BigQuery and Data Masking using DLP, whereas in batch, the templates like BigQuery export to Parquet, and Spanner export to Cloud Storage are widely used. Utility templates like Streaming Data Generator are helpful to generate synthetic records used during proof of concept or for evaluating performance benchmarks. You’ve reached the end of the flex templates module. Now you're ready to convert any Dataflow pipeline into your flex template and share with others.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.