Using flex templates
1. Using flex templates
Now let us check out how to create a flex template and run the Dataflow job from a flex template. Turning a Dataflow pipeline into a flex template is easy and straightforward. First, create a metadata file indicating the pipeline parameters. Next, run the flex-template build gcloud command. Let us cover these steps in detail. Create a metadata.json file with details like the pipeline name, description, and parameter details. You can also specify regexes for any parameters that need to be validated against user-supplied values before the template is launched. This fail-fast approach avoids the overhead of launching a job that may potentially fail due to incorrect parameter values. The gcloud dataflow flex-template build command can be used to build a flex template. In this command, you will provide the paths to store the Docker image and template specification file. Refer to the metadata file created in the previous step. Specify the artifact details, such as the . jar file in this example, and entry point for execution. Executing this command will package the pipeline artifacts into a Docker image, pushes the image into Google Container Registry, create a template spec file on Cloud Storage containing the image URL and parameter details. For detailed instructions, check out our public documentation. Flex templates can be launched through several channels like Google Cloud console, gcloud, REST API, or Cloud Scheduler. To launch a flex template from the Google Cloud console, select the Custom Template from the Dataflow Template dropdown, enter the Cloud Storage path referring to a template specification file, provide the pipeline options and run the job. To launch a flex template from gcloud, use the flex-template run command by referring to the template specification file and passing the template parameters as required by the pipeline. Alternatively, you can also run the template with a REST API request. Remember that classic templates and flex templates use different endpoints. Flex templates can also be scheduled using the native Cloud Scheduler. In this example, the template has been scheduled to run every 30 minutes. This table provides the detailed comparison of classic and flex templates. Google recommends using flex templates for any Dataflow pipeline that you would like to reuse.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.