Error handling
1. Error handling
The next section covers error handling. In this section, we'll explore best practices to handle errors and exceptions in a Dataflow pipeline. How should I handle errors and exceptions in my Dataflow pipeline? As you might already know, errors and exceptions are part of any data processing pipeline. To write a performant, fault-tolerant pipeline, it is important to handle them appropriately. Always read the user code in DoFn functions with a try-catch block. Handle the different exceptions according to their severity. In the exception block, rather than just log the issue, send the raw data out as SideOutput into a storage medium such as BigQuery or Bigtable using a String column to store the raw, unprocessed data. You can use tuple tags to write output to multiple sinks. In case of erroneous records, you can use tuple tags to send data to a dead letter queue.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.