BigQueryIO
1. BigQueryIO
BigQueryIO is a useful connector for BigQuery, a scalable and serverless data warehouse. In this example, you read from BigQuery and use a standard SQL statement to retrieve results from BigQuery. When using standard SQL, Dataflow will submit the query to BigQuery and first retrieve metadata. BigQuery will then export the results to a temporary staging location in Cloud Storage and then Dataflow will read the contents from there, prioritizing throughput. Once the data is read from Cloud Storage, you can then map the results to be used in the data pipeline. The BigQuery Storage API is built to facilitate consumption from distributed processing frameworks, such as Beam. This Storage API allows you to achieve very high throughput when reading from BigQuery. To invoke the Storage API read method, use the DIRECT_READ method. In this example, you are using the column projection feature, withSelectedFields, to reduce the number of columns accessed by BigQuery. There is also a simple predicate filter that you can pass through for row filtering by invoking the withRowRestriction clause. You can write to BigQuery with BigQueryIO. There are multiple ways to specify schema, but using the built-in schema functionality is a quick and efficient way to specify it. You will be exploring schemas in more depth later on in this course. In this Java example, you are utilizing dynamic destinations to route your writes to various tables in BigQuery. Dynamic destinations allow you to write to multiple destinations-- in this case tables-- with varying schemas as well. BigQueryIO can write to BigQuery with streams or batches. Your streaming job can write to BigQuery in batches by windowing your data and using the FILE_LOADS method. By default, your streaming job will default to the streaming write method. Here's a Python example of a BigQuery dynamic destination write. You can define the function to return the destination you would like the data routed to.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.