High Availability
1. High Availability
person: In this final section we will look at different high availability configurations for your dataflow pipelines. This section only applies to streaming pipelines. High availability is a hard requirement for some use cases. If you are processing financial transactions or identifying cybersecurity threats in an event stream, there are very real external risks if your pipeline goes down. your specific needs should be assessed when considering how to implement a highly available architecture of your own. When considering high availability, you need to take three factors into consideration. First downtime. How much downtime can your operation tolerate without breaking business continuity? Many organizations define recovery time objectives, or RTO to articulate this upper link. Second data loss. How much of your data can your application afford to lose in the event of an outage? ET managers will often use the term recovery point objectives or RPOs to describe this requirement. Third, cost. Running in a highly available configuration doesn't come for free, and it is important to consider how much your business is willing to pay to ensure that their data pipelines reach sufficient reliability standards. Now that we've discussed the consideration, let's look at a couple of possible configurations on dataflow. You can choose to make redundant sources that are available in multiple regions. In this example architecture, you can maintain two independent subscriptions in two different regions that are reading from the same topic. If a regional outage occurs, you can start a replacement pipeline in the second region and have the pipeline consume data from the backup subscription. If a region goes offline, you can start a new pipeline in a different region immediately to continue processing. Your application might drop data in the process as the intermediate data in the original pipeline will be dropped. However, you can replay the backup subscription to an appropriate time to keep data loss at a minimum if you're coordinating pub/sub snapshots between the two subscriptions. Using a multi-regional sync can also ensure that your new pipeline will be able to write to the sync without degrading latency. Downstream applications must know how to switch to the running pipelines output. Since only the source data is duplicated, it is more cost efficient than other alternative high availability configurations. If your application cannot tolerate data loss, run duplicate web pipelines in parallel in two different regions. Your pipelines will consume the same data from two different subscriptions, process data using workers in different regions, and write to multi-regional sinks in each location. This architectures provides geographical redundancy and fault tolerance. We have already discussed how pub/sub's global architecture makes setting up redundant subscriptions in different regions really easy. Dataflow workers can only work in one zone per job. By running parallel pipelines in separate Google Cloud regions, you can insulate your jobs from failures that affect a single region. Using multi-regional storage locations for your data syncs is not a requirement, but provides you one extra degree of fault tolerance. Applications that feed from the process data sets must have a way to switch to the running pipelines output. In the example above, we are running pipelines in two different continents, America and Europe. However, this would be the approach if you were running on the same continent, but in different regions. For example, running redundant pipelines in US Central one and US East one. This architecture basically offers you zero downtime, even where we'll have multiple instances of your pipeline running. Similarly, as your data is being processed in multiple regions, data loss is extremely unlikely. However, since you are duplicating resources across the entire stack, this approach is the most expensive high availability configuration. This is the end of this module. You should now be able to take snapshots of your data fill pipeline for disaster recovery requirements.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.