Shared VPC
1. Shared VPC
person: Another security feature you can use with Dataflow is shared VPC. Dataflow can run in networks that are either in the same project or in a separate project which we call the host project. When a network exists in a host project, we call the networking setup shared VPC. Shared VPC lets organization admins delegate administrative responsibilities, such as creating and managing instances, to others while maintaining centralized control over network resources like subnets, routes, and firewalls. When set to run in a shared VPC, Dataflow works in either a default or a custom network. The default network is the one automatically set by Google Cloud when you create a project. A custom network is one where you create the network and define the regions and the subnets in the network. When setting the number of workers to use, remember to have enough IP addresses available. For example, if you have a subnet with a /29 subnet and no other VMs running in it, the maximum number of Dataflow workers that you can launch is four. Finally, the Dataflow service account needs the Compute Network user role in the host project on either a project level or a subnet level. We show the difference between using the network and subnetwork flags here. In the Python example, the Dataflow service account has a compute network role set at the project level, and the user wants to deploy to the default network. In the Java example, the Dataflow service account's permissions are defined at the subnet level, and the user is launching the job in a custom subnet.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.