Parameter server strategy
1. Parameter server strategy
person: Earlier, we explored the asynchronous parameter server architecture. A parameter server training cluster consists of Workers and ParameterServers. Variables are created on ParameterServers, and they are read and updated by Workers in each step. By default, Workers read and update these variables independently without synchronizing with each other. The TensorFlow parameter server strategy introduces a central coordinator. The Coordinator is a special task type that creates resources, dispatches training tasks, writes checkpoints and deals with task failures. You can create your parameter server strategy object just like you would for the other strategies. Note that you will need to parse in the ClusterResolver argument, and if training with AI platform, this is just a simple TFConfigClusterResolver. Using model.fit with parameter server training, requires that the input data be provided in a call of an object that takes a single argument of type TF distribute input context and returns TF data dataset. We then need to wrap our dataset function in tf.keras.utils.experimental. DatasetCreator. The code in dataset_fn will be invoked on the input device which is usually the CPU on each of the Worker machines. When using parameter server strategy, it is recommended that you shuffle and repeat your dataset and parse in the steps per epoch argument to model.fit.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.