Training design decisions
1. Training design decisions
person: One of the key decisions you'll need to make about your production ML system concerns training. Here's a question. How is physics unlike fashion? If we assume that science is about discovering relationships that already exist in the world, the answer is that physics is constant whereas fashion isn't. To see some proof, just look at some old pictures of yourself. Now you might be asking, why is this relevant? Well, when making decisions about training, you have to decide whether the phenomenon you're modeling is more like physics or like fashion. When training your model, there are two paradigms: static training and dynamic training. In static training, we do what we did in the last specialization. We gather our data, we partition it, we train our model, and then we deploy it. In dynamic training, we do this repeatedly as more data arrives. This leads to the fundamental trade-off between static and dynamic. Static is simpler to build and test, but will probably become stale. Whereas dynamic is harder to build and test, but will adapt to changes. And the tendency to become or not become stale is what we alluded to earlier when we contrasted physics and fashion. If the relationship you're trying to model is constant, like physics, a statically trained model may be sufficient. If, in contrast, the relationship you're trying to model is one that changes, like fashion, the dynamically trained model might be more appropriate. Part of the reason the dynamic is harder to build and test is that new data may have all sorts of bugs in it. And that's something we'll talk about more deeply in a later module on designing adaptable ML systems. Engineering might also be harder, because we need more monitoring, model rollback, and data quarantine capabilities. Let's explore some use cases and think about which sort of training style would be most appropriate. The first use case concerns spam detection. And the question you should ask yourself is, how fresh does spam detection need to be? You could do this as static, but spammers are a crafty and determined bunch. They will probably discover ways of passing whatever filter you impose within a short time. So dynamic is likely to be more effective over time. What about Android voice to text? Note that this question has some subtlety. For a global model, training off-line is probably fine. But if you want to personalize the voice recognition, you may need to do something online, or at least different, on the phone. So this could be static or dynamic, depending on whether you want global or personalized transcription. What about ad conversion rate? The interesting subtlety here is that conversions may come in very late. For example, if I'm shopping for a car online, I'm unlikely to buy for a very long time. This system could use dynamic training. Then regularly going back at different intervals to catch up on new conversion data that has arrived for the past. So in practice, most of the time you'll need to use dynamic, but you might start with static because it's simpler. In a reference architecture for static training, models are trained once and then pushed to AI platform. Now for dynamic training, there are three potential architectures to explore: Cloud Functions, App Engine, or Dataflow. In a general architecture for dynamic training using Cloud Functions, a new data file appears in Cloud Storage, and then the Cloud Function is launched. After that, the Cloud Function starts the AI Platform training job. And then the AI Platform writes out a new model. In a general architecture for dynamic training using App Engine, when a user makes a web request from a dashboard to App Engine, an AI Platform training job is launched, and the AI Platform job writes a new model to Cloud Storage. From there, the statistics of the training job are displayed to the user when the job is complete. The Dataflow pipeline is also possibly invoking the model for predictions. Here, the streaming topic is ingested into Pub/Sub from subscribers. Messages are then aggregated with Dataflow, and aggregated data is stored in BigQuery. AI Platform is launched on the arrival of new data in BigQuery, and then an updated model is deployed.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.