Predictions
1. Predictions
person: We've talked about the time to train, but there is another aspect to performance: predictions. During inference, you'll have performance considerations as well. If you're doing batch prediction, the considerations are very similar to that of training. You're concerned with things such as time. How long does it take for you to do all of your predictions? And this might be driven by a business need as well. So, for example, if you're doing product recommendations for the next day, you might want recommendations for the top 20% of users precomputed and available in, say, five hours if it takes 18 hours to do the full training. You'll also want to consider cost. "What predictions are you doing, and how much do you precompute" is going to be driven by cost considerations. And then there's scale. Do you have to do all of this on a single machine, or can you distribute it, say, to multiple workers? What kind of hardware is available on these workers? Do they, for example, have GPUs? If you are doing online prediction, the performance considerations are quite different. This is because the end user is actually waiting for the prediction. So let's take a look at how it's different. You typically cannot distribute the prediction graph. Instead, you carry out the computation for one end user on one machine. However, you almost always scale out the predictions onto multiple workers. Essentially, each prediction is handled by a microservice, and you can replicate and scale out the predictions using Kubernetes or App Engine. Cloud ML Engine predictions are a higher-level abstraction, but they are equivalent to doing this. The performance consideration is not how many training stamps you can carry out per minute, but how many queries you can handle per second. The unit of this, queries per second, is often called QPS. That's the performance target that you need to hit. When you design for higher performance, you want to consider training and performance separately, especially if you will be doing online predictions. As I kind of suggested in my line about precomputing batch predictions for the top 20% of users and handling the rest of your users via online prediction, performance considerations will also involve striking the right balance, and ultimately, you will know the exact trade-off-- is it 20% or 10% or 25%?-- only after you build your system and start to measure things. However, unless you plan to be able to do both batch predictions and online predictions, you will be stuck with a solution that doesn't meet all of your needs. The idea behind this module and this course in general is so that you're aware of all of the possibilities. Once you're aware that it can be done, it's not actually all that difficult to accomplish. The technical part is usually quite straightforward, especially if you're using TensorFlow on a capable Cloud platform.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.