Serving design decisions
1. Serving design decisions
Just as the use case determines appropriate training architecture, it also determines the appropriate serving architecture. In designing our serving architecture, one of our goals is to minimize average latency. Just as in operating systems where we don't want to be bottlenecked by slow disk IO, when serving models, we don't want to be bottlenecked by slow to decide models. Remarkably, the solution for serving models is very similar to what we do to optimize IO performance. We use a cache. In this case, instead of faster memory, we'll use a table. Static serving then computes the label ahead of time and serves by looking it up in the table. Dynamic serving in contrast computes the label on demand. There's a space-time trade-off. Static serving is space intensive, resulting in higher storage costs because we store precomputed predictions with a low, fixed latency and lower maintenance costs. Dynamic serving, however, is compute intensive. It has lower storage costs, higher maintenance, and variable latency. The choice of whether to use static or dynamic serving is determined by considering how important costs are with regard to latency, storage, and CPU. Sometimes it can be hard to express the relative important of these three areas. As a result, it might be helpful to consider static and dynamic serving through another lens, peakedness and cardinality. Peakedness in a data distribution is a degree to which data values are concentrated around the mean, or in this case, how concentrated the distribution of the prediction workload is. You can also think of it as inverse entropy. For example, a model that predicts the next word based on the current word, which you might find in your mobile phone keyboard app would be highly peaked because a small number of words account for the majority of words used. In contrast, a model that predicted quarterly revenue for all sales verticals in order to populate a report would be right on the same verticals. And with the same frequently for each. And so, it would be very flat. Cardinality refers to the number of values in a set. In this case, the set is composed of all the possible things we might have to make predictions for. So, a model predicting sales revenue given organization division number would have fairly low cardinality. A model predicting lifetime value given a user friendly e-commerce platform would have high cardinality because the number of users and the number of characters of each user are probably quite large. Taken together peakedness and cardinality create a space. When the cardinality is sufficiently low, we can store the entire expected prediction workload. For example, the predicted sales revenue for all divisions in a table and use static serving. When the cardinality is high because the size of the input space is large and the workload is not very peaked, you probably want to use dynamic training. In practice though, a hybrid of static and dynamic is often chosen, where you statically cache some of the predictions while responding on demand for the long tail. This works best when the distribution is sufficiently peaked. The striped area above the curve and not inside the blue rectangle is suitable for a hybrid solution, with the most frequently requested predictions cached and the tail computed on demand. Let's try to estimate training and inference needs for the same use cases that we saw in the previous lesson. The first use case is predicting whether an e-mail is spam. What inference style is needed? Well, first we need to consider how peaked the distribution is. The answer is not at all. Most e-mails are probably different, although they may be very similar if generated programmatically. Depending on the choice of representation, the cardinality might be enormous, so this would be dynamic. The second use case is Android voice-to-text. This is, again, subtle. Inference is almost certain on line because there's such a long tail of possible voice clips. But maybe with sufficient signal processing, some key phrases like, "Okay, Google," may have pre-computed answers. So, this would be dynamic or hybrid. And the third use case is shopping ad conversion rate. The set of all ads doesn't change much from day to day, assuming users are comfortable waiting for a short while after uploading their ads. This could be done statically. And then the back script could be run at regular intervals throughout the day. This would be static. In practice, you'll often use a hybrid approach. You might not have realized it, but dynamic serving is what we have learned so far. Think back to the architecture of the systems we've used to make predictions. A model that lived in AI Platform was sent one or more instances and returned predictions for each. If you want to build a static serving system, you want to make three design changes. First, you need to change your call to AI Platform from an online prediction job to a batch prediction job. Second, you need to make sure that your model accepted and passed through keys as input. These keys will allow you to join your request to prediction at serving time. And third, you write the predictions to a data warehouse like BigQuery, and create an API to read from it.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.