How does Vertex AI help with the MLOps workflow, part 2?

1. How does Vertex AI help with the MLOps workflow, part 2?

Let’s continue with MLOps capabilities of Vertex AI. The next capability is understanding model behavior which reveals the “why” behind your model and predictions. You can use Vertex Explainable AI to understand your model's outputs for classification and regression tasks. Vertex Explainable AI tells you how much each feature in the data contributes to the predicted result and identifies model bias. Vertex Explainable AI is a fully managed service on Vertex AI that offers feature-based explanations. By integrating “feature attributions” or “feature importance” into Vertex AI, Vertex Explainable AI provides a better understanding of model decision making. Feature attributions are an explainability method that shows you how much each input feature contributed to your model’s predictions and to the model’s overall predictive power by using sampled Shapley, Integrated gradients, and eXplanation with Ranked Area Integrals (XRAI). When you request predictions, you get predicted values that are appropriate for your model. When you request explanations, you get the predictions along with feature attribution information. Explainable AI is built into multiple Vertex AI services. You can currently get feature attributions in: Vertex AI Prediction AutoML Tables And Vertex AI Workbench Feature attribution is supported for all types of models, both AutoML and custom-trained, frameworks like TensorFlow, scikit-learn, or XGBoost, and modalities such as images, text, tabular, or video. Vertex Explainable AI offers three methods to use for feature attributions: sampled Shapley, integrated gradients, and eXplanation with Ranked Area Integrals, or XRAI. Each feature attribution method is based on Shapley values, which are a cooperative game theory algorithm that assigns credit to each player in a game for a particular outcome. Applying Shapley values to ML models means that each model feature is treated as a "player" in the game. Vertex Explainable AI assigns proportional credit to each feature for the outcome of a particular prediction. For a thorough comparison of attribution methods, see the AI Explanations Whitepaper and check the Feature-based explanations documentation. The next MLOps capability of Vertex AI is proactively monitoring your model performance by using Vertex AI Model Monitoring. A model deployed in production performs best on prediction when the input data is similar to the training data. When the input data deviates from the data used to train the model, the model's performance can deteriorate, even if the model itself hasn't changed. To help you maintain a model's performance, Vertex AI provides a host of products to monitor and govern your models. Vertex AI Model Monitoring maintains your model's performance by monitoring the model's prediction input data for feature skew and drift: Training-serving skew occurs when the feature data distribution in production deviates from the feature data distribution used to train the model. If the original training data is available, you can enable skew detection to monitor your models for training-serving skew. Prediction drift occurs when feature data distribution in production changes significantly over time. If the original training data isn't available, you can enable drift detection to monitor the input data for changes over time. You can enable both skew and drift detection on Vertex AI Model Monitoring, but you should consider skew detection if you provide the original training dataset for your model. Otherwise, you should enable drift detection. For more information, see Introduction to Vertex AI Model Monitoring. The next capability is tracking and comparing multiple experiment runs and analyzing main model metrics. The goal when developing a model for a problem is to identify the best model for that particular use case. Vertex AI hosts different products to monitor and govern your models. For example, you can use Vertex AI Experiments to track, analyze, compare, and search across different model architectures, different ML frameworks, such as TensorFlow, PyTorch or scikit-learn, different hyperparameters and different training environments. In order to understand the nature of Vertex AI Experiments, let’s look at some crucial terms. Recall that an artifact is a discrete entity or piece of data produced and consumed by an ML workflow. A context is used to group artifacts and executions together under a single, queryable, and typed category. Contexts can be used to represent sets of metadata. An example of a “context” can be a run of ML pipeline. Vertex AI Experiments is a context in Vertex ML Metadata where an experiment can contain n experiment runs in addition to n pipeline runs. An experiment run consists of parameters, summary metrics, time series metrics, artifacts, executions and Vertex AI resources such as PipelineJob, which is created when users want to run an ML pipeline on Vertex AI Pipelines. Another tool that lets you track, visualize, and compare ML experiments and share them with your team is Vertex AI TensorBoard. Open source TensorBoard (TB) is a Google open source project for ML experiment visualization. Vertex AI TensorBoard is an enterprise-ready managed version of TensorBoard. Executions and artifacts of a pipeline run are viewable in the Google Cloud console. Vertex AI TensorBoard provides various detailed visualizations that include: Tracking and visualizing metrics such as loss and accuracy over time. Visualizing model computational graphs such as ops and layers. Viewing histograms of weights, biases, or other tensors as they change over time. Projecting embeddings to a lower dimensional space. And Displaying image, text, and audio samples. Check out the documentation to learn more about Vertex AI TensorBoard features and capabilities. You can also simplify your MLOps processes with Vertex AI Tabular Workflows. Vertex AI Tabular Workflow for End-to-End AutoML is a managed instance of Vertex AI Pipelines. It is a set of integrated, fully managed, and scalable pipelines for end-to-end ML with tabular data that uses Google's AutoML technology for model development and providing customization options to fit your needs. The following are some of the benefits of Tabular Workflow for End-to-End AutoML : Supports large datasets that are multiple TB in size. Lets you improve stability and lower training time by limiting the search space of architecture types or skipping architecture search. Lets you improve training speed by manually selecting the hardware used for training and architecture search. Lets you reduce model size and improve latency with distillation or by changing the ensemble size. Additionally, each AutoML component can be inspected in a powerful pipelines graph interface that lets you see the transformed data tables, evaluated model architectures, and many more details. These AutoML components also provide extended flexibility and transparency, such as being able to customize parameters, hardware, view process status, logs and more. In this section, you learned about Vertex AI MLOps tools that help you collaborate across ML teams and improve your models through predictive model monitoring, alerting, diagnosis, and actionable explanations. All of these tools are modular, so you can integrate them into your existing systems as needed. In the next section, which is the final section of the course, you review the main concepts you learned throughout this course.

2. Let's practice!

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.