The first chapter will explain why businesses need to monitor your machine  learning models in production. You will learn about the ideal monitoring  workflow and the steps involved, as well as some of the challenges that  monitoring systems can face in production.

Why you need to monitor your model

Why models fail?

The benefits of monitoring systems

The ideal monitoring workflow

The importance of monitoring KPIs

Ideal monitoring workflow

Monitoring workflow in real-life scenario

Challenges of monitoring ML models

Delayed ground truth

Covariate shift vs concept drift

What is ML Monitoring 

In Chapter 2, you'll discover the fundamental importance of performance  monitoring in a reliable monitoring system. We'll explore the common challenges faced in real-world production environments, such as the availability of ground truth. By the end of the chapter, you'll know how to handle situations when ground truth data is delayed or absent , using performance estimation algorithms.

Monitoring technical performance directly

Why performance first?

Different covariate shifts

Availability of ground truth

Is performance estimation required?

Options

True

False

Performance monitoring in production

Performance estimation

CBPE considerations

Performance estimation in production

CBPE

Algorithms for performance estimation

Theoretical Concepts of monitoring 

Now that you know the basics of covariate shift and concept drift in production, let''s dive a little bit deeper. At the end of this chapter, you will know the different ways to detect and handle them in real-world scenarios.

What is covariate shift?

Swapped features

Drift detection method

How to detect covariate shift

Subtle changes

Different methods

What is concept drift?

Type of concept drift

Concept drift and covariate shift interaction

How to handle concept drift?

Real-time adaptation

Concept drift detection and resolution

Wrap-up

Covariate Shift and Concept Drift Detection

To ensure the long-term success of machine learning models, it's essential to understand how to effectively monitor them in production. As machine learning becomes more widely adopted in the business world, post-deployment data science is emerging as an important field. This course covers all the essential concepts related to monitoring machine learning systems in production, in order to maintain business value, reduce the risk of failure, and increase visibility.

<h2> Machine Learning Monitoring Concepts</h2>
Machine learning models influence more and more decisions in the real world. These models need monitoring to prevent failure and ensure that they provide business value to your company. This course will introduce you to the fundamental concepts of creating a robust monitoring system for your models in production.
<br><br>
<h2>Discover the Ideal Monitoring Workflow</h2>
The course starts with the blueprint of where to begin monitoring in production and how to structure the processes around it. We will cover basic workflow by showing you how to detect the issues, identify root causes, and resolve them with real-world examples.
<br><br>
<h2>Explore the Challenges of Monitoring Models in Production</h2>
Deploying a model in production is just the beginning of the model lifecycle. Even if it performs well during development, it can fail due to continuously changing production data. In this course, you will explore the difficulties of monitoring a model’s performance, especially when there’s no ground truth.
<br><br>
<h2> Understand in Detail Covariate Shift and Concept Drift</h2>
The last part of this course will focus on two types of silent model failure. You will understand in detail the different kinds of covariate shifts and concept drift, their influence on the model performance, and how to detect and prevent them.

MLOps Concepts

Supervised Learning with scikit-learn

Learn about the challenges of monitoring machine learning models in production, including data and concept drift, and methods to address model degradation.

Machine Learning Monitoring Concepts

Monitoring Machine Learning Concepts

Ingenieur für maschinelles Lernen

Machine Learning in der Produktion in Python

Monitoring performance is at the core of monitoring workflow

Car arrival prediction is an example of when ground truth is immediately available

Realized performance is calculated using ground truth

Insurance pricing is an example of when ground truth is immediately available

Shifts close to the decision boundary don't have a negative impact on performance

If the ground truth in production is absent, then monitoring the performance is not necessary

Performance monitoring in production

Monitoring Machine Learning Concepts

Interaktive Übung