In this chapter, you will be introduced to the NannyML library and its fundamental functions. Initially, you will learn the process of preparing raw data to create reference and analysis sets ready for production monitoring. As a practical example, you will investigate predicting the tip amount for taxi rides in New York. Toward the end of the chapter, you will also discover how to estimate the performance of the tip prediction model using NannyML.

What is NannyML?

Key features of NannyML

Load the dataset

Data preparation for NannyML

Options

Reference period 

Analysis period

Reference or analysis period?

Loading and splitting the data

Creating reference and analysis set

Performance estimation

Specify the algorithm and problem type

Interpreting results

CBPE and DLE workflow

Performance estimation for tip prediction

Data Preparation and Performance Estimation

In this chapter, you will be introduced to realized performance calculators used when ground truth becomes available. You will learn about the more advanced methods for handling results, including filtering, plotting, converting them to data frames, chunking, and establishing custom thresholds. Lastly, you'll apply this knowledge to calculate the business value of a model trained on the hotel booking dataset.

When labels are available

When performance estimation is off

Comparing estimated and realized performance

Working with calculated and estimated results

Different chunking methods

Modifying the thresholds

Interacting with results

Business value calculation and estimation

Business value calculation

Drop in monetary value

Business calculation for hotel booking dataset

Monitoring Performance and Business Value

Having detected the performance degradation in the hotel booking model, you will now learn how to identify the underlying issue causing it. In this chapter, you will be introduced to multivariate and univariate drift detection methods. You will also learn how to identify data quality issues and how to address the underlying problems you detect.

Multivariate drift detection

Identifying relevant drifts

Drift in hotel booking dataset

Univariate drift detection

Univariate drift detection for hotel booking dataset

Ranking the univariate results

Visualizing drifting features

Data quality and statistic checks

Data quality checks

Summary statistics

Issue resolution

What is the resolution?

Should you do nothing or not?

Implementing a monitoring workflow

Congratulations

Root Cause Analysis and Issue Resolution

Green Taxi Dataset

Hotel Booking Analysis Dataset

Hotel Booking Reference Dataset

Monitoring machine learning models ensures the long-term success of your machine learning projects. Monitoring can be very complex. However, there are Python packages to help us understand how our models are performing, what data has changed that might have led to a drop in performance, and give us clues on what we need to do to get our models back on track. This course covers everything you need to know to build a basic monitoring system in Python, using the popular monitor package, nannyml.


<h2>Learn how to monitor your ML Models in Python</h2>

Monitoring machine learning models ensures the long-term success of your machine learning projects. Monitoring can be very complex, however, there are Python packages to help us understand how our models are performing, what data has changed that might have led to a drop in performance, and give us clues on what we need to do to get our models back on track. This course covers everything you need to know to build a basic monitoring system in Python, using the popular monitor package, nannyml.

<h2>Understand the optimal monitoring workflow</h2>

Model monitoring is not only about simply calculating model performance in production. Unfortunately, it is not that easy. Especially when labels are hard to come by. This course will teach you about the optimal monitoring workflow. It will ensure that you always catch model failures, avoid alert fatigue, and quickly get to the root of the issue.

<h2>Learn how to find the root cause of model performance issues</h2>

Another important component to model monitoring is root cause analysis. This course will dive into how to use data drift detection techniques to get to the root cause of model performance issues. You will learn how to use both univariate and multivariate data drift detection techniques to uncover potential root causes of model issues.



Monitoring Machine Learning Concepts

This course covers everything you need to know to build a basic machine learning monitoring system in Python

Monitoring Machine Learning in Python

Machine Learning Engineer

Machine Learning in Production in Python

Specify the algorithm and problem type

Monitoring Machine Learning in Python

Hands-on interactive exercise