This chapter covers techniques to detect outliers in 1-dimensional data using histograms, scatterplots, box plots, z-scores, and modified z-scores.

What are anomalies and outliers?

Print a 5-number summary

Histograms for outlier detection

Scatterplots for outlier detection

Box plots and IQR

Boxplots for outlier detection

Calculating outlier limits with IQR

Using outlier limits for filtering

Using z-scores for Anomaly Detection

Finding outliers with z-scores

Using modified z-scores with PyOD

Detecting Univariate Outliers

In this chapter, you’ll learn the ins and outs of how the Isolation Forest algorithm works. Explore how Isolation Trees are built, the essential parameters of PyOD's IForest and how to tune them, and how to interpret the output of IForest using outlier probability scores.

Getting started with Isolation Forests

The difference between univariate and multivariate anomalies

Detecting outliers with IForest

Overview of Isolation Forest hyperparameters

Most important IForest parameters

Choosing contamination

Choosing n_estimators

Checking the theory

Hyperparameter tuning of Isolation Forest

Tuning contamination

Tuning multiple hyperparameters

Interpreting the output of IForest

Alternative way of classifying with IForest

Using outlier probabilities

Isolation Forests with PyOD

After a tree-based outlier classifier, you will explore a class of distance and density-based detectors. KNN and Local Outlier Factor classifiers have been proven highly effective in this area, and you will learn how to use them.

KNN for outlier detection

KNN for the first time

KNN with outlier probabilities

Outlier-robust feature scaling

Finding the euclidean distance manually

Finding the euclidean distance with SciPy

Practicing standardization

Testing QuantileTransformer

Hyperparameters of KNN

Differentiating distance metrics

Calculating manhattan distance manually

Tuning n_neighbors

Tuning the aggregation method

Local Outlier Factor

LOF for the first time

LOF with outlier probabilities

Distance and Density-based Algorithms

In this chapter, you’ll learn how to perform anomaly detection on time series datasets and make your predictions more stable and trustworthy using outlier ensembles.

Introduction to time series

Working with DateTime columns

Creating a DateTimeIndex

MAD on time series

Isolation Forest on time series

Time Series Decomposition for Outlier Detection

Practicing decomposition

Fitting on residuals

Outlier classifier ensembles

Scaling parts of a dataset

Manual outlier ensembles - creating the arrays

Storing outlier probabilities

Aggregating and thresholding the probabilities

How to deal with identified outliers

Classifying the reasons for outlier presence

When to drop outliers

Non-aggressive methods of dealing with outliers

Congratulations!

Time Series Anomaly Detection and Outlier Ensembles

Anomalies are present in almost any dataset, and it is critical to detect and deal with them before continuing statistical exploration. This course will teach you to use Python for various anomaly detection methods. You'll visually identify outliers and apply statistical methods and techniques for univariate and multivariate data. Additionally, you'll discover how to combine multiple outlier classifiers for a reliable final estimate. Adding this skill to your existing Python repertoire will help you with data cleaning, fraud detection, and identifying system disturbances.

<h2>Spot Anomalies in Your Data Analysis</h2><br>
Extreme values or anomalies are present in almost any dataset, and it is critical to detect and deal with them before continuing statistical exploration. When left untouched, anomalies can easily disrupt your analyses and skew the performance of machine learning models.<br>
<br>
<h2>Learn to Use Estimators Like Isolation Forest and Local Outlier Factor</h2><br>
In this course, you'll leverage Python to implement a variety of anomaly detection methods. You'll spot extreme values visually and use tested statistical techniques like Median Absolute Deviation for univariate datasets. For multivariate data, you'll learn to use estimators such as Isolation Forest, k-Nearest-Neighbors, and Local Outlier Factor. You'll also learn how to ensemble multiple outlier classifiers into a low-risk final estimator. You'll walk away with an essential data science tool in your belt: anomaly detection with Python.<br>
<br>
<h2>Expand Your Python Statistical Toolkit</h2><br>
Better anomaly detection means better understanding of your data, and particularly, better root cause analysis and communication around system behavior. Adding this skill to your existing Python repertoire will help you with data cleaning, fraud detection, and identifying system disturbances.

Supervised Learning with scikit-learn

Use Python and estimators like isolation forest and local outlier factor to spot anomalies in your data, with this four-hour course on anomaly detection. 


Anomaly Detection in Python

Detect anomalies in your data analysis and expand your Python statistical toolkit in this four-hour course.


Kaggle Master, Data Science Content Creator

Survey example

Big Mart sales data

Encode categoricals

evaluate_outlier_classifier

evaluate_regressor

Look at the output

Cartesian product

Inside the loop

Looking at the output

Parallel execution

Let's practice!

https://s3.amazonaws.com/assets.datacamp.com/production/course_29802/subtitles/course_29802_d254085abe58c75e6ac2b8023169fc4f.vtt

Isolation Forests with PyOD - Hyperparameter tuning of Isolation Forest

Hyperparameter tuning of Isolation Forest

Create Your Free Account