1. Why deal with missing data?
Welcome to the course on dealing with missing data in python.
I'm Suraj Donthi, a Deep Learning and Computer Vision Consultant. While I specialize in solving computer vision problems like vision for self-driving cars, video analytics of traffic on roads, people analytics in retail and public spaces, or biomedical image analysis, I've also extensively worked on analyzing and back testing trading strategies using time-series data.
In data science, the first and foremost task while working with any data for analysis is to clean the messy data.
2. Why does missing data exist?
Almost all real world data is messy data and a large portion of it includes missing values.
For instance, did you know that 72% of the organizations believe that data quality issues hinders their analysis, customer trust and perception!
3. Why does missing data exist?
Values might go missing during the data acquisition process, whether it is due to faulty sensors or due to unfilled information by humans.
Another prominent reason can be due to accidental data loss or deletion of records by ill-informed users.
There can be several other reasons for missingness. In this course you will dig deep into analyzing the causes of missingness and appropriately treat them.
4. In this course, you'll learn
This course will cover the significance of missing values, detecting missing values, analyzing the type of missingness and treating the missing values for all the frequently encountered data types namely, numerical, time-series and categorical values.
5. In this course, you'll learn
Lastly, you'll learn the most important step in dealing with missing data which is imputing them. You'll learn both the simple techniques as well as advanced techniques to deal with missing data.
Finally, you'll also learn to compare between various imputation techniques both statistically and visually.
6. Workflow for treating missing values
To be concise, the workflow for dealing with missing data is detect and convert all missing values to null values, analyze the amount and type of missingness, delete or impute them accordingly and finally choose the best imputation method by evaluating their performance.
Before we start of with treating missing values, let's get familiar with the NULL value operations
7. NULL value Operations
There are two types of null values that are to be considered. One is the 'None' type, which is a built-in python DataType. And the other is 'np.nan' of Numpy, which stands for Not a Number.
Both Numpy and Pandas libraries use 'np.nan' extensively for working with null values.
Let's compare the differences between the two.
Performing logical operations as 'None or True' returns True while 'np.nan or True' returns np.nan itself as the output.
'None' does not support arithmetic operations like addition or division and returns a 'TypeError'. However, 'np.nan' does not show any difference in its output.
Note here that 'np.nan' is equivalent to undefined and any operation on undefined is undefined that is 'np.nan'
The reason the types 'None' and 'np.nan' behave this way can be figured out by checking their types.
While 'None' is of NoneType and supports logical operations only, surprisingly 'np.nan' is float and as a result we see that it supports both logical and arithmetic operations.
8. NULL value operations
Checking for null values is a very important task in our course, hence checking for 'None' equals to None' returns 'True' while this is not the case of 'np.nan' which states that an undefined number cannot be equal and thereby returns 'False'.
Instead, the correct way to check for 'nan' is by using the function 'np.isnan()'. You must note that finding null values this way does not work for 'None' type.
9. Let's practice!
Now, let's dive in to practice!