1. Uniformity
Stellar work on chapter 2! You're now an expert at handling categorical and text variables.
2. In this chapter
In this chapter, we're looking at more advanced data cleaning problems, such as uniformity, cross field validation and dealing with missing data.
3. Data range constraints
In chapter 1, we saw how out of range values are a common problem when cleaning data, and that when left untouched, can skew your analysis.
4. Uniformity
In this lesson, we're going to tackle a problem that could similarly skew our data, which is unit uniformity.
For example, we can have temperature data that has values in both Fahrenheit and Celsius, weight data in Kilograms and in stones, dates in multiple formats, and so on.
Verifying unit uniformity is imperative to having accurate analysis.
5. An example
Here's a dataset with average temperature data throughout the month of March in New York City.
The dataset was collected from different sources with temperature data in Celsius and Fahrenheit merged together.
We can see that unless a major climate event occurred,
6. An example
this value here is most likely Fahrenheit, not Celsius. Let's confirm the presence of these values visually.
7. An example
We can do so by plotting a scatter plot of our data. We can do this using matplotlib.pyplot, which was imported as plt.
We use the plt dot scatter function, which takes in what to plot on the x axis, the y axis, and which data source to use.
We set the title, axis labels with the helper functions seen here, show the plot with plt dot show,
8. Insert title here...
and voila.
9. Insert title here...
Notice these values here? They all must be fahrenheit.
10. Treating temperature data
A simple web search returns the formula for converting Fahrenheit to Celsius.
To convert our temperature data, we isolate all rows of temperature column where it is above 40 using the loc method.
We chose 40 because it's a common sense maximum for Celsius temperatures in New York City.
We then convert these values to Celsius using the formula above, and reassign them to their respective Fahrenheit values in temperatures.
We can make sure that our conversion was correct with an assert statement, by making sure the maximum value of temperature is less than 40.
11. Treating date data
Here's another common uniformity problem with date data. This is a DataFrame called birthdays containing birth dates for a variety of individuals.
It has been collected from a variety of sources and merged into one.
12. Treating date data
Notice the dates here? The one in blue has the month, day, year format, whereas the one in orange has the month written out. The one in red is obviously an error, with what looks like a day day year format. We'll learn how to deal with that one as well.
13. Datetime formatting
We already discussed datetime objects. Without getting too much into detail, datetime accepts different formats that help you format your dates as pleased.
The pandas to datetime function automatically accepts most date formats, but could raise errors when certain formats are unrecognizable.
You don't have to memorize these formats, just know that they exist and are easily searchable!
14. Treating date data
You can treat these date inconsistencies easily by converting your date column to datetime.
We can do this in pandas with the to_datetime function. However this isn't enough and will most likely return an error, since we have dates in multiple formats, especially the weird day/day/format which triggers an error with months.
Instead we set the infer_datetime_format argument to True, and set errors equal to coerce. This will infer the format and return missing value for dates that couldn't be identified and converted instead of a value error.
15. Treating date data
This returns the birthday column with aligned formats, with the initial ambiguous format of day day year, being set to NAT, which represents missing values in Pandas for datetime objects.
16. Treating date data
We can also convert the format of a datetime column using the dt dot strftime method, which accepts a datetime format of your choice.
For example, here we convert the Birthday column to day month year, instead of year month day.
17. Treating ambiguous date data
However a common problem is having ambiguous dates with vague formats.
For example, is this date value set in March or August?
Unfortunately there's no clear cut way to spot this inconsistency or to treat it.
Depending on the size of the dataset and suspected ambiguities, we can either convert these dates to NAs and deal with them accordingly.
If you have additional context on the source of your data, you can probably infer the format.
If the majority of subsequent or previous data is of one format, you can probably infer the format as well.
All in all, it is essential to properly understand where your data comes from, before trying to treat it, as it will make making these decisions much easier.
18. Let's practice!
Now let's make our data uniform!