1. Cleaning text data
Good job on the previous lesson. In the final lesson of this chapter, we'll talk about text data and regular expressions.
2. What is text data?
Text data is one of the most common types of data types.
Examples of it range from names, phone numbers, addresses, emails and more.
Common text data problems include handling inconsistencies, making sure text data is of a certain length, typos and others.
3. Example
Let's take a look at the following example. Here's a DataFrame named phones containing the full name and phone numbers of individuals. Both are string columns. Notice the phone number column.
4. Example
We can see that there are phone number values, that begin with 00 or +. We also see that there is one entry where the phone number is 4 digits, which is non-existent.
Furthermore, we can see that there are dashes across the phone number column.
If we wanted to feed these phone numbers into an automated call system, or create a report discussing the distribution of users by area code, we couldn't really do so without uniform phone numbers.
5. Example
Ideally, we'd want to the phone number column as such. Where all phone numbers are aligned to begin with 00, where any number below the 10 digit value is replaced with NaN to represent a missing value, and where all dashes have been removed. Let's see how that's done!
6. Fixing the phone number column
Let's first begin by replacing the plus sign with 00, to do this, we use the dot str dot replace method which takes in two values, the string being replaced, which is in this case the plus sign and the string to replace it with which is in this case 00. We can see that the column has been updated.
7. Fixing the phone number column
We use the same exact technique to remove the dashes, by replacing the dash symbol with an empty string.
8. Fixing the phone number column
Now finally we're going to replace all phone numbers below 10 digits to NaN. We can do this by chaining the Phone number column with the dot str dot len method, which returns the string length of each row in the column. We can then use the dot loc method, to index rows where digits is below 10, and replace the value of Phone number with numpy's nan object, which is here imported as np.
9. Fixing the phone number column
We can also write assert statements to test whether the phone number column has a specific length,and whether it contains the symbols we removed.
The first assert statement tests that the minimum length of the strings in the phone number column, found through str dot len, is bigger than or equal to 10.
In the second assert statement, we use the str dot contains method to test whether the phone number column contains a specific pattern. It returns a series of booleans for that are True for matches and False for non-matches.
We set the pattern plus bar pipe minus, the bar pipe here is basically an or statement, so we're trying to find matches for either symbols.
We chain it with the any method which returns True if any element in the output of our dot-str-contains is True, and test whether the it returns False.
10. But what about more complicated examples?
But what about more complicated examples? How can we clean a phone number column that looks like this for example? Where phone numbers can contain a range of symbols from plus signs, dashes, parenthesis and maybe more.
This is where regular expressions come in. Regular expressions give us the ability to search for any pattern in text data, like only digits for example. They are like control + find in your browser, but way more dynamic and robust.
11. Regular expressions in action
Let's a look at this example. Here we are attempting to only extract digits from the phone number column. To do this, we use the dot str dot replace method with the pattern we want to replace with an empty string.
Notice the pattern fed into the method. This is essentially us telling pandas to replace anything that is not a digit with nothing.
We won't get into the specifics of regular expressions, and how to construct them, but they are immensely useful for difficult string cleaning tasks, so make sure to check out DataCamp's course library on regular expressions.
12. Let's practice!
Now that we know how to clean text data, let's get to practice!