Common cleaning functions from tm

Now that you know two ways to make a corpus, you can focus on cleaning, or preprocessing, the text. First, you'll clean a small piece of text, then you will move on to larger corpora.

In bag of words text mining, cleaning helps aggregate terms. For example, it might make sense for the words "miner", "mining" and "mine" to be considered one term. Specific preprocessing steps will vary based on the project. For example, the words used in tweets are vastly different than those used in legal documents, so the cleaning process can also be quite different.

Common preprocessing functions include:

tolower(): Make all characters lowercase
removePunctuation(): Remove all punctuation marks
removeNumbers(): Remove numbers
stripWhitespace(): Remove excess whitespace

tolower() is part of base R, while the other three functions come from the tm package. Going forward, we'll load tm and qdap for you when they are needed. Every time we introduce a new package, we'll have you load it the first time.

The variable text, containing a sentence, is shown in the script.

Apply each of the following functions to text, simply printing results to the console:

- `tolower()`
- `removePunctuation()`
- `removeNumbers()`
- `stripWhitespace()`

Take Hint (-30 XP)

Exercise

Common cleaning functions from tm

Instructions