1. The TFIDF
A simple bag-of-words representation is a great start.
In this lesson however, we will expand this idea and introduce the infamous TFIDF.
2. Bag-of-word pitfalls
Consider these three pieces of text.
We learn about John and Joe, that they are best friends, and that they like tacos. We can clean this text by
removing stop words and punctuation, and lower casing all words. This is a simple example, but if we just compare the words they have in common, text 1 and 2 would be a lot more similar than text 1 and 3.
3. Sharing common words
Let's compare the words from each text.
When comparing t1 and t2, they share almost every single word.
However, when comparing t1 and t3, they don't seem to share that many words. So t1 and t2 must be more similar than t1 and t3!
4. Tacos matter
Let's return to the original text. One word that should matter here, above all others, is tacos.
All three texts have John and Joe, but only two texts share
John, Joe, and tacos. This is the main idea behind why we use what is known as the term frequency-inverse document frequency matrix.
5. TFIDF
The TF-IDF is a way of representing word counts by considering two components. The term frequency,
which represents the proportion of words in a text that are that specific term. For example, clean_t1 has 4 words total, John is one of those 4, so the term frequency is .25.
Next, the inverse document frequency, which considers how frequent words appear relative to the full collection of text. For example, John appears in all 3 clean texts, so the IDF for John is 0. Let's look at how the IDF portion can be calculated.
6. IDF Equation
The inverse document frequency has several different calculation methods, but we will use the most common form.
Which is the log of the total number of documents, divided by the number of documents that contain the word.
We can quickly calculate the IDF for some of the words we saw on the previous slide. Taco appears in 2 out of 3 documents, so it gets an IDF weight of .405.
7. TF + IDF
Putting these two calculations together is straight forward.
The TFIDF value for Tacos for each text can be calculated. Consider clean text 3. Tacos is 1 out of 6 total words, and tacos has an IDF of .405. The TFIDF is the multiplication of these two values, which is .0675.
8. Calculating the TFIDF matrix
In order to quickly implement a TFIDF calculation, we can use tidytext's bind_tf_idf function.
This builds off of our previous steps of tokenization, removing stop words, and counting words. We add the bind_tf_idf function at the end, and tell it to calculate the weights
based on the word,
the document id,
and the word counts, which in this case comes from the count function. This function will calculate the term frequency and inverse document frequency values, and create a tibble.
9. bind_tf_idf output
The bind_tf_idf function quickly adds n, tf, idf, and tf_idf as columns to our output. We can see the tf and idf weights that have discussed on the previous slides. To get the tf_idf value, you simply multiple the two weights together.
10. TFIDF Practice
Let's look at a couple examples of to solidify our understanding of the TFIDF.