Get startedGet started for free

TFIDF Practice

Earlier you looked at a bag-of-words representation of articles on crude oil. Calculating TFIDF values relies on this bag-of-words representation, but takes into account how often a word appears in an article, and how often that word appears in the collection of articles.

To determine how meaningful words would be when comparing different articles, calculate the TFIDF weights for the words in crude, a collection of 20 articles about crude oil.

This exercise is part of the course

Introduction to Natural Language Processing in R

View Course

Exercise instructions

  • Calculate TFIDF values for crude by article_id and by word. Save the resulting tibble as crude_weights.
  • Sort crude_weights with the arrange() function by descending tf_idf values.
  • Filter crude_weights to the lowest non-zero tf_idf values. Again, use the arrange function.

Hands-on interactive exercise

Have a go at this exercise by completing this sample code.

# Create a tibble with TFIDF values
___ <- crude_tibble %>%
  unnest_tokens(output = "word", token = "words", input = text) %>%
  anti_join(stop_words) %>%
  count(article_id, word) %>%
  ___(___, ___, n)

# Find the highest TFIDF values
crude_weights %>%
  ___(desc(___))

# Find the lowest non-zero TFIDF values
crude_weights %>%
  filter(___ != ___) %>%
  ___(___)
Edit and Run Code