CommencerCommencer gratuitement

TFIDF Practice

Earlier you looked at a bag-of-words representation of articles on crude oil. Calculating TFIDF values relies on this bag-of-words representation, but takes into account how often a word appears in an article, and how often that word appears in the collection of articles.

To determine how meaningful words would be when comparing different articles, calculate the TFIDF weights for the words in crude, a collection of 20 articles about crude oil.

Cet exercice fait partie du cours

Introduction to Natural Language Processing in R

Afficher le cours

Instructions

  • Calculate TFIDF values for crude by article_id and by word. Save the resulting tibble as crude_weights.
  • Sort crude_weights with the arrange() function by descending tf_idf values.
  • Filter crude_weights to the lowest non-zero tf_idf values. Again, use the arrange function.

Exercice interactif pratique

Essayez cet exercice en complétant cet exemple de code.

# Create a tibble with TFIDF values
___ <- crude_tibble %>%
  unnest_tokens(output = "word", token = "words", input = text) %>%
  anti_join(stop_words) %>%
  count(article_id, word) %>%
  ___(___, ___, n)

# Find the highest TFIDF values
crude_weights %>%
  ___(desc(___))

# Find the lowest non-zero TFIDF values
crude_weights %>%
  filter(___ != ___) %>%
  ___(___)
Modifier et exécuter le code