1. Learn
    2. /
  2. Courses
    3. /
  3. Feature Engineering for NLP in Python
  • 1

    Basic features and readability scores

    Free

    Learn to compute basic features such as number of words, number of characters, average word length and number of special characters (such as Twitter hashtags and mentions). You will also learn to compute readability scores and determine the amount of education required to comprehend a piece of text.

  • 2

    Text preprocessing, POS tagging and NER

    In this chapter, you will learn about tokenization and lemmatization. You will then learn how to perform text cleaning, part-of-speech tagging, and named entity recognition using the spaCy library. Upon mastering these concepts, you will proceed to make the Gettysburg address machine-friendly, analyze noun usage in fake news, and identify people mentioned in a TechCrunch article.

  • 3

    N-Gram models

    Learn about n-gram modeling and use it to perform sentiment analysis on movie reviews.

  • 4

    TF-IDF and similarity scores

    Learn how to compute tf-idf weights and the cosine similarity score between two vectors. You will use these concepts to build a movie and a TED Talk recommender. Finally, you will also learn about word embeddings and using word vector representations, you will compute similarities between various Pink Floyd songs.

Initializing

Exercise

Word count of TED talks

ted is a dataframe that contains the transcripts of 500 TED talks. Your job is to compute a new feature word_count which contains the approximate number of words for each talk. Consequently, you also need to compute the average word count of the talks. The transcripts are available as the transcript feature in ted.

In order to complete this task, you will need to define a function count_words that takes in a string as an argument and returns the number of words in the string. You will then need to apply this function to the transcript feature of ted to create the new feature word_count and compute its mean.

Instructions

100 XP
  • Split string into a list of words using the split() method.
  • Return the number of elements in words using len().
  • Apply your function to the transcript column of ted to create the new feature word_count.
  • Compute the average word count of the talks using mean().