1. Learn
    2. /
  2. Courses
    3. /
  3. Case Study: School Budgeting with Machine Learning in Python

Exercise

What's in a token?

Now you will use combine_text_columns to convert all training text data in your DataFrame to a single vector that can be passed to the vectorizer object and made into a bag-of-words using the .fit_transform() method.

You'll compare the effect of tokenizing using any non-whitespace characters as a token and using only alphanumeric characters as a token.

Instructions

100 XP
  • Import CountVectorizer from sklearn.feature_extraction.text.
  • Instantiate vec_basic and vec_alphanumeric using, respectively, the TOKENS_BASIC and TOKENS_ALPHANUMERIC patterns.
  • Create the text vector by using the combine_text_columns() function on df.
  • Using the .fit_transform() method with text_vector, fit and transform first vec_basic and then vec_alphanumeric. Print the number of tokens they contain.