1. Learn
    2. /
  2. Courses
    3. /
  3. Introduction to Spark SQL in Python

Exercise

Practicing array column

The SQL function udf is available, as well as a dataframe df_before is available, of type DataFrame[doc: array<string>, in: array<string>, out: array<string>].

The TRIVIAL_TOKENS variable is a set. It contains certain words that we want to remove.

Instructions

100 XP
  • Show the rows of df_before where doc contains the item 5.
  • Create a udf that removes items in TRIVIAL_TOKENS from an array column. The order does not need to be preserved.
  • Remove tokens from the in and out columns in df2 that appear in TRIVIAL_TOKENS.