CommencerCommencer gratuitement

Label the data

A dataframe df is available having columns endword: string, features: vector, and outvec: vector. You are to select the rows where endword equals "him", and add a column label having the integer value 1. Then, use the union operation to add an equal number of rows having endword not equals to him, such that these additional rows have label = 0.

As a reminder, in SQL the not equals comparison is done using <>.

Cet exercice fait partie du cours

Introduction to Spark SQL in Python

Afficher le cours

Instructions

  • Import the lit function.
  • Select the rows where endword is 'him' and add a integer column label having the value 1.
  • Select the rows where endword is not 'him' and add a integer column label having the value 0.
  • Union these two sets, using a number of negative examples equal to the number of positive examples.

Exercice interactif pratique

Essayez cet exercice en complétant cet exemple de code.

# Import the lit function
from pyspark.____ import lit

# Select the rows where endword is 'him' and label 1
df_pos = df.where("____ = 'him'")\
           .withColumn('label', lit(____))

# Select the rows where endword is not 'him' and label 0
df_neg = df.where("endword <> '____'")\
           .withColumn('label', ____(0))

# Union pos and neg in equal number
df_examples = df_pos.____(df_neg.limit(df_pos.count()))
print("Number of examples: ", df_examples.count())
df_examples.where("endword <> 'him'").sample(False, .1, 42).show(5)
Modifier et exécuter le code