Label the data
A dataframe df
is available having columns endword
: string, features
: vector, and outvec
: vector. You are to select the rows where endword
equals "him", and add a column label
having the integer value 1. Then, use the union
operation to add an equal number of rows having endword
not equals to him, such that these additional rows have label
= 0.
As a reminder, in SQL the not equals comparison is done using <>
.
This exercise is part of the course
Introduction to Spark SQL in Python
Exercise instructions
- Import the
lit
function. - Select the rows where endword is 'him' and add a integer column
label
having the value 1. - Select the rows where endword is not 'him' and add a integer column
label
having the value 0. - Union these two sets, using a number of negative examples equal to the number of positive examples.
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
# Import the lit function
from pyspark.____ import lit
# Select the rows where endword is 'him' and label 1
df_pos = df.where("____ = 'him'")\
.withColumn('label', lit(____))
# Select the rows where endword is not 'him' and label 0
df_neg = df.where("endword <> '____'")\
.withColumn('label', ____(0))
# Union pos and neg in equal number
df_examples = df_pos.____(df_neg.limit(df_pos.count()))
print("Number of examples: ", df_examples.count())
df_examples.where("endword <> 'him'").sample(False, .1, 42).show(5)