IniziaInizia gratis

Label the data

A dataframe df is available having columns endword: string, features: vector, and outvec: vector. You are to select the rows where endword equals "him", and add a column label having the integer value 1. Then, use the union operation to add an equal number of rows having endword not equals to him, such that these additional rows have label = 0.

As a reminder, in SQL the not equals comparison is done using <>.

Questo esercizio fa parte del corso

Introduction to Spark SQL in Python

Visualizza il corso

Istruzioni dell'esercizio

  • Import the lit function.
  • Select the rows where endword is 'him' and add a integer column label having the value 1.
  • Select the rows where endword is not 'him' and add a integer column label having the value 0.
  • Union these two sets, using a number of negative examples equal to the number of positive examples.

Esercizio pratico interattivo

Prova a risolvere questo esercizio completando il codice di esempio.

# Import the lit function
from pyspark.____ import lit

# Select the rows where endword is 'him' and label 1
df_pos = df.where("____ = 'him'")\
           .withColumn('label', lit(____))

# Select the rows where endword is not 'him' and label 0
df_neg = df.where("endword <> '____'")\
           .withColumn('label', ____(0))

# Union pos and neg in equal number
df_examples = df_pos.____(df_neg.limit(df_pos.count()))
print("Number of examples: ", df_examples.count())
df_examples.where("endword <> 'him'").sample(False, .1, 42).show(5)
Modifica ed esegui il codice