ComeçarComece de graça

Aggregating in RDDs

Now that you have conducted analytics with DataFrames in PySpark, let's briefly do a similar task with an RDD. Using the provided code, get the sum of the values of an RDD in PySpark.

A Spark session called spark has already been made for you.

Este exercício faz parte do curso

Introduction to PySpark

Ver curso

Instruções do exercício

  • Create an RDD from the provided DataFrame.
  • Apply the provided Lambda Function to the keys of the RDD.
  • Collect the results of the aggregation.

Exercício interativo prático

Experimente este exercício completando este código de exemplo.

# DataFrame Creation
data = [("HR", "3000"), ("IT", "4000"), ("Finance", "3500")]
columns = ["Department", "Salary"]
df = spark.createDataFrame(data, schema=columns)

# Map the DataFrame to an RDD
rdd = df.rdd.____(lambda row: (row["Department"], row["Salary"]))

# Apply a lambda function to get the sum of the DataFrame
rdd_aggregated = rdd.____(lambda x, y: x + y)

# Show the collected Results
print(rdd_aggregated.____())
Editar e executar o código