Infer and filter
Imagine you have a census dataset that you know has a header and a schema. Let's load that dataset and let PySpark infer the schema. What do you see if you filter on adults over 40?
Remember, there's already a SparkSession called spark in your workspace!
Deze oefening maakt deel uit van de cursus
Introduction to PySpark
Oefeninstructies
- Load a JSON file
adults.json. - Filter the data to include adults over the
ageof40. - Show the results.
Praktische interactieve oefening
Probeer deze oefening eens door deze voorbeeldcode in te vullen.
# Load the dataframe
census_df = spark.read.json("adults.json")
# Filter rows based on age condition
salary_filtered_census = census_df.____(census_df[____]____)
# Show the result
____