Infer and filter
Imagine you have a census dataset that you know has a header and a schema. Let's load that dataset and let PySpark infer the schema. What do you see if you filter on adults over 40?
Remember, there's already a SparkSession called spark in your workspace!
This exercise is part of the course
Introduction to PySpark
Exercise instructions
- Load a JSON file
adults.json. - Filter the data to include adults over the
ageof40. - Show the results.
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
# Load the dataframe
census_df = spark.read.json("adults.json")
# Filter rows based on age condition
salary_filtered_census = census_df.____(census_df[____]____)
# Show the result
____