Complex Aggregations
To get you familiar with more of the built in aggregation methods, let's do a slightly more complex aggregation! The goal is to merge all these commands into a single line.
Remember, a SparkSession called spark is already in your workspace, along the Spark DataFrame salaries_df.
This exercise is part of the course
Introduction to PySpark
Exercise instructions
- Calculate the average salaries of large US companies using the
"salary_in_usd"column. - Calculate the total salaries of large US companies.
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
# Average salaries at large us companies
large_companies=salaries_df.filter(salaries_df.company_size == "L").filter(salaries_df.company_location == "US").groupBy().____
#set a large companies variable for other analytics
large_companies=salaries_df.filter(salaries_df.company_size == "L").filter(salaries_df.company_location == "US")
# Total salaries in usd
large_companies.groupBy().____.show()