Reading a CSV and performing aggregations
You have a spreadsheet of Data Scientist salaries from companies ranging is size from small to large. You want to see if there is a major difference between average salaries grouped by company size.
Remember, there's already a SparkSession called spark in your workspace!
Este exercício faz parte do curso
Introduction to PySpark
Instruções do exercício
- Load a csv file as a DataFrame and infer the schema.
- Return the count of the number of rows.
- Group by the column
company_sizeand calculate the average salary withsalary_in_usd.
Exercício interativo prático
Experimente este exercício completando este código de exemplo.
# Load the CSV file into a DataFrame
salaries_df = ____("salaries.csv", header=True, inferSchema=____)
# Count the total number of rows
row_count = salaries_df.____
print(f"Total rows: {row_count}")
# Group by company size and calculate the average of salaries
salaries_df.____("company_size").____({"salary_in_usd": "avg"}).show()
salaries_df.show()