Cleaning the online retail dataset
You're a data engineer at Global Retail Analytics. Your manager has flagged quality issues in the online_retail dataset - missing customer IDs, duplicated rows, and cancelled orders are skewing weekly reports.
Before any analysis can begin, you need to define an explicit schema, diagnose the data quality problems, and build a cleaning pipeline that removes invalid records.
This exercise is part of the course
Data Transformation with Spark SQL in Databricks
Hands-on interactive exercise
Turn theory into action with one of our interactive exercises
Start Exercise