1. Learn
    2. /
  2. Courses
    3. /
  3. Feature Engineering with PySpark

Exercise

Verifying Data Load

Let's suppose each month you get a new file. You know to expect a certain number of records and columns. In this exercise we will create a function that will validate the file loaded.

Instructions

100 XP
  • Create a data validation function check_load() with parameters df a dataframe, num_records as the number of records and num_columns the number of columns.
  • Using num_records create a check to see if the input dataframe df has the same amount with count().
  • Compare input number of columns the input dataframe has withnum_columns by using len() on columns.
  • If both of these return True, then print Validation Passed