Get startedGet started for free

Caching a DataFrame

You've been assigned a task that requires running several analysis operations on a DataFrame. You've learned that caching can improve performance when reusing DataFrames and would like to implement it.

You'll be working with a new dataset consisting of airline departure information. It may have repetitive data and will need to be de-duplicated.

The DataFrame departures_df is defined, but no actions have been performed.

This exercise is part of the course

Cleaning Data with PySpark

View Course

Exercise instructions

  • Cache the unique rows in the departures_df DataFrame.
  • Perform a count query on departures_df, noting how long the operation takes.
  • Count the rows again, noting the variance in time of a cached DataFrame.

Hands-on interactive exercise

Have a go at this exercise by completing this sample code.

start_time = time.time()

# Add caching to the unique rows in departures_df
departures_df = departures_df.____.____

# Count the unique rows in departures_df, noting how long the operation takes
print("Counting %d rows took %f seconds" % (departures_df.____, time.time() - start_time))

# Count the rows again, noting the variance in time of a cached DataFrame
start_time = time.time()
print("Counting %d rows again took %f seconds" % (____, time.time() - start_time))
Edit and Run Code