Session Ready
Exercise

Pandafy a Spark DataFrame

Suppose you've run a query on your huge dataset and aggregated it down to something a little more manageable.

Sometimes it makes sense to then take that table and work with it locally using a tool like pandas. Spark DataFrames make that easy with the .toPandas() method. Calling this method on a Spark DataFrame returns the corresponding pandas DataFrame. It's as simple as that!

This time the query counts the number of flights to each airport from SEA and PDX.

Remember, there's already a SparkSession called spark in your workspace!

Instructions
100 XP
  • Run the query using the .sql() method. Save the result in flight_counts.
  • Use the .toPandas() method on flight_counts to create a pandas DataFrame called pd_counts.
  • Print the .head() of pd_counts to the console.