Aan de slagGa gratis aan de slag

SQL and Parquet

Parquet files are perfect as a backing data store for SQL queries in Spark. While it is possible to run the same queries directly via Spark's Python functions, sometimes it's easier to run SQL queries alongside the Python options.

For this example, we're going to read in the Parquet file we created in the last exercise and register it as a SQL table. Once registered, we'll run a quick query against the table (aka, the Parquet file).

The spark object and the AA_DFW_ALL.parquet file are available for you automatically.

Deze oefening maakt deel uit van de cursus

Cleaning Data with PySpark

Cursus bekijken

Oefeninstructies

  • Import the AA_DFW_ALL.parquet file into flights_df.
  • Use the createOrReplaceTempView method to alias the flights table.
  • Run a Spark SQL query against the flights table.

Praktische interactieve oefening

Probeer deze oefening eens door deze voorbeeldcode in te vullen.

# Read the Parquet file into flights_df
flights_df = spark.read.____(____)

# Register the temp table
flights_df.____('flights')

# Run a SQL query of the average flight duration
avg_duration = spark.____('SELECT avg(flight_duration) from flights').collect()[0]
print('The average flight time is: %d' % avg_duration)
Code bewerken en uitvoeren