1. Learn
    2. /
  2. Courses
    3. /
  3. Big Data Fundamentals with PySpark
Connected

Exercise

Part 2: SQL Queries on DataFrame

The fifa_df DataFrame that we created has additional information about datatypes and names of columns associated with it. This additional information allows PySpark SQL to run SQL queries on DataFrame. SQL queries are concise and easy to run compared to DataFrame operations. But in order to apply SQL queries on DataFrame first, you need to create a temporary view of DataFrame as a table and then apply SQL queries on the created table (Running SQL Queries Programmatically).

In the second part, you'll create a temporary table of fifa_df DataFrame and run SQL queries to extract the 'Age' column of players from Germany.

You already have a SparkContext spark and fifa_df available in your workspace.

Instructions

100 XP
  • Create temporary table fifa_df_table from fifa_df DataFrame.
  • Construct a "query" to extract the "Age" column from Germany players in fifa_df_table.
  • Apply the SQL "query" and create a new DataFrame fifa_df_germany_age.
  • Computes basic statistics of the created DataFrame.