1. Learn
    2. /
  2. Courses
    3. /
  3. Foundations of PySpark
  • 1

    Introduction to Apache Spark and PySpark

    Free

    A General introduction to PySpark and distributed computing. This section introduces PySpark, PySpark DataFrames, and RDDs.

  • 2

    PySpark in Python

    A continuation of DataFrames and complex datatypes. This section expands on what DataFrames offer in PySpark and introduces some Spark SQL concepts.

  • 3

    Introduction to PySpark SQL

    Delve into leveraging Spark SQL and PySpark for scalable data processing, combining SQL's simplicity with PySpark's distributed computing power to handle large datasets efficiently.

Initializing

Exercise

Bringing it all together II

Create a DataFrame, apply transformations, cache it, and check if it’s cached. Then, uncache it to release memory. For this exercise a spark session has been made for you! Look carefully at the outcome of the .explain() method to understand what the outcome is!

Instructions

100 XP
  • Cache the df DataFrame.
  • Explain the processing of the agg_result DataFrame.
  • Unpersist the cached df DataFrame after processing.