1. Learn
    2. /
  2. Courses
    3. /
  3. Cleaning Data with PySpark

Exercise

Reading Spark configurations

You've recently configured a cluster via a cloud provider. Your only access is via the command shell or your python code. You'd like to verify some Spark settings to validate the configuration of the cluster.

The spark object is available for use.

Instructions

100 XP
  • Check the name of the Spark application instance ('spark.app.name').
  • Determine the TCP port the driver runs on ('spark.driver.port').
  • Determine how many partitions are configured for joins.
  • Show the results.