เริ่มต้นใช้งานเริ่มต้นใช้งานได้ฟรี

Pandafy a Spark DataFrame

Suppose you've run a query on your huge dataset and aggregated it down to something a little more manageable.

Sometimes it makes sense to then take that table and work with it locally using a tool like pandas. Spark DataFrames make that easy with the .toPandas() method. Calling this method on a Spark DataFrame returns the corresponding pandas DataFrame. It's as simple as that!

This time the query counts the number of flights to each airport from SEA and PDX.

Remember, there's already a SparkSession called spark in your workspace!

แบบฝึกหัดนี้เป็นส่วนหนึ่งของหลักสูตร

Foundations of PySpark

ดูคอร์ส

คำแนะนำการฝึกหัด

  • Run the query using the .sql() method. Save the result in flight_counts.
  • Use the .toPandas() method on flight_counts to create a pandas DataFrame called pd_counts.
  • Print the .head() of pd_counts to the console.

แบบฝึกหัดเชิงโต้ตอบแบบลงมือทำ

ลองทำแบบฝึกหัดนี้โดยเติมโค้ดตัวอย่างนี้ให้สมบูรณ์

# Don't change this query
query = "SELECT origin, dest, COUNT(*) as N FROM flights GROUP BY origin, dest"

# Run the query
flight_counts = ____

# Convert the results to a pandas DataFrame
pd_counts = ____

# Print the head of pd_counts
print(____)
แก้ไขและรันโค้ด