เริ่มต้นใช้งานเริ่มต้นใช้งานได้ฟรี

Correcting Right Skew Data

In the slides we showed how you might use log transforms to fix positively skewed data (data whose distribution is mostly to the left). To correct negative skew (data mostly to the right) you need to take an extra step called "reflecting" before you can apply the inverse of \(\log\), written as (1/\(\log\)) to make the data look more like normal a normal distribution. Reflecting data uses the following formula to reflect each value: \((x_{\text{max}} +1) – x\).

แบบฝึกหัดนี้เป็นส่วนหนึ่งของหลักสูตร

Feature Engineering with PySpark

ดูคอร์ส

คำแนะนำการฝึกหัด

  • Use the aggregate function skewness() to verify that 'YEARBUILT' has negative skew.
  • Use the withColumn() to create a new column 'Reflect_YearBuilt' and reflect the values of 'YEARBUILT'.
  • Using 'Reflect_YearBuilt' column, create another column 'adj_yearbuilt' by taking 1/log() of the values.

แบบฝึกหัดเชิงโต้ตอบแบบลงมือทำ

ลองทำแบบฝึกหัดนี้โดยเติมโค้ดตัวอย่างนี้ให้สมบูรณ์

from pyspark.sql.functions import log

# Compute the skewness
print(df.____({____: ____}).____())

# Calculate the max year
max_year = df.____({____: ____}).____()[0][0]

# Create a new column of reflected data
df = df.____(____, (max_year + 1) - df[____])

# Create a new column based reflected data
df = df.____(____, 1 / ____(df[____]))
แก้ไขและรันโค้ด