Menerapkan UDF pada data vektor

Sebuah dataframe tersedia dengan nama df yang memiliki kolom output bertipe vector. Lima baris pertamanya ditampilkan di konsol.

Sebuah UDF get_first_udf tersedia untuk memilih elemen pertama dari kolom vektor.

Latihan ini adalah bagian dari kursus

Pengantar Spark SQL dalam Python

Lihat Kursus

Petunjuk latihan

Buat dataframe baru bernama df_new dengan menambahkan kolom baru ke df. Beri nama kolom baru tersebut label.
Tampilkan lima baris pertama dari df_new.

Latihan interaktif praktis

Cobalah latihan ini dengan menyelesaikan kode contoh berikut.

# Add label by applying the get_first_udf to output column
df_new = df.____('____', ____('____'))

# Show the first five rows 
df_new.____

Edit dan Jalankan Kode

Latihan ini adalah bagian dari kursus

Pengantar Spark SQL dalam Python

SkillTag.level.advancedSkillTag.label

4.7+

95 reviews

Mulai Kursus Gratis

In this chapter you will learn how to create and query a SQL table in Spark. Spark SQL brings the expressiveness of SQL to Spark. You will also learn how to use SQL window functions in Spark. Window functions perform a calculation across rows that are related to the current row. They greatly simplify achieving results that are difficult to express using only joins and traditional aggregations. We'll use window functions to perform running sums, running differences, and other operations that are challenging to perform in basic SQL.

Exercise 1: Creating and querying a SQL table in Spark Exercise 2: Create a SQL table from a dataframe Exercise 3: Determine the column names of a table Exercise 4: Window function SQL Exercise 5: Running sums using window function SQL Exercise 6: Fix the broken query Exercise 7: Dot notation and SQL Exercise 8: Aggregation, step by step Exercise 9: Aggregating the same column twice Exercise 10: Aggregate dot SQL Exercise 11: Convert window function from dot notation to SQL

In this chapter, you will be loading natural language text. Then you will apply a moving window analysis to find frequent word sequences.

Exercise 1: Loading natural language text Exercise 2: Loading a dataframe from a parquet file Exercise 3: Split and explode a text column Exercise 4: Using monotonically_increasing_id()Exercise 5: Moving window analysis Exercise 6: Creating context window feature data Exercise 7: Repartitioning the data Exercise 8: Common word sequences Exercise 9: What type of data is this Exercise 10: Finding common word sequences Exercise 11: Unique 5-tuples in sorted order Exercise 12: Most frequent 3-tuples per chapter

In the previous chapters you learned how to use the expressiveness of window function SQL. However, this expressiveness now makes it important that you understand how to properly cache dataframes and cache SQL tables. It is also important to know how to evaluate your application. You learn how to do do this using the Spark UI. You'll also learn a best practice for logging in Spark. Spark SQL brings with it another useful tool for tuning query performance issues, the query execution plan. You will learn how to use the execution plan for evaluating the provenance of a dataframe.

Exercise 1: Caching Exercise 2: Practice query plans 2 Exercise 3: Practicing caching: part 1 Exercise 4: Practicing caching: the SQL Exercise 5: Practicing caching: putting it all together Exercise 6: Caching and uncaching tables Exercise 7: The Spark UI Exercise 8: Spark UI storage tab Exercise 9: Inspecting cache in the Spark UI Exercise 10: Logging Exercise 11: Practice logging Exercise 12: Practice logging 2 Exercise 13: Query plans Exercise 14: Practice query plans Exercise 15: Practice reading query plans 2

Previous chapters provided you with the tools for loading raw text, tokenizing it, and extracting word sequences. This is already very useful for analysis, but it is also useful for machine learning. What you've learned now comes together by using logistic regression to classify text. By the conclusion of this chapter, you will have loaded raw natural language text data and used it to train a text classifier.

Exercise 1: Extract Transform Select Exercise 2: Latihan membuat UDF Exercise 3: Latihan kolom array Exercise 4: Membuat data fitur untuk klasifikasi Exercise 5: Membuat UDF untuk data vektor Exercise 6: Menerapkan UDF pada data vektor

Latihan Saat Ini

Exercise 7: Mengonversi teks ke format vektor Exercise 8: Klasifikasi Teks Exercise 9: Memberi label pada data Exercise 10: Membagi data Exercise 11: Latih pengklasifikasi Exercise 12: Memprediksi dan mengevaluasi Exercise 13: Evaluasi classifier Exercise 14: Prediksi data uji Exercise 15: Rekap