Oefenen met loggen 2

De volgende code wordt bij het opstarten uitgevoerd:

import logging
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG,
                    format='%(levelname)s - %(message)s')

In de les leerden we dat Spark-bewerkingen die een action starten, met zorg gelogd moeten worden om stille verspilling van rekenbronnen te voorkomen. Je gaat nu oefenen met het herkennen van logregels die een action op een dataframe of tabel starten.

Er is een dataframe text_df beschikbaar. Dit dataframe is geregistreerd als een tabel met de naam table1.

Deze oefening maakt deel uit van de cursus

Introductie tot Spark SQL in Python

Cursus bekijken

Oefeninstructies

Er zijn verschillende logstatements voorzien. Ze zijn allemaal aanvankelijk uitcommentarieerd. Haal bij de vijf statements die géén action op text_df starten het commentaar weg.

Praktische interactieve oefening

Probeer deze oefening eens door deze voorbeeldcode in te vullen.

# Uncomment the 5 statements that do NOT trigger text_df
# logging.debug("text_df columns: %s", text_df.columns)
# logging.info("table1 is cached: %s", spark.catalog.isCached(tableName="table1"))
# logging.warning("The first row of text_df: %s", text_df.first())
# logging.error("Selected columns: %s", text_df.select("id", "word"))
# logging.info("Tables: %s", spark.sql("show tables").collect())
# logging.debug("First row: %s", spark.sql("SELECT * FROM table1 limit 1"))
# logging.debug("Count: %s", spark.sql("SELECT COUNT(*) AS count FROM table1").collect())

Code bewerken en uitvoeren

Deze oefening maakt deel uit van de cursus

Introductie tot Spark SQL in Python

SkillTag.level.advancedSkillTag.label

4.7+

Begin de cursus gratis

In this chapter you will learn how to create and query a SQL table in Spark. Spark SQL brings the expressiveness of SQL to Spark. You will also learn how to use SQL window functions in Spark. Window functions perform a calculation across rows that are related to the current row. They greatly simplify achieving results that are difficult to express using only joins and traditional aggregations. We'll use window functions to perform running sums, running differences, and other operations that are challenging to perform in basic SQL.

Exercise 1: Creating and querying a SQL table in Spark Exercise 2: Create a SQL table from a dataframe Exercise 3: Determine the column names of a table Exercise 4: Window function SQL Exercise 5: Running sums using window function SQL Exercise 6: Fix the broken query Exercise 7: Dot notation and SQL Exercise 8: Aggregation, step by step Exercise 9: Aggregating the same column twice Exercise 10: Aggregate dot SQL Exercise 11: Convert window function from dot notation to SQL

In this chapter, you will be loading natural language text. Then you will apply a moving window analysis to find frequent word sequences.

Exercise 1: Loading natural language text Exercise 2: Loading a dataframe from a parquet file Exercise 3: Split and explode a text column Exercise 4: Using monotonically_increasing_id()Exercise 5: Moving window analysis Exercise 6: Creating context window feature data Exercise 7: Repartitioning the data Exercise 8: Common word sequences Exercise 9: What type of data is this Exercise 10: Finding common word sequences Exercise 11: Unique 5-tuples in sorted order Exercise 12: Most frequent 3-tuples per chapter

In the previous chapters you learned how to use the expressiveness of window function SQL. However, this expressiveness now makes it important that you understand how to properly cache dataframes and cache SQL tables. It is also important to know how to evaluate your application. You learn how to do do this using the Spark UI. You'll also learn a best practice for logging in Spark. Spark SQL brings with it another useful tool for tuning query performance issues, the query execution plan. You will learn how to use the execution plan for evaluating the provenance of a dataframe.

Exercise 1: Cachen Exercise 2: Oefenen met cachen: deel 1 Exercise 3: Oefenen met cachen: de SQL Exercise 4: Oefenen met cachen: alles bij elkaar brengen Exercise 5: Tabellen cachen en uit de cache halen Exercise 6: De Spark UI Exercise 7: Spark UI-tabblad Storage Exercise 8: Cache inspecteren in de Spark UI Exercise 9: Loggen Exercise 10: Oefenen met logging Exercise 11: Oefenen met loggen 2

Huidige oefening

Exercise 12: Queryplannen Exercise 13: Oefenen met queryplannen Exercise 14: Oefen met het lezen van queryplannen 2

Previous chapters provided you with the tools for loading raw text, tokenizing it, and extracting word sequences. This is already very useful for analysis, but it is also useful for machine learning. What you've learned now comes together by using logistic regression to classify text. By the conclusion of this chapter, you will have loaded raw natural language text data and used it to train a text classifier.

Exercise 1: Extract Transform Select Exercise 2: Practicing creating a UDF Exercise 3: Practicing array column Exercise 4: Creating feature data for classification Exercise 5: Creating a UDF for vector data Exercise 6: Applying a UDF to vector data Exercise 7: Transforming text to vector format Exercise 8: Text Classification Exercise 9: Label the data Exercise 10: Split the data Exercise 11: Train the classifier Exercise 12: Predicting and evaluating Exercise 13: Evaluate the classifier Exercise 14: Predict test data Exercise 15: Recap