Leveraging Auto Loader

So far, we have shown how to ingest individual files, but how can we do this more efficiently and intelligently? This is where Auto Loader enters the conversation, as it provides the capability to incrementally process new files as they land in a specified location.

Which of the following statements best describes how Auto Loader works when ingesting new data?

Cet exercice fait partie du cours

Introduction to Databricks SQL

Afficher le cours

Exercice interactif pratique

Passez de la théorie à la pratique avec l’un de nos exercices interactifs

Commencer l’exercice

Cet exercice fait partie du cours

Introduction to Databricks SQL

IntermédiaireNiveau de compétence

4.7+

Commencer le cours gratuitement

Build a strong data warehousing foundation with the Databricks Data Intelligence Platform! You will learn how Databricks and the lakehouse architecture set your organization up for a modern SQL and BI stack. You will also explore the key components of the Databricks SQL product, ranging from data storage techniques to compute optimizations.

Exercise 1: SQL in the Data Intelligence Platform Exercise 2: Benefits of the lakehouse for SQL Exercise 3: Exploring Databricks SQL Exercise 4: Exploring some data Exercise 5: Partner Connect Exercise 6: Databricks SQL key assets Exercise 7: Classifying Databricks SQL assets Exercise 8: Create a query Exercise 9: Build a visualization

Build a strong data foundation for the lakehouse architecture! You will learn how to ingest, transform, and model your data using the capabilities in Databricks SQL. You will explore GUI-based and programmatic approaches to building out the medallion architecture, and creating a data ecosystem ready for any analytics workload.

Exercise 1: Ingesting Data Exercise 2: GUI-based data ingest Exercise 3: Hydrating the lakehouse Exercise 4: Upload data manually Exercise 5: Using COPY INTO Exercise 6: Leveraging Auto Loader

Exercice en cours

Exercise 7: Transforming data Exercise 8: SQL and the medallion architecture Exercise 9: Creating a coffee data layer Exercise 10: Cleaning up raw tables Exercise 11: Creating the silver layer Exercise 12: Creating a table of large premium claims

Create powerful data insights using Databricks SQL! You will learn how to write queries, create visualizations, and power dashboards using in-platform Databricks capabilities. You will practice leveraging SQL code, filters, and parameters to create a robust analytical application for your end users.

Exercise 1: Querying in the Data Intelligence Platform Exercise 2: Analytical assets in Databricks SQL Exercise 3: Querying our coffee dataset Exercise 4: Analyzing insurance claims Exercise 5: Supplementing queries with functions Exercise 6: Visualizing query results Exercise 7: Creating an analytics application Exercise 8: The role of Partner Connect Exercise 9: A coffee data dashboard Exercise 10: Adding filters to dashboards Exercise 11: Using parameters in queries Exercise 12: Create an executive dashboard

In the final chapter, you will learn some more advanced techniques that leverage the key differentiators of the Databricks platform. You will learn how to handle high-velocity and fast-changing data using window functions, and will be able to merge datasets as they come in.

Exercise 1: Common data engineering patterns Exercise 2: Append vs. CDC Exercise 3: Updating coffee sales data Exercise 4: Optimizing data with SQL Exercise 5: Using INSERT Exercise 6: Using MERGE Exercise 7: Advanced data analysis patterns Exercise 8: Understanding window functions and sub-queries Exercise 9: Analyzing coffee sales by store Exercise 10: Writing a sub-query Exercise 11: Using a window function Exercise 12: Using windows and sub-queries together Exercise 13: Course recap