Resuming after a restart
Your streaming pipeline at Global Retail Analytics picks up new CSV files from a Unity Catalog volume and writes them to Delta Lake. After a weekend outage, the cluster restarts on Monday morning. Three new files landed over the weekend. The stream processes only those three files — none of the files from the previous week are reloaded.
What allows the stream to skip previously processed files and resume exactly where it left off?
Cet exercice fait partie du cours
Data Transformation with Spark SQL in Databricks
Exercice interactif pratique
Passez de la théorie à la pratique avec l’un de nos exercices interactifs
Commencer l’exercice