Resuming after a restart
Your streaming pipeline at Global Retail Analytics picks up new CSV files from a Unity Catalog volume and writes them to Delta Lake. After a weekend outage, the cluster restarts on Monday morning. Three new files landed over the weekend. The stream processes only those three files — none of the files from the previous week are reloaded.
What allows the stream to skip previously processed files and resume exactly where it left off?
Questo esercizio fa parte del corso
Data Transformation with Spark SQL in Databricks
Esercizio pratico interattivo
Passa dalla teoria alla pratica con uno dei nostri esercizi interattivi
Inizia esercizio