Sorgenti dati

Nel video precedente hai visto tre modi per estrarre i dati:

Estrazione da file di testo, come .txt o .csv
Estrazione da API di servizi web, come la Hacker News API
Estrazione da un database, ad esempio un database applicativo SQL con dati dei clienti

Abbiamo anche accennato brevemente ai database orientati alle righe e all'OLTP.

Sai selezionare l'affermazione su questi argomenti che non è vera?

Questo esercizio fa parte del corso

Introduzione al Data Engineering

Visualizza il corso

Esercizio pratico interattivo

Passa dalla teoria alla pratica con uno dei nostri esercizi interattivi

Inizia esercizio

Questo esercizio fa parte del corso

Introduzione al Data Engineering

IntermediárioNível de habilidade

4.8+

Inizia il corso gratis

In this first chapter, you will be exposed to the world of data engineering! Explore the differences between a data engineer and a data scientist, get an overview of the various tools data engineers use and expand your understanding of how cloud technology plays a role in data engineering.

Exercise 1: What is data engineering?Exercise 2: Tasks of the data engineer Exercise 3: Data engineer or data scientist?Exercise 4: Data engineering problems Exercise 5: Tools of the data engineer Exercise 6: Kinds of databases Exercise 7: Processing tasks Exercise 8: Scheduling tools Exercise 9: Cloud providers Exercise 10: Why cloud computing?Exercise 11: Big players in cloud computing Exercise 12: Cloud services

Now that you know the primary differences between a data engineer and a data scientist, get ready to explore the data engineer's toolbox! Learn in detail about different types of databases data engineers use, how parallel computing is a cornerstone of the data engineer's toolkit, and how to schedule data processing jobs using scheduling frameworks.

Exercise 1: Databases Exercise 2: SQL vs NoSQL Exercise 3: The database schema Exercise 4: Joining on relations Exercise 5: Star schema diagram Exercise 6: What is parallel computing Exercise 7: Why parallel computing?Exercise 8: From task to subtasks Exercise 9: Using a DataFrame Exercise 10: Parallel computation frameworks Exercise 11: Spark, Hadoop and Hive Exercise 12: A PySpark groupby Exercise 13: Running PySpark files Exercise 14: Workflow scheduling frameworks Exercise 15: Airflow, Luigi and cron Exercise 16: Airflow DAGs

Having been exposed to the toolbox of data engineers, it's now time to jump into the bread and butter of a data engineer's workflow! With ETL, you will learn how to extract raw data from various sources, transform this raw data into actionable insights, and load it into relevant databases ready for consumption!

Exercise 1: Estrai Exercise 2: Sorgenti dati

Esercizio in corso

Exercise 3: Recupera dati da un'API Exercise 4: Leggere da un database Exercise 5: Trasforma Exercise 6: Suddividere il rental rate Exercise 7: Preparati alle trasformazioni Exercise 8: Join con le valutazioni Exercise 9: Caricamento Exercise 10: OLAP o OLTP Exercise 11: Scrivere su un file Exercise 12: Caricare in Postgres Exercise 13: Mettiamo tutto insieme Exercise 14: Definire un DAG Exercise 15: Configurare Airflow Exercise 16: Interpretare il DAG

Cap off all that you've learned in the previous three chapters by completing a real-world data engineering use case from DataCamp! You will perform and schedule an ETL process that transforms raw course rating data, into actionable course recommendations for DataCamp students!

Exercise 1: Course ratings Exercise 2: Exploring the schema Exercise 3: Querying the table Exercise 4: Average rating per course Exercise 5: From ratings to recommendations Exercise 6: Filter out corrupt data Exercise 7: Using the recommender transformation Exercise 8: Scheduling daily jobs Exercise 9: The target table Exercise 10: Defining the DAG Exercise 11: Enable the DAG Exercise 12: Querying the recommendations Exercise 13: Congratulations