Modern data engineering with Snowflake
1. Modern data engineering with Snowflake
A few years back, Snowflake published Cloud Data Engineering for Dummies, Snowflake Special Edition. Among many of the useful insights in it, it did a great job of capturing the essence of data engineering and what data engineers do. It states a couple of things that I think are worth calling out here. Data engineering isn't something you do once. It's an ongoing practice that involves collecting, preparing, transforming, and delivering data. Data engineers oversee the ingestion, transformation, delivery, and movement of data throughout every part of the organization. Did you notice any common words or phrases among these two sentences? If so, you might have noticed the following terms. Collecting or ingesting, transforming, and delivery. The concepts of data ingestion, data transformation, and data delivery are core to data pipelines. The practice of applying these concepts in concert to build data pipelines is known as data engineering. In this course, we'll use these three core concepts, data ingestion, transformation, and delivery, to form a framework for understanding data engineering with Snowflake. For the purposes of brevity, you might hear me refer to this framework as ITD, and you should think of these three core concepts when you hear this acronym. This framework will help us contextualize many of the specific technical concepts and Snowflake features that you'll learn about in the course. Let's dive a little deeper into each component in this framework. Ingestion refers to the gathering or collection of data, often into a central platform, in this case, Snowflake. Transformation refers to the cleaning, changing, wrangling, or processing of data. And delivery refers to the delivery of a data product, like a data set, for example, to a consumer or system. A consumer could be someone like an analyst or application developer on your team. And a system could be something like a dashboard or some other application requiring a specific set of data. In this course, we'll use a single platform, the Snowflake AI Data Cloud, to ingest, transform, and deliver data products to build modern data engineering pipelines. And finally, let's talk a little more about the term modern in this context. Early data pipelines were fraught with challenges that today are solved with modern approaches and platforms, like Snowflake. Here are some examples of those challenges. Siloed data, meaning different data sets were housed in different data environments, and navigating those different environments was often challenging. Siloed and complex management of compute resources, specifically having to manage individual, different development environments, many times with different programming languages, just to be able to process data. And finally, loose governance over data, which meant an increased security risk against that data. Many of these data silos reflect legacy approaches to data engineering, namely because if you're doing modern data engineering, all of your data can live within the same platform with no silos. Within Snowflake, your data can also be transformed easily using multiple languages, like SQL, Python, and more. And you can do this without needing to manage different development environments. Your data can also be processed with powerful compute resources, without any management overhead. And finally, your data can be governed consistently, resulting in decreased security risks. All of these capabilities make it much easier to remove friction points and perform modern data engineering. You can take data from ingestion, through transformation, to delivery, all on the same platform. But keep in mind that a single platform won't solve the challenge of picking an approach to building a data pipeline. As a data engineer, you'll of course still need to make choices around how you perform data engineering, namely around which specific tools and features you'll use, among many other considerations. Coming up, let's talk a little about how you've likely done data engineering before.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.