Delivery of data products
1. Delivery of data products
There's so much that is possible with clean, transformed data produced by a strong pipeline. Things like well-trained machine learning models, applications that present critical data to end-users, datasets that enrich other use cases, and much more. In short, with ready-to-use transformed data, we're at the stage of being able to deliver value and insights to consumers and other systems. And in this module, we'll dive into how to do this by exploring the third phase of our data engineering framework, delivery. What exactly do we mean by delivery? Before we get into the details, let's step back a little and contextualize everything that you've learned. You're learning to build data pipelines. If we oversimplify things for just a bit, we know that, given an input, a data pipeline should produce an output. More tactically, the pipelines that you're building will take in raw data as an input, perform transformations against that data, and output, or deliver, something of value. For example, common use cases for data pipelines include feeding dashboards with important data that will be used by other teams, like teams of, say, analysts or product managers. Or serving data to web applications. Perhaps there are datasets that are important to surface to an end user within an app for the purposes of taking action or making a decision. Or creating and delivering enterprise-grade datasets used for training machine learning models and performing imprints. Or feeding data into other data systems. Maybe your pipeline is just one part of the journey, and its output must be fed into another data system or pipeline for some broader purpose. These are just a few examples, and your use case will vary, of course. In any case, many times pipelines are delivering, or are helping to deliver, a high-quality, highly polished data product to be used for a specific use case. And that's what we mean by delivery of data products. And here's another important note. Earlier, I mentioned that we've arrived at the third phase of our framework. I didn't say final. And that's intentional. This is because although delivering a data product is an outcome, it isn't necessarily the final thing that you'll do with the pipeline. In fact, there's a good chance that you'll continue to maintain a pipeline throughout its lifetime. For the simple reason that, well, things change. Things like the source data. The data structure might evolve in some new way that impacts your logic, and you may need to update your ingestion and or transformation logic to account for that. There may be new data sources. More data could be introduced from new sources that may need to be combined with existing data you're ingesting, possibly for enrichment purposes. And last but not least, there could be new pipeline requirements. Perhaps new requirements to extract more insights from the data are introduced, so you might need to update existing logic or create new user-defined functions, stored procedures, and more. With that, here are the techniques that we'll cover in this module for delivering data products with Snowflake. Data sharing on Snowflake Marketplace, Streamlit in Snowflake, and Snowflake native applications. Let's dive in. Thank you for watching. And I'll see you in the next video.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.