1. SQL in the Data Intelligence Platform
Hello there! Welcome to this course on Databricks SQL, where we will cover all the different capabilities within the Databricks platform. We will start by going over the core benefits of running SQL workloads in the Databricks Data Intelligence Platform.
2. Instructor Introduction
Before we begin, let me take some time to introduce myself! I am Kevin Barlow, and I am currently a manager of a team of data professionals working with large enterprise companies. I have over a decade of experience in the data analytics industry, and have delivered analytical solutions with a variety of products and to companies of all sizes.
Most importantly for this course, I have extensive experience with the Databricks platform, and have first-hand knowledge of how to implement these solutions well.
3. Motivation
All organizations have need for SQL and BI capabilities, and many have suffered from supporting two technology stacks.
Data warehouses are great for SQL workloads, but are often very expensive and proprietary systems that limit the kind of work you can do in them.
Data lakes, on the other hand, are cost effective with unlimited capabilities, but can often lack suitable performance for business intelligence workloads.
4. Data warehousing in the Lakehouse
In short, this is a significant driving factor in Databricks creating the Lakehouse paradigm. In a single architecture, organizations can benefit from support for all kinds of data workloads, while not sacrificing on performance and governance for business critical operations.
5. Data warehousing in the Lakehouse
In this course, we will be focusing on the data warehouse portion of the lakehouse architecture. As you can see in this diagram, the lakehouse architecture can support your traditional data warehouse architectures, and allows you to migrate your data marts, data vaults, or whichever design you want. As we move through the medallion architecture, our Bronze data, raw, is cleaned and joined to our silver, analytical, datasets, and finally aggregated and filtered to our Gold, BI ready, layer.
6. Benefits
Databricks delivers excellent performance for SQL and BI workloads, leveraging open-source technologies for seamless integration with other platforms.
Built on a data lake architecture, the lakehouse also offers a highly cost-effective solution.
7. Business Intelligence ecosystem
While Databricks SQL is a powerful tool, many organizations prefer to keep using their existing tools. You can seamlessly connect your preferred BI tool to a Databricks SQL warehouse using various methods, which we'll cover later.
This allows end users to enjoy improved report performance and scalability without switching from the tools they already know.
8. Let's practice!
Now, let's practice!