Introduction to the Databricks Lakehouse Platform

1. Introduction to Databricks Lakehouse

Hello! Welcome to the course. In this video, we will be discussing an overview of Databricks and why it has become a popular option for enterprise data architectures.

2. The Data Warehouse

Before we talk about Databricks specifically, let's start by reviewing some of the common platforms available for enterprise data. Historically, there have been two main options: the data warehouse and the data lake. The data warehouse has been around for decades, starting with on-premise databases and now evolving into cloud data warehouses hosted on one of the various cloud platforms. The data warehouse has several benefits. As they are designed to hold structured data, data warehouses are very performant, allowing for fast queries and reports. This structure also keeps data relatively clean through different data quality enforcement. Data warehouses, however, are often very expensive due to their proprietary nature and cannot support more modern applications such as machine learning workloads.

3. The Data Lake

Data lakes are a more recent architectural paradigm and are based around unstructured object stores in the cloud. Data lakes are well-suited for any data workload and can handle all kinds of data. The underlying technology is scalable and flexible and generally will be the most cost-effective solution in the cloud. Data lakes come with their downsides, however. While flexibility is great, data stored in the data lake becomes messy and often results in a "data swamp". Data lakes also require tuning and optimization since data processing can be slow.

4. Birth of the Lakehouse

To satisfy their needs, enterprises often would have two separate "stacks" in their architecture. A data warehouse stack would be supported for their SQL and Business Intelligence workloads, while a data lake stack would be maintained for machine learning and more advanced applications. This requires a significant amount of organizational administration, often leading to poor data analysis results.

5. Birth of the Lakehouse

This prompted the need for a new architectural design: the lakehouse. The lakehouse builds on top of the data lake, giving your architecture flexibility for all data types and workloads while getting the performance and governance benefits from the data warehouse design.

6. The Databricks Lakehouse

This is precisely why Databricks has become a very popular platform for data architectures. Databricks provides a single platform that can deliver the lakehouse architecture simply and at scale. By using the lakehouse, data teams can deliver every use case on any dataset without having to worry about how to manage these different technology stacks.

7. Databricks Architecture Benefits

So what specifically makes Databricks unique when compared to other lakehouse platforms? Databricks was the first lakehouse company and pioneered the concept. They developed a leading platform in this space. Firstly, Databricks unifies your entire data stack into a single location. Since the platform can handle use cases from AI to BI, you get the benefits of both the data warehouse and data lake architectures in one. Secondly, the lakehouse is a multi-cloud technology that runs on all of the leading cloud platforms. This means you can bring compute to your data without feeling locked in to a particular vendor.

8. Databricks Development Benefits

Thirdly, Databricks creates a collaborative platform for your data teams. Any data persona will find a user interface that aligns with how they work and can even work together on the same artifacts in real time. Finally, Databricks is built on open-source technologies. Apache Spark is the leading framework for data processing and powers the engine. In the platform, users can use any of the leading languages for data processing and analysis without any special configuration.

9. Let's practice!

With that overview complete, let's review some of the key benefits of Databricks and the lakehouse architecture!

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.