After seeing this chapter, you will be able to explain what modern data architecture is, and what are their key requirements and components. You'll also review different types of existing data architectures, ranging from Lambda to Data Mesh.

Modern Data Architectures

What is a modern data architecture?

Modern vs traditional architectures

Examples of the modern data architecture

Lambda architecture

Lambda architecture layers

Kappa architecture

Kappa: What about the batch layer?

Data Mesh and Data Fabric

Data Mesh

Data Fabric

Data architecture & its use cases

Introduction to Modern Data Architecture

Learn about the different components and layers of modern data architectures; from data ingestion to serving. Get familiar with the different paradigms and approaches in each layer, and some of the cloud services used to implement modern data solutions to get field experience!

Data Storage: Storage types

What are blob storages?

When should you use blob storage?

SQL vs. NoSQL

Storage use cases

Data Ingestion

What is ingestion?

What is the landing zone?

Ingesting new data

Data Processing

Batch vs. Streaming

What is a window in streaming?

Processing paradigms

Data serving

Serving technologies

Sort the layers

Modern Data Architecture Components

Dive into the transversal components that modern data architectures consist of, covering key aspects like data governance, metadata, data security, and observability. By understanding these vital elements, you'll be prepared to design robust, secure, and efficient data platforms while maintaining high standards for the data and overall service.

Data governance

What is data governance?

Data governance roles

Metadata Management

The role of metadata

Metadata types

The business glossary is composed of...

Data Security

What is a data breach?

Virtual Private Clouds (VPC)

Effective data security

Classifying Data Security Measures

Observability

Troubleshooting the source of a problem

Data security measures

Transversal Components of Data Architectures

Learn how to manage workflows, understand costs in cloud services, and create modern data designs. This chapter helps you work with tools like Apache Airflow and teaches you ways to save money on cloud use. By the end, you'll know how to build and judge strong data systems for businesses.

Orchestration

Options

Scheduling

Orchestration vs. Scheduling

Orchestration sensors

Storage & processing costs best practices

Reserved capacity in cloud services

Standard 

Glacier/Coldline

Storage classes

Designing a modern data architecture

Asking the right questions

Ingesting the application data

Evaluating modern data architecture solutions

Why a quarantine zone?

Appropriate for this use case

Inappropriate for this use case

Designing a modern data platform

Wrap-up

Putting it All Together

In this course, you'll gain a deep understanding of the key components that make up modern data solutions and explore different existing architectures with their unique benefits and drawbacks. From data ingestion and serving to governance and orchestration, you'll explore each layer and discover how they contribute to building robust and scalable data solutions. By the end of the course, you'll have the knowledge and skills to design and implement modern data architectures that not only drive valuable insights but also take cost and security considerations into account.

<h2>Introduction to modern data architectures</h2>
<br>
The world is constantly changing thanks to data, and every day the amount of data to be processed is even bigger. Thus, this course gives you an overview of modern data architectures, guiding you through the fundamental components of data solutions.

<h2>Components of modern data architectures</h2>
<br>
This is an introductory course to modern data solutions. During the first half of this course you'll learn what modern data architectures are, will review some popular solutions like Data Mesh and Data Fabric, and will explore the different elements of data architectures such as storage, data ingestion methods, processing options, and serving techniques.

<h2>Transversal Components of modern data architectures</h2>
<br>
In the second half, the course shows the transversal components that ensure the cohesion and enable the management of data architectures; explaining concepts like data governance, orchestration, observability, and even cost optimization within cloud providers. With all these elements, you'll be able to design data solutions that highly align with your organization's needs.

<h2>Designing and evaluating modern data architectures</h2>
<br>
The final part of the course integrates the knowledge you've gained, using a business case to design a data solution that satisfies the requirements of the business.

Data Warehousing Concepts

Understanding Data Engineering

Discover modern data architecture's key components, from ingestion and serving to governance and orchestration.

Understanding Modern Data Architecture

Professional Data Engineer in Python

Senior Software Engineer

Ingestion

Storage

Processing

Processing: The model scores

Serving the data

Some other details

Let's practice!

https://s3.amazonaws.com/assets.datacamp.com/production/course_32326/subtitles/course_32326_f4c6d5800d18911654e28d8f8a078208.vtt

Putting it All Together - Evaluating modern data architecture solutions

Evaluating modern data architecture solutions

Create Your Free Account