In this chapter, you’ll be given an introduction to basic concepts in batch data processing, revolving around implementation, scaling, and potential challenges during use.

Intro to batch processing

Batch characteristics

Batch ordering

Scaling batch processing

On the scale

Horizontally opposed

Batch issues

Batch problems

Batch scenarios

Methods for Processing Data

Here, you’ll learn about the basics of streaming, event-based data processing, and the advantages  over traditional batch-based data processing.

Intro to event-based computing

Statements

True

False

In the event of...

Actions

Event

Not an event

Welcome to the event!

Queuing

Characteristics

Describes a queue

Does not describe a queue

Queue characteristics

Scenario

Use a queue

Don't use a queue

To queue or not to queue

Single system data streaming

Single stream event log

Log stream order

Projects

Logging

Something else

Log options

Batching vs. streaming

Scenarios

Batch

Queue

Stream

Batch, queue, or stream?

Streaming Log Process

Log stream processor

Intro to Streaming

This chapter introduces the concepts behind modern streaming systems, how to scale them, and some of the issues found in using streaming processes.

Intro to real-time streaming

Real-time?

Is it real this time?

Vertically scaling streaming systems

Scaling reasons

To vertically scale...?

Horizontally scaling streaming systems

Upscaled out

SLA guarantees

Streaming roadblocks

Streaming attributes

Issue types

Streaming challenges

Streaming Systems

In this final chapter, you’ll learn about common streaming systems and consider the implementation of streaming data processing in different use cases.

Popular streaming systems

Streaming truths

Crossing the streams...

Real-world use case: streaming music service

Message components

Answer me this...

Real-world use case: sensor data

Great order of the SLAs

Sensor scaling considerations

Real-world use case: vaccination clinic

Vaccination clinic - classify areas

A new problem...

Congratulations!

Real-World Use Cases

Streaming is a huge aspect of the data world right now and is being used by nearly every industry from manufacturing to healthcare. Would you like to learn more about the general concepts behind data pipelines and how the processes work?

This course provides a general introduction to streaming concepts including batching, queuing, and stream processing along with where they fit into data processing frameworks. It covers real-world examples of how streaming is implemented in production. It is designed as a general introduction to these concepts and does not require an extensive background in data processing.


Understanding Data Engineering

Learn about the difference between batching and streaming, scaling streaming systems, and real-world applications.

Streaming Concepts

Professional Data Engineer in Python

A client requires an order handling system for digital items that are always in-stock. They want each sale fulfilled in the order it was received.

An online game that can only support a single user, but your manager asks to allow each user to click a `I want to play` button and wait their turn.

A client wants to run a raffle where user information is centrally stored. Once the raffle starts, they want to obtain the winner info on demand.

Develop an image processing system, where you must process as many items as possible at a time. Ordering does not matter in this case.

To queue or not to queue

Streaming Concepts

Hands-on interactive exercise