在本章中，您将了解批处理与流处理的区别，创建您的第一个数据流，管理其权限，并向数据流写入与读取数据。

什么是流式处理？为什么重要？

选项

批处理

流处理

批处理 vs 流处理

数据生产者

..将数据放入 Firehose 流

...由其传送至目标

生产者与目标

管理 Firehose Delivery Streams

准备创建第一个流

创建角色

IAM 用户

角色

IAM 用户与角色（均适用）

IAM 用户与角色

Options

Permissions for User only

Permissions for User and Role

权限练习

创建 S3 存储桶

使用 Firehose 传输流

# _setup.py: Create clients, load data, etc. No need to edit.

import boto3

# Create firehose client
firehose = boto3.client('firehose', 
    aws_access_key_id="None", 
    aws_secret_access_key="None", 
    region_name='us-east-1', 
    endpoint_url="http://localhost:4573")

# Create s3 client
s3 = boto3.client('s3', 
    aws_access_key_id="None", 
    aws_secret_access_key="None", 
    region_name='us-east-1', 
    endpoint_url="http://localhost:4572")

# Prep variables for export
ex_vars = [firehose, s3]


# create_firehose.py: Create firehose stream. No need to edit.
import _setup
firehose, s3 = _setup.ex_vars

# Create s3 bucket
s3.create_bucket(Bucket='sd-vehicle-data')

# Create Firehose delivery stream
res = firehose.____(
    DeliveryStreamName="gps-delivery-stream",
    DeliveryStreamType="DirectPut",
    # Specify configuration of the destination S3 bucket
    ____ = {
        "BucketARN": "arn:aws:s3:::sd-vehicle-data",
        "RoleARN": "arn:aws:iam::0000000:role/firehoseDeliveryRole"
    })
    
# Print the stream ARN
print("Created Firehose Stream ARN: {}".format(res['DeliveryStreamARN']))


# _setup.py: Create clients, load data, etc. No need to edit.
import boto3

# Create firehose client
firehose = boto3.client('firehose', 
    aws_access_key_id="None", 
    aws_secret_access_key="None", 
    region_name='us-east-1', 
    endpoint_url="http://localhost:4573")

# Create s3 client
s3 = boto3.client('s3', 
    aws_access_key_id="None", 
    aws_secret_access_key="None", 
    region_name='us-east-1', 
    endpoint_url="http://localhost:4572")

# Prep variables for export
ex_vars = [firehose, s3]


# create_firehose.py: Create firehose stream. No need to edit.
import _setup
firehose, s3 = _setup.ex_vars

# Create s3 bucket
s3.create_bucket(Bucket='sd-vehicle-data')

# Create Firehose delivery stream
res = firehose.create_delivery_stream(
    DeliveryStreamName =  "gps-delivery-stream",
    DeliveryStreamType = "DirectPut",
    
    # Specify configuration of the destination S3 bucket
    S3DestinationConfiguration = {
        "BucketARN": "arn:aws:s3:::sd-vehicle-data",
        "RoleARN": "arn:aws:iam::0000000:role/firehoseDeliveryRole"
    })
    
# Print the stream ARN
print("Created Firehose Stream ARN: {}".format(res['DeliveryStreamARN']))


创建您的第一个 Firehose 流

# _setup.py: Create clients, load data, etc. No need to edit.
import boto3, pandas as pd

# Load Data
data_url = "https://assets.datacamp.com/production/repositories/5668/datasets/6bba555e0e42ae31d1d634256679db718cfb8d76/vehicles.csv"
records = pd.read_csv(data_url).head(100)

# Create firehose client
firehose = boto3.client('firehose', 
    aws_access_key_id="None", 
    aws_secret_access_key="None", 
    region_name='us-east-1', 
    endpoint_url="http://localhost:4573")

# Create s3 client
s3 = boto3.client('s3', 
    aws_access_key_id="None", 
    aws_secret_access_key="None", 
    region_name='us-east-1', 
    endpoint_url="http://localhost:4572")
    
# Prep variables for export
ex_vars = [firehose, s3, records]
    


# create_firehose.py: Create firehose stream. No need to edit.
import _setup
firehose, s3, records = _setup.ex_vars

# Create s3 bucket
s3.create_bucket(Bucket='sd-vehicle-data')

# Create Firehose delivery stream
res = firehose.create_delivery_stream(
    DeliveryStreamName =  "gps-delivery-stream",
    DeliveryStreamType = "DirectPut",
    
    # Specify configuration of the destination S3 bucket
    S3DestinationConfiguration = {
        "BucketARN": "arn:aws:s3:::sd-vehicle-data",
        "RoleARN": "arn:aws:iam::0000000:role/firehoseDeliveryRole"
    })
    


# OBD2_sensors.py: Write to Firehose stream. EDIT HERE.
import _setup, create_firehose
firehose, s3, records = _setup.ex_vars
for idx, row in records.iterrows(): 

    # Create a payload string that ends with a newline
    payload = ' '.join(str(value) for value in row) 
    payload = payload + "\n"
    print("Sending payload: {}".format(payload))

    # Send the payload string to Firehose stream
    res = firehose.____(
        DeliveryStreamName = 'gps-delivery-stream',
        Record = {'____': payload})

    # Print the written RecordId
    print("Wrote to RecordId: {}".format(res['RecordId']))


# OBD2_sensors.py: Write to Firehose stream. EDIT HERE.
import _setup, create_firehose
firehose, s3, records = _setup.ex_vars
for idx, row in records.iterrows(): 

    # Create a payload string that ends with a newline
    payload = ' '.join(str(value) for value in row) 
    payload = payload + "\n"
    print("Sending payload: {}".format(payload))

    # Send the payload string to Firehose stream
    res = firehose.put_record(
        DeliveryStreamName = 'gps-delivery-stream',
        Record = {'Data': payload})

    # Print the written RecordId
    print("Wrote to RecordId: {}".format(res['RecordId']))


写入 Firehose 流

# _setup.py: Create clients, load data, etc. No need to edit.
import boto3, pandas as pd

# Load Data
data_url = "https://assets.datacamp.com/production/repositories/5668/datasets/6bba555e0e42ae31d1d634256679db718cfb8d76/vehicles.csv"
records = pd.read_csv(data_url).sample(100)

# Create firehose client
firehose = boto3.client('firehose', 
    aws_access_key_id="None", 
    aws_secret_access_key="None", 
    region_name='us-east-1', 
    endpoint_url="http://localhost:4573")

# Create s3 client
s3 = boto3.client('s3', 
    aws_access_key_id="None", 
    aws_secret_access_key="None", 
    region_name='us-east-1', 
    endpoint_url="http://localhost:4572")
    
# Prep variables for export
ex_vars = [firehose, s3, records]
    


# _run_deps.py: Execute dependencies. No need to edit.
import create_firehose
import OBD2_sensors


# create_firehose.py: Create firehose stream and s3 bucket. No need to edit.
import _setup
firehose, s3, records = _setup.ex_vars

# Create s3 bucket
s3.create_bucket(Bucket='sd-vehicle-data')

# Create Firehose delivery stream
res = firehose.create_delivery_stream(
    DeliveryStreamName =  "gps-delivery-stream",
    DeliveryStreamType = "DirectPut",
    
    # Specify configuration of the destination S3 bucket
    S3DestinationConfiguration = {
        "BucketARN": "arn:aws:s3:::sd-vehicle-data",
        "RoleARN": "arn:aws:iam::0000000:role/firehoseDeliveryRole"
    })
    


# OBD2_sensors.py: Write OBD2 sensor data. No need to edit.
import _setup
firehose, s3, records = _setup.ex_vars
for idx, row in records.iterrows(): 

    # Create a payload string that ends with a newline
    payload = ' '.join(str(value) for value in row) 
    payload = payload + "\n"

    # Send the payload string to Firehose stream
    res = firehose.put_record(
        DeliveryStreamName = 'gps-delivery-stream',
        Record = {'Data': payload})
    


# analyze_data.py: Analyze written sensor data. EDIT HERE.
import _setup, _run_deps, pandas as pd
firehose, s3, records = _setup.ex_vars

# List the objects that have been written to the S3 bucket
objects = s3.____(Bucket='sd-vehicle-data')['Contents']

# Create list for collecting dataframes from read files.
dfs = []

# For every object, load it from S3
for obj in objects:
    data_file = s3.____(Bucket='sd-vehicle-data', ____=obj['Key'])

    # Load it into a dataframe, specifying a delimiter and column names
    dfs.append(pd.read_csv(data_file['Body'], 
                           delimiter = " ", 
                           names=["record_id", "timestamp", "vin", "lon", "lat", "speed"]))

# Concatenate the resulting dataframes.
data = pd.concat(dfs)
print(data.groupby(['vin'])['speed'].max())

# OBD2_sensors.py: Analyze written sensor data. EDIT HERE.
import _setup, _run_deps, pandas as pd
firehose, s3, records = _setup.ex_vars

# List the objects that have been written to the S3 bucket
objects = s3.list_objects(Bucket='sd-vehicle-data')['Contents']

# Create list for collecting dataframes from read files.
dfs = []

# For every object, load it from S3
for obj in objects:
    data_file = s3.get_object(Bucket='sd-vehicle-data', Key=obj['Key'])

    # Load it into a dataframe, specifying a delimiter and column names
    dfs.append(pd.read_csv(data_file['Body'], 
                           delimiter = " ", 
                           names=["record_id", "timestamp", "vin", "lon", "lat", "speed"]))

# Concatenate the resulting dataframes.
data = pd.concat(dfs)
print(data.groupby(['vin'])['speed'].max())


读取 Firehose 数据

云端流处理

在流数据学习之旅的下一步，您将学习使用具有转换能力的 Lambda 函数来实现无服务器架构。通过动手练习，您将添加 Lambda 层，并在满足特定条件时触发 Lambda 函数。

无服务器架构入门

创建并运行 Lambda 函数

Serverless 与传统服务器

Lambda 函数组件

基于事件从 S3 读取数据

您的第一个实时 Lambda！

添加 Lambda 层

Lambda 流程

使用环境变量

无服务器数据工作流

快速与慢速 Lambda

按时间触发的 Lambda

按时间触发的 Lambda 代码

无服务器 API

创建一个 API lambda

走向无服务器

现在，您已准备好对流数据进行编码与解码，并直接在数据流中分析数据。您甚至会使用多个数据流来获取车辆的每日最高车速。

一个用于转换的 Lambda

在流中转换数据

编码与解码 base64

创建一个转换型 Lambda

一个精简的转换型 lambda

在流中分析数据

创建 Kinesis Data Analytics 应用程序

Kinesis Data Analytics 与转换型 Lambda 的对比

构建 Kinesis 数据分析应用

Kinesis Data Analytics 的 SQL 组件

使用多个流

从 Kinesis Analytics 交付数据

获取每日最高车速

使用 Kinesis 数据分析发送告警

分析流数据

在最后一章中，您将学习如何通过日志、指标、告警和仪表板来监控数据流的性能。您将使用 Elasticsearch，并构建自己的 Kibana 仪表板。

流式数据案例研究

创建 Elasticsearch 集群

RedShift 与 Elasticsearch

监控性能

CloudWatch 仪表板与告警

CloudWatch 组件

CloudWatch 监控流程

可视化流式数据

使用 Kibana 操作 ElasticSearch

CloudWatch 与 Elasticsearch

Elasticsearch 可视化流程

另一种实现思路

另一种可选方案

监控与可视化流数据

是什么在悄然驱动我们日常使用的系统，例如保障交易安全的欺诈检测，或让交通顺畅的信号控制？答案是流数据（由不同来源持续产生的数据）以及无服务器技术——如 Amazon Kinesis 和 AWS Lambda。
<br /> <br />
在本课程中，您将通过协助一位名叫 Cody 的虚构数据工程师来学习如何运用这些强大的技术。您的目标是帮助她收集来自城市公务车辆的实时流数据，对数据进行分析，并向驾驶员发送相关告警，例如超速提醒。借助 Amazon Kinesis 和 Firehose，您将学习如何从数百万个来源摄取数据；随后使用 Kinesis Analytics 在数据流动过程中进行分析。您还将使用 AWS Lambda 启动无服务器函数，根据接收到的数据有条件地触发相应操作。完成本课程后，您将掌握如何使用 AWS QuickSight 和 CloudWatch 创建实时的 ElasticSearch 仪表板——也希望能帮助 Cody 完成她的雄心勃勃的项目。


Introduction to AWS Boto in Python

Introduction to Shell

Streaming Concepts

学习如何在AWS上使用无服务器技术处理流数据。

使用 AWS Kinesis 与 Lambda 处理流数据

学习如何使用 AWS 上的无服务器技术处理流数据。

每日夜间统计超速前 5 名驾驶员的报告。

当驾驶员长时间怠速时进行实时告警。

车辆离开指定区域时通知车队管理者。

当车辆制动液不足时提醒驾驶员。

批处理 vs 流处理

使用 AWS Kinesis 与 Lambda 处理流数据

动手互动练习