Managing Firehose delivery streams
Data engineering sometimes involves managing permissions, users, and using GUI tools to do so instead of scripts. In this course, we will show you how to use boto3 to use Kinesis and Lambda, but also how to run certain operations on AWS.
Throughout the course, the operations on AWS are taken care of for you, so that you can focus on the programming aspect of things.
The screencasts are here to give you complete context on how things work. Feel free to try these operations on AWS on your own, but this is completely optional:
- You won't be tested on the AWS platform
- The AWS operations are taken care of for you throughout the course, so we strongly suggest to focus on finishing the course first
- If you ever decide to try things on AWS, make sure to always stop your instances as soon as you leave or you will end up with a hefty bill
For now, you are beginning to learn how to interact with Firehose streams. Your predecessor has created a few Firehose streams, and you want to delete them all to have the account start fresh.
Let's make a fresh start in the city's AWS account!
This exercise is part of the course
Streaming Data with AWS Kinesis and Lambda
Exercise instructions
- Import boto3 and create boto3 Firehose client.
- Get the list of Firehose Delivery streams.
- Iterate over the response contents and delete every stream.
- Verify that all the delivery streams have been deleted by printing the results of
.list_delivery_streams()
.
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
# Import boto3 and create boto3 Firehose client
import ____
firehose = boto3.____('____',
aws_access_key_id=AWS_KEY_ID, aws_secret_access_key=AWS_SECRET,
region_name='us-east-1', endpoint_url=endpoints['FIREHOSE'])
# Get list of delivery streams
response = firehose.____()
# Iterate over the response contents and delete every stream
for stream_name in response['DeliveryStreamNames']:
firehose.____(DeliveryStreamName=stream_name)
print(f"Deleted stream: {stream_name}")
# Print list of delivery streams
print(firehose.____())