Getting ready for the first stream
1. Getting ready for the first stream
Hola! Welcome to Lesson 2! Last lesson we learned streaming vs batch and managing streams. In this lesson, we'll get to make our first one! Let's go!2. End goal
Until the end of the chapter, we'll work on a simple data ingestion system.3. End goal
Vehicle sensors4. End goal
will write vehicle speed information and other telemetry to a Firehose delivery stream5. End goal
which will deliver data to an S3 bucket. Let's set it up!6. dcUser
In the boto3 course, we used dcUser's AWS key and secret.7. dcUser
We gave him permission to create s3_buckets. Let's make sd-vehicle-data bucket - the stream's destination.8. A destination S3 bucket
Create the S3 client, then call create_bucket_method(), with 'sd-vehicle-data' as the Bucket argument.9. dcUser
But dcUser doesn't have permission to create Firehose delivery streams.10. dcUser
We will use his keys to put data into the stream too - and he doesn't have permission for that either.11. Add permissions to user
Let's give dcUser an additional permission. Login to AWS Console, then Identity and Access Management (or IAM),12. Add permissions to user
users13. Add permissions to user
dcUser14. Add permissions to user
You will see the policies we assigned in the last class. Click add permissions,15. Add permissions to user
Select "Attach existing policies directly."16. Add permissions to user
Search for, and add AmazonKinesisFirehoseFullAccess Click Review, then Add, and you're done!17. New powers
Now, dcUser has the power to create, manage and write to Firehose streams.18. A note on security
There is an AdministratorAccess policy in IAM. It's a superUser policy that will allow any action on any service. Why not just assign that policy?19. A note on security
It would allow ANYONE with dcUser's AWS_Key and Secret to have free reign, leaving us with a hefty bill.20. A note on security
Data engineers balance security and getting things done. We're walking a balance by using service-specific full access permissions. As you're building for production, you may need even more granularity.21. New powers
Now, dcUser has all the permissions to execute this. One small snag.22. Firehose stream permissions
The firehose stream can't write to the S3 bucket.23. Firehose stream permissions
After receiving records from the vehicles, the stream is acting on its own to write records to S3, not as dcUser. So it needs its own permissions.24. A role
To let the Firehose stream write to the S3 bucket, we're going to create a role.25. How roles work
A role is a hat that an Amazon service puts on.26. How roles work
Roles have their own permissions. When the AWS service puts on the hat, it acquires those permissions.27. How roles work
But this hat can only be worn by a trusted entity, which is specified in the role definition. Let's make our own.28. Roles vs users
Roles, like users live in IAM. Like users, they can have permissions policies attached. But roles cannot act on their own or have keys or logins.29. Review - end goal
Our end goal is to build a system that collects data from vehicle sensors and streams it into an S3 bucket.30. Review
To create the stream and write data to it, dcUser needed an additional permission.31. Review
We had him create a bucket that will serve as the destination.32. Review
We now need to create a role, or a hat, that only fits Firehose streams, letting them write to S3 buckets.33. Let's do it!
Let's do that right now!Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.