Get startedGet started for free

Getting ready for the first stream

1. Getting ready for the first stream

Hola! Welcome to Lesson 2! Last lesson we learned streaming vs batch and managing streams. In this lesson, we'll get to make our first one! Let's go!

2. End goal

Until the end of the chapter, we'll work on a simple data ingestion system.

3. End goal

Vehicle sensors

4. End goal

will write vehicle speed information and other telemetry to a Firehose delivery stream

5. End goal

which will deliver data to an S3 bucket. Let's set it up!

6. dcUser

In the boto3 course, we used dcUser's AWS key and secret.

7. dcUser

We gave him permission to create s3_buckets. Let's make sd-vehicle-data bucket - the stream's destination.

8. A destination S3 bucket

Create the S3 client, then call create_bucket_method(), with 'sd-vehicle-data' as the Bucket argument.

9. dcUser

But dcUser doesn't have permission to create Firehose delivery streams.

10. dcUser

We will use his keys to put data into the stream too - and he doesn't have permission for that either.

11. Add permissions to user

Let's give dcUser an additional permission. Login to AWS Console, then Identity and Access Management (or IAM),

12. Add permissions to user

users

13. Add permissions to user

dcUser

14. Add permissions to user

You will see the policies we assigned in the last class. Click add permissions,

15. Add permissions to user

Select "Attach existing policies directly."

16. Add permissions to user

Search for, and add AmazonKinesisFirehoseFullAccess Click Review, then Add, and you're done!

17. New powers

Now, dcUser has the power to create, manage and write to Firehose streams.

18. A note on security

There is an AdministratorAccess policy in IAM. It's a superUser policy that will allow any action on any service. Why not just assign that policy?

19. A note on security

It would allow ANYONE with dcUser's AWS_Key and Secret to have free reign, leaving us with a hefty bill.

20. A note on security

Data engineers balance security and getting things done. We're walking a balance by using service-specific full access permissions. As you're building for production, you may need even more granularity.

21. New powers

Now, dcUser has all the permissions to execute this. One small snag.

22. Firehose stream permissions

The firehose stream can't write to the S3 bucket.

23. Firehose stream permissions

After receiving records from the vehicles, the stream is acting on its own to write records to S3, not as dcUser. So it needs its own permissions.

24. A role

To let the Firehose stream write to the S3 bucket, we're going to create a role.

25. How roles work

A role is a hat that an Amazon service puts on.

26. How roles work

Roles have their own permissions. When the AWS service puts on the hat, it acquires those permissions.

27. How roles work

But this hat can only be worn by a trusted entity, which is specified in the role definition. Let's make our own.

28. Roles vs users

Roles, like users live in IAM. Like users, they can have permissions policies attached. But roles cannot act on their own or have keys or logins.

29. Review - end goal

Our end goal is to build a system that collects data from vehicle sensors and streams it into an S3 bucket.

30. Review

To create the stream and write data to it, dcUser needed an additional permission.

31. Review

We had him create a bucket that will serve as the destination.

32. Review

We now need to create a role, or a hat, that only fits Firehose streams, letting them write to S3 buckets.

33. Let's do it!

Let's do that right now!

Create Your Free Account

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.