Get startedGet started for free

Going serverless

1. Going serverless

Hey-o! Welcome back to chapter 2!

2. Last chapter

In the last chapter, we built our first streaming data pipeline using Firehose and S3. At the end, we wrote a script that pulls the written data from S3 and analyzes it.

3. Sensors question

But, if the sensors send data at various intervals,

4. Sensors questions

And firehose is sending data at various intervals to S3

5. Right back to batch

To read and analyze these records, we have to call analyze_data.py manually. So it's almost like we're right back to batch. Boo.

6. How do we run code

So how do we run code

7. In response to each record insertion

In response to each new record being inserted?

8. Lambda

Using AWS Lambda, you can execute code. In the cloud. Triggered by an event.

9. Serverless vs servers

Lambda is an example of Serverless infrastructure. With serverless, you outsource managing the machine where your code runs to AWS. You pay per code execution instead of paying for machine run time. As your workload increases, AWS scales up to run your functions. In Traditional architecture, that job is yours. Because serverless has memory and execution time limits, it's best for quick functions like event responses. Traditional architecture is better for long jobs like training models.

10. 1. Trigger

Lambda functions execute in response to a trigger. This can be an S3 event, an API request, a time trigger, a manual invocation or others.

11. 2. Handler

The function code that defines the logic is a handler.

12. 3. Layer

The environment where the code executes is bare-bones. Layers add additional libraries like pandas and numpy.

13. 4. Destination

Finally there's an optional destination for use cases like writing to another Firehose stream.

14. Powerful combination

With lambda, we can transform data, build APIs, send alerts, trigger Alexa actions, and much more.

15. Sample handler

The handler takes an event argument from the trigger. For example, S3 will send a different event than an API. We won't worry about the context argument yet. The handler returns an object that is sent back to the trigger, and optionally, a destination.

16. Review

Whew! What a lesson!

17. Review

You learned some things we can build with lambda.

18. Review

You learned about serverless concepts and differences between serverless and traditional architecture.

19. Review

Finally, you learned the anatomy of a lambda function.

20. Let's practice!

Now let's learn how to make and test your own lambda function!