1. Going serverless
Hey-o! Welcome back to chapter 2!
2. Last chapter
In the last chapter, we built our first streaming data pipeline using Firehose and S3. At the end, we wrote a script that pulls the written data from S3 and analyzes it.
3. Sensors question
But, if the sensors send data at various intervals,
4. Sensors questions
And firehose is sending data at various intervals to S3
5. Right back to batch
To read and analyze these records, we have to call analyze_data.py manually. So it's almost like we're right back to batch. Boo.
6. How do we run code
So how do we run code
7. In response to each record insertion
In response to each new record being inserted?
8. Lambda
Using AWS Lambda, you can execute code. In the cloud. Triggered by an event.
9. Serverless vs servers
Lambda is an example of Serverless infrastructure.
With serverless, you outsource managing the machine where your code runs to AWS.
You pay per code execution instead of paying for machine run time.
As your workload increases, AWS scales up to run your functions. In Traditional architecture, that job is yours.
Because serverless has memory and execution time limits, it's best for quick functions like event responses. Traditional architecture is better for long jobs like training models.
10. 1. Trigger
Lambda functions execute in response to a trigger. This can be an S3 event, an API request, a time trigger, a manual invocation or others.
11. 2. Handler
The function code that defines the logic is a handler.
12. 3. Layer
The environment where the code executes is bare-bones. Layers add additional libraries like pandas and numpy.
13. 4. Destination
Finally there's an optional destination for use cases like writing to another Firehose stream.
14. Powerful combination
With lambda, we can transform data, build APIs, send alerts, trigger Alexa actions, and much more.
15. Sample handler
The handler takes an event argument from the trigger. For example, S3 will send a different event than an API.
We won't worry about the context argument yet.
The handler returns an object that is sent back to the trigger, and optionally, a destination.
16. Review
Whew! What a lesson!
17. Review
You learned some things we can build with lambda.
18. Review
You learned about serverless concepts and differences between serverless and traditional architecture.
19. Review
Finally, you learned the anatomy of a lambda function.
20. Let's practice!
Now let's learn how to make and test your own lambda function!