Your first live lambda!
1. Your first live lambda!
Welcome to Lesson 2. This is getting exciting!2. Last lesson
In the last exercise, we created a lambda function that prints out the object keys as they're added to S3.3. This lesson
In this lesson, we will determine speeders, and send Cody a text!4. Common log format
In the last chapter, we wrote sensor data in common log format - like a CSV, but separated by spaces.5. Reading S3 files
To read these files, we need pandas. We will use a Lambda layer to add pandas to the Lambda runtime, since it's not available out of the box. We will show you how to add a layer in the next demo.6. Updating the handler
After adding a layer, we can use pandas. Let's tweak lambda function. Import libraries and initialize boto3 clients. Define speeding threshold, phone number to send SMS. get_new_data() will accept the event dictionary, read the incoming records and return a single pandas dataframe. record_created_handler() will determine speeders and send an SMS.7. get_new_data()
get_new_data() will iterate over written records, getting event_name, bucket_name and obj_key.8. get_new_data()
Verify the event is ObjectCreated:Put and the bucket is "sd-vehicle-data." Read the object into a dataframe. Append the dataframe to to written_objects list. Concatenate all the dataframes in the list and return the results.9. record_created_handler()
In the handler, we call the get_new_data() function to get new records in a dataframe. We calculate each vehicle's top speed. And create a dataframe of vehicles who have exceeded the SPEED_ALERT_THRESHOLD.10. record_created_handler()
Next, we trigger an SMS with a speeding alert. Finally, we return an aggregated CSV.11. Test the Lambda function
Testing the function gets us an SMS on the phone!12. Adding environment variables
Data engineers need to think about code maintenance and security. Environment variables are a great tool for both. These can be defined in AWS. Once environment variables have been added on AWS, in the lambda code, import the os module. Use os.environ.get() method to retrieve environment variables, and provide a default if the variable is not set. Let's use this method for SPEED_ALERT_THRESHOLD and ALERT_PHONE_NUMBER.13. Adding environment variables
Let's take another look at the PHONE_NUMBER and SPEED_ALERT_THRESHOLD variables we defined. Click on Manage Environment Variables in the main function screen.14. Adding environment variables
Add ALERT_PHONE_NUMBER and SPEED_ALERT_THRESHOLD as environment variables and save.15. Adding a trigger
Our function is almost production ready. But - it's missing a trigger. No one expects you to sit there and press the test button over and over again.16. Adding a trigger
Click add trigger.17. Adding a trigger
Select S3. Then, select sd-vehicle-data as the bucket. Select "All object create events" for event type. This will ensure that our Lambda will fire any time an object is created. Finally, Add!18. Adding a trigger
Our function is live. When firehose writes sensor data to S3, our function will execute. It could be once an hour, or once every half a second - AWS will scale. Let's review.19. Review
Vehicle sensors are continuously writing measurements to the Firehose stream using put_record().20. Review
Firehose assumes the firehoseDeliveryRole which allows it to write to the sd-vehicle-data bucket.21. Review
When an object is written to sd-vehicle-data, an event triggers our Lambda function,22. Review
using pandas powers that we gave it by adding a layer, reads the written file, and sends an SMS with speeders to Cody using Amazon SNS.23. Let's practice!
Let's see how to add a layer that gives our function pandas powers!Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.