Transforming data inside a stream

1. Transforming data inside a stream

First, we head over to the AWS console, then Lambda, then we create a new function. Let's call it timeStampTransformer. Next let's add a test event that contains two records. We can use the kinesis-kinesis-firehose as a template. Then, we add in the code for the lambda function. We also need to update the timeout to be over one minute. That's a requirement for firehose transformational lambdas. Now, we test the function. All good Next, let's make a few tweaks to our stream. First, we have to give the Firehose role permission to execute our Lambda. We do this by granting it the AwsLambdaFullAccess policy. Next, under transform source records, we select our lambda function, and we're good to go! Now we can start writing to the Firehose stream. When we navigate to see the incoming data in S3, we see that Firehose has written our records with the full datetime string!

2. Let's practice!

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.