Intro
It's very common to have a requirement for web applications to allow uploads to S3, and then have some post-processing take place on those files. A useful pattern is to have S3 fire an event once an object has been created (uploaded), and to invoke a Lambda function each time this event is fired.
This is a basic example of using the AWS Cloud Development Kit (CDK) in a serverless application to create an S3 bucket with those events configured, and a Lambda function with some basic code to respond to those object creation events.
This example is written in Javascript and we'll be using Jest as we take a Test Driven Development (TDD) approach to building both the application and infrastructure code.
How it works
Take a look at this diagram and you'll see that it's fairly straightforward.
When objects are created in S3, the bucket emits an event that triggers the Lambda function. Each time the function is invoked, details about the objects that caused the event are included in the event argument to the Lambda function.
In this case the Lambda won't do very much - just log the name of the uploaded file to Cloudwatch - but you can implement your own logic in the Lambda depending on your use case.
Creating the project
If you want to see the project in its entirety you can check it out on Github.
Feel free to clone it and skip to the deployment section if you don't want to work through each step.
If you want to do it yourself, there are essentially 3 steps that we'll talk through. They are:
- Define an S3 bucket
- Write the function code for our Lambda
- Define a Lambda function that will execute our code
For each step, we'll first write tests and then implement the code so that the tests pass.
Prerequisites
You'll need the AWS CDK installed and configured on your machine. If you don't have have this already, follow these instructions to get set up.
Generate a CDK app
The first thing we need to do is initialise a new application. So we create a new directory called cdk-s3-event-trigger-lambda and run cdk init app from within this directory.
$ mkdir cdk-s3-event-trigger-lambda && cd cdk-s3-event-trigger-lambda
$ cdk init app --language javascript
This creates the project and structure, which should now look like this:
$ tree --gitignore
.
├── README.md
├── bin
│ └── cdk-s3-event-trigger-lambda.js
├── cdk.json
├── jest.config.js
├── lib
│ └── cdk-s3-event-trigger-lambda-stack.js
├── package-lock.json
├── package.json
└── test
└── cdk-s3-event-trigger-lambda.test.js
3 directories, 8 files
S3 Bucket
We're going to write the code to create the S3 bucket, but first we'll create the test that will let us know when we've done it correctly. CDK uses Cloudformation under the hood, and gives us a set of Jest assertions we can import to check the resources that will be defined in the Cloudformation stack.
// test/cdk-s3-event-trigger-lambda.test.js
const cdk = require('aws-cdk-lib');
const { Template } = require('aws-cdk-lib/assertions');
const CdkS3EventTriggerLambda = require('../lib/cdk-s3-event-trigger-lambda-stack');
test('S3 Bucket Created', () => {
const app = new cdk.App();
// WHEN
const stack = new CdkS3EventTriggerLambda.CdkS3EventTriggerLambdaStack(app, 'MyTestStack');
// THEN
const template = Template.fromStack(stack);
template.hasResourceProperties('AWS::S3::Bucket', {});
});
In this case we're using Template.hasResourceProperties() to assert that there is an S3 bucket (AWS::S3::Bucket) in the output template.
Now to create the bucket:
// lib/cdk-s3-event-trigger-lambda-stack.js
const { Stack, RemovalPolicy } = require('aws-cdk-lib');
const s3 = require('aws-cdk-lib/aws-s3');
class CdkS3EventTriggerLambdaStack extends Stack {
/**
*
* @param {Construct} scope
* @param {string} id
* @param {StackProps=} props
*/
constructor(scope, id, props) {
super(scope, id, props);
const myBucket = new s3.Bucket(this, 'ExampleBucket', {
removalPolicy: RemovalPolicy.DESTROY,
autoDeleteObjects: true
});
}
}
module.exports = { CdkS3EventTriggerLambdaStack }
Notice that I created a bucket with a removal policy. This is so the bucket is removed when I use CDK to destroy the application. The default behaviour is for the bucket and data to remain when the stack is destroyed (essentially orphaning the bucket), which is probably a good idea if your application has critical data. As this is just an example though, I've set it to destroy the bucket and everything inside it when I tear down the app.
With that code in place we can check that the test passes by running npm run test.
$ npm run test
> cdk-s3-event-trigger-lambda@0.1.0 test
> jest
PASS test/cdk-s3-event-trigger-lambda.test.js
✓ S3 Bucket Created (47 ms)
Test Suites: 1 passed, 1 total
Tests: 1 passed, 1 total
Snapshots: 0 total
Time: 2.142 s
Ran all test suites.
Function code
To create the code for our Lambda function, first make a new directory called lambda/ and create a file called example-function.js inside it.
For now, we're just going to define an empty function that we can come back and edit later.
// lambda/example-function.js
const handler = async (event, context ) => {}
module.exports = { handler }
Next we're going to create a test that will check that our code correctly handles the S3 event data it will receive as part of the event notification when an object is created.
AWS provides a number of examples of the notification payloads a Lambda function can receive. You can see an example for S3 notifications here.
We're going to use that sample in our tests, so create a file called test/sample-event.json and paste the JSON provided:
{
"Records":[
{
"eventVersion":"2.1",
"eventSource":"aws:s3",
"awsRegion":"us-west-2",
"eventTime":"1970-01-01T00:00:00.000Z",
"eventName":"ObjectCreated:Put",
"userIdentity":{
"principalId":"AIDAJDPLRKLG7UEXAMPLE"
},
"requestParameters":{
"sourceIPAddress":"127.0.0.1"
},
"responseElements":{
"x-amz-request-id":"C3D13FE58DE4C810",
"x-amz-id-2":"FMyUVURIY8/IgAtTv8xRjskZQpcIZ9KG4V5Wp6S7S/JRWeUWerMUE5JgHvANOjpD"
},
"s3":{
"s3SchemaVersion":"1.0",
"configurationId":"testConfigRule",
"bucket":{
"name":"mybucket",
"ownerIdentity":{
"principalId":"A3NL1KOZZKExample"
},
"arn":"arn:aws:s3:::mybucket"
},
"object":{
"key":"HappyFace.jpg",
"size":1024,
"eTag":"d41d8cd98f00b204e9800998ecf8427e",
"versionId":"096fKKXTRTtl3on89fVO.nfljtsv6qko",
"sequencer":"0055AED6DCD90281E5"
}
}
}
]
}
As you can see from the JSON, this sample event was for an object with the key HappyFace.jpg, so if our code is working correctly it should output the name of that file to the logs.
Let's create a test to check for this behaviour.
First edit test/cdk-s3-event-trigger-lambda.test.js to import our (empty) function code, and the sample event from above.
const { handler } = require('../lambda/example-function');
const sampleEvent = require('./sample-event.json');
Then we'll add a test that will check that our code is calling console.log() with the text 'HappyFace.jpg'
test('Lambda handles sample event', async () => {
const consoleSpy = jest.spyOn(console, 'log');
await handler(sampleEvent, {});
expect(consoleSpy).toHaveBeenCalledWith('HappyFace.jpg');
})
Obviously in the real world this would be a more extensive set of tests to check that your function code was processing these events in a way that makes sense for your use case.
The code to output the names of the objects in the event is quite simple. Just update the function code at test/cdk-s3-event-trigger-lambda.test.js like so:
const handler = async (event, context ) => {
event.Records.map( record => console.log(record.s3.object.key) );
}
module.exports = { handler }
And check the tests pass.
Lambda
In a similar way to the S3 bucket above, we're going to write a test to check that a resource is being output as part of the Cloudformation stack. CDK deploys some of its own helper Lambda functions to help with application deployment, so we can't just test if any Lambda exists, we need to specifically test that a Lambda is being deployed with our function code as the handler.
Add the following test to test/cdk-s3-event-trigger-lambda.test.js
test('Lambda function created', () => {
const app = new cdk.App();
// WHEN
const stack = new CdkS3EventTriggerLambda.CdkS3EventTriggerLambdaStack(app, 'MyTestStack');
// THEN
const template = Template.fromStack(stack);
template.hasResourceProperties('AWS::Lambda::Function', {
Handler: 'example-function.handler'
});
})
As you can see, this time we're passing properties to Template.hasResourceProperties() to check for a AWS::Lambda::Function resource that has a handler of example-function.handler.
Time to create the Lambda in our stack. We need to make the following changes to lib/cdk-s3-event-trigger-lambda-stack.js.
First import aws-lambda:
const lambda = require('aws-cdk-lib/aws-lambda');
Then create the Lambda function with the NODEJS_16_X runtime, and our function code as the handler:
const myFunction = new lambda.Function(this, 'ExampleFunction', {
runtime: lambda.Runtime.NODEJS_16_X,
handler: 'example-function.handler',
code: lambda.Code.fromAsset('lambda')
})
Next, we'll update our S3 bucket to send event notifications to Lambda when new objects are created. To do this, import aws-s3-notifications:
const notifications = require('aws-cdk-lib/aws-s3-notifications');
And finally, add an event notification to the S3 bucket with the Lambda set as the destination:
myBucket.addEventNotification(s3.EventType.OBJECT_CREATED, new notifications.LambdaDestination(myFunction))
The file lib/cdk-s3-event-trigger-lambda-stack.js should now look like this:
const { Stack, RemovalPolicy } = require('aws-cdk-lib');
const s3 = require('aws-cdk-lib/aws-s3');
const lambda = require('aws-cdk-lib/aws-lambda');
const notifications = require('aws-cdk-lib/aws-s3-notifications');
class CdkS3EventTriggerLambdaStack extends Stack {
/**
*
* @param {Construct} scope
* @param {string} id
* @param {StackProps=} props
*/
constructor(scope, id, props) {
super(scope, id, props);
const myBucket = new s3.Bucket(this, 'ExampleBucket', {
removalPolicy: RemovalPolicy.DESTROY,
autoDeleteObjects: true
});
const myFunction = new lambda.Function(this, 'ExampleFunction', {
runtime: lambda.Runtime.NODEJS_16_X,
handler: 'example-function.handler',
code: lambda.Code.fromAsset('lambda')
})
myBucket.addEventNotification(s3.EventType.OBJECT_CREATED, new notifications.LambdaDestination(myFunction))
}
}
module.exports = { CdkS3EventTriggerLambdaStack }
Once again, check that the tests pass.
Stack outputs
Before we can deploy and end-to-end test our application we need to configure CDK to output the names of the S3 bucket and Lambda function when they are created.
If you are familiar with Cloudformation then you will probably know about stack outputs, we can configure these by importing CfnOutput from the CDK library.
const { CfnOutput, Stack, RemovalPolicy } = require('aws-cdk-lib');
And then we can create them for both the bucket name and the function name like this:
new CfnOutput(this, 'bucketName', {
value: myBucket.bucketName,
description: 'The name of the s3 bucket'
});
new CfnOutput(this, 'functionName', {
value: myFunction.functionName,
description: 'The name of the lambda function'
});
Putting it all together your final stack file should look like this:
// lib/cdk-s3-event-trigger-lambda-stack.js
const { Stack, RemovalPolicy } = require('aws-cdk-lib');
const s3 = require('aws-cdk-lib/aws-s3');
const lambda = require('aws-cdk-lib/aws-lambda');
const notifications = require('aws-cdk-lib/aws-s3-notifications');
class CdkS3EventTriggerLambdaStack extends Stack {
/**
*
* @param {Construct} scope
* @param {string} id
* @param {StackProps=} props
*/
constructor(scope, id, props) {
super(scope, id, props);
const myBucket = new s3.Bucket(this, 'ExampleBucket', {
removalPolicy: RemovalPolicy.DESTROY,
autoDeleteObjects: true
});
const myFunction = new lambda.Function(this, 'ExampleFunction', {
runtime: lambda.Runtime.NODEJS_16_X,
handler: 'example-function.handler',
code: lambda.Code.fromAsset('lambda')
})
myBucket.addEventNotification(s3.EventType.OBJECT_CREATED, new notifications.LambdaDestination(myFunction))
}
}
module.exports = { CdkS3EventTriggerLambdaStack }
Deploy the application to AWS
At this point we are ready to deploy our application and test it end to end.
To deploy the application we need to run cdk deploy and hit yes to confirm you'd like to create the resources listed.
$ cdk deploy
✨ Synthesis time: 2.04s
This deployment will make potentially sensitive changes according to your current security approval level (--require-approval broadening).
Please confirm you intend to make the following modifications:
IAM Statement Changes
┌───┬───────────────────┬────────┬───────────────────┬───────────────────┬───────────────────┐
│ │ Resource │ Effect │ Action │ Principal │ Condition │
├───┼───────────────────┼────────┼───────────────────┼───────────────────┼───────────────────┤
│ + │ ${BucketNotificat │ Allow │ sts:AssumeRole │ Service:lambda.am │ │
│ │ ionsHandler050a05 │ │ │ azonaws.com │ │
│ │ 87b7544547bf325f0 │ │ │ │ │
│ │ 94a3db834/Role.Ar │ │ │ │ │
│ │ n} │ │ │ │ │
├───┼───────────────────┼────────┼───────────────────┼───────────────────┼───────────────────┤
│ + │ ${Custom::S3AutoD │ Allow │ sts:AssumeRole │ Service:lambda.am │ │
│ │ eleteObjectsCusto │ │ │ azonaws.com │ │
│ │ mResourceProvider │ │ │ │ │
│ │ /Role.Arn} │ │ │ │ │
├───┼───────────────────┼────────┼───────────────────┼───────────────────┼───────────────────┤
│ + │ ${ExampleBucket.A │ Allow │ s3:DeleteObject* │ AWS:${Custom::S3A │ │
│ │ rn} │ │ s3:GetBucket* │ utoDeleteObjectsC │ │
│ │ ${ExampleBucket.A │ │ s3:List* │ ustomResourceProv │ │
│ │ rn}/* │ │ │ ider/Role.Arn} │ │
├───┼───────────────────┼────────┼───────────────────┼───────────────────┼───────────────────┤
│ + │ ${ExampleFunction │ Allow │ lambda:InvokeFunc │ Service:s3.amazon │ "ArnLike": { │
│ │ .Arn} │ │ tion │ aws.com │ "AWS:SourceArn" │
│ │ │ │ │ │ : "${ExampleBucke │
│ │ │ │ │ │ t.Arn}" │
│ │ │ │ │ │ }, │
│ │ │ │ │ │ "StringEquals": { │
│ │ │ │ │ │ "AWS:SourceAcco │
│ │ │ │ │ │ unt": "${AWS::Acc │
│ │ │ │ │ │ ountId}" │
│ │ │ │ │ │ } │
├───┼───────────────────┼────────┼───────────────────┼───────────────────┼───────────────────┤
│ + │ ${ExampleFunction │ Allow │ sts:AssumeRole │ Service:lambda.am │ │
│ │ /ServiceRole.Arn} │ │ │ azonaws.com │ │
├───┼───────────────────┼────────┼───────────────────┼───────────────────┼───────────────────┤
│ + │ * │ Allow │ s3:PutBucketNotif │ AWS:${BucketNotif │ │
│ │ │ │ ication │ icationsHandler05 │ │
│ │ │ │ │ 0a0587b7544547bf3 │ │
│ │ │ │ │ 25f094a3db834/Rol │ │
│ │ │ │ │ e} │ │
└───┴───────────────────┴────────┴───────────────────┴───────────────────┴───────────────────┘
IAM Policy Changes
┌───┬───────────────────────────────────────────┬────────────────────────────────────────────┐
│ │ Resource │ Managed Policy ARN │
├───┼───────────────────────────────────────────┼────────────────────────────────────────────┤
│ + │ ${BucketNotificationsHandler050a0587b7544 │ arn:${AWS::Partition}:iam::aws:policy/serv │
│ │ 547bf325f094a3db834/Role} │ ice-role/AWSLambdaBasicExecutionRole │
├───┼───────────────────────────────────────────┼────────────────────────────────────────────┤
│ + │ ${Custom::S3AutoDeleteObjectsCustomResour │ {"Fn::Sub":"arn:${AWS::Partition}:iam::aws │
│ │ ceProvider/Role} │ :policy/service-role/AWSLambdaBasicExecuti │
│ │ │ onRole"} │
├───┼───────────────────────────────────────────┼────────────────────────────────────────────┤
│ + │ ${ExampleFunction/ServiceRole} │ arn:${AWS::Partition}:iam::aws:policy/serv │
│ │ │ ice-role/AWSLambdaBasicExecutionRole │
└───┴───────────────────────────────────────────┴────────────────────────────────────────────┘
(NOTE: There may be security-related changes not in this list. See https://github.com/aws/aws-cdk/issues/1299)
Do you wish to deploy these changes (y/n)? y
CdkS3EventTriggerLambdaStack: deploying...
[0%] start: Publishing ac9d3642b3e0626d3a6d8c4f5f56507a478c329febdf1dcfbbf7c7db9812cd30:current_account-current_region
[0%] start: Publishing 7de16f08a760bf3773f0e41238bde8c85e5c09c4da9aae0a67257800acdda6e9:current_account-current_region
[0%] start: Publishing 051e534a39681330e15be7581d2a8b7ac8655075406ced1c15a87946e4f7d83d:current_account-current_region
[33%] success: Published 7de16f08a760bf3773f0e41238bde8c85e5c09c4da9aae0a67257800acdda6e9:current_account-current_region
[66%] success: Published ac9d3642b3e0626d3a6d8c4f5f56507a478c329febdf1dcfbbf7c7db9812cd30:current_account-current_region
[100%] success: Published 051e534a39681330e15be7581d2a8b7ac8655075406ced1c15a87946e4f7d83d:current_account-current_region
CdkS3EventTriggerLambdaStack: creating CloudFormation changeset...
✅ CdkS3EventTriggerLambdaStack
✨ Deployment time: 79.71s
Outputs:
CdkS3EventTriggerLambdaStack.bucketName = cdks3eventtriggerlambdastac-examplebucketdc717cf4-fccxjh0xv9jv
CdkS3EventTriggerLambdaStack.functionName = CdkS3EventTriggerLambdaSta-ExampleFunctionB28997EC-0d2wDCcV247P
Stack ARN:
arn:aws:cloudformation:eu-west-1:483567639850:stack/CdkS3EventTriggerLambdaStack/fa0f90e0-00b8-11ed-ab4c-022d51d75423
✨ Total time: 81.75s
You can see the outputs at the bottom - the bucket name and function name outputs that we created earlier. We can use these to test the process by uploading a file to that bucket, and checking the function logs the filename to Cloudwatch.
Lets create hello.txt and upload it to the bucket with the AWS CLI:
$ touch hello.txt
$ aws s3 cp hello.txt s3://cdks3eventtriggerlambdastac-examplebucketdc717cf4-fccxjh0xv9jv
upload: ./hello.txt to s3://cdks3eventtriggerlambdastac-examplebucketdc717cf4-fccxjh0xv9jv/hello.txt
And finally, check the logs. The Cloudwatch log path will be in the form /aws/lambda/[function_name], and you can tail the logs with aws logs tail --follow like this:
aws logs tail --follow /aws/lambda/CdkS3EventTriggerLambdaSta-ExampleFunctionB28997EC-0d2wDCcV247P
2022-07-11T01:35:30.436000+00:00 2022/07/11/[$LATEST]71c9e0701dc04ee38eb5050b2d4ce36d START RequestId: a8e39867-1a8c-4066-a8dd-b6956df38dbe Version: $LATEST
2022-07-11T01:35:30.443000+00:00 2022/07/11/[$LATEST]71c9e0701dc04ee38eb5050b2d4ce36d 2022-07-11T01:35:30.438Z a8e39867-1a8c-4066-a8dd-b6956df38dbe INFO hello.txt
2022-07-11T01:35:30.444000+00:00 2022/07/11/[$LATEST]71c9e0701dc04ee38eb5050b2d4ce36d END RequestId: a8e39867-1a8c-4066-a8dd-b6956df38dbe
2022-07-11T01:35:30.444000+00:00 2022/07/11/[$LATEST]71c9e0701dc04ee38eb5050b2d4ce36d REPORT RequestId: a8e39867-1a8c-4066-a8dd-b6956df38dbe Duration: 7.45 ms Billed Duration: 8 ms Memory Size: 128 MB Max Memory Used: 56 MBInit Duration: 143.56 ms
Success!
I hope this post has been helpful to you, if it has then let me know!