S3 AWS Lambda

Suppose you want to process a file that is uploaded to a bucket. You can create a Lambda function (BucketFileTransfer) that Amazon S3 can invoke when objects are created. Then, the Lambda function can read the image object from the source bucket and create a copy image target bucket.

In this post I will show you how to create a Java Lambda function in order to copy a file (HappyFace.jpg) from one bucket to another. First we need to create a Java basic project this time using lazybones.

lazybones create java-basic s3-aws-lambda

Previous command will create this structure

<proj>
      |
      +- src
          |
          +- main
          |     |
          |     +- java
          |
          +- test
          |   |
          |   +- java

Edit your build.gradle file to create a make a Zip task and add lambda AWS dependencies.

apply plugin: "java"
apply plugin: "application"

version = '0.0.1'

task buildZip(type: Zip) {
    from compileJava
    from processResources
    into('lib') {
        from configurations.runtime
    }
}

repositories {
  mavenCentral()
}

dependencies {
  compile 'com.amazonaws:aws-lambda-java-core:1.1.0'
  compile 'com.amazonaws:aws-lambda-java-events:1.1.0'
}

Then define Java classes under project-dir/src/main/java/example/

BucketFileTransfer

package example;

import java.io.InputStream;
import com.amazonaws.services.lambda.runtime.events.S3Event;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.LambdaLogger;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.s3.model.ObjectMetadata;

public class BucketFileTransfer implements RequestHandler<S3Event, Integer> {

  @Override
  public Integer handleRequest(S3Event event, Context context) {
    LambdaLogger logger = context.getLogger();
    logger.log("STARTING to copy file");

    MetadataFileHelper metadataFileHelper = new MetadataFileHelper(event);
    String sourceBucket = metadataFileHelper.getSourceBucketName();
    String sourceKey = metadataFileHelper.getSourceBucketKey();
    String destinationBucket = metadataFileHelper.getDestinationBucketName();;
    String destinationKey = metadataFileHelper.getDestinationBucketKey();

    AWSClient awsClient = new AWSClient(sourceBucket, sourceKey);
    InputStream objectData = awsClient.getS3Object().getObjectContent();
    ObjectMetadata meta = awsClient.getS3Object().getObjectMetadata();
    awsClient.getS3Client().putObject(destinationBucket, destinationKey, objectData, meta);
    return ResultCode.OK;
  }

}

Basically in this piece of code we are getting the source bucket, file and metatada. We are getting destination bucket as well, finally we are writing the file in the destination bucket.

MetadataFileHelper

package example;

import com.amazonaws.services.lambda.runtime.events.S3Event;
import com.amazonaws.services.s3.event.S3EventNotification.S3Entity;

public class MetadataFileHelper {

  private S3Event event;

  public MetadataFileHelper(S3Event event){
    this.event = event;
  }

  private S3Entity getBucketEntity(){
    return this.event.getRecords().get(0).getS3();
  }

  public String getSourceBucketName() {
    return getBucketEntity().getBucket().getName();
  }

  public String getSourceBucketKey() {
    return getBucketEntity().getObject().getKey();
  }

  public String getDestinationBucketName() {
    return "bucket-s3-destination";
  }

  public String getDestinationBucketKey() {
    return "copied-" + getBucketEntity().getObject().getKey();
  }

}

We are using this helper to get source and destination buckets name and keys.

AWSClient

package example;

import com.amazonaws.ClientConfiguration;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.S3Object;
import com.amazonaws.services.s3.model.GetObjectRequest;

public class AWSClient {

  String sourceBucket;
  String sourceKey;
  AmazonS3 s3Client;

  public AWSClient(String sourceBucket, String sourceKey){
    this.sourceBucket = sourceBucket;
    this.sourceKey = sourceKey;
    AWSCredentials credentials = new BasicAWSCredentials("Access Key ID", "Secret Access Key");
    ClientConfiguration clientConfig = new ClientConfiguration();
    this.s3Client = new AmazonS3Client(credentials, clientConfig);
  }

  public S3Object getS3Object() {
    return getS3Client().getObject(new GetObjectRequest(sourceBucket, sourceKey));
  }

  public AmazonS3 getS3Client(){
    return this.s3Client;
  }
}

We are using this client to get AWS connection, Amazon S3 client and Amazon S3 Object.

Use the following gradle command to generate your standalone .zip deployment file

gradle buildZip

Then go to your AWS console and create a new Lambda AWS function to upload this project.

You can invoke your lambda function installing command line interface AWS CLI

aws lambda invoke \
--invocation-type Event \
--function-name s3-aws-lambda \
--region us-west-1 \
--payload file://inputfile.txt \
outputfile.txt

Where inputfile.txt is:

{
  "Records": [
    {
      "eventVersion": "2.0",
      "eventTime": "1970-01-01T00:00:00.000Z",
      "requestParameters": {
        "sourceIPAddress": "127.0.0.1"
      },
      "s3": {
        "configurationId": "testConfigRule",
        "object": {
          "eTag": "0123456789abcdef0123456789abcdef",
          "sequencer": "0A1B2C3D4E5F678901",
          "key": "HappyFace.jpg",
          "size": 1024
        },
        "bucket": {
          "arn": "arn:aws:s3:::josdem-s3-source",
          "name": "josdem-s3-source",
          "ownerIdentity": {
            "principalId": "EXAMPLE"
          }
        },
        "s3SchemaVersion": "1.0"
      },
      "responseElements": {
        "x-amz-id-2": "EXAMPLE123/5678abcdefghijklambdaisawesome/mnopqrstuvwxyzABCDEFGH",
        "x-amz-request-id": "EXAMPLE123456789"
      },
      "awsRegion": "us-east-1",
      "eventName": "ObjectCreated:Put",
      "userIdentity": {
        "principalId": "EXAMPLE"
      },
      "eventSource": "aws:s3"
    }
  ]
}

After invoke command you should see this output:

{
  "StatusCode": 202
}

And the HappyFace.jpg image copied.

To download the code:

git clone https://github.com/josdem/java-topics.git
git fetch
git checkout feature/copy-bucket-file
cd s3-aws-lambda

Return to the main article