Skip to content
The Chat Archive
llama-4-maverick-03-26-experimental· Meta LMArena Human Preference 140K

implement lambda to read a file from s3 and load to dynamo db, in typescript

  1. User

    implement lambda to read a file from s3 and load to dynamo db, in typescript

  2. Assistant

    Here's a step-by-step guide on implementing an AWS Lambda function in TypeScript to read a file from S3 and load its contents into DynamoDB:

    Prerequisites:

    1. AWS Account: You need an AWS account with access to S3, DynamoDB, and Lambda.
    2. AWS CLI: Installed and configured on your machine.
    3. Node.js (14.x or higher) and npm (6.x or higher) installed.
    4. TypeScript (4.x or higher) installed globally (npm install -g typescript).
    5. AWS SAM CLI (optional but recommended for local testing).

    Step 1: Set up your project

    Create a new directory for your project and initialize it with npm:

    mkdir s3-to-dynamodb-lambda
    cd s3-to-dynamodb-lambda
    npm init -y
    

    Install required dependencies:

    npm install --save aws-sdk typescript @types/aws-sdk @types/node
    npm install --save-dev @types/aws-lambda
    

    Initialize a TypeScript project:

    tsc --init
    

    Update your tsconfig.json to include the following settings:

    {
      "compilerOptions": {
        "target": "es2020",
        "module": "commonjs",
        "rootDir": "./",
        "outDir": "./build",
        "esModuleInterop": true,
        "strict": true
      }
    }
    

    Step 2: Write the Lambda function code

    Create a file named index.ts. This will contain the Lambda function code:

    import * as AWS from 'aws-sdk';
    import { S3Event } from 'aws-lambda';
    
    const s3 = new AWS.S3({ region: 'your-region' }); // e.g., 'us-east-1'
    const dynamoDb = new AWS.DynamoDB.DocumentClient({ region: 'your-region' });
    const tableName = 'your-dynamodb-table-name'; // Replace with your DynamoDB table name
    
    export const handler = async (event: S3Event): Promise<void> => {
      try {
        console.log('Event:', JSON.stringify(event));
    
        // Get the S3 bucket and object key from the event
        const bucket = event.Records[0].s3.bucket.name;
        const key = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, ' '));
    
        console.log(`Reading file from S3: ${bucket}/${key}`);
    
        // Read the file from S3
        const params = { Bucket: bucket, Key: key };
        const s3Data = await s3.getObject(params).promise();
        const fileContent = s3Data.Body?.toString('utf-8');
    
        if (!fileContent) {
          throw new Error('File is empty or could not be read.');
        }
    
        // Assuming the file is a JSON array, parse it
        const items = JSON.parse(fileContent);
    
        if (!Array.isArray(items)) {
          throw new Error('File content is not a valid JSON array.');
        }
    
        // Batch write items to DynamoDB
        const putRequests = items.map(item => ({
          PutRequest: {
            Item: item,
          },
        }));
    
        const dynamoParams = {
          RequestItems: {
            [tableName]: putRequests,
          },
        };
    
        console.log(`Writing ${items.length} items to DynamoDB table: ${tableName}`);
    
        await dynamoDb.batchWrite(dynamoParams).promise();
    
        console.log('Data successfully loaded into DynamoDB');
      } catch (error) {
        console.error('Error:', error);
        throw error; // Let Lambda retry or handle the error based on your setup
      }
    };
    

    Important points:

    1. Event Trigger: This Lambda function is designed to be triggered by an S3 ObjectCreated event. The S3Event type reflects this.
    2. File Format: The example assumes your S3 file is a JSON array. Adjust the parsing logic if your file format differs (e.g., CSV, newline-delimited JSON).
    3. DynamoDB Table: Replace your-dynamodb-table-name with the actual name of your DynamoDB table.
    4. IAM Permissions: Ensure your Lambda execution role has permissions to:
      • Read from the S3 bucket (s3:GetObject).
      • Write to the DynamoDB table (dynamodb:BatchWriteItem).

    Step 3: Build and package the Lambda function

    Add a build script to your package.json:

    "scripts": {
      "build": "tsc"
    }
    

    Run the build command:

    npm run build
    

    This compiles your index.ts into JavaScript in the build folder.

    Step 4: Deploy the Lambda function

    You can deploy using the AWS SAM CLI or directly via the AWS Management Console. Here's a quick SAM approach:

    1. Install SAM CLI: https://aws.amazon.com/serverless/sam/
    2. Create a template.yaml in your project root:
    AWSTemplateFormatVersion: '2010-09-09'
    Transform: AWS::Serverless-2016-10-31
    Description: S3 to DynamoDB Lambda function
    
    Resources:
      S3ToDynamoLambda:
        Type: AWS::Serverless::Function
        Properties:
          FunctionName: !Sub 's3-to-dynamo-${AWS::Region}'
          Handler: index.handler
          Runtime: nodejs14.x
          CodeUri: build/
          Policies:
            - S3ReadPolicy:
                BucketName: 'your-s3-bucket-name' # Replace with your S3 bucket
            - DynamoDBCrudPolicy:
                TableName: 'your-dynamodb-table-name' # Replace with your DynamoDB table
          Events:
            S3Trigger:
              Type: S3
              Properties:
                Bucket: !Ref S3Bucket
                Events: s3:ObjectCreated:*
      S3Bucket:
        Type: AWS::S3::Bucket
        Properties:
          BucketName: 'your-s3-bucket-name' # Ensure this matches the bucket you're using
    
    Outputs:
      LambdaFunction:
        Description: 'Lambda Function ARN'
        Value: !GetAtt S3ToDynamoLambda.Arn
        Export:
          Name: !Sub '${AWS::StackName}-LambdaArn'
    
    1. Deploy using SAM:
    sam deploy --guided
    

    Follow the prompts to set your stack name, region, etc. Once deployed:

    • Go to the S3 Console and upload a JSON file (e.g., data.json) to the specified bucket:
    [
      { "id": "1", "name": "Alice" },
      { "id": "2", "name": "Bob" }
    ]
    
    • The Lambda function will automatically trigger, read data.json, and load the items into your DynamoDB table.

    Troubleshooting Tips:

    • CloudWatch Logs: Check Lambda execution logs for errors.
    • S3 Event Test: Manually trigger the Lambda via the S3 console or test event in the Lambda console.
    • IAM Permissions: Validate the Lambda role has s3:GetObject and dynamodb:BatchWriteItem permissions.

    That's it! You've now implemented a TypeScript Lambda function to move data from S3 to DynamoDB. Happy coding! 🚀

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.