implement lambda to read a file from s3 and load to dynamo db, in typescript
- User
implement lambda to read a file from s3 and load to dynamo db, in typescript
- Assistant
Here's a step-by-step guide on implementing an AWS Lambda function in TypeScript to read a file from S3 and load its contents into DynamoDB:
Prerequisites:
- AWS Account: You need an AWS account with access to S3, DynamoDB, and Lambda.
- AWS CLI: Installed and configured on your machine.
- Node.js (14.x or higher) and npm (6.x or higher) installed.
- TypeScript (4.x or higher) installed globally (
npm install -g typescript). - AWS SAM CLI (optional but recommended for local testing).
Step 1: Set up your project
Create a new directory for your project and initialize it with
npm:mkdir s3-to-dynamodb-lambda cd s3-to-dynamodb-lambda npm init -yInstall required dependencies:
npm install --save aws-sdk typescript @types/aws-sdk @types/node npm install --save-dev @types/aws-lambdaInitialize a TypeScript project:
tsc --initUpdate your
tsconfig.jsonto include the following settings:{ "compilerOptions": { "target": "es2020", "module": "commonjs", "rootDir": "./", "outDir": "./build", "esModuleInterop": true, "strict": true } }Step 2: Write the Lambda function code
Create a file named
index.ts. This will contain the Lambda function code:import * as AWS from 'aws-sdk'; import { S3Event } from 'aws-lambda'; const s3 = new AWS.S3({ region: 'your-region' }); // e.g., 'us-east-1' const dynamoDb = new AWS.DynamoDB.DocumentClient({ region: 'your-region' }); const tableName = 'your-dynamodb-table-name'; // Replace with your DynamoDB table name export const handler = async (event: S3Event): Promise<void> => { try { console.log('Event:', JSON.stringify(event)); // Get the S3 bucket and object key from the event const bucket = event.Records[0].s3.bucket.name; const key = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, ' ')); console.log(`Reading file from S3: ${bucket}/${key}`); // Read the file from S3 const params = { Bucket: bucket, Key: key }; const s3Data = await s3.getObject(params).promise(); const fileContent = s3Data.Body?.toString('utf-8'); if (!fileContent) { throw new Error('File is empty or could not be read.'); } // Assuming the file is a JSON array, parse it const items = JSON.parse(fileContent); if (!Array.isArray(items)) { throw new Error('File content is not a valid JSON array.'); } // Batch write items to DynamoDB const putRequests = items.map(item => ({ PutRequest: { Item: item, }, })); const dynamoParams = { RequestItems: { [tableName]: putRequests, }, }; console.log(`Writing ${items.length} items to DynamoDB table: ${tableName}`); await dynamoDb.batchWrite(dynamoParams).promise(); console.log('Data successfully loaded into DynamoDB'); } catch (error) { console.error('Error:', error); throw error; // Let Lambda retry or handle the error based on your setup } };Important points:
- Event Trigger: This Lambda function is designed to be triggered by an S3
ObjectCreatedevent. TheS3Eventtype reflects this. - File Format: The example assumes your S3 file is a JSON array. Adjust the parsing logic if your file format differs (e.g., CSV, newline-delimited JSON).
- DynamoDB Table: Replace
your-dynamodb-table-namewith the actual name of your DynamoDB table. - IAM Permissions: Ensure your Lambda execution role has permissions to:
- Read from the S3 bucket (
s3:GetObject). - Write to the DynamoDB table (
dynamodb:BatchWriteItem).
- Read from the S3 bucket (
Step 3: Build and package the Lambda function
Add a build script to your
package.json:"scripts": { "build": "tsc" }Run the build command:
npm run buildThis compiles your
index.tsinto JavaScript in thebuildfolder.Step 4: Deploy the Lambda function
You can deploy using the AWS SAM CLI or directly via the AWS Management Console. Here's a quick SAM approach:
- Install SAM CLI: https://aws.amazon.com/serverless/sam/
- Create a
template.yamlin your project root:
AWSTemplateFormatVersion: '2010-09-09' Transform: AWS::Serverless-2016-10-31 Description: S3 to DynamoDB Lambda function Resources: S3ToDynamoLambda: Type: AWS::Serverless::Function Properties: FunctionName: !Sub 's3-to-dynamo-${AWS::Region}' Handler: index.handler Runtime: nodejs14.x CodeUri: build/ Policies: - S3ReadPolicy: BucketName: 'your-s3-bucket-name' # Replace with your S3 bucket - DynamoDBCrudPolicy: TableName: 'your-dynamodb-table-name' # Replace with your DynamoDB table Events: S3Trigger: Type: S3 Properties: Bucket: !Ref S3Bucket Events: s3:ObjectCreated:* S3Bucket: Type: AWS::S3::Bucket Properties: BucketName: 'your-s3-bucket-name' # Ensure this matches the bucket you're using Outputs: LambdaFunction: Description: 'Lambda Function ARN' Value: !GetAtt S3ToDynamoLambda.Arn Export: Name: !Sub '${AWS::StackName}-LambdaArn'- Deploy using SAM:
sam deploy --guidedFollow the prompts to set your stack name, region, etc. Once deployed:
- Go to the S3 Console and upload a JSON file (e.g.,
data.json) to the specified bucket:
[ { "id": "1", "name": "Alice" }, { "id": "2", "name": "Bob" } ]- The Lambda function will automatically trigger, read
data.json, and load the items into your DynamoDB table.
Troubleshooting Tips:
- CloudWatch Logs: Check Lambda execution logs for errors.
- S3 Event Test: Manually trigger the Lambda via the S3 console or test event in the Lambda console.
- IAM Permissions: Validate the Lambda role has
s3:GetObjectanddynamodb:BatchWriteItempermissions.
That's it! You've now implemented a TypeScript Lambda function to move data from S3 to DynamoDB. Happy coding! 🚀
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.