Skip to content
The Chat Archive
claude-opus-4-20250514· Anthropic LMArena Human Preference 140K

implement lambda to read a file from s3 and load to dynamo db, in typescript

  1. User

    implement lambda to read a file from s3 and load to dynamo db, in typescript

  2. Assistant

    Here's a TypeScript Lambda function that reads a file from S3 and loads it to DynamoDB:

    import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";
    import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
    import { DynamoDBDocumentClient, BatchWriteCommand } from "@aws-sdk/lib-dynamodb";
    import { Context, S3Event } from "aws-lambda";
    import { Readable } from "stream";
    
    const s3Client = new S3Client({});
    const dynamoClient = new DynamoDBClient({});
    const docClient = DynamoDBDocumentClient.from(dynamoClient);
    
    const TABLE_NAME = process.env.DYNAMO_TABLE_NAME || "YourTableName";
    const BATCH_SIZE = 25; // DynamoDB batch write limit
    
    export const handler = async (event: S3Event, context: Context) => {
      try {
        // Process each S3 event record
        for (const record of event.Records) {
          const bucket = record.s3.bucket.name;
          const key = decodeURIComponent(record.s3.object.key.replace(/\+/g, " "));
          
          console.log(`Processing file: s3://${bucket}/${key}`);
          
          // Get object from S3
          const getObjectCommand = new GetObjectCommand({
            Bucket: bucket,
            Key: key,
          });
          
          const s3Response = await s3Client.send(getObjectCommand);
          
          // Convert stream to string
          const bodyContents = await streamToString(s3Response.Body as Readable);
          
          // Parse the file content (assuming JSON format)
          let items: any[];
          try {
            items = JSON.parse(bodyContents);
            
            // Handle both single object and array of objects
            if (!Array.isArray(items)) {
              items = [items];
            }
          } catch (parseError) {
            console.error("Error parsing JSON:", parseError);
            throw new Error(`Failed to parse JSON from file ${key}`);
          }
          
          // Process items in batches
          await processBatches(items);
          
          console.log(`Successfully processed ${items.length} items from ${key}`);
        }
        
        return {
          statusCode: 200,
          body: JSON.stringify({ message: "File processed successfully" }),
        };
        
      } catch (error) {
        console.error("Error processing S3 event:", error);
        throw error;
      }
    };
    
    async function processBatches(items: any[]): Promise<void> {
      const batches = [];
      
      // Split items into batches of 25
      for (let i = 0; i < items.length; i += BATCH_SIZE) {
        batches.push(items.slice(i, i + BATCH_SIZE));
      }
      
      // Process each batch
      for (const [index, batch] of batches.entries()) {
        console.log(`Processing batch ${index + 1} of ${batches.length}`);
        
        const putRequests = batch.map(item => ({
          PutRequest: {
            Item: {
              ...item,
              // Add any additional fields or transformations here
              uploadedAt: new Date().toISOString(),
            }
          }
        }));
        
        const batchWriteCommand = new BatchWriteCommand({
          RequestItems: {
            [TABLE_NAME]: putRequests
          }
        });
        
        try {
          const response = await docClient.send(batchWriteCommand);
          
          // Handle unprocessed items
          if (response.UnprocessedItems && Object.keys(response.UnprocessedItems).length > 0) {
            console.warn("Unprocessed items detected, retrying...");
            await handleUnprocessedItems(response.UnprocessedItems);
          }
        } catch (error) {
          console.error(`Error writing batch ${index + 1}:`, error);
          throw error;
        }
      }
    }
    
    async function handleUnprocessedItems(unprocessedItems: any, retries = 3): Promise<void> {
      if (retries === 0) {
        throw new Error("Max retries reached for unprocessed items");
      }
      
      // Exponential backoff
      await new Promise(resolve => setTimeout(resolve, Math.pow(2, 4 - retries) * 100));
      
      const retryCommand = new BatchWriteCommand({
        RequestItems: unprocessedItems
      });
      
      const response = await docClient.send(retryCommand);
      
      if (response.UnprocessedItems && Object.keys(response.UnprocessedItems).length > 0) {
        await handleUnprocessedItems(response.UnprocessedItems, retries - 1);
      }
    }
    
    async function streamToString(stream: Readable): Promise<string> {
      const chunks: Buffer[] = [];
      
      return new Promise((resolve, reject) => {
        stream.on("data", (chunk) => chunks.push(Buffer.from(chunk)));
        stream.on("error", (err) => reject(err));
        stream.on("end", () => resolve(Buffer.concat(chunks).toString("utf-8")));
      });
    }
    

    You'll also need the corresponding package.json:

    {
      "name": "s3-to-dynamodb-lambda",
      "version": "1.0.0",
      "description": "Lambda function to read from S3 and write to DynamoDB",
      "main": "index.js",
      "scripts": {
        "build": "tsc",
        "package": "npm run build && zip -r function.zip dist/ node_modules/"
      },
      "dependencies": {
        "@aws-sdk/client-s3": "^3.x.x",
        "@aws-sdk/client-dynamodb": "^3.x.x",
        "@aws-sdk/lib-dynamodb": "^3.x.x"
      },
      "devDependencies": {
        "@types/aws-lambda": "^8.x.x",
        "@types/node": "^20.x.x",
        "typescript": "^5.x.x"
      }
    }
    

    And the tsconfig.json:

    {
      "compilerOptions": {
        "target": "ES2022",
        "module": "commonjs",
        "lib": ["ES2022"],
        "outDir": "./dist",
        "rootDir": "./src",
        "strict": true,
        "esModuleInterop": true,
        "skipLibCheck": true,
        "forceConsistentCasingInFileNames": true,
        "resolveJsonModule": true,
        "moduleResolution": "node"
      },
      "include": ["src/**/*"],
      "exclude": ["node_modules", "dist"]
    }
    

    To deploy this Lambda function, you'll also need to set up:

    1. IAM Role with permissions for:

      • S3 read access to your bucket
      • DynamoDB write access to your table
      • CloudWatch Logs for logging
    2. Environment Variables:

      • DYNAMO_TABLE_NAME: Your DynamoDB table name
    3. S3 Event Trigger: Configure your S3 bucket to trigger this Lambda on object creation

    The function handles:

    • Reading files from S3 triggered by S3 events
    • Parsing JSON content (you can modify for CSV or other formats)
    • Batch writing to DynamoDB (respecting the 25-item limit)
    • Retry logic for unprocessed items
    • Error handling and logging

    You can customize the data transformation logic in the processBatches function based on your specific requirements.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.