claude-opus-4-20250514· Anthropic LMArena Human Preference 140K
implement lambda to read a file from s3 and load to dynamo db, in typescript
- User
implement lambda to read a file from s3 and load to dynamo db, in typescript
- Assistant
Here's a TypeScript Lambda function that reads a file from S3 and loads it to DynamoDB:
import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3"; import { DynamoDBClient } from "@aws-sdk/client-dynamodb"; import { DynamoDBDocumentClient, BatchWriteCommand } from "@aws-sdk/lib-dynamodb"; import { Context, S3Event } from "aws-lambda"; import { Readable } from "stream"; const s3Client = new S3Client({}); const dynamoClient = new DynamoDBClient({}); const docClient = DynamoDBDocumentClient.from(dynamoClient); const TABLE_NAME = process.env.DYNAMO_TABLE_NAME || "YourTableName"; const BATCH_SIZE = 25; // DynamoDB batch write limit export const handler = async (event: S3Event, context: Context) => { try { // Process each S3 event record for (const record of event.Records) { const bucket = record.s3.bucket.name; const key = decodeURIComponent(record.s3.object.key.replace(/\+/g, " ")); console.log(`Processing file: s3://${bucket}/${key}`); // Get object from S3 const getObjectCommand = new GetObjectCommand({ Bucket: bucket, Key: key, }); const s3Response = await s3Client.send(getObjectCommand); // Convert stream to string const bodyContents = await streamToString(s3Response.Body as Readable); // Parse the file content (assuming JSON format) let items: any[]; try { items = JSON.parse(bodyContents); // Handle both single object and array of objects if (!Array.isArray(items)) { items = [items]; } } catch (parseError) { console.error("Error parsing JSON:", parseError); throw new Error(`Failed to parse JSON from file ${key}`); } // Process items in batches await processBatches(items); console.log(`Successfully processed ${items.length} items from ${key}`); } return { statusCode: 200, body: JSON.stringify({ message: "File processed successfully" }), }; } catch (error) { console.error("Error processing S3 event:", error); throw error; } }; async function processBatches(items: any[]): Promise<void> { const batches = []; // Split items into batches of 25 for (let i = 0; i < items.length; i += BATCH_SIZE) { batches.push(items.slice(i, i + BATCH_SIZE)); } // Process each batch for (const [index, batch] of batches.entries()) { console.log(`Processing batch ${index + 1} of ${batches.length}`); const putRequests = batch.map(item => ({ PutRequest: { Item: { ...item, // Add any additional fields or transformations here uploadedAt: new Date().toISOString(), } } })); const batchWriteCommand = new BatchWriteCommand({ RequestItems: { [TABLE_NAME]: putRequests } }); try { const response = await docClient.send(batchWriteCommand); // Handle unprocessed items if (response.UnprocessedItems && Object.keys(response.UnprocessedItems).length > 0) { console.warn("Unprocessed items detected, retrying..."); await handleUnprocessedItems(response.UnprocessedItems); } } catch (error) { console.error(`Error writing batch ${index + 1}:`, error); throw error; } } } async function handleUnprocessedItems(unprocessedItems: any, retries = 3): Promise<void> { if (retries === 0) { throw new Error("Max retries reached for unprocessed items"); } // Exponential backoff await new Promise(resolve => setTimeout(resolve, Math.pow(2, 4 - retries) * 100)); const retryCommand = new BatchWriteCommand({ RequestItems: unprocessedItems }); const response = await docClient.send(retryCommand); if (response.UnprocessedItems && Object.keys(response.UnprocessedItems).length > 0) { await handleUnprocessedItems(response.UnprocessedItems, retries - 1); } } async function streamToString(stream: Readable): Promise<string> { const chunks: Buffer[] = []; return new Promise((resolve, reject) => { stream.on("data", (chunk) => chunks.push(Buffer.from(chunk))); stream.on("error", (err) => reject(err)); stream.on("end", () => resolve(Buffer.concat(chunks).toString("utf-8"))); }); }You'll also need the corresponding
package.json:{ "name": "s3-to-dynamodb-lambda", "version": "1.0.0", "description": "Lambda function to read from S3 and write to DynamoDB", "main": "index.js", "scripts": { "build": "tsc", "package": "npm run build && zip -r function.zip dist/ node_modules/" }, "dependencies": { "@aws-sdk/client-s3": "^3.x.x", "@aws-sdk/client-dynamodb": "^3.x.x", "@aws-sdk/lib-dynamodb": "^3.x.x" }, "devDependencies": { "@types/aws-lambda": "^8.x.x", "@types/node": "^20.x.x", "typescript": "^5.x.x" } }And the
tsconfig.json:{ "compilerOptions": { "target": "ES2022", "module": "commonjs", "lib": ["ES2022"], "outDir": "./dist", "rootDir": "./src", "strict": true, "esModuleInterop": true, "skipLibCheck": true, "forceConsistentCasingInFileNames": true, "resolveJsonModule": true, "moduleResolution": "node" }, "include": ["src/**/*"], "exclude": ["node_modules", "dist"] }To deploy this Lambda function, you'll also need to set up:
-
IAM Role with permissions for:
- S3 read access to your bucket
- DynamoDB write access to your table
- CloudWatch Logs for logging
-
Environment Variables:
DYNAMO_TABLE_NAME: Your DynamoDB table name
-
S3 Event Trigger: Configure your S3 bucket to trigger this Lambda on object creation
The function handles:
- Reading files from S3 triggered by S3 events
- Parsing JSON content (you can modify for CSV or other formats)
- Batch writing to DynamoDB (respecting the 25-item limit)
- Retry logic for unprocessed items
- Error handling and logging
You can customize the data transformation logic in the
processBatchesfunction based on your specific requirements. -
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.