Skip to content
TheChatArchiveThe Chat Archive
Qwenqwen3-235b-a22b· Alibaba (Qwen)Hugging Face LMArena Human Preference 140K

I’m using boto3 to upload file objects to an S3 compatible storage. I’m using…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-3-7-sonnet-20250219-thinking-32k’s answer.
  1. User

    I’m using boto3 to upload file objects to an S3 compatible storage. I’m using send_fileobj to upload large directories compressed as tar archives. I’m getting HTTP 500 errors. How to retry these errors? Won’t my file cursor position prevent retries from being correct? What if I wanted to retry HTTP 400 as well?

  2. Qwen Assistant

    To handle retries when uploading large files with boto3's upload_fileobj while addressing HTTP 500/400 errors and cursor position issues, follow these steps:


    Key Considerations

    1. Cursor Position: If a retry occurs, the file cursor must reset to the start (seek(0)), or the upload will resume mid-file, corrupting the object.
    2. Seekable File Objects: Ensure your file object supports seek() (e.g., files opened in binary mode). Non-seekable streams (e.g., sockets) require alternative approaches.
    3. Retry Logic: Implement retries for HTTP 500 errors (server-side) and selective 400 errors (only if transient).
    4. Multipart Uploads: Large files use multipart uploads internally. Failed parts may retry automatically, but full-upload failures require manual handling.

    Solution Code

    import boto3
    import time
    from botocore.exceptions import ClientError
    
    def upload_with_retries(file_path, bucket, key, max_retries=3, retry_delay=5):
        s3_client = boto3.client('s3')
        
        for attempt in range(max_retries + 1):
            try:
                # Open the file in binary mode for each attempt to ensure a fresh cursor
                with open(file_path, 'rb') as file_obj:
                    s3_client.upload_fileobj(file_obj, bucket, key)
                print("Upload successful")
                return
            except ClientError as e:
                error_code = e.response['Error']['Code']
                print(f"Attempt {attempt + 1} failed with error: {error_code}")
                
                if error_code in ['500', '503', '400'] and attempt < max_retries:
                    print(f"Retrying in {retry_delay} seconds...")
                    time.sleep(retry_delay)
                else:
                    print("Max retries reached or non-retryable error. Aborting.")
                    raise
    

    Explanation

    1. File Reopening: The file is reopened in each retry iteration (with open(...)) to reset the cursor automatically. This avoids issues with non-seekable streams.
    2. Retry Conditions:
      • Retries HTTP 500/503 (server errors) unconditionally.
      • Retries HTTP 400 only if you explicitly trust it to be transient (e.g., rate limiting).
    3. Exponential Backoff: Add retry_delay *= 2 inside the loop for exponential backoff.
    4. Error Handling: Catches ClientError to inspect error codes and re-raise non-retryable errors.

    Best Practices

    • Avoid Retrying 400 Errors: HTTP 400 (Bad Request) usually indicates a client-side issue (e.g., invalid headers). Only retry if certain the error is transient (e.g., temporary throttling).
    • Monitor Multipart Uploads: For very large files, use create_multipart_upload/upload_part/complete_multipart_upload for finer control.
    • Use seek(0) if Reusing File Objects: If you reuse the same file object (not reopened), call file_obj.seek(0) before each retry.
    • Memory Constraints: For massive files, avoid loading into memory (e.g., use streaming or disk-based reads).

    Example Usage

    upload_with_retries(
        file_path='path/to/large_archive.tar',
        bucket='my-bucket',
        key='uploads/large_archive.tar',
        max_retries=5,
        retry_delay=10
    )
    

    When to Use Built-in Retries

    Boto3/botocore automatically retries:

    • 5xx errors (server errors).
    • 400 errors like RequestTimeout, Throttling, or ProvisionedThroughputExceededException.

    Custom retry logic is needed if:

    • You need to retry non-default 4xx errors.
    • Your file object lacks seek() support.
    • You want explicit control over backoff logic.

    By combining file reopening/resetting and explicit error checking, you ensure reliable retries for S3 uploads.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.