Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-2.5-pro· GoogleHugging Face LMArena Human Preference 140K

Write me a bash script to migrate an aws glue table

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Googlegemma-3-27b-it’s answer.
  1. User

    Write me a bash script to migrate an aws glue table

  2. Gemini Assistant

    Of course! Here is a comprehensive and robust bash script to migrate an AWS Glue table definition from a source to a target. This script can handle migrations within the same AWS account/region or across different accounts and regions.

    Key Features of the Script

    • Cross-Account & Cross-Region: Easily specify different AWS profiles and regions for the source and target.
    • Dependency Checks: Ensures aws CLI and jq are installed before running.
    • Error Handling: Checks the exit status of AWS CLI commands and provides clear error messages.
    • User-Friendly: Provides a clear usage/help message and informative output.
    • Metadata Cleaning: Uses jq to automatically clean the table definition, removing read-only fields that would cause the create-table command to fail.

    The Script: migrate_glue_table.sh

    #!/bin/bash
    
    # ===================================================================================
    # AWS Glue Table Migration Script
    #
    # Description:
    # This script migrates an AWS Glue table definition from a source database/table
    # to a target database/table. It can operate within the same AWS account/region
    # or across different accounts and regions by using AWS CLI profiles.
    #
    # Note: This script migrates the TABLE METADATA ONLY. It does NOT move the
    # underlying data in S3. If the data location needs to change, you must handle
    # that separately. It also does not migrate partitions; they must be recreated.
    #
    # Dependencies:
    # - aws-cli: The AWS Command Line Interface. Must be configured.
    # - jq: A lightweight and flexible command-line JSON processor.
    #
    # Usage:
    # ./migrate_glue_table.sh --source-db <db> --source-table <tbl> \
    #                         --target-db <db> --target-table <tbl> \
    #                         [--source-profile <prof>] [--source-region <reg>] \
    #                         [--target-profile <prof>] [--target-region <reg>]
    #
    # Examples:
    # 1. Same account, same region:
    #    ./migrate_glue_table.sh --source-db raw_db --source-table users \
    #                            --target-db processed_db --target-table users_v2
    #
    # 2. Cross-region migration:
    #    ./migrate_glue_table.sh --source-db sales --source-table daily_sales \
    #                            --target-db sales --target-table daily_sales \
    #                            --source-region us-east-1 --target-region us-west-2
    #
    # 3. Cross-account migration:
    #    ./migrate_glue_table.sh --source-db analytics --source-table events \
    #                            --target-db analytics --target-table events \
    #                            --source-profile acct1-profile --target-profile acct2-profile
    #
    # ===================================================================================
    
    set -e # Exit immediately if a command exits with a non-zero status.
    set -o pipefail # The return value of a pipeline is the status of the last command to exit with a non-zero status.
    
    # --- Function to display usage information ---
    usage() {
        echo "Usage: $0 --source-db <db> --source-table <tbl> --target-db <db> --target-table <tbl> [--source-profile <prof>] [--source-region <reg>] [--target-profile <prof>] [--target-region <reg>]"
        echo "Migrates an AWS Glue table definition."
        echo ""
        echo "Required arguments:"
        echo "  --source-db       Name of the source Glue database."
        echo "  --source-table    Name of the source Glue table."
        echo "  --target-db       Name of the target Glue database."
        echo "  --target-table    Name of the new target Glue table."
        echo ""
        echo "Optional arguments:"
        echo "  --source-profile  AWS CLI profile for the source account."
        echo "  --source-region   AWS region for the source table."
        echo "  --target-profile  AWS CLI profile for the target account."
        echo "  --target-region   AWS region for the target table."
        echo "  --help            Display this help message."
        exit 1
    }
    
    # --- Parse Command Line Arguments ---
    while [[ "$#" -gt 0 ]]; do
        case $1 in
            --source-db) SOURCE_DB="$2"; shift ;;
            --source-table) SOURCE_TABLE="$2"; shift ;;
            --target-db) TARGET_DB="$2"; shift ;;
            --target-table) TARGET_TABLE="$2"; shift ;;
            --source-profile) SOURCE_PROFILE="$2"; shift ;;
            --source-region) SOURCE_REGION="$2"; shift ;;
            --target-profile) TARGET_PROFILE="$2"; shift ;;
            --target-region) TARGET_REGION="$2"; shift ;;
            --help) usage ;;
            *) echo "Unknown parameter passed: $1"; usage ;;
        esac
        shift
    done
    
    # --- Validate required arguments ---
    if [ -z "$SOURCE_DB" ] || [ -z "$SOURCE_TABLE" ] || [ -z "$TARGET_DB" ] || [ -z "$TARGET_TABLE" ]; then
        echo "Error: Missing required arguments."
        usage
    fi
    
    # --- Check for dependencies ---
    if ! command -v aws &> /dev/null; then
        echo "Error: aws-cli is not installed. Please install and configure it."
        exit 1
    fi
    if ! command -v jq &> /dev/null; then
        echo "Error: jq is not installed. Please install it (e.g., 'sudo apt-get install jq' or 'brew install jq')."
        exit 1
    fi
    
    # --- Construct AWS CLI options ---
    SOURCE_AWS_OPTS=""
    [ -n "$SOURCE_PROFILE" ] && SOURCE_AWS_OPTS+="--profile $SOURCE_PROFILE "
    [ -n "$SOURCE_REGION" ] && SOURCE_AWS_OPTS+="--region $SOURCE_REGION"
    
    TARGET_AWS_OPTS=""
    [ -n "$TARGET_PROFILE" ] && TARGET_AWS_OPTS+="--profile $TARGET_PROFILE "
    [ -n "$TARGET_REGION" ] && TARGET_AWS_OPTS+="--region $TARGET_REGION"
    
    # --- Main Script Logic ---
    
    echo "▶️  Starting Glue table migration..."
    echo "--------------------------------------------------"
    echo "Source:           $SOURCE_DB.$SOURCE_TABLE"
    [ -n "$SOURCE_PROFILE" ] && echo "Source Profile:   $SOURCE_PROFILE"
    [ -n "$SOURCE_REGION" ] && echo "Source Region:    $SOURCE_REGION"
    echo "Target:           $TARGET_DB.$TARGET_TABLE"
    [ -n "$TARGET_PROFILE" ] && echo "Target Profile:   $TARGET_PROFILE"
    [ -n "$TARGET_REGION" ] && echo "Target Region:    $TARGET_REGION"
    echo "--------------------------------------------------"
    
    # 1. Get the source table definition
    echo "➡️  Fetching definition for source table '$SOURCE_DB.$SOURCE_TABLE'..."
    SOURCE_TABLE_JSON=$(aws glue get-table --database-name "$SOURCE_DB" --name "$SOURCE_TABLE" $SOURCE_AWS_OPTS)
    
    if [ $? -ne 0 ]; then
        echo "❌ Error: Failed to get source table definition. Please check parameters and permissions."
        exit 1
    fi
    
    # 2. Transform the JSON for the create-table command
    #    We need to extract the 'Table' object, remove read-only fields,
    #    update the name, and wrap it in the structure required by 'create-table'.
    echo "➡️  Transforming table definition for target..."
    TARGET_TABLE_INPUT_JSON=$(echo "$SOURCE_TABLE_JSON" | jq \
        --arg target_db "$TARGET_DB" \
        --arg target_table "$TARGET_TABLE" \
        '{
            "DatabaseName": $target_db,
            "TableInput": (
                .Table |
                del(.DatabaseName, .CreateTime, .UpdateTime, .CreatedBy, .IsRegisteredWithLakeFormation, .CatalogId, .VersionId) |
                .Name = $target_table
            )
        }')
    
    # 3. Create the target table
    echo "➡️  Creating target table '$TARGET_DB.$TARGET_TABLE'..."
    aws glue create-table --cli-input-json "$TARGET_TABLE_INPUT_JSON" $TARGET_AWS_OPTS
    
    if [ $? -eq 0 ]; then
        echo "✅ Successfully migrated table '$SOURCE_DB.$SOURCE_TABLE' to '$TARGET_DB.$TARGET_TABLE'."
    else
        echo "❌ Error: Failed to create the target table. Check the AWS CLI output above for details."
        exit 1
    fi
    
    

    How to Use the Script

    1. Save the Code: Save the script above as a file, for example, migrate_glue_table.sh.

    2. Make it Executable: Open your terminal and run:

      chmod +x migrate_glue_table.sh
      
    3. Prerequisites:

      • AWS CLI: Make sure you have the AWS CLI installed and configured. If you are doing a cross-account migration, ensure you have set up named profiles in your ~/.aws/credentials and ~/.aws/config files.
      • jq: Make sure you have jq installed.
        • On macOS (using Homebrew): brew install jq
        • On Debian/Ubuntu: sudo apt-get install jq
        • On RHEL/CentOS: sudo yum install jq
    4. Run the Script: Execute the script with the appropriate arguments.

      • Example 1: Same Account, Different Database

        ./migrate_glue_table.sh \
            --source-db my_raw_data \
            --source-table customer_events_2023 \
            --target-db my_processed_data \
            --target-table customer_events
        
      • Example 2: Cross-Region

        ./migrate_glue_table.sh \
            --source-db production_db --source-table logs \
            --target-db dr_db --target-table logs \
            --source-region us-east-1 \
            --target-region us-west-2
        
      • Example 3: Cross-Account and Cross-Region

        ./migrate_glue_table.sh \
            --source-db sales_db --source-table transactions \
            --target-db analytics_db --target-table transactions \
            --source-profile dev-account --source-region us-east-1 \
            --target-profile prod-account --target-region eu-west-1
        

    Script Breakdown and Explanation

    1. Argument Parsing: The script uses a while loop and a case statement to parse named arguments like --source-db. This is more robust and user-friendly than relying on positional arguments.
    2. Dependency Checks: command -v checks if aws and jq exist in the system's PATH. This prevents the script from failing with cryptic errors later on.
    3. aws glue get-table: This command retrieves the complete metadata for the source table as a JSON object.
    4. jq Transformation (The Magic): This is the core of the script.
      • echo "$SOURCE_TABLE_JSON" | jq ...: We pipe the JSON output from the get-table command into jq.
      • --arg target_db "$TARGET_DB": This safely passes the bash variable $TARGET_DB into the jq script as a jq variable named $target_db.
      • .Table: The JSON from get-table has a top-level key named Table. We select its value.
      • del(...): The create-table API call will fail if you provide fields that are generated by AWS (like CreateTime, CatalogId, etc.). The del() function removes these keys from the JSON object.
      • .Name = $target_table: We update the Name of the table to the new target name.
      • {"DatabaseName": ..., "TableInput": ...}: Finally, we wrap the modified table definition in the exact JSON structure that the aws glue create-table command expects for its --cli-input-json parameter.
    5. aws glue create-table: This command takes the clean, properly formatted JSON and creates the new table in the target location. The --cli-input-json argument is perfect for this, as it avoids complex command-line escaping issues.

    Important Considerations

    • IAM Permissions: The AWS credentials/profiles used must have the necessary IAM permissions:
      • Source Profile: glue:GetTable
      • Target Profile: glue:CreateTable, glue:GetDatabase
    • Data Location: This script only migrates the table's metadata. It does not move or copy the underlying data in S3. The new table will point to the exact same S3 location as the old one unless you modify the Location field in the jq transformation.
    • Partitions: This script does not migrate table partitions. If your source table is partitioned, you will need to re-run your partitioning process (e.g., MSCK REPAIR TABLE in Athena) or script a separate partition migration using aws glue get-partitions and aws glue batch-create-partition.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.