Write me a bash script to migrate an aws glue table
- User
Write me a bash script to migrate an aws glue table
Assistant
Of course! Here is a comprehensive and robust bash script to migrate an AWS Glue table definition from a source to a target. This script can handle migrations within the same AWS account/region or across different accounts and regions.
Key Features of the Script
- Cross-Account & Cross-Region: Easily specify different AWS profiles and regions for the source and target.
- Dependency Checks: Ensures
awsCLI andjqare installed before running. - Error Handling: Checks the exit status of AWS CLI commands and provides clear error messages.
- User-Friendly: Provides a clear usage/help message and informative output.
- Metadata Cleaning: Uses
jqto automatically clean the table definition, removing read-only fields that would cause thecreate-tablecommand to fail.
The Script:
migrate_glue_table.sh#!/bin/bash # =================================================================================== # AWS Glue Table Migration Script # # Description: # This script migrates an AWS Glue table definition from a source database/table # to a target database/table. It can operate within the same AWS account/region # or across different accounts and regions by using AWS CLI profiles. # # Note: This script migrates the TABLE METADATA ONLY. It does NOT move the # underlying data in S3. If the data location needs to change, you must handle # that separately. It also does not migrate partitions; they must be recreated. # # Dependencies: # - aws-cli: The AWS Command Line Interface. Must be configured. # - jq: A lightweight and flexible command-line JSON processor. # # Usage: # ./migrate_glue_table.sh --source-db <db> --source-table <tbl> \ # --target-db <db> --target-table <tbl> \ # [--source-profile <prof>] [--source-region <reg>] \ # [--target-profile <prof>] [--target-region <reg>] # # Examples: # 1. Same account, same region: # ./migrate_glue_table.sh --source-db raw_db --source-table users \ # --target-db processed_db --target-table users_v2 # # 2. Cross-region migration: # ./migrate_glue_table.sh --source-db sales --source-table daily_sales \ # --target-db sales --target-table daily_sales \ # --source-region us-east-1 --target-region us-west-2 # # 3. Cross-account migration: # ./migrate_glue_table.sh --source-db analytics --source-table events \ # --target-db analytics --target-table events \ # --source-profile acct1-profile --target-profile acct2-profile # # =================================================================================== set -e # Exit immediately if a command exits with a non-zero status. set -o pipefail # The return value of a pipeline is the status of the last command to exit with a non-zero status. # --- Function to display usage information --- usage() { echo "Usage: $0 --source-db <db> --source-table <tbl> --target-db <db> --target-table <tbl> [--source-profile <prof>] [--source-region <reg>] [--target-profile <prof>] [--target-region <reg>]" echo "Migrates an AWS Glue table definition." echo "" echo "Required arguments:" echo " --source-db Name of the source Glue database." echo " --source-table Name of the source Glue table." echo " --target-db Name of the target Glue database." echo " --target-table Name of the new target Glue table." echo "" echo "Optional arguments:" echo " --source-profile AWS CLI profile for the source account." echo " --source-region AWS region for the source table." echo " --target-profile AWS CLI profile for the target account." echo " --target-region AWS region for the target table." echo " --help Display this help message." exit 1 } # --- Parse Command Line Arguments --- while [[ "$#" -gt 0 ]]; do case $1 in --source-db) SOURCE_DB="$2"; shift ;; --source-table) SOURCE_TABLE="$2"; shift ;; --target-db) TARGET_DB="$2"; shift ;; --target-table) TARGET_TABLE="$2"; shift ;; --source-profile) SOURCE_PROFILE="$2"; shift ;; --source-region) SOURCE_REGION="$2"; shift ;; --target-profile) TARGET_PROFILE="$2"; shift ;; --target-region) TARGET_REGION="$2"; shift ;; --help) usage ;; *) echo "Unknown parameter passed: $1"; usage ;; esac shift done # --- Validate required arguments --- if [ -z "$SOURCE_DB" ] || [ -z "$SOURCE_TABLE" ] || [ -z "$TARGET_DB" ] || [ -z "$TARGET_TABLE" ]; then echo "Error: Missing required arguments." usage fi # --- Check for dependencies --- if ! command -v aws &> /dev/null; then echo "Error: aws-cli is not installed. Please install and configure it." exit 1 fi if ! command -v jq &> /dev/null; then echo "Error: jq is not installed. Please install it (e.g., 'sudo apt-get install jq' or 'brew install jq')." exit 1 fi # --- Construct AWS CLI options --- SOURCE_AWS_OPTS="" [ -n "$SOURCE_PROFILE" ] && SOURCE_AWS_OPTS+="--profile $SOURCE_PROFILE " [ -n "$SOURCE_REGION" ] && SOURCE_AWS_OPTS+="--region $SOURCE_REGION" TARGET_AWS_OPTS="" [ -n "$TARGET_PROFILE" ] && TARGET_AWS_OPTS+="--profile $TARGET_PROFILE " [ -n "$TARGET_REGION" ] && TARGET_AWS_OPTS+="--region $TARGET_REGION" # --- Main Script Logic --- echo "▶️ Starting Glue table migration..." echo "--------------------------------------------------" echo "Source: $SOURCE_DB.$SOURCE_TABLE" [ -n "$SOURCE_PROFILE" ] && echo "Source Profile: $SOURCE_PROFILE" [ -n "$SOURCE_REGION" ] && echo "Source Region: $SOURCE_REGION" echo "Target: $TARGET_DB.$TARGET_TABLE" [ -n "$TARGET_PROFILE" ] && echo "Target Profile: $TARGET_PROFILE" [ -n "$TARGET_REGION" ] && echo "Target Region: $TARGET_REGION" echo "--------------------------------------------------" # 1. Get the source table definition echo "➡️ Fetching definition for source table '$SOURCE_DB.$SOURCE_TABLE'..." SOURCE_TABLE_JSON=$(aws glue get-table --database-name "$SOURCE_DB" --name "$SOURCE_TABLE" $SOURCE_AWS_OPTS) if [ $? -ne 0 ]; then echo "❌ Error: Failed to get source table definition. Please check parameters and permissions." exit 1 fi # 2. Transform the JSON for the create-table command # We need to extract the 'Table' object, remove read-only fields, # update the name, and wrap it in the structure required by 'create-table'. echo "➡️ Transforming table definition for target..." TARGET_TABLE_INPUT_JSON=$(echo "$SOURCE_TABLE_JSON" | jq \ --arg target_db "$TARGET_DB" \ --arg target_table "$TARGET_TABLE" \ '{ "DatabaseName": $target_db, "TableInput": ( .Table | del(.DatabaseName, .CreateTime, .UpdateTime, .CreatedBy, .IsRegisteredWithLakeFormation, .CatalogId, .VersionId) | .Name = $target_table ) }') # 3. Create the target table echo "➡️ Creating target table '$TARGET_DB.$TARGET_TABLE'..." aws glue create-table --cli-input-json "$TARGET_TABLE_INPUT_JSON" $TARGET_AWS_OPTS if [ $? -eq 0 ]; then echo "✅ Successfully migrated table '$SOURCE_DB.$SOURCE_TABLE' to '$TARGET_DB.$TARGET_TABLE'." else echo "❌ Error: Failed to create the target table. Check the AWS CLI output above for details." exit 1 fiHow to Use the Script
-
Save the Code: Save the script above as a file, for example,
migrate_glue_table.sh. -
Make it Executable: Open your terminal and run:
chmod +x migrate_glue_table.sh -
Prerequisites:
- AWS CLI: Make sure you have the AWS CLI installed and configured. If you are doing a cross-account migration, ensure you have set up named profiles in your
~/.aws/credentialsand~/.aws/configfiles. jq: Make sure you havejqinstalled.- On macOS (using Homebrew):
brew install jq - On Debian/Ubuntu:
sudo apt-get install jq - On RHEL/CentOS:
sudo yum install jq
- On macOS (using Homebrew):
- AWS CLI: Make sure you have the AWS CLI installed and configured. If you are doing a cross-account migration, ensure you have set up named profiles in your
-
Run the Script: Execute the script with the appropriate arguments.
-
Example 1: Same Account, Different Database
./migrate_glue_table.sh \ --source-db my_raw_data \ --source-table customer_events_2023 \ --target-db my_processed_data \ --target-table customer_events -
Example 2: Cross-Region
./migrate_glue_table.sh \ --source-db production_db --source-table logs \ --target-db dr_db --target-table logs \ --source-region us-east-1 \ --target-region us-west-2 -
Example 3: Cross-Account and Cross-Region
./migrate_glue_table.sh \ --source-db sales_db --source-table transactions \ --target-db analytics_db --target-table transactions \ --source-profile dev-account --source-region us-east-1 \ --target-profile prod-account --target-region eu-west-1
-
Script Breakdown and Explanation
- Argument Parsing: The script uses a
whileloop and acasestatement to parse named arguments like--source-db. This is more robust and user-friendly than relying on positional arguments. - Dependency Checks:
command -vchecks ifawsandjqexist in the system's PATH. This prevents the script from failing with cryptic errors later on. aws glue get-table: This command retrieves the complete metadata for the source table as a JSON object.jqTransformation (The Magic): This is the core of the script.echo "$SOURCE_TABLE_JSON" | jq ...: We pipe the JSON output from theget-tablecommand intojq.--arg target_db "$TARGET_DB": This safely passes the bash variable$TARGET_DBinto thejqscript as ajqvariable named$target_db..Table: The JSON fromget-tablehas a top-level key namedTable. We select its value.del(...): Thecreate-tableAPI call will fail if you provide fields that are generated by AWS (likeCreateTime,CatalogId, etc.). Thedel()function removes these keys from the JSON object..Name = $target_table: We update theNameof the table to the new target name.{"DatabaseName": ..., "TableInput": ...}: Finally, we wrap the modified table definition in the exact JSON structure that theaws glue create-tablecommand expects for its--cli-input-jsonparameter.
aws glue create-table: This command takes the clean, properly formatted JSON and creates the new table in the target location. The--cli-input-jsonargument is perfect for this, as it avoids complex command-line escaping issues.
Important Considerations
- IAM Permissions: The AWS credentials/profiles used must have the necessary IAM permissions:
- Source Profile:
glue:GetTable - Target Profile:
glue:CreateTable,glue:GetDatabase
- Source Profile:
- Data Location: This script only migrates the table's metadata. It does not move or copy the underlying data in S3. The new table will point to the exact same S3 location as the old one unless you modify the
Locationfield in thejqtransformation. - Partitions: This script does not migrate table partitions. If your source table is partitioned, you will need to re-run your partitioning process (e.g.,
MSCK REPAIR TABLEin Athena) or script a separate partition migration usingaws glue get-partitionsandaws glue batch-create-partition.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.