Skip to content
The Chat Archive
claude-sonnet-4-20250514· Anthropic LMArena Human Preference 140K

I've never used NCO before. Can you show me how to make a shell script to run these…

  1. User

    I've never used NCO before. Can you show me how to make a shell script to run these commands from?

  2. Assistant

    Absolutely! Here's how to create and run a shell script with NCO commands:

    1. Create the shell script

    Create a new file called concatenate_yearly.sh:

    #!/bin/bash
    
    # Set your paths and variables
    labspace="/dfs9/baldwij1-lab/"
    workdir="${labspace}hstaudmy/chapter1/"
    scenario="ssp585"
    warming_level="onepointfive"
    model="ACCESS-CM2"
    
    # Build the data path
    path_data="${workdir}output/${scenario}/${model}/04_apply_deltas_and_trim_to_90/${scenario}/${warming_level}/"
    
    # Create output directory for yearly files
    output_dir="${path_data}yearly_files"
    mkdir -p "$output_dir"
    
    echo "Starting concatenation for $model, $scenario, $warming_level"
    echo "Input directory: $path_data"
    echo "Output directory: $output_dir"
    
    # Loop through years 1980-2010
    for year in {1980..2010}; do
        echo "Processing year $year..."
        
        # Find all files for this year and concatenate them
        ncrcat -h "${path_data}"*_${year}_*.nc "${output_dir}/${model}_${scenario}_${warming_level}_${year}.nc"
        
        # Check if the command succeeded
        if [ $? -eq 0 ]; then
            echo "✅ Year $year completed successfully"
        else
            echo "❌ Error processing year $year"
        fi
    done
    
    echo "🎉 All years processed!"
    

    2. Make the script executable

    chmod +x concatenate_yearly.sh
    

    3. Run the script

    ./concatenate_yearly.sh
    

    Key NCO flags explained:

    • -h: Suppress history attribute (keeps output files cleaner)
    • ncrcat: Concatenates files along the record dimension (usually time)

    Optional enhancements:

    If you want to add error checking and logging:

    #!/bin/bash
    
    # ... (same variables as above)
    
    # Create log file
    log_file="${output_dir}/concatenation_log_$(date +%Y%m%d_%H%M%S).txt"
    
    echo "Starting concatenation at $(date)" | tee "$log_file"
    
    for year in {1980..2010}; do
        echo "Processing year $year..." | tee -a "$log_file"
        
        # Check if input files exist for this year
        input_files=("${path_data}"*_${year}_*.nc)
        if [ ! -e "${input_files[0]}" ]; then
            echo "⚠️  No files found for year $year" | tee -a "$log_file"
            continue
        fi
        
        # Run ncrcat
        ncrcat -h "${path_data}"*_${year}_*.nc "${output_dir}/${model}_${scenario}_${warming_level}_${year}.nc" 2>&1 | tee -a "$log_file"
        
        if [ $? -eq 0 ]; then
            echo "✅ Year $year completed successfully" | tee -a "$log_file"
        else
            echo "❌ Error processing year $year" | tee -a "$log_file"
        fi
    done
    
    echo "Finished at $(date)" | tee -a "$log_file"
    

    This will create a log file with timestamps and error messages, which is helpful for long-running processes with large datasets.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.