Howdy! I have 30 years of climate model data (1980-2010). Currently, it's broken up into…
- User
Howdy!
I have 30 years of climate model data (1980-2010). Currently, it's broken up into months. I'd like to concatenate my files to be yearly. What's the best way to do this in Python?
Here's the code I have so far, to get you acquainted with the file paths and filename structure:
import xarray as xr
labspace = '/dfs9/baldwij1-lab/' workdir = labspace + 'hstaudmy/chapter1/'
scenario = 'ssp585' warming_level = 'onepointfive'
model = 'ACCESS-CM2'
path_data = workdir + 'output/' + scenario + '/' + model + '/04_apply_deltas_and_trim_to_90/' + scenario + '/' +
warming_level + '/' - Assistant
Howdy!
You're on the right track using xarray — it's ideal for working with climate model data in NetCDF format. If your files are divided by month and you'd like to merge them into annual files, the overall process looks like:
- Open all monthly NetCDF files.
- Group them by year.
- Concatenate all months within each year into one dataset.
- Save each year's dataset into a new NetCDF file.
✅ Assumptions:
- Files are monthly (e.g.,
tas_1980_01.nc,tas_1980_02.nc, ...). - Variables and coordinates are consistent across monthly files.
- Time information is present in each file, allowing for time-wise concatenation.
🔁 Here's a complete and flexible script:
import xarray as xr import os import glob import pandas as pd labspace = '/dfs9/baldwij1-lab/' workdir = labspace + 'hstaudmy/chapter1/' scenario = 'ssp585' warming_level = 'onepointfive' model = 'ACCESS-CM2' # Path to your monthly files path_data = os.path.join(workdir, 'output', scenario, model, '04_apply_deltas_and_trim_to_90', scenario, warming_level) # Output path for yearly files path_out = os.path.join(path_data, 'yearly_files') os.makedirs(path_out, exist_ok=True) # Grab all netCDF files monthly_files = sorted(glob.glob(os.path.join(path_data, '*.nc'))) # Group files by year files_by_year = {} for f in monthly_files: # Extract year from filename assuming format like 'tas_1980_01.nc' base = os.path.basename(f) parts = base.split('_') year = [p for p in parts if p.isdigit() and len(p) == 4] if year: year = int(year[0]) files_by_year.setdefault(year, []).append(f) # Loop over each year and combine files for year, year_files in files_by_year.items(): print(f'Processing year {year} with {len(year_files)} files...') # Sort by month (optional - ensures proper order) year_files = sorted(year_files) # Open and concatenate along time dimension ds = xr.open_mfdataset(year_files, combine='by_coords', parallel=True) # Optional: load into memory to make save more stable ds.load() # Write to new yearly NetCDF output_filename = os.path.join(path_out, f'{model}_{scenario}_{warming_level}_{year}.nc') ds.to_netcdf(output_filename) ds.close() print("✅ All yearly files created!")
🛠 Notes:
- This assumes that each month's dataset has a
timecoordinate — xarray will handle concatenation automatically. open_mfdatasetwithcombine='by_coords'should work if time coordinates are correct.- Consider using
daskif your data is large and might not fit into memory.
🚀 Example:
If your filenames look like
tas_ACCESS-CM2_ssp585_onepointfive_1980_01.nc, extracting the year might be slightly different. You can modify the year-extraction logic accordingly.
Let me know if you want this tailored for specific variable names, or if your filenames differ in format!
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.