Build a heatwave excess mortality attribution with counterfactual weather.
- User
Build a heatwave excess mortality attribution with counterfactual weather.
Assistant
The user wants a model to estimate excess mortality attributable to a heatwave, using a counterfactual weather scenario (e.g., what the weather would have been without the heatwave).
This is a complex task usually done in epidemiology and climate science. I need to provide a conceptual framework and a Python code example.
1. Conceptual Framework:
- Observed Data: Daily mortality counts, daily temperature (and potentially humidity, air pollution, etc.).
- The Heatwave: A defined period of extreme heat.
- Counterfactual Scenario: What would the temperature have been? This is the tricky part.
- Simple approach: Replace heatwave temps with the long-term average for those calendar days.
- Complex approach: Statistical modeling (e.g., removing the trend, using a specific year as baseline).
- Exposure-Response Function (ERF): A mathematical relationship between temperature and mortality. Usually U-shaped or J-shaped (higher risk at very cold and very hot temps).
- Attribution Calculation:
- Observed Mortality:
- Counterfactual Mortality: (calculated using in the ERF).
- Excess Mortality: . Wait, no. We don't usually just subtract raw counts because mortality varies by day of week, season, etc.
- Standard Epidemiological approach:
- Fit a model (e.g., DLNM - Distributed Lag Non-Linear Model) to historical data to get the ERF.
- Predict mortality for the heatwave period using observed temperatures.
- Predict mortality for the heatwave period using counterfactual temperatures.
- The difference is the attributable number (AN).
2. Python Implementation Strategy:
- I need to simulate data because I don't have access to real private mortality datasets.
- Libraries:
pandas,numpy,matplotlib,statsmodels(orscikit-learn, but DLNMs are specific). Actually, writing a full DLNM from scratch in a snippet is too hard and prone to errors. I will use a simplified regression model (e.g., Generalized Linear Model with splines) to demonstrate the logic, while mentioning that DLNMs are the standard tool. - Correction: To make it actually robust and modern, I should probably stick to a standard GLM with a spline for temperature.
3. Step-by-Step Code Structure:
- Step 1: Generate Synthetic Data.
- Create a date range.
- Create a seasonal temperature pattern.
- Inject a "heatwave".
- Create a seasonal mortality pattern (baseline).
- Link mortality to temperature using a quadratic or exponential relationship (the "truth").
- Step 2: Define the Exposure-Response Relationship (The Model).
- Fit a model on a "training" period (or just assume the whole period except the heatwave is the baseline).
- Use
statsmodelsorpatsyfor the formula.Mortality ~ ns(Date, df) + ns(Temp, df).
- Step 3: Define Counterfactual Weather.
- Calculate the historical average temperature for those specific calendar days (e.g., average of June 15th across 10 years).
- Step 4: Predict and Calculate Excess.
- Predict with observed temps.
- Predict with counterfactual temps.
- Subtract.
4. Refining the "Counterfactual" Concept:
- Simply averaging past days is the most intuitive "Counterfactual Weather".
- However, sometimes "Counterfactual" means "What if the heatwave happened but climate change hadn't shifted the baseline?" -> This implies detrending. I will stick to the "Average Historical Weather" definition as it's the standard for calculating excess mortality during a specific event.
5. Drafting the Code (Mental or Scratchpad):
- Need
numpy,pandas,matplotlib. - Simulation:
- Days: 1000.
- Temp: .
- Heatwave: Days 500-510. Temp adds +10 degrees.
- Deaths: Baseline + function(Temp).
- Model:
- Since I'm simulating, I know the ground truth. But in reality, you estimate.
- I will fit a Generalized Additive Model (GAM) or a GLM with splines. Let's stick to
statsmodelsGLM.
- The Attribution Equation:
- . Excess = .
- Actually, it's easier to just predict and .
- .
6. Constraints & Safety:
- This is a statistical modeling task. No safety issues.
- The user likely wants a clear, reproducible example.
7. Refined Plan:
- Introduction: Explain the methodology (DLNM/GLM, Counterfactual definition).
- Part 1: Data Simulation: Create realistic noisy data with a mortality-temperature relationship.
- Part 2: Modeling: Fit a regression model (Temperature + Seasonality).
- Part 3: Counterfactual Generation: Replace heatwave temps with climatology.
- Part 4: Calculation: Predict difference.
- Part 5: Visualization: Plot the heatwave and the excess deaths.
Self-Correction during thought process:
- Lag effects: Heat mortality often has displacement (harvesting) or delayed effects (lags 0-3). Implementing lags in a simple script complicates the "Counterfactual Weather" generation (because you need the history of the counterfactual).
- Decision: I will explicitly mention that this example ignores lag effects for simplicity, but note that a real study uses DLNMs (Distributed Lag Non-Linear Models).
Code Specifics:
- Use
scipy.interpolateorstatsmodelsfor smoothing. - Formula:
Deaths ~ bs(Temperature, df=4) + bs(Time, df=10). (Cubic splines). - Heatwave definition: e.g., 3 consecutive days > 95th percentile.
Final Output Structure:
- Concept: Explanation of the methodology.
- Python Code: A complete, runnable script generating data, modeling, and attributing.
- Interpretation: How to read the results.
Let's write the code. Check: Does
statsmodelscome with standard Anaconda? Yes. Check: Is the logic sound?- Observed Mortality is driven by Observed Temp.
- Predicted Mortality (Obs) should be close to Observed Mortality (if the model is good).
- Predicted Mortality (Counterfactual) uses Normal Temps.
- Excess = Pred(Obs) - Pred(Counterfactual).
- Wait, strictly speaking, we compare Actual Observed Mortality vs Baseline Expected Mortality.
- Alternative View: We want to attribute the specific excess during the heatwave.
- If the model fits perfectly, Pred(Obs) == Actual Observed.
- Excess = Actual Observed - Expected (Counterfactual).
- This is the standard "Attributable Fraction" approach.
Let's proceed. This approach estimates excess mortality during a heatwave by comparing the observed mortality (driven by observed extreme weather) against a counterfactual scenario (what the weather would have been under normal, non-heatwave conditions).
This method effectively "subtracts" the seasonal baseline trend to isolate the specific impact of the heat.
The Methodology
- Baseline Model: We establish a relationship between historical weather and mortality. This accounts for the natural seasonality of death rates (deaths are typically higher in winter, lower in summer) and captures the non-linear "U-shaped" relationship between temperature and health (where both extreme cold and extreme heat are dangerous).
- Counterfactual Generation: We replace the extreme temperatures observed during the heatwave with "expected" temperatures (e.g., the 10-year average for those specific calendar days).
- Attribution:
- Scenario A (Factual): Predict mortality using the observed extreme temperatures.
- Scenario B (Counterfactual): Predict mortality using the "normal" average temperatures.
- Excess Mortality: The difference between Scenario A and Scenario B.
Python Implementation
Here is a complete, reproducible simulation in Python.
import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns import statsmodels.api as sm import statsmodels.formula.api as smf from scipy.interpolate import BSpline # --------------------------------------------------------- # 1. DATA SIMULATION # --------------------------------------------------------- np.random.seed(42) dates = pd.date_range(start='2010-01-01', end='2020-12-31', freq='D') n = len(dates) # Generate Seasonal Temperature (Sinusoidal pattern) day_of_year = dates.dayofyear # Base temp varies between 5C (winter) and 30C (summer) base_temp = 17.5 + 12.5 * np.sin(2 * np.pi * (day_of_year - 120) / 365) # Add random noise to make it look real observed_temp = base_temp + np.random.normal(0, 3, n) # Generate Seasonal Mortality Baseline # Deaths are typically higher in winter. We create a baseline unrelated to immediate temp. # Baseline ~ 120 deaths/day in summer, 150 in winter seasonal_baseline = 135 + 15 * np.cos(2 * np.pi * (day_of_year - 120) / 365) # Add a Non-Linear Temperature Effect (The "Hidden Truth") # Risk rises sharply above 28C def temp_risk_function(t): # Minimal risk below 25, exponential risk above 30 if t < 25: return 1.0 return 1.0 + 0.15 * (t - 25)**1.5 temp_effect = np.array([temp_risk_function(t) for t in observed_temp]) # Generate Observed Mortality # Mortality = Baseline * Temp_Risk + Random Noise observed_mortality = seasonal_baseline * temp_effect + np.random.normal(0, 5, n) df = pd.DataFrame({ 'date': dates, 'year': dates.year, 'doy': day_of_year, 'temp_obs': observed_temp, 'mortality_obs': observed_mortality }) # --------------------------------------------------------- # 2. ARTIFICIALLY INDUCE A MASSIVE HEATWAVE (2020) # --------------------------------------------------------- # We create a ground truth heatwave to test if the model catches it heatwave_start = '2020-07-15' heatwave_end = '2020-07-25' # Spike temps by 10 degrees during the heatwave mask_hw = (df['date'] >= heatwave_start) & (df['date'] <= heatwave_end) df.loc[mask_hw, 'temp_obs'] = df.loc[mask_hw, 'temp_obs'] + 10 # Recalculate mortality for the heatwave period based on the new spiked temps new_risk = np.array([temp_risk_function(t) for t in df.loc[mask_hw, 'temp_obs']]) # Use the baseline from that time (approx 130) * new risk df.loc[mask_hw, 'mortality_obs'] = 130 * new_risk + np.random.normal(0, 5, sum(mask_hw)) # --------------------------------------------------------- # 3. COUNTERFACTUAL WEATHER # --------------------------------------------------------- # Calculate the "Normal" weather for every day of the year # (Average of 2010-2019 for each specific Day of Year) climatology = df[df['year'] < 2020].groupby('doy')['temp_obs'].mean().reset_index() climatology.columns = ['doy', 'temp_counterfactual'] # Merge counterfactual weather into the main dataframe df = pd.merge(df, climatology, on='doy', how='left') # --------------------------------------------------------- # 4. MODELING THE EXPOSURE-RESPONSE # --------------------------------------------------------- # We use a Generalized Linear Model (GLM) with a Natural Spline for Temperature. # This captures the U-shape of mortality vs temp. # We train on all data (or pre-2020 data). Here we use the whole dataset for stability. # Formula: Mortality depends on Temperature (spline) and Seasonality (spline) formula = "mortality_obs ~ bs(temp_obs, df=4) + bs(doy, df=6)" model = smf.glm(formula=formula, data=df, family=sm.families.Poisson()).fit() # --------------------------------------------------------- # 5. ATTRIBUTION CALCULATION # --------------------------------------------------------- # Predict Mortality under FACTUAL conditions (Observed Heatwave Temps) df['mortality_pred_factual'] = model.predict(df) # Predict Mortality under COUNTERFACTUAL conditions (Normal Temps) # We swap the temperature column in the prediction logic df_temp_swap = df.copy() df_temp_swap['temp_obs'] = df_temp_swap['temp_counterfactual'] df['mortality_pred_counterfactual'] = model.predict(df_temp_swap) # Calculate Excess Mortality # Excess = What happened - What WOULD have happened df['excess_mortality'] = df['mortality_pred_factual'] - df['mortality_pred_counterfactual'] # --------------------------------------------------------- # 6. RESULTS & VISUALIZATION # --------------------------------------------------------- # Filter for the heatwave period for reporting results_hw = df[mask_hw] total_excess = results_hw['excess_mortality'].sum() print(f"Analysis of Heatwave: {heatwave_start} to {heatwave_end}") print(f"Total Observed Mortality: {results_hw['mortality_obs'].sum():.0f}") print(f"Total Excess Mortality (Attributable to Heat): {total_excess:.0f}") print("-" * 30) # Visualization plt.figure(figsize=(14, 6)) # Plot 1: Temperature plt.subplot(2, 1, 1) plt.plot(df['date'], df['temp_obs'], color='red', alpha=0.5, label='Observed Temp') plt.plot(df['date'], df['temp_counterfactual'], color='blue', linestyle='--', alpha=0.5, label='Counterfactual (Normal) Temp') plt.axvspan(pd.to_datetime(heatwave_start), pd.to_datetime(heatwave_end), color='red', alpha=0.1) plt.title('Observed vs Counterfactual Temperature (2020)') plt.ylabel('Temperature (C)') plt.legend() # Plot 2: Mortality plt.subplot(2, 1, 2) plt.plot(df['date'], df['mortality_pred_factual'], color='black', label='Predicted Mortality (with Heat)') plt.plot(df['date'], df['mortality_pred_counterfactual'], color='green', linestyle='--', label='Counterfactual Mortality (without Heat)') plt.fill_between(df['date'], df['mortality_pred_counterfactual'], df['mortality_pred_factual'], where=df['excess_mortality'] > 0, color='red', alpha=0.3, label='Excess Mortality') plt.axvspan(pd.to_datetime(heatwave_start), pd.to_datetime(heatwave_end), color='red', alpha=0.1) plt.title(f'Excess Mortality Attribution (Total Excess: {total_excess:.0f})') plt.ylabel('Deaths') plt.legend() plt.tight_layout() plt.show()Key Components of the Code
- The Exposure-Response Function (ERF):
We used a Poisson Generalized Linear Model (GLM) with B-Splines (
bs()). This is critical. A simple linear regression implies that every degree increase adds the same number of deaths. In reality, going from 20°C to 25°C usually adds zero deaths, but going from 35°C to 40°C adds many. The spline allows the model to bend and fit this curve. - The Counterfactual Weather:
We calculated the
doy(Day of Year) climatology. For July 15th, the counterfactual temperature is the average of every July 15th from the previous decade. This represents the "Weather without Climate Change/Event." - The Difference:
By predicting
mu_factualandmu_counterfactualand subtracting them, we isolate the specific contribution of the temperature anomaly to mortality, removing the underlying seasonal trend.
How to Interpret the Output
- Top Plot: You will see a red spike in temperature during the summer of 2020. The blue dashed line is what the temperature should have been.
- Bottom Plot: The black line represents expected deaths based on the heat. The green dashed line is the baseline deaths. The shaded red area between them represents the Excess Mortality.
- The Number: The printed "Total Excess Mortality" sums up the daily deaths attributable solely to the difference between the heatwave weather and the normal weather.