Skip to content
TheChatArchiveThe Chat Archive
step-1o-turbo-202506Hugging Face Arena Expert 5K

This code is forecasting then number of mortgage contract, but it is grouping by weeks,…

  1. User

    This code is forecasting then number of mortgage contract, but it is grouping by weeks, I need it to be divided proportionally on each day of week, by the proprtions according to the last year before forecast import pandas as pd import numpy as np from prophet import Prophet import optuna import matplotlib.pyplot as plt

    ------------------------------------------------------------------

    1. Load your data (ds = date, y = daily contract count)

    ------------------------------------------------------------------

    Example data (replace with your actual data)

    df['ds'] = pd.to_datetime(df['ds'])

    ------------------------------------------------------------------

    2. Keep only business days (drop Sat/Sun)

    ------------------------------------------------------------------

    df = df[df['ds'].dt.weekday < 5].reset_index(drop=True)

    ------------------------------------------------------------------

    3. Aggregate daily data into weekly sums

    ------------------------------------------------------------------

    Add a 'week_start' column (Monday of each week)

    df['week_start'] = df['ds'] - pd.to_timedelta(df['ds'].dt.weekday, unit='D')

    Group by week_start and sum the 'y' values

    weekly_df = df.groupby('week_start').agg({'y': 'sum'}).reset_index() weekly_df.columns = ['ds', 'y'] # Prophet expects 'ds' and 'y'

    ------------------------------------------------------------------

    4. Train / test split -> last 52 weeks as hold-out

    ------------------------------------------------------------------

    HOLDOUT_WEEKS = 52 train_df = weekly_df.iloc[:-HOLDOUT_WEEKS] test_df = weekly_df.iloc[-HOLDOUT_WEEKS:]

    ------------------------------------------------------------------

    5. SMAPE helper

    ------------------------------------------------------------------

    def smape(y_true, y_pred): y_true, y_pred = np.array(y_true), np.array(y_pred) return 100 * np.mean(2 * np.abs(y_pred - y_true) / (np.abs(y_true) + np.abs(y_pred) + 1e-8))

    ------------------------------------------------------------------

    6. Optuna objective

    ------------------------------------------------------------------

    def objective(trial): params = { 'changepoint_prior_scale': trial.suggest_float('changepoint_prior_scale', 0.001, 0.5, log=True), 'seasonality_prior_scale': trial.suggest_float('seasonality_prior_scale', 0.01, 10, log=True), 'changepoint_range': trial.suggest_float('changepoint_range', 0.8, 0.95), 'n_changepoints': trial.suggest_int('n_changepoints', 10, 30), # Fewer changepoints for weekly data 'seasonality_mode': trial.suggest_categorical('seasonality_mode', ['additive', 'multiplicative']), 'interval_width': trial.suggest_float('interval_width', 0.90, 0.95) }

    m = Prophet(
        daily_seasonality=False,
        weekly_seasonality=False,  # Disable weekly seasonality (data is already weekly)
        yearly_seasonality=True,
        **params
    )
    m.fit(train_df)
    
    # Forecast exactly the test period
    future = pd.DataFrame({'ds': test_df['ds']})
    fcst = m.predict(future)
    
    # Align and compute SMAPE
    merged = pd.merge(test_df, fcst[['ds', 'yhat']], on='ds', how='inner')
    return smape(merged['y'], merged['yhat'])
    

    ------------------------------------------------------------------

    7. Run Bayesian optimisation

    ------------------------------------------------------------------

    study = optuna.create_study(direction='minimize') study.optimize(objective, n_trials=30, show_progress_bar=True)

    print("Best SMAPE:", round(study.best_value, 2)) print("Best params:", study.best_params)

    ------------------------------------------------------------------

    8. Final model on best parameters

    ------------------------------------------------------------------

    best = study.best_params m_final = Prophet( daily_seasonality=False, weekly_seasonality=False, # Disable weekly seasonality (data is weekly) yearly_seasonality=True, **best ) m_final.fit(train_df)

    future = pd.DataFrame({'ds': test_df['ds']}) fcst = m_final.predict(future)

    merged = pd.merge(test_df, fcst[['ds', 'yhat', 'yhat_lower', 'yhat_upper']], on='ds')

    fcst_backtest_CNT = merged.drop('y', axis = 1)

  2. Assistant

    To modify the code so that the forecast is divided proportionally to each day of the week based on the distribution from the last year (before the forecast), we need to do the following:

    1. Forecast at the weekly level (as currently done), but then break down the weekly forecast into daily values.
    2. Use the distribution of daily counts from the last year (before the forecast period) to allocate the weekly forecast to each day of the week.
    3. Ensure that the weekends (Saturday and Sunday) are handled correctly, as the original data only includes business days (Monday to Friday).

    Here’s the step-by-step process and the modified code:

    Steps:

    1. Forecast at the weekly level (as in the current code).
    2. Get the daily distribution from the last year (before the forecast period) for each day of the week (Monday to Friday).
    3. For each week in the forecast, allocate the total weekly forecast to each day of the week proportionally based on the daily distribution from the last year.
    4. Exclude weekends (Saturday and Sunday) since the original data only includes business days.

    Modified Code:

    import pandas as pd
    import numpy as np
    from prophet import Prophet
    import optuna
    import matplotlib.pyplot as plt
    
    # ------------------------------------------------------------------
    # 1. Load your data (ds = date, y = daily contract count)
    # ------------------------------------------------------------------
    # Example data (replace with your actual data)
    # df = pd.read_csv('your_data.csv')  # Load your data here
    # For this example, let's assume 'ds' is the date and 'y' is the count
    
    # Convert 'ds' to datetime
    df['ds'] = pd.to_datetime(df['ds'])
    
    # ------------------------------------------------------------------
    # 2. Keep only business days (drop Sat/Sun)
    # ------------------------------------------------------------------
    df = df[df['ds'].dt.weekday < 5].reset_index(drop=True)  # 0 = Monday, 4 = Friday
    
    # Sort by date (just in case)
    df = df.sort_values(by='ds').reset_index(drop=True)
    
    # ------------------------------------------------------------------
    # 3. Aggregate daily data into weekly sums
    # ------------------------------------------------------------------
    # Add a 'week_start' column (Monday of each week)
    df['week_start'] = df['ds'] - pd.to_timedelta(df['ds'].dt.weekday, unit='D')
    
    # Group by week_start and sum the 'y' values
    weekly_df = df.groupby('week_start').agg({'y': 'sum'}).reset_index()
    weekly_df.columns = ['ds', 'y']  # Prophet expects 'ds' and 'y'
    
    # ------------------------------------------------------------------
    # 4. Determine the last year before the forecast for daily distribution
    # ------------------------------------------------------------------
    # Get the last year of data (for the daily distribution)
    last_year_end = df['ds'].max()  # Last date in the data
    last_year_start = last_year_end - pd.DateOffset(years=1)
    
    # Filter the data for the last year
    last_year_df = df[(df['ds'] >= last_year_start) & (df['ds'] <= last_year_end)].copy()
    
    # Calculate the average daily distribution for each day of the week (Monday to Friday)
    daily_dist = last_year_df.groupby(last_year_df['ds'].dt.weekday).agg({'y': 'sum'}).reset_index()
    daily_dist['day_of_week'] = daily_dist['ds'].dt.weekday
    daily_dist['proportion'] = daily_dist['y'] / daily_dist['y'].sum()
    
    # Map days of the week (0 = Monday, 1 = Tuesday, ..., 4 = Friday)
    day_order = {0: 'Monday', 1: 'Tuesday', 2: 'Wednesday', 3: 'Thursday', 4: 'Friday'}
    daily_dist['day_name'] = daily_dist['day_of_week'].map(day_order)
    daily_dist = daily_dist[['day_of_week', 'proportion']]
    
    # Check the daily distribution
    print("Daily distribution from last year:")
    print(daily_dist)
    
    # ------------------------------------------------------------------
    # 5. Train / test split -> last 52 weeks as hold-out
    # ------------------------------------------------------------------
    HOLDOUT_WEEKS = 52
    train_df = weekly_df.iloc[:-HOLDOUT_WEEKS]
    test_df = weekly_df.iloc[-HOLDOUT_WEEKS:]
    
    # ------------------------------------------------------------------
    # 6. SMAPE helper
    # ------------------------------------------------------------------
    def smape(y_true, y_pred):
        y_true, y_pred = np.array(y_true), np.array(y_pred)
        return 100 * np.mean(2 * np.abs(y_pred - y_true) /
                             (np.abs(y_true) + np.abs(y_pred) + 1e-8))
    
    # ------------------------------------------------------------------
    # 7. Optuna objective
    # ------------------------------------------------------------------
    def objective(trial):
        params = {
            'changepoint_prior_scale': trial.suggest_float('changepoint_prior_scale', 0.001, 0.5, log=True),
            'seasonality_prior_scale': trial.suggest_float('seasonality_prior_scale', 0.01, 10, log=True),
            'changepoint_range': trial.suggest_float('changepoint_range', 0.8, 0.95),
            'n_changepoints': trial.suggest_int('n_changepoints', 10, 30),  # Fewer changepoints for weekly data
            'seasonality_mode': trial.suggest_categorical('seasonality_mode', ['additive', 'multiplicative']),
            'interval_width': trial.suggest_float('interval_width', 0.90, 0.95)
        }
    
        m = Prophet(
            daily_seasonality=False,
            weekly_seasonality=False,  # Disable weekly seasonality (data is already weekly)
            yearly_seasonality=True,
            **params
        )
        m.fit(train_df)
    
        # Forecast exactly the test period
        future = pd.DataFrame({'ds': test_df['ds']})
        fcst = m.predict(future)
    
        # Align and compute SMAPE
        merged = pd.merge(test_df, fcst[['ds', 'yhat']], on='ds', how='inner')
        return smape(merged['y'], merged['yhat'])
    
    # ------------------------------------------------------------------
    # 8. Run Bayesian optimisation
    # ------------------------------------------------------------------
    study = optuna.create_study(direction='minimize')
    study.optimize(objective, n_trials=30, show_progress_bar=True)
    
    print("Best SMAPE:", round(study.best_value, 2))
    print("Best params:", study.best_params)
    
    # ------------------------------------------------------------------
    # 9. Final model on best parameters
    # ------------------------------------------------------------------
    best = study.best_params
    m_final = Prophet(
        daily_seasonality=False,
        weekly_seasonality=False,  # Disable weekly seasonality (data is weekly)
        yearly_seasonality=True,
        **best
    )
    m_final.fit(weekly_df)  # Use the full data for the final model
    
    # Forecast future (e.g., next 52 weeks)
    future_weeks = 52
    future = m_final.make_future_dataframe(periods=future_weeks, freq='W')
    fcst = m_final.predict(future)
    
    # Keep only the forecasted part
    fcst_forecast = fcst[fcst['ds'] > weekly_df['ds'].max()]
    
    # ------------------------------------------------------------------
    # 10. Break down the weekly forecast to daily using the daily distribution
    # ------------------------------------------------------------------
    def break_down_weekly_forecast(weekly_fcst, daily_dist):
        result = []
        for _, row in weekly_fcst.iterrows():
            week_start = row['ds']  # Monday of the week
            week_end = week_start + pd.DateOffset(days=4)  # Friday of the week
            total_forecast = row['yhat']
            
            # Generate dates for Monday to Friday
            dates = pd.date_range(start=week_start, end=week_end)
            
            # Allocate the forecast to each day
            for date in dates:
                day_of_week = date.dayofweek  # 0 = Monday, 1 = Tuesday, ..., 4 = Friday
                proportion = daily_dist.loc[daily_dist['day_of_week'] == day_of_week, 'proportion'].values[0]
                daily_forecast = total_forecast * proportion
                result.append({
                    'ds': date,
                    'yhat': daily_forecast
                })
        
        return pd.DataFrame(result)
    
    # Break down the weekly forecast to daily
    daily_fcst = break_down_weekly_forecast(fcst_forecast, daily_dist)
    
    # ------------------------------------------------------------------
    # 11. Visualize the daily forecast (optional)
    # ------------------------------------------------------------------
    plt.figure(figsize=(12, 6))
    plt.plot(daily_fcst['ds'], daily_fcst['yhat'], label='Forecast', color='b')
    plt.title('Daily Mortgage Contract Forecast')
    plt.xlabel('Date')
    plt.ylabel('Contracts')
    plt.legend()
    plt.grid(True)
    plt.show()
    
    # ------------------------------------------------------------------
    # 12. Save or return the daily forecast
    # ------------------------------------------------------------------
    print("Daily forecast:")
    print(daily_fcst.head())
    
    # Save to CSV (optional)
    daily_fcst.to_csv('daily_mortgage_forecast.csv', index=False)
    

    Key Modifications:

    1. Daily Distribution Calculation:

      • We extract the last year of data (before the forecast) and calculate the sum of contracts for each day of the week (Monday to Friday).
      • The proportion for each day is calculated as the ratio of that day’s sum to the total weekly sum.
    2. Weekly to Daily Breakdown:

      • After getting the weekly forecast, we break it down to daily values using the daily proportions calculated from the last year.
      • For each week in the forecast, we allocate the total forecast to each day (Monday to Friday) based on the pre-calculated proportions.
    3. Excluding Weekends:

      • Since the original data only includes business days (Monday to Friday), we only forecast for these days and ignore weekends.

    Example Output:

    The output daily_fcst will be a DataFrame with columns ds (date) and yhat (daily forecasted contracts), broken down from the weekly forecast.

    Visualization:

    The code includes an optional plot to visualize the daily forecast.

    Saving the Result:

    The daily forecast can be saved to a CSV file for further analysis.

    Notes:

    • If the last year’s data has missing days or irregular patterns, you might want to smooth the daily distribution (e.g., using a rolling average).
    • You can adjust the forecast horizon (future_weeks) as needed.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.