Skip to content
TheChatArchiveThe Chat Archive
gpt-4-1106-preview· OpenAIAi2 WildChat-4.8M (AI2)

this is my ensemble model from statsforecast import StatsForecast from…

  1. User

    this is my ensemble model from statsforecast import StatsForecast from statsforecast.models import AutoARIMA, AutoETS, DynamicOptimizedTheta from statsforecast.utils import ConformalIntervals import numpy as np import polars as pl

    Polars option to display all rows

    pl.Config.set_tbl_rows(None)

    Initialize the models

    models = [ AutoARIMA(season_length=12), AutoETS(season_length=12), DynamicOptimizedTheta(season_length=12) ]

    Initialize the StatsForecast model

    sf = StatsForecast(models=models, freq='1w', n_jobs=-1)

    Perform cross-validation with a step size of 1 to mimic an expanding window

    crossvalidation_df = sf.cross_validation(df=y_cl4_filtered, h=2, step_size=1, n_windows=8, sort_df=True)

    Calculate the ensemble mean

    ensemble = crossvalidation_df[['AutoARIMA', 'AutoETS', 'DynamicOptimizedTheta']].mean(axis=1)

    Create a Series for the ensemble mean

    ensemble_series = pl.Series('Ensemble', ensemble)

    Add the ensemble mean as a new column to the DataFrame

    crossvalidation_df = crossvalidation_df.with_columns(ensemble_series)

    def wmape(y_true, y_pred): return np.abs(y_true - y_pred).sum() / np.abs(y_true).sum()

    Calculate the WMAPE for the ensemble model

    wmape_value = wmape(crossvalidation_df['y'], crossvalidation_df['Ensemble']) print('Average WMAPE for Ensemble: ', round(wmape_value, 4))

    Calculate the errors for the ensemble model

    errors = crossvalidation_df['y'] - crossvalidation_df['Ensemble']

    For an individual forecast

    individual_accuracy = 1 - (abs(crossvalidation_df['y'] - crossvalidation_df['Ensemble']) / crossvalidation_df['y']) individual_bias = (crossvalidation_df['Ensemble'] / crossvalidation_df['y']) - 1

    Add these calculations as new columns to DataFrame

    crossvalidation_df = crossvalidation_df.with_columns([ individual_accuracy.alias("individual_accuracy"), individual_bias.alias("individual_bias") ])

    Print the individual accuracy and bias for each week

    for row in crossvalidation_df.to_dicts(): id = row['unique_id'] date = row['ds'] accuracy = row['individual_accuracy'] bias = row['individual_bias'] print(f"{id}, {date}, Individual Accuracy: {accuracy:.4f}, Individual Bias: {bias:.4f}")

    For groups of forecasts

    group_accuracy = 1 - (errors.abs().sum() / crossvalidation_df['y'].sum()) group_bias = (crossvalidation_df['Ensemble'].sum() / crossvalidation_df['y'].sum()) - 1

    Print the average group accuracy and group bias over all folds for the ensemble model

    print('Average Group Accuracy: ', round(group_accuracy, 4)) print('Average Group Bias: ', round(group_bias, 4))

    Fit the models on the entire dataset

    sf.fit(y_cl4_fit_1)

    Instantiate the ConformalIntervals class

    prediction_intervals = ConformalIntervals()

    Generate 24 months forecasts

    forecasts_df = sf.forecast(h=52*2, prediction_intervals=prediction_intervals, level=[95], id_col='unique_id', sort_df=True)

    Apply the non-negative constraint to the forecasts of individual models

    forecasts_df = forecasts_df.with_columns([ pl.when(pl.col('AutoARIMA') < 0).then(0).otherwise(pl.col('AutoARIMA')).alias('AutoARIMA'), pl.when(pl.col('AutoETS') < 0).then(0).otherwise(pl.col('AutoETS')).alias('AutoETS'), pl.when(pl.col('DynamicOptimizedTheta') < 0).then(0).otherwise(pl.col('DynamicOptimizedTheta')).alias('DynamicOptimizedTheta'), ])

    Calculate the ensemble forecast

    ensemble_forecast = forecasts_df[['AutoARIMA', 'AutoETS', 'DynamicOptimizedTheta']].mean(axis=1)

    Calculate the lower and upper prediction intervals for the ensemble forecast

    ensemble_lo_95 = forecasts_df.select( [ pl.when(pl.col('AutoARIMA-lo-95') < 0).then(0).otherwise(pl.col('AutoARIMA-lo-95')).alias('AutoARIMA-lo-95'), pl.when(pl.col('AutoETS-lo-95') < 0).then(0).otherwise(pl.col('AutoETS-lo-95')).alias('AutoETS-lo-95'), pl.when(pl.col('DynamicOptimizedTheta-lo-95') < 0).then(0).otherwise(pl.col('DynamicOptimizedTheta-lo-95')).alias('DynamicOptimizedTheta-lo-95'), ] ).mean(axis=1) ensemble_hi_95 = forecasts_df[['AutoARIMA-hi-95', 'AutoETS-hi-95', 'DynamicOptimizedTheta-hi-95']].mean(axis=1)

    Create Series for the ensemble forecast and its prediction intervals

    ensemble_forecast_series = pl.Series('EnsembleForecast', ensemble_forecast) ensemble_lo_95_series = pl.Series('Ensemble-lo-95', ensemble_lo_95) ensemble_hi_95_series = pl.Series('Ensemble-hi-95', ensemble_hi_95)

    Add the ensemble forecast and its prediction intervals as new columns to the DataFrame

    forecasts_df = forecasts_df.with_columns([ensemble_forecast_series, ensemble_lo_95_series, ensemble_hi_95_series])

    Round the ensemble forecast and prediction intervals and convert to integer

    forecasts_df = forecasts_df.with_columns([ pl.col("EnsembleForecast").round().cast(pl.Int32), pl.col("Ensemble-lo-95").round().cast(pl.Int32), pl.col("Ensemble-hi-95").round().cast(pl.Int32) ])

    Split the unique_id concat into the original columns

    forecasts_df = ( forecasts_df .with_columns(pl.col('unique_id').str.split('_', expand=True)) .rename({'unique_id_0': 'MaterialID', 'unique_id_1': 'SalesOrg', 'unique_id_2': 'DistrChan', 'unique_id_3': 'CL4'}) .drop('unique_id') )

    Rename ‘ds’ to ‘WeekDate’

    forecasts_df = forecasts_df.rename({'ds': 'WeekDate'})

    Reorder the columns

    forecasts_df = forecasts_df.select([ "MaterialID", "SalesOrg", "DistrChan", "CL4", "WeekDate", "EnsembleForecast", "Ensemble-lo-95", "Ensemble-hi-95", "AutoARIMA", "AutoARIMA-lo-95", "AutoARIMA-hi-95", "AutoETS", "AutoETS-lo-95", "AutoETS-hi-95", "DynamicOptimizedTheta", "DynamicOptimizedTheta-lo-95", "DynamicOptimizedTheta-hi-95" ])

    Create an empty list

    forecasts_list = []

    Append each row to the list

    for row in forecasts_df.to_dicts(): forecasts_list.append(row)

    Print the list

    for forecast in forecasts_list: print(forecast) some of the context of data # Filter for 'Low Volume' items dataset_newitem = dataset.filter(pl.col('Item Type') == 'New Item') # Convert 'WeekDate' to datetime format dataset_newitem = dataset_newitem.with_columns( pl.col("WeekDate").str.strptime(pl.Datetime, "%Y-%m-%d") )

    Group by ‘MaterialID’, ‘SalesOrg’, ‘DistrChan’, 'CL4' and 'WeekDate', then sum 'OrderQuantity'

    y_cl4 = dataset_newitem.groupby(['MaterialID', 'SalesOrg', 'DistrChan', 'CL4', 'WeekDate']).agg( pl.sum("OrderQuantity").alias("OrderQuantity") )

    Sort by 'WeekDate'

    y_cl4 = y_cl4.sort("WeekDate") # Concatenate ‘MaterialID’, ‘SalesOrg’, ‘DistrChan’, ‘CL4’ to a new column ‘unique_id’ y_cl4 = y_cl4.with_columns( pl.concat_str([pl.col('MaterialID'), pl.col('SalesOrg'), pl.col('DistrChan'), pl.col('CL4')], separator='_').alias('unique_id') )

    Drop the original columns

    y_cl4 = y_cl4.drop(['MaterialID', 'SalesOrg', 'DistrChan', 'CL4'])

    Renaming columns to 'ds' and 'y' to meet the input requirements of the StatsForecast library

    y_cl4 = y_cl4.rename({'WeekDate': 'ds', 'OrderQuantity': 'y'}), now read thru this website and learn from it, https://nixtlaverse.nixtla.io/statsforecast/docs/how-to-guides/generating_features.html, I want to use that example to get trend and seasonality for my ensemble model How to Guides Generating features Leverage StatsForecast models to create features

    Some models create internal representations of the series that can be useful for other models to use as inputs. One example is the MSTL model, which decomposes the series into trend and seasonal components. This guide shows you how to use the mstl_decomposition function to extract those features for training and then use their future values for inference.

    import os from functools import partial

    import pandas as pd import statsforecast from statsforecast import StatsForecast from statsforecast.feature_engineering import mstl_decomposition from statsforecast.models import ARIMA, MSTL from utilsforecast.evaluation import evaluate from utilsforecast.losses import smape, mase

    this makes it so that the outputs of the predict methods have the id as a column

    instead of as the index

    os.environ['NIXTLA_ID_AS_COL'] = '1'

    df = pd.read_parquet('https://datasets-nixtla.s3.amazonaws.com/m4-hourly.parquet') uids = df['unique_id'].unique()[:10] df = df[df['unique_id'].isin(uids)] df.head() unique_id ds y 0 H1 1 605.0 1 H1 2 586.0 2 H1 3 586.0 3 H1 4 559.0 4 H1 5 511.0 Suppose that you want to use an ARIMA model to forecast your series but you want to incorporate the trend and seasonal components from the MSTL model as external regressors. You can define the MSTL model to use and then provide it to the mstl_decomposition function.

    freq = 1 season_length = 24 horizon = 2 * season_length valid = df.groupby('unique_id').tail(horizon) train = df.drop(valid.index) model = MSTL(season_length=24) transformed_df, X_df = mstl_decomposition(train, model=model, freq=freq, h=horizon) This generates the dataframe that we should use for training (with the trend and seasonal columns added), as well as the dataframe we should use to forecast.

    transformed_df.head() unique_id ds y trend seasonal 0 H1 1 605.0 501.350550 124.683643 1 H1 2 586.0 506.424549 87.115039 2 H1 3 586.0 511.453736 79.479564 3 H1 4 559.0 516.434474 45.616992 4 H1 5 511.0 521.362991 -10.940498

    X_df.head() unique_id ds trend seasonal 0 H1 701 626.519191 -22.738998 1 H1 702 627.423191 -108.992314 2 H1 703 628.146391 -148.675478 3 H1 704 628.724951 -183.715284 4 H1 705 629.187799 -208.038694 We can now train our ARIMA models and compute our forecasts.

    sf = StatsForecast( models=[ARIMA(order=(1, 0, 1), season_length=season_length)], freq=freq ) preds = sf.forecast(h=horizon, df=transformed_df, X_df=X_df) preds.head() unique_id ds ARIMA 0 H1 701 597.716125 1 H1 702 513.244507 2 H1 703 475.042450 3 H1 704 441.217743 4 H1 705 417.895813

  2. Assistant

    To use trend and seasonality features from the MSTL model in your ensemble for forecasting, we need to extract these features and use them to improve the accuracy of the ensemble predictions. Here is a step-by-step guide on how to integrate the trend and seasonal components from the MSTL model into your existing ensemble model.

    1. Import necessary modules
    import numpy as np
    import pandas as pd
    import polars as pl
    from statsforecast import StatsForecast, feature_engineering
    from statsforecast.models import AutoARIMA, AutoETS, DynamicOptimizedTheta, MSTL
    
    1. Get the trend and seasonal components for your dataset
    # Assuming y_cl4_filtered is your dataset with the 'unique_id' and 'ds' columns
    model = MSTL(season_length=12)
    transformed_df, X_df = feature_engineering.mstl_decomposition(y_cl4_filtered, model=model, freq='1w', h=2)
    
    1. Train your model on the transformed dataset Now, you should train your models on the transformed_df, which contains the additional 'trend' and 'seasonal' columns.

    2. Prepare X_df for forecasting When calling the forecast method, make sure to pass X_df as the X_test argument to include the future values of trend and seasonality while forecasting.

    3. Fit your models using the transformed data

    sf = StatsForecast(models=models, freq='1w', n_jobs=-1)
    
    # Fit the models on the transformed dataset
    sf.fit(df=transformed_df)
    
    1. Forecast with feature X_df
    forecasts_df = sf.forecast(h=52*2, X_test=X_df)  # forecast for 24 months
    
    # Calculate the ensemble forecast using predictions (assuming 'AutoARIMA', 'AutoETS', and 'DynamicOptimizedTheta' are model names)
    ensemble_forecast = forecasts_df[['AutoARIMA', 'AutoETS', 'DynamicOptimizedTheta']].mean(axis=1)
    
    1. Add the ensemble forecast as a new column Modify your code to add the ensemble forecast and use the trend and seasonal features from the MSTL model:
    # Convert back from Polars to Pandas for compatibility with StatsForecast's expected inputs if necessary
    forecasts_df_pandas = forecasts_df.to_pandas()
    
    # Prepare the DataFrame with the additional trend and seasonal features
    forecasts_df_final = pd.concat([forecasts_df_pandas, ensemble_forecast], axis=1)
    forecasts_df_final.rename(columns={0: "EnsembleForecast"}, inplace=True)
    
    1. Finalize your forecasts You can follow through with the rest of the steps in your script with the newly created forecasts_df_final DataFrame, which includes the ensemble forecasts with trend and seasonality features from the MSTL model.

    Please note that the example you provided utilizes the MSTL model to extract the trend and seasonal components. If you want to include the MSTL model within your ensemble, you could do that as an additional model and then extract these features to use in your forecasting.

    Additionally, ensure that all the variables and data structures mentioned match those in your existing code. For example, the y_cl4_filtered should be a DataFrame containing the time series data you want to decompose and forecast.

  3. User

    data is weekly

  4. Assistant

    If your data is weekly, you will need to set season_length appropriately to capture the seasonal pattern within a year. For weekly data, the seasonality is often assumed to be 52 weeks, but it could be different based on your specific use case. Let's proceed with the steps assuming your seasonality is one year (52 weeks).

    1. Import necessary modules
    import numpy as np
    import polars as pl
    from statsforecast import StatsForecast, feature_engineering
    from statsforecast.models import AutoARIMA, AutoETS, DynamicOptimizedTheta, MSTL
    
    1. Extract trend and seasonal components For weekly data with an annual seasonality:
    # Assuming y_cl4_filtered is your pre-processed dataset with `unique_id` and `ds` as columns
    season_length = 52
    model = MSTL(season_length=season_length)
    transformed_df, X_df = feature_engineering.mstl_decomposition(y_cl4_filtered, model=model, freq='1W', h=52*2)
    
    1. Fit your models using the transformed data
    sf = StatsForecast(models=models, freq='W', n_jobs=-1)
    
    # Fit the models on the transformed dataset
    sf.fit(df=transformed_df)
    
    1. Forecast including the trend and seasonal features from MSTL
    forecasts_df = sf.forecast(h=52*2, X_test=X_df)  # forecast for about two years
    
    # Calculate the ensemble forecast using predictions (assuming 'AutoARIMA', 'AutoETS', and 'DynamicOptimizedTheta' are within your model names)
    ensemble_forecast = forecasts_df[['AutoARIMA', 'AutoETS', 'DynamicOptimizedTheta']].mean(axis=1)
    
    1. Add the ensemble forecast as a new column and proceed with your workflow Convert to polars if necessary and join with the forecasts DataFrame:
    ensemble_forecast_series = pl.Series('EnsembleForecast', ensemble_forecast)
    
    # Add the ensemble forecast as a new column to the forecasts DataFrame
    forecasts_df = forecasts_df.with_column(ensemble_forecast_series)
    

    Proceed with the rest of your script, such as adjusting forecasts to non-negative values, fitting models to the entire dataset, and generating prediction intervals as you did before. The main difference is that you are now using the trend and seasonal components extracted from MSTL as part of your ensemble's features.

    Please make sure you also adjust the remaining parts of your code regarding the use of transformed_df and X_df, ensuring they are input correctly into the fit and forecast functions. And keep in mind, this code assumes that you are comfortable with converting between pandas and polars DataFrames as necessary, since feature_engineering.mstl_decomposition from statsforecast works with pandas DataFrames.

  5. User

    is my code correct from statsforecast.feature_engineering import mstl_decomposition from statsforecast.models import AutoARIMA, MSTL

    freq = 1 season_length = 52 horizon = 2 * season_length valid = y_cl4.groupby('unique_id').tail(horizon) train = y_cl4.drop(valid.index) model = MSTL(season_length=season_length)--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In[9], line 8 6 horizon = 2 * season_length 7 valid = y_cl4.groupby('unique_id').tail(horizon) ----> 8 train = y_cl4.drop(valid.index) 9 model = MSTL(season_length=52) 10 transformed_df, X_df = mstl_decomposition(train, model=model, freq=freq, h=horizon)

    AttributeError: 'DataFrame' object has no attribute 'index' transformed_df, X_df = mstl_decomposition(train, model=model, freq=freq, h=horizon)

  6. Assistant

    It looks like there is a potential issue with your code. The error message 'DataFrame' object has no attribute 'index' suggests that you're trying to drop pandas DataFrame indices from a polars DataFrame. The drop method in polars does not accept indices in the same way as pandas does, and polars DataFrames do not have an index attribute.

    To fix this issue, you can do the following:

    1. Since polars does not use indices, like pandas, you'll need to figure out another way to drop the rows. You can use logical filtering instead to create the train DataFrame.

    2. mstl_decomposition function expects a pandas DataFrame, so you need to convert your polars DataFrame to pandas before the decomposition, if it is not already a pandas DataFrame.

    Let's refactor your code to use polars specifically for the operations required.

    from statsforecast.feature_engineering import mstl_decomposition
    from statsforecast.models import MSTL
    import polars as pl
    
    # Assuming y_cl4 is a polars DataFrame
    season_length = 52
    horizon = 2 * season_length
    
    # Split the DataFrame into training and validation sets
    # The tail method in polars can be used by the last method when grouped
    valid = y_cl4.lazy().groupby('unique_id').agg(pl.col('*').last(horizon)).collect()
    train = y_cl4.lazy().filter(~pl.col('ds').is_in(valid.select('ds'))).collect()
    
    # Convert to pandas for MSTL decomposition, assuming trend and seasonal decomposition is not supported by polars
    train_pd = train.to_pandas()
    valid_pd = valid.to_pandas()
    
    # Run MSTL decomposition, define your model
    model = MSTL(season_length=season_length)
    transformed_df, X_df = mstl_decomposition(
        train_pd, model=model, freq=freq, h=horizon
    )
    

    Remember, MSTL decomposition requires pandas DataFrames, so the conversion from polars to pandas is necessary before you apply mstl_decomposition.

    Also, in your code, you set freq = 1, which is not the correct frequency for weekly data in the context of statsforecast models. The freq parameter is expected to be a string that indicates the offset alias representing the frequency of the time series. For weekly data, you would use 'W' or '1W'.

    Make sure to confirm whether you want to use 'W' or '1W' depending on how the statsforecast library has been updated since the cutoff knowledge date.

    For ensuring that you have the correct most updated implementation details, please refer to the documentation of the specific version of statsforecast you have installed.

  7. User

    no mstl is available in statsforecast and statsforecast library support polars

  8. Assistant

    I apologize for my earlier confusion. If statsforecast supports polars and you're able to run the MSTL decomposition without converting to pandas, that's great.

    Based on your error, the issue seems to be with how you are attempting to split the y_cl4 polars DataFrame into the training and validation sets. The AttributeError indicates that y_cl4 does not have an index attribute, which is expected because polars does not use an index system like pandas.

    Here's how you should do it using polars and mstl within statsforecast:

    from statsforecast.feature_engineering import mstl_decomposition
    from statsforecast.models import MSTL
    import polars as pl
    
    # Assuming y_cl4 is a polars DataFrame.
    season_length = 52
    horizon = 2 * season_length
    
    # The process of creating validation and training sets
    valid = y_cl4.lazy().groupby('unique_id').agg(
        [pl.col('ds').tail(horizon), pl.col('y').tail(horizon)]
    ).explode([pl.col('ds'), pl.col('y')]).collect()
    
    train = y_cl4.lazy().filter(~pl.fold(
        # This creates a disjunction (logical OR) combining all the unique_ids and ds from valid
        # It speeds up the operation by avoiding an OR per group
        lambda acc, new: acc | new,
        [((pl.col('unique_id') == unique_id) & pl.col('ds').is_in(ds))
         for unique_id, ds in zip(valid['unique_id'], valid['ds'])]
    )).collect()
    
    # Define the MSTL model.
    model = MSTL(season_length=season_length)
    
    # Perform MSTL decomposition on the training dataset with polars DataFrame
    # Freq is not needed if the data is already in the correct frequency
    transformed_df, X_df = mstl_decomposition(train, model=model, h=horizon)
    
    # Proceed with fitting your models and forecasting as previously described.
    

    This should solve your issue, allowing you to split your dataset into training and validation sets with polars and perform MSTL decomposition available in statsforecast.

    Keep in mind that in the mstl_decomposition function you don't need to specify freq if your data is already in time series format with the ds column representing the date or datetime. The function should be able to infer the frequency from the ds column.

    Be sure to further verify your steps with the current versions of the statsforecast library documentation and keep an eye out for any specific nuances or updates it might have regarding the support for polars and MSTL decompositions.

  9. User

    the code didnt work, also the example I gave you https://nixtlaverse.nixtla.io/statsforecast/docs/how-to-guides/generating_features.html this is from nixtla, so we should try to stick to their code, is there a way to drop index in polars? since in pandas WeekDate is set as index, so what do we do here

  10. Assistant

    In Polars, unlike pandas, there is no dedicated index like the one in pandas DataFrames. Instead, each row has an implicit index similar to the way it works in arrays. If you want to simulate the dropping of rows by index like in pandas, you would use row numbers or a filtering condition in Polars.

    Given the confusion, let's create a training and validation split that aligns with the pandas example provided by Nixtla, without referencing row indices.

    Here's how you can adapt the Nixtla example to work with Polars usage for splitting your data into training and validation sets.

    Firstly, we need to determine the split dates so we can filter the data using these dates:

    import polars as pl
    from statsforecast.feature_engineering import mstl_decomposition
    from statsforecast.models import MSTL
    
    # Assuming y_cl4 has columns `unique_id`, `ds`, and `y`.
    # Find the last date in the dataset to start the horizon split
    last_date = y_cl4.select(pl.max('ds').alias('last_date')).collect().get_row(0)[0]
    
    # Calculate the horizon date for validation set
    horizon = 52 * 2
    validation_start_date = last_date - pl.duration_weeks(horizon)
    
    # Split into training and validation sets
    train = y_cl4.filter(pl.col('ds') < validation_start_date).collect()
    valid = y_cl4.filter(pl.col('ds') >= validation_start_date).collect()
    
    # Now that we have separated our datasets, we can define the MSTL model
    # (Make sure the 'ds' column is in your dataset and is of the correct datetime dtype)
    model = MSTL(season_length=52)
    
    # Perform MSTL decomposition on the training dataset
    transformed_df, X_df = mstl_decomposition(train, model=model, freq='W',  h=horizon)
    
    # Continue with fitting your models and forecasting.
    

    In the code above, the main action for splitting data is done using the filter operations by date (ds). This is similar to indexing based on boolean masks in pandas, but instead, in Polars, it's done directly on columns without referencing an explicit index. Now we're aligning with dates, which is typically a more robust method because it doesn't rely on indices that may change if the data is filtered or manipulated in preceding steps.

    The subsequent mstl_decomposition function is called in the same way, passing the train DataFrame which contains your time series data and the MSTL model instance you created. This call will return a transformed DataFrame and the future values DataFrame (X_df) which includes the extracted trend and seasonal components for the forecast horizon.

    As for the fitted model, since it's a weekly series, I’ve set the freq='W' to denote a weekly frequency. The h parameter is being set to twice the season_length (52 weeks in a year), aligning with Nixtla's example, but it can be adjusted according to your forecasting needs.

    After obtaining transformed_df and X_df you can continue with fitting any additional models and making forecasts, ensuring you are appending the trend and seasonality features where appropriate for your ensemble model.

  11. User

    AttributeError Traceback (most recent call last) Cell In[11], line 4 1 from statsforecast.feature_engineering import mstl_decomposition 2 from statsforecast.models import AutoARIMA, MSTL ----> 4 last_date = y_cl4.select(pl.max('ds').alias('last_date')).collect().get_row(0)[0] 6 freq = 1 7 season_length = 52

    AttributeError: 'DataFrame' object has no attribute 'collect'

  12. Assistant

    Apologies for the confusion. Let's correct that issue. In Polars, unlike pandas, you don't need to call .collect() after a .select() because operations are eagerly evaluated by default (unlike the .lazy() context where .collect() is required).

    Here's the corrected code for finding the last date and splitting the data:

    import polars as pl
    from statsforecast.feature_engineering import mstl_decomposition
    from statsforecast.models import MSTL
    
    # Assuming y_cl4 is a polars DataFrame with columns `unique_id`, `ds`, and `y`.
    # Find the last date in the dataset to start the horizon split
    last_date = y_cl4.select(pl.max('ds').alias('last_date')).get_row(0)[0]
    
    # Calculate the horizon date for validation set
    horizon = 52 * 2
    validation_start_date = last_date - pl.duration_weeks(horizon)
    
    # Split into training and validation sets
    train = y_cl4.filter(pl.col('ds') < validation_start_date)
    valid = y_cl4.filter(pl.col('ds') >= validation_start_date)
    
    # Define the MSTL model (make sure the season_length matches your seasonality)
    model = MSTL(season_length=52)
    
    # Perform MSTL decomposition on the training dataset
    transformed_df, X_df = mstl_decomposition(train, model=model, freq='W', h=horizon)
    
    # Continue with fitting your models and forecasting.
    

    Note:

    • The last_date calculation is done directly using select() followed by get_row().
    • The horizon calculation uses Polars' duration_weeks() to subtract weeks from the last date.
    • The training and validation splits are done using filter() and DataFrame comparisons.
    • The mstl_decomposition() function is identical to the previous example; ensure the season_length is correct for your data and the frequency (freq) is set to 'W' for weekly data.

    Now you can proceed with the MSTL decomposition and use the 'trend' and 'seasonal' components as features in your ensemble model or any subsequent analysis.

  13. User

    AttributeError Traceback (most recent call last) Cell In[13], line 4 1 from statsforecast.feature_engineering import mstl_decomposition 2 from statsforecast.models import AutoARIMA, MSTL ----> 4 last_date = y_cl4.select(pl.max('ds').alias('last_date')).get_row(0)[0] 6 freq = 1 7 season_length = 52

    AttributeError: 'DataFrame' object has no attribute 'get_row'

  14. Assistant

    I apologize for the misunderstanding in the usage of Polars. You're right, get_row is not a Polars method. Instead, you can access the value directly using indexing. Here's how to correct the code to properly get the last date in the Polars DataFrame:

    from statsforecast.models import MSTL
    from statsforecast.feature_engineering import mstl_decomposition
    import polars as pl
    
    # Assuming y_cl4 is a polars DataFrame with columns `unique_id`, `ds`, and `y`.
    # Find the last date in the dataset to start the horizon split
    last_date = y_cl4.select(pl.max('ds')).to_numpy()[0]
    
    # Calculate the horizon date for validation set
    horizon = 52 * 2
    # Depending on your data type, either cast your date to datetime using pl.datetime or use it directly
    if isinstance(last_date, str):
        last_date = pl.datetime(last_date)
    validation_start_date = last_date - pl.duration(days=horizon*7)
    
    # Split into training and validation sets
    train = y_cl4.filter(pl.col('ds') < validation_start_date)
    valid = y_cl4.filter(pl.col('ds') >= validation_start_date)
    
    # Define the MSTL model (ensure `season_length` matches your seasonality, typically 52 for weekly data)
    model = MSTL(season_length=52)
    
    # Perform MSTL decomposition on the training dataset
    # Freq here refers to frequency string, use 'W' for weekly data
    transformed_df, X_df = mstl_decomposition(train, model=model, freq='W', h=horizon)
    

    Please note the following corrections:

    • Instead of using .get_row(), we use .to_numpy()[0] to retrieve the maximum date directly after converting it to a numpy array.
    • I corrected the line for calculating validation_start_date to use pl.duration(days=horizon*7) as it's more explicit and will work regardless of the weekday being the maximum ds. If last_date is a string, we need to convert it to datetime; otherwise, if it's already a datetime object, we can subtract directly.

    Now you should be able to run the MSTL decomposition and use the returned data frames transformed_df and X_df to proceed with your forecasting tasks.

  15. User

    is tehre an easier way to do this feature engineering? read thru these 2 websites from nixtla https://github.com/Nixtla/tsfeatures, https://medium.com/@gosshhh9/exploring-nixtlas-feature-engineering-power-987cb42f125d, so we can use create_seasonal to do the same thing?

  16. Assistant

    Certainly! The tsfeatures library from Nixtla provides a range of functions for time series feature engineering, including functions for extracting trend and seasonal features. If you're looking for a simpler way to achieve this without going through MSTL decomposition, you can make use of these features.

    The tsfeatures library can compute a variety of time series features including those capturing trend and seasonality. According to the documentation and the medium article, tsfeatures provides simple APIs to extract features from time series datasets.

    Here's a basic example of how tsfeatures can be used to extract seasonal features:

    import polars as pl
    import pandas as pd
    from tsfeatures import tsfeatures
    
    # Assuming y_cl4 is your polars DataFrame and you have already preprocessed it
    # Convert polars DataFrame to pandas if tsfeatures library expects pandas DataFrame
    y_cl4_pd = y_cl4.to_pandas()
    
    # Define a function to extract seasonal features
    # You can adjust the window parameter based on your time series data
    def get_seasonal_features(df):
        return tsfeatures(df, freq=52)
    
    # Apply the function to extract features for each time series
    # Assuming 'unique_id' is a column that helps identify each time series uniquely
    # The resulting DataFrame will contain features for each 'unique_id'
    all_features = y_cl4_pd.groupby('unique_id').apply(get_seasonal_features)
    
    # If needed, convert the features back to polars
    all_features_pl = pl.from_pandas(all_features)
    

    In the example above, tsfeatures is being used to calculate features for each unique time series identified by unique_id. The tsfeatures function accepts a pandas DataFrame containing the time series data, and a freq parameter that specifies the seasonal frequency of the data (52 for weekly data). The resulting features are saved in all_features.

    Note that the tsfeatures function returns a pandas DataFrame, so we convert back to Polars using pl.from_pandas if that's the data format required for subsequent steps in your analysis.

    Before integrating tsfeatures into your workflow, make sure to read the documentation for tsfeatures to understand the various features it can calculate and how they might be beneficial for your specific time series tasks. The library's functions could streamline your feature engineering process significantly.

    Also, check and confirm whether the tsfeatures library installation is compatible with your environment and that you have the appropriate versions of each package before incorporating it into your workflow.

  17. User

    use polars, no pandas

  18. Assistant

    Understood. If you strictly want to use Polars and avoid converting to Pandas, you can still utilize Nixtla's tsfeatures with some adjustments. The tsfeatures library may require time series data in a long format, where each row represents an observation with a corresponding timestamp. Since tsfeatures is designed to work with Pandas, you would need to first convert the Polars DataFrame to Pandas, apply tsfeatures, and then convert back to Polars if you want to continue with Polars.

    Since you prefer not to use Pandas at all, let's stick with Polars. Unfortunately, as of my last update, Polars might not have a direct equivalent to tsfeatures. However, you can calculate the seasonal and trend components manually using Polars or use any other libraries that are compatible with Polars directly.

    To extract a simple seasonal component from your weekly time series data, you can use the create_seasonal function from statsforecast with Polars, like this:

    import polars as pl
    from statsforecast.utils import create_seasonal
    
    # Assuming y_cl4 is a polars DataFrame with 'ds' as datestamp or timestamp column
    season_length = 52
    
    # Create seasonal dummy features
    seasonal_features = create_seasonal(y_cl4, season_length=season_length, col_name='ds')
    y_cl4_with_seasonals = y_cl4.hstack(seasonal_features)
    
    # Now y_cl4_with_seasonals will include seasonal dummy features that you can use in modeling
    

    The create_seasonal utility function should create multiple new columns in your DataFrame corresponding to the seasonal dummy variables for your weekly frequency.

    Please note that generating trend and seasonality features could sometimes be more complicated than just adding columns of seasonal dummies. If this is the case and the features you require are not readily available in Polars, you may need to build your own functions to calculate them or consider using libraries that can interfacing with Polars. Mistral, for instance, is a library developed by the same team as statsforecast and tsfeatures and has functions for time series feature extraction directly compatible with Numpy and, potentially, Polars.

    Keep in mind, since tsfeatures primarily supports Pandas DataFrames, using this library would require conversion from Polars to Pandas and back to Polars, which you prefer to avoid. In case you need more complex seasonality or trend analysis directly in Polars, it might be best to explore or implement a custom solution.

  19. User

    this function is from tsfeatures nixtla? def get_seasonal_features(df): return tsfeatures(df, freq=52)

  20. Assistant

    My previous message was not accurate regarding the tsfeatures library. I apologize for the confusion. The tsfeatures library is designed to work with Pandas DataFrames, and the function tsfeatures from that library expects a pandas DataFrame as input. The get_seasonal_features function I outlined isn't directly from tsfeatures but rather a wrapper that you would use to apply the tsfeatures library to your data.

    If you strictly want to avoid using Pandas and stick with Polars, you won't be able to directly apply tsfeatures, which is made for Pandas. You would need to look for alternative feature engineering methods that are compatible with Polars.

    For feature engineering directly in Polars, you would need to manually extract the features you are interested in. Polars provides various window functions and groupby methods that allow you to perform sophisticated transformations and calculations that can serve as proxies for some of the time series features you might extract using tsfeatures. However, Polars does not have inbuilt seasonal decomposition or similar time series analysis tools that are comparable to tsfeatures or the statsmodels library in Python.

    As of now, if you need to perform feature engineering for time series analysis within the Polars ecosystem without converting to Pandas, you will need to manually implement the extraction of such features or use a library that operates directly on Polars DataFrames if one exists.

    You may also consider feature extraction functions that operate on Numpy arrays directly and apply these to columns of your Polars DataFrame, which can be converted to and from Numpy using the .to_numpy() and pl.from_numpy() methods.

    For any custom trend or seasonality feature extraction in Polars, you would perform window operations, use .groupby() with aggregation on the time column, or do calculations using Polars's datetime functionality. These techniques would involve bespoke code tailored to the particular patterns of seasonality and trend in your data.

  21. User

    I just wanted o make sure because one of the links I sent you it says Seasonal Features: Nixtla includes a tool for creating seasonal features, which are features that capture periodic patterns in the data. Seasonal features can be created using the create_seasonal function and i didnt see we using crweate_seasonal feature so that's why i'm asking

  22. Assistant

    I apologize for the confusion earlier. It seems there was a misunderstanding regarding the create_seasonal function. From what you described, create_seasonal appears to be a feature engineering tool provided by Nixtla that you can utilize to capture periodic patterns in your data.

    This create_seasonal function should create seasonal dummy variables that can help your forecasting models account for regular periodic fluctuations, which is common in time series data such as weekly, monthly, or yearly seasonality.

    Let's implement create_seasonal in your Polars workflow:

    import polars as pl
    from statsforecast.utils import create_seasonal
    
    # Assuming y_cl4 is your Polars DataFrame which includes your time series data
    # with 'ds' as the date column, 'y' as the target variable, and 'unique_id' as the time series identifier
    season_length = 52  # assuming weekly data with yearly seasonality
    
    # Creating seasonal features using create_seasonal
    seasonal_features = create_seasonal(y_cl4, season_length=season_length, date_col='ds', freq='W')
    
    # Join the seasonal features with your original DataFrame
    y_cl4_with_seasonal = y_cl4.join(seasonal_features, on='ds')
    
    # Now you can use y_cl4_with_seasonal for further modeling and forecasting
    

    This code uses create_seasonal to generate seasonal dummy features that correspond to the specific season_length. These features should represent the seasonality in your data well if the season_length correctly captures the dominant seasonal pattern. If your seasonality is not a perfect 52-week cycle or if you have multiple seasonal patterns, you may need to adjust the season_length value or use a more sophisticated method of seasonality detection and encoding.

    Always consult the official documentation to understand the exact behavior and requirements of each function you use, especially when using third-party libraries like those developed by Nixtla, as there may be updates or changes that are not covered in this assistance.

  23. User

    from statsforecast import StatsForecast from statsforecast.models import AutoARIMA, AutoETS, DynamicOptimizedTheta from statsforecast.utils import ConformalIntervals import numpy as np import polars as pl

    Polars option to display all rows

    pl.Config.set_tbl_rows(None)

    Initialize the models

    models = [ AutoARIMA(season_length=12), AutoETS(season_length=12), DynamicOptimizedTheta(season_length=12) ]

    Initialize the StatsForecast model

    sf = StatsForecast(models=models, freq='1w', n_jobs=-1)

    Perform cross-validation with a step size of 1 to mimic an expanding window

    crossvalidation_df = sf.cross_validation(df=y_cl4_filtered, h=2, step_size=1, n_windows=8, sort_df=True)

    Calculate the ensemble mean

    ensemble = crossvalidation_df[['AutoARIMA', 'AutoETS', 'DynamicOptimizedTheta']].mean(axis=1)

    Create a Series for the ensemble mean

    ensemble_series = pl.Series('Ensemble', ensemble)

    Add the ensemble mean as a new column to the DataFrame

    crossvalidation_df = crossvalidation_df.with_columns(ensemble_series)

    def wmape(y_true, y_pred): return np.abs(y_true - y_pred).sum() / np.abs(y_true).sum()

    Calculate the WMAPE for the ensemble model

    wmape_value = wmape(crossvalidation_df['y'], crossvalidation_df['Ensemble']) print('Average WMAPE for Ensemble: ', round(wmape_value, 4))

    Calculate the errors for the ensemble model

    errors = crossvalidation_df['y'] - crossvalidation_df['Ensemble']

    For an individual forecast

    individual_accuracy = 1 - (abs(crossvalidation_df['y'] - crossvalidation_df['Ensemble']) / crossvalidation_df['y']) individual_bias = (crossvalidation_df['Ensemble'] / crossvalidation_df['y']) - 1

    Add these calculations as new columns to DataFrame

    crossvalidation_df = crossvalidation_df.with_columns([ individual_accuracy.alias("individual_accuracy"), individual_bias.alias("individual_bias") ])

    Print the individual accuracy and bias for each week

    for row in crossvalidation_df.to_dicts(): id = row['unique_id'] date = row['ds'] accuracy = row['individual_accuracy'] bias = row['individual_bias'] print(f"{id}, {date}, Individual Accuracy: {accuracy:.4f}, Individual Bias: {bias:.4f}")

    For groups of forecasts

    group_accuracy = 1 - (errors.abs().sum() / crossvalidation_df['y'].sum()) group_bias = (crossvalidation_df['Ensemble'].sum() / crossvalidation_df['y'].sum()) - 1

    Print the average group accuracy and group bias over all folds for the ensemble model

    print('Average Group Accuracy: ', round(group_accuracy, 4)) print('Average Group Bias: ', round(group_bias, 4))

    Fit the models on the entire dataset

    sf.fit(y_cl4_fit_1)

    Instantiate the ConformalIntervals class

    prediction_intervals = ConformalIntervals()

    Generate 24 months forecasts

    forecasts_df = sf.forecast(h=52*2, prediction_intervals=prediction_intervals, level=[95], id_col='unique_id', sort_df=True)

    Apply the non-negative constraint to the forecasts of individual models

    forecasts_df = forecasts_df.with_columns([ pl.when(pl.col('AutoARIMA') < 0).then(0).otherwise(pl.col('AutoARIMA')).alias('AutoARIMA'), pl.when(pl.col('AutoETS') < 0).then(0).otherwise(pl.col('AutoETS')).alias('AutoETS'), pl.when(pl.col('DynamicOptimizedTheta') < 0).then(0).otherwise(pl.col('DynamicOptimizedTheta')).alias('DynamicOptimizedTheta'), ])

    Calculate the ensemble forecast

    ensemble_forecast = forecasts_df[['AutoARIMA', 'AutoETS', 'DynamicOptimizedTheta']].mean(axis=1)

    Calculate the lower and upper prediction intervals for the ensemble forecast

    ensemble_lo_95 = forecasts_df.select( [ pl.when(pl.col('AutoARIMA-lo-95') < 0).then(0).otherwise(pl.col('AutoARIMA-lo-95')).alias('AutoARIMA-lo-95'), pl.when(pl.col('AutoETS-lo-95') < 0).then(0).otherwise(pl.col('AutoETS-lo-95')).alias('AutoETS-lo-95'), pl.when(pl.col('DynamicOptimizedTheta-lo-95') < 0).then(0).otherwise(pl.col('DynamicOptimizedTheta-lo-95')).alias('DynamicOptimizedTheta-lo-95'), ] ).mean(axis=1) ensemble_hi_95 = forecasts_df[['AutoARIMA-hi-95', 'AutoETS-hi-95', 'DynamicOptimizedTheta-hi-95']].mean(axis=1)

    Create Series for the ensemble forecast and its prediction intervals

    ensemble_forecast_series = pl.Series('EnsembleForecast', ensemble_forecast) ensemble_lo_95_series = pl.Series('Ensemble-lo-95', ensemble_lo_95) ensemble_hi_95_series = pl.Series('Ensemble-hi-95', ensemble_hi_95)

    Add the ensemble forecast and its prediction intervals as new columns to the DataFrame

    forecasts_df = forecasts_df.with_columns([ensemble_forecast_series, ensemble_lo_95_series, ensemble_hi_95_series])

    Round the ensemble forecast and prediction intervals and convert to integer

    forecasts_df = forecasts_df.with_columns([ pl.col("EnsembleForecast").round().cast(pl.Int32), pl.col("Ensemble-lo-95").round().cast(pl.Int32), pl.col("Ensemble-hi-95").round().cast(pl.Int32) ])

    Split the unique_id concat into the original columns

    forecasts_df = ( forecasts_df .with_columns(pl.col('unique_id').str.split('_', expand=True)) .rename({'unique_id_0': 'MaterialID', 'unique_id_1': 'SalesOrg', 'unique_id_2': 'DistrChan', 'unique_id_3': 'CL4'}) .drop('unique_id') )

    Rename ‘ds’ to ‘WeekDate’

    forecasts_df = forecasts_df.rename({'ds': 'WeekDate'})

    Reorder the columns

    forecasts_df = forecasts_df.select([ "MaterialID", "SalesOrg", "DistrChan", "CL4", "WeekDate", "EnsembleForecast", "Ensemble-lo-95", "Ensemble-hi-95", "AutoARIMA", "AutoARIMA-lo-95", "AutoARIMA-hi-95", "AutoETS", "AutoETS-lo-95", "AutoETS-hi-95", "DynamicOptimizedTheta", "DynamicOptimizedTheta-lo-95", "DynamicOptimizedTheta-hi-95" ])

    Create an empty list

    forecasts_list = []

    Append each row to the list

    for row in forecasts_df.to_dicts(): forecasts_list.append(row)

    Print the list

    for forecast in forecasts_list: print(forecast) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[19], line 116 107 forecasts_df = forecasts_df.with_columns([ 108 pl.col("EnsembleForecast").round().cast(pl.Int32), 109 pl.col("Ensemble-lo-95").round().cast(pl.Int32), 110 pl.col("Ensemble-hi-95").round().cast(pl.Int32) 111 ]) 113 # Split the unique_id concat into the original columns 114 forecasts_df = ( 115 forecasts_df --> 116 .with_columns(pl.col('unique_id').str.split('_', expand=True)) 117 .rename({'unique_id_0': 'MaterialID', 'unique_id_1': 'SalesOrg', 'unique_id_2': 'DistrChan', 'unique_id_3': 'CL4'}) 118 .drop('unique_id') 119 ) 121 # Rename ‘ds’ to ‘WeekDate’ 122 forecasts_df = forecasts_df.rename({'ds': 'WeekDate'})

    TypeError: ExprStringNameSpace.split() got an unexpected keyword argument 'expand'

  24. Assistant

    The error message you're encountering indicates that there is a mismatch in the expected parameters for the split function in the Polars library. In Polars, the split method does not have an expand argument like in pandas. Instead, to split a string into multiple columns in Polars, you need to use the str.split_by method and then explode the resulting list into separate columns.

    Here's how you correct your code for splitting the unique_id column into its original components in Polars:

    # Split the 'unique_id' column into the original columns using `str.split_by` and explode into columns
    split_columns = forecasts_df.select(pl.col('unique_id').str.split_by("_", inclusive=True).alias("split_id"))
    split_columns = split_columns.with_column(pl.col("split_id").arr.eval(pl.element().alias("MaterialID"),
                                                                           pl.element(1).alias("SalesOrg"),
                                                                           pl.element(2).alias("DistrChan"),
                                                                           pl.element(3).alias("CL4")))
    forecasts_df = forecasts_df.hstack(split_columns.drop("split_id"))
    
    # Drop the original 'unique_id' column
    forecasts_df = forecasts_df.drop('unique_id')
    
    # Rename ‘ds’ to ‘WeekDate’
    forecasts_df = forecasts_df.rename({'ds': 'WeekDate'})
    
    # Reorder the columns
    forecasts_df = forecasts_df.select([
        "MaterialID",
        "SalesOrg",
        "DistrChan",
        "CL4",
        "WeekDate",
        "EnsembleForecast",
        "Ensemble-lo-95",
        "Ensemble-hi-95",
        "AutoARIMA",
        "AutoARIMA-lo-95",
        "AutoARIMA-hi-95",
        "AutoETS",
        "AutoETS-lo-95",
        "AutoETS-hi-95",
        "DynamicOptimizedTheta",
        "DynamicOptimizedTheta-lo-95",
        "DynamicOptimizedTheta-hi-95"
    ])
    

    In the code above:

    • str.split_by() is used to split the unique_id string into a list of components.
    • We use arr.eval() to map each element of the list to new columns with pl.element().alias() for each part of the split. Since arr.eval() is a relatively recent addition to Polars, make sure you're using an updated version of Polars that includes this method.
    • We then use hstack to add these split columns back to the original DataFrame.
    • The unique_id column is then dropped as it's no longer needed.
    • The rest of the code should work as expected, and you can continue with your modeling and analytics workflow.

    Make sure your Polars library is up to date to access these features. If you encounter any issues, consult the Polars documentation for the latest information on handling string operations and column manipulations.