this is my ensemble model from statsforecast import StatsForecast from…
- User
this is my ensemble model from statsforecast import StatsForecast from statsforecast.models import AutoARIMA, AutoETS, DynamicOptimizedTheta from statsforecast.utils import ConformalIntervals import numpy as np import polars as pl
Polars option to display all rows
pl.Config.set_tbl_rows(None)
Initialize the models
models = [ AutoARIMA(season_length=12), AutoETS(season_length=12), DynamicOptimizedTheta(season_length=12) ]
Initialize the StatsForecast model
sf = StatsForecast(models=models, freq='1w', n_jobs=-1)
Perform cross-validation with a step size of 1 to mimic an expanding window
crossvalidation_df = sf.cross_validation(df=y_cl4_filtered, h=2, step_size=1, n_windows=8, sort_df=True)
Calculate the ensemble mean
ensemble = crossvalidation_df[['AutoARIMA', 'AutoETS', 'DynamicOptimizedTheta']].mean(axis=1)
Create a Series for the ensemble mean
ensemble_series = pl.Series('Ensemble', ensemble)
Add the ensemble mean as a new column to the DataFrame
crossvalidation_df = crossvalidation_df.with_columns(ensemble_series)
def wmape(y_true, y_pred): return np.abs(y_true - y_pred).sum() / np.abs(y_true).sum()
Calculate the WMAPE for the ensemble model
wmape_value = wmape(crossvalidation_df['y'], crossvalidation_df['Ensemble']) print('Average WMAPE for Ensemble: ', round(wmape_value, 4))
Calculate the errors for the ensemble model
errors = crossvalidation_df['y'] - crossvalidation_df['Ensemble']
For an individual forecast
individual_accuracy = 1 - (abs(crossvalidation_df['y'] - crossvalidation_df['Ensemble']) / crossvalidation_df['y']) individual_bias = (crossvalidation_df['Ensemble'] / crossvalidation_df['y']) - 1
Add these calculations as new columns to DataFrame
crossvalidation_df = crossvalidation_df.with_columns([ individual_accuracy.alias("individual_accuracy"), individual_bias.alias("individual_bias") ])
Print the individual accuracy and bias for each week
for row in crossvalidation_df.to_dicts(): id = row['unique_id'] date = row['ds'] accuracy = row['individual_accuracy'] bias = row['individual_bias'] print(f"{id}, {date}, Individual Accuracy: {accuracy:.4f}, Individual Bias: {bias:.4f}")
For groups of forecasts
group_accuracy = 1 - (errors.abs().sum() / crossvalidation_df['y'].sum()) group_bias = (crossvalidation_df['Ensemble'].sum() / crossvalidation_df['y'].sum()) - 1
Print the average group accuracy and group bias over all folds for the ensemble model
print('Average Group Accuracy: ', round(group_accuracy, 4)) print('Average Group Bias: ', round(group_bias, 4))
Fit the models on the entire dataset
sf.fit(y_cl4_fit_1)
Instantiate the ConformalIntervals class
prediction_intervals = ConformalIntervals()
Generate 24 months forecasts
forecasts_df = sf.forecast(h=52*2, prediction_intervals=prediction_intervals, level=[95], id_col='unique_id', sort_df=True)
Apply the non-negative constraint to the forecasts of individual models
forecasts_df = forecasts_df.with_columns([ pl.when(pl.col('AutoARIMA') < 0).then(0).otherwise(pl.col('AutoARIMA')).alias('AutoARIMA'), pl.when(pl.col('AutoETS') < 0).then(0).otherwise(pl.col('AutoETS')).alias('AutoETS'), pl.when(pl.col('DynamicOptimizedTheta') < 0).then(0).otherwise(pl.col('DynamicOptimizedTheta')).alias('DynamicOptimizedTheta'), ])
Calculate the ensemble forecast
ensemble_forecast = forecasts_df[['AutoARIMA', 'AutoETS', 'DynamicOptimizedTheta']].mean(axis=1)
Calculate the lower and upper prediction intervals for the ensemble forecast
ensemble_lo_95 = forecasts_df.select( [ pl.when(pl.col('AutoARIMA-lo-95') < 0).then(0).otherwise(pl.col('AutoARIMA-lo-95')).alias('AutoARIMA-lo-95'), pl.when(pl.col('AutoETS-lo-95') < 0).then(0).otherwise(pl.col('AutoETS-lo-95')).alias('AutoETS-lo-95'), pl.when(pl.col('DynamicOptimizedTheta-lo-95') < 0).then(0).otherwise(pl.col('DynamicOptimizedTheta-lo-95')).alias('DynamicOptimizedTheta-lo-95'), ] ).mean(axis=1) ensemble_hi_95 = forecasts_df[['AutoARIMA-hi-95', 'AutoETS-hi-95', 'DynamicOptimizedTheta-hi-95']].mean(axis=1)
Create Series for the ensemble forecast and its prediction intervals
ensemble_forecast_series = pl.Series('EnsembleForecast', ensemble_forecast) ensemble_lo_95_series = pl.Series('Ensemble-lo-95', ensemble_lo_95) ensemble_hi_95_series = pl.Series('Ensemble-hi-95', ensemble_hi_95)
Add the ensemble forecast and its prediction intervals as new columns to the DataFrame
forecasts_df = forecasts_df.with_columns([ensemble_forecast_series, ensemble_lo_95_series, ensemble_hi_95_series])
Round the ensemble forecast and prediction intervals and convert to integer
forecasts_df = forecasts_df.with_columns([ pl.col("EnsembleForecast").round().cast(pl.Int32), pl.col("Ensemble-lo-95").round().cast(pl.Int32), pl.col("Ensemble-hi-95").round().cast(pl.Int32) ])
Split the unique_id concat into the original columns
forecasts_df = ( forecasts_df .with_columns(pl.col('unique_id').str.split('_', expand=True)) .rename({'unique_id_0': 'MaterialID', 'unique_id_1': 'SalesOrg', 'unique_id_2': 'DistrChan', 'unique_id_3': 'CL4'}) .drop('unique_id') )
Rename ‘ds’ to ‘WeekDate’
forecasts_df = forecasts_df.rename({'ds': 'WeekDate'})
Reorder the columns
forecasts_df = forecasts_df.select([ "MaterialID", "SalesOrg", "DistrChan", "CL4", "WeekDate", "EnsembleForecast", "Ensemble-lo-95", "Ensemble-hi-95", "AutoARIMA", "AutoARIMA-lo-95", "AutoARIMA-hi-95", "AutoETS", "AutoETS-lo-95", "AutoETS-hi-95", "DynamicOptimizedTheta", "DynamicOptimizedTheta-lo-95", "DynamicOptimizedTheta-hi-95" ])
Create an empty list
forecasts_list = []
Append each row to the list
for row in forecasts_df.to_dicts(): forecasts_list.append(row)
Print the list
for forecast in forecasts_list: print(forecast) some of the context of data # Filter for 'Low Volume' items dataset_newitem = dataset.filter(pl.col('Item Type') == 'New Item') # Convert 'WeekDate' to datetime format dataset_newitem = dataset_newitem.with_columns( pl.col("WeekDate").str.strptime(pl.Datetime, "%Y-%m-%d") )
Group by ‘MaterialID’, ‘SalesOrg’, ‘DistrChan’, 'CL4' and 'WeekDate', then sum 'OrderQuantity'
y_cl4 = dataset_newitem.groupby(['MaterialID', 'SalesOrg', 'DistrChan', 'CL4', 'WeekDate']).agg( pl.sum("OrderQuantity").alias("OrderQuantity") )
Sort by 'WeekDate'
y_cl4 = y_cl4.sort("WeekDate") # Concatenate ‘MaterialID’, ‘SalesOrg’, ‘DistrChan’, ‘CL4’ to a new column ‘unique_id’ y_cl4 = y_cl4.with_columns( pl.concat_str([pl.col('MaterialID'), pl.col('SalesOrg'), pl.col('DistrChan'), pl.col('CL4')], separator='_').alias('unique_id') )
Drop the original columns
y_cl4 = y_cl4.drop(['MaterialID', 'SalesOrg', 'DistrChan', 'CL4'])
Renaming columns to 'ds' and 'y' to meet the input requirements of the StatsForecast library
y_cl4 = y_cl4.rename({'WeekDate': 'ds', 'OrderQuantity': 'y'}), now read thru this website and learn from it, https://nixtlaverse.nixtla.io/statsforecast/docs/how-to-guides/generating_features.html, I want to use that example to get trend and seasonality for my ensemble model How to Guides Generating features Leverage StatsForecast models to create features
Some models create internal representations of the series that can be useful for other models to use as inputs. One example is the MSTL model, which decomposes the series into trend and seasonal components. This guide shows you how to use the mstl_decomposition function to extract those features for training and then use their future values for inference.
import os from functools import partial
import pandas as pd import statsforecast from statsforecast import StatsForecast from statsforecast.feature_engineering import mstl_decomposition from statsforecast.models import ARIMA, MSTL from utilsforecast.evaluation import evaluate from utilsforecast.losses import smape, mase
this makes it so that the outputs of the predict methods have the id as a column
instead of as the index
os.environ['NIXTLA_ID_AS_COL'] = '1'
df = pd.read_parquet('https://datasets-nixtla.s3.amazonaws.com/m4-hourly.parquet') uids = df['unique_id'].unique()[:10] df = df[df['unique_id'].isin(uids)] df.head() unique_id ds y 0 H1 1 605.0 1 H1 2 586.0 2 H1 3 586.0 3 H1 4 559.0 4 H1 5 511.0 Suppose that you want to use an ARIMA model to forecast your series but you want to incorporate the trend and seasonal components from the MSTL model as external regressors. You can define the MSTL model to use and then provide it to the mstl_decomposition function.
freq = 1 season_length = 24 horizon = 2 * season_length valid = df.groupby('unique_id').tail(horizon) train = df.drop(valid.index) model = MSTL(season_length=24) transformed_df, X_df = mstl_decomposition(train, model=model, freq=freq, h=horizon) This generates the dataframe that we should use for training (with the trend and seasonal columns added), as well as the dataframe we should use to forecast.
transformed_df.head() unique_id ds y trend seasonal 0 H1 1 605.0 501.350550 124.683643 1 H1 2 586.0 506.424549 87.115039 2 H1 3 586.0 511.453736 79.479564 3 H1 4 559.0 516.434474 45.616992 4 H1 5 511.0 521.362991 -10.940498
X_df.head() unique_id ds trend seasonal 0 H1 701 626.519191 -22.738998 1 H1 702 627.423191 -108.992314 2 H1 703 628.146391 -148.675478 3 H1 704 628.724951 -183.715284 4 H1 705 629.187799 -208.038694 We can now train our ARIMA models and compute our forecasts.
sf = StatsForecast( models=[ARIMA(order=(1, 0, 1), season_length=season_length)], freq=freq ) preds = sf.forecast(h=horizon, df=transformed_df, X_df=X_df) preds.head() unique_id ds ARIMA 0 H1 701 597.716125 1 H1 702 513.244507 2 H1 703 475.042450 3 H1 704 441.217743 4 H1 705 417.895813
- Assistant
To use trend and seasonality features from the MSTL model in your ensemble for forecasting, we need to extract these features and use them to improve the accuracy of the ensemble predictions. Here is a step-by-step guide on how to integrate the trend and seasonal components from the MSTL model into your existing ensemble model.
- Import necessary modules
import numpy as np import pandas as pd import polars as pl from statsforecast import StatsForecast, feature_engineering from statsforecast.models import AutoARIMA, AutoETS, DynamicOptimizedTheta, MSTL- Get the trend and seasonal components for your dataset
# Assuming y_cl4_filtered is your dataset with the 'unique_id' and 'ds' columns model = MSTL(season_length=12) transformed_df, X_df = feature_engineering.mstl_decomposition(y_cl4_filtered, model=model, freq='1w', h=2)-
Train your model on the transformed dataset Now, you should train your models on the
transformed_df, which contains the additional 'trend' and 'seasonal' columns. -
Prepare
X_dffor forecasting When calling the forecast method, make sure to passX_dfas theX_testargument to include the future values of trend and seasonality while forecasting. -
Fit your models using the transformed data
sf = StatsForecast(models=models, freq='1w', n_jobs=-1) # Fit the models on the transformed dataset sf.fit(df=transformed_df)- Forecast with feature
X_df
forecasts_df = sf.forecast(h=52*2, X_test=X_df) # forecast for 24 months # Calculate the ensemble forecast using predictions (assuming 'AutoARIMA', 'AutoETS', and 'DynamicOptimizedTheta' are model names) ensemble_forecast = forecasts_df[['AutoARIMA', 'AutoETS', 'DynamicOptimizedTheta']].mean(axis=1)- Add the ensemble forecast as a new column Modify your code to add the ensemble forecast and use the trend and seasonal features from the MSTL model:
# Convert back from Polars to Pandas for compatibility with StatsForecast's expected inputs if necessary forecasts_df_pandas = forecasts_df.to_pandas() # Prepare the DataFrame with the additional trend and seasonal features forecasts_df_final = pd.concat([forecasts_df_pandas, ensemble_forecast], axis=1) forecasts_df_final.rename(columns={0: "EnsembleForecast"}, inplace=True)- Finalize your forecasts
You can follow through with the rest of the steps in your script with the newly created
forecasts_df_finalDataFrame, which includes the ensemble forecasts with trend and seasonality features from the MSTL model.
Please note that the example you provided utilizes the MSTL model to extract the trend and seasonal components. If you want to include the MSTL model within your ensemble, you could do that as an additional model and then extract these features to use in your forecasting.
Additionally, ensure that all the variables and data structures mentioned match those in your existing code. For example, the
y_cl4_filteredshould be a DataFrame containing the time series data you want to decompose and forecast. - User
data is weekly
- Assistant
If your data is weekly, you will need to set
season_lengthappropriately to capture the seasonal pattern within a year. For weekly data, the seasonality is often assumed to be 52 weeks, but it could be different based on your specific use case. Let's proceed with the steps assuming your seasonality is one year (52 weeks).- Import necessary modules
import numpy as np import polars as pl from statsforecast import StatsForecast, feature_engineering from statsforecast.models import AutoARIMA, AutoETS, DynamicOptimizedTheta, MSTL- Extract trend and seasonal components For weekly data with an annual seasonality:
# Assuming y_cl4_filtered is your pre-processed dataset with `unique_id` and `ds` as columns season_length = 52 model = MSTL(season_length=season_length) transformed_df, X_df = feature_engineering.mstl_decomposition(y_cl4_filtered, model=model, freq='1W', h=52*2)- Fit your models using the transformed data
sf = StatsForecast(models=models, freq='W', n_jobs=-1) # Fit the models on the transformed dataset sf.fit(df=transformed_df)- Forecast including the trend and seasonal features from MSTL
forecasts_df = sf.forecast(h=52*2, X_test=X_df) # forecast for about two years # Calculate the ensemble forecast using predictions (assuming 'AutoARIMA', 'AutoETS', and 'DynamicOptimizedTheta' are within your model names) ensemble_forecast = forecasts_df[['AutoARIMA', 'AutoETS', 'DynamicOptimizedTheta']].mean(axis=1)- Add the ensemble forecast as a new column and proceed with your workflow Convert to polars if necessary and join with the forecasts DataFrame:
ensemble_forecast_series = pl.Series('EnsembleForecast', ensemble_forecast) # Add the ensemble forecast as a new column to the forecasts DataFrame forecasts_df = forecasts_df.with_column(ensemble_forecast_series)Proceed with the rest of your script, such as adjusting forecasts to non-negative values, fitting models to the entire dataset, and generating prediction intervals as you did before. The main difference is that you are now using the trend and seasonal components extracted from MSTL as part of your ensemble's features.
Please make sure you also adjust the remaining parts of your code regarding the use of
transformed_dfandX_df, ensuring they are input correctly into thefitandforecastfunctions. And keep in mind, this code assumes that you are comfortable with converting between pandas and polars DataFrames as necessary, sincefeature_engineering.mstl_decompositionfromstatsforecastworks with pandas DataFrames. - User
is my code correct from statsforecast.feature_engineering import mstl_decomposition from statsforecast.models import AutoARIMA, MSTL
freq = 1 season_length = 52 horizon = 2 * season_length valid = y_cl4.groupby('unique_id').tail(horizon) train = y_cl4.drop(valid.index) model = MSTL(season_length=season_length)--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In[9], line 8 6 horizon = 2 * season_length 7 valid = y_cl4.groupby('unique_id').tail(horizon) ----> 8 train = y_cl4.drop(valid.index) 9 model = MSTL(season_length=52) 10 transformed_df, X_df = mstl_decomposition(train, model=model, freq=freq, h=horizon)
AttributeError: 'DataFrame' object has no attribute 'index' transformed_df, X_df = mstl_decomposition(train, model=model, freq=freq, h=horizon)
- Assistant
It looks like there is a potential issue with your code. The error message
'DataFrame' object has no attribute 'index'suggests that you're trying to drop pandas DataFrame indices from a polars DataFrame. Thedropmethod in polars does not accept indices in the same way as pandas does, and polars DataFrames do not have anindexattribute.To fix this issue, you can do the following:
-
Since polars does not use indices, like pandas, you'll need to figure out another way to drop the rows. You can use logical filtering instead to create the
trainDataFrame. -
mstl_decompositionfunction expects a pandas DataFrame, so you need to convert your polars DataFrame to pandas before the decomposition, if it is not already a pandas DataFrame.
Let's refactor your code to use polars specifically for the operations required.
from statsforecast.feature_engineering import mstl_decomposition from statsforecast.models import MSTL import polars as pl # Assuming y_cl4 is a polars DataFrame season_length = 52 horizon = 2 * season_length # Split the DataFrame into training and validation sets # The tail method in polars can be used by the last method when grouped valid = y_cl4.lazy().groupby('unique_id').agg(pl.col('*').last(horizon)).collect() train = y_cl4.lazy().filter(~pl.col('ds').is_in(valid.select('ds'))).collect() # Convert to pandas for MSTL decomposition, assuming trend and seasonal decomposition is not supported by polars train_pd = train.to_pandas() valid_pd = valid.to_pandas() # Run MSTL decomposition, define your model model = MSTL(season_length=season_length) transformed_df, X_df = mstl_decomposition( train_pd, model=model, freq=freq, h=horizon )Remember, MSTL decomposition requires pandas DataFrames, so the conversion from polars to pandas is necessary before you apply
mstl_decomposition.Also, in your code, you set
freq = 1, which is not the correct frequency for weekly data in the context ofstatsforecastmodels. Thefreqparameter is expected to be a string that indicates the offset alias representing the frequency of the time series. For weekly data, you would use'W'or'1W'.Make sure to confirm whether you want to use
'W'or'1W'depending on how thestatsforecastlibrary has been updated since the cutoff knowledge date.For ensuring that you have the correct most updated implementation details, please refer to the documentation of the specific version of
statsforecastyou have installed. -
- User
no mstl is available in statsforecast and statsforecast library support polars
- Assistant
I apologize for my earlier confusion. If
statsforecastsupportspolarsand you're able to run the MSTL decomposition without converting to pandas, that's great.Based on your error, the issue seems to be with how you are attempting to split the
y_cl4polarsDataFrame into the training and validation sets. The AttributeError indicates thaty_cl4does not have anindexattribute, which is expected becausepolarsdoes not use an index system likepandas.Here's how you should do it using polars and mstl within statsforecast:
from statsforecast.feature_engineering import mstl_decomposition from statsforecast.models import MSTL import polars as pl # Assuming y_cl4 is a polars DataFrame. season_length = 52 horizon = 2 * season_length # The process of creating validation and training sets valid = y_cl4.lazy().groupby('unique_id').agg( [pl.col('ds').tail(horizon), pl.col('y').tail(horizon)] ).explode([pl.col('ds'), pl.col('y')]).collect() train = y_cl4.lazy().filter(~pl.fold( # This creates a disjunction (logical OR) combining all the unique_ids and ds from valid # It speeds up the operation by avoiding an OR per group lambda acc, new: acc | new, [((pl.col('unique_id') == unique_id) & pl.col('ds').is_in(ds)) for unique_id, ds in zip(valid['unique_id'], valid['ds'])] )).collect() # Define the MSTL model. model = MSTL(season_length=season_length) # Perform MSTL decomposition on the training dataset with polars DataFrame # Freq is not needed if the data is already in the correct frequency transformed_df, X_df = mstl_decomposition(train, model=model, h=horizon) # Proceed with fitting your models and forecasting as previously described.This should solve your issue, allowing you to split your dataset into training and validation sets with polars and perform MSTL decomposition available in
statsforecast.Keep in mind that in the
mstl_decompositionfunction you don't need to specifyfreqif your data is already in time series format with thedscolumn representing the date or datetime. The function should be able to infer the frequency from thedscolumn.Be sure to further verify your steps with the current versions of the
statsforecastlibrary documentation and keep an eye out for any specific nuances or updates it might have regarding the support forpolarsand MSTL decompositions. - User
the code didnt work, also the example I gave you https://nixtlaverse.nixtla.io/statsforecast/docs/how-to-guides/generating_features.html this is from nixtla, so we should try to stick to their code, is there a way to drop index in polars? since in pandas WeekDate is set as index, so what do we do here
- Assistant
In Polars, unlike pandas, there is no dedicated index like the one in pandas DataFrames. Instead, each row has an implicit index similar to the way it works in arrays. If you want to simulate the dropping of rows by index like in pandas, you would use row numbers or a filtering condition in Polars.
Given the confusion, let's create a training and validation split that aligns with the pandas example provided by Nixtla, without referencing row indices.
Here's how you can adapt the Nixtla example to work with Polars usage for splitting your data into training and validation sets.
Firstly, we need to determine the split dates so we can filter the data using these dates:
import polars as pl from statsforecast.feature_engineering import mstl_decomposition from statsforecast.models import MSTL # Assuming y_cl4 has columns `unique_id`, `ds`, and `y`. # Find the last date in the dataset to start the horizon split last_date = y_cl4.select(pl.max('ds').alias('last_date')).collect().get_row(0)[0] # Calculate the horizon date for validation set horizon = 52 * 2 validation_start_date = last_date - pl.duration_weeks(horizon) # Split into training and validation sets train = y_cl4.filter(pl.col('ds') < validation_start_date).collect() valid = y_cl4.filter(pl.col('ds') >= validation_start_date).collect() # Now that we have separated our datasets, we can define the MSTL model # (Make sure the 'ds' column is in your dataset and is of the correct datetime dtype) model = MSTL(season_length=52) # Perform MSTL decomposition on the training dataset transformed_df, X_df = mstl_decomposition(train, model=model, freq='W', h=horizon) # Continue with fitting your models and forecasting.In the code above, the main action for splitting data is done using the filter operations by date (
ds). This is similar to indexing based on boolean masks in pandas, but instead, in Polars, it's done directly on columns without referencing an explicit index. Now we're aligning with dates, which is typically a more robust method because it doesn't rely on indices that may change if the data is filtered or manipulated in preceding steps.The subsequent
mstl_decompositionfunction is called in the same way, passing thetrainDataFrame which contains your time series data and theMSTLmodel instance you created. This call will return a transformed DataFrame and the future values DataFrame (X_df) which includes the extracted trend and seasonal components for the forecast horizon.As for the fitted model, since it's a weekly series, I’ve set the
freq='W'to denote a weekly frequency. Thehparameter is being set to twice theseason_length(52 weeks in a year), aligning with Nixtla's example, but it can be adjusted according to your forecasting needs.After obtaining
transformed_dfandX_dfyou can continue with fitting any additional models and making forecasts, ensuring you are appending the trend and seasonality features where appropriate for your ensemble model. - User
AttributeError Traceback (most recent call last) Cell In[11], line 4 1 from statsforecast.feature_engineering import mstl_decomposition 2 from statsforecast.models import AutoARIMA, MSTL ----> 4 last_date = y_cl4.select(pl.max('ds').alias('last_date')).collect().get_row(0)[0] 6 freq = 1 7 season_length = 52
AttributeError: 'DataFrame' object has no attribute 'collect'
- Assistant
Apologies for the confusion. Let's correct that issue. In Polars, unlike pandas, you don't need to call
.collect()after a.select()because operations are eagerly evaluated by default (unlike the.lazy()context where.collect()is required).Here's the corrected code for finding the last date and splitting the data:
import polars as pl from statsforecast.feature_engineering import mstl_decomposition from statsforecast.models import MSTL # Assuming y_cl4 is a polars DataFrame with columns `unique_id`, `ds`, and `y`. # Find the last date in the dataset to start the horizon split last_date = y_cl4.select(pl.max('ds').alias('last_date')).get_row(0)[0] # Calculate the horizon date for validation set horizon = 52 * 2 validation_start_date = last_date - pl.duration_weeks(horizon) # Split into training and validation sets train = y_cl4.filter(pl.col('ds') < validation_start_date) valid = y_cl4.filter(pl.col('ds') >= validation_start_date) # Define the MSTL model (make sure the season_length matches your seasonality) model = MSTL(season_length=52) # Perform MSTL decomposition on the training dataset transformed_df, X_df = mstl_decomposition(train, model=model, freq='W', h=horizon) # Continue with fitting your models and forecasting.Note:
- The
last_datecalculation is done directly usingselect()followed byget_row(). - The
horizoncalculation uses Polars'duration_weeks()to subtract weeks from the last date. - The training and validation splits are done using
filter()and DataFrame comparisons. - The
mstl_decomposition()function is identical to the previous example; ensure theseason_lengthis correct for your data and the frequency (freq) is set to'W'for weekly data.
Now you can proceed with the MSTL decomposition and use the
'trend'and'seasonal'components as features in your ensemble model or any subsequent analysis. - The
- User
AttributeError Traceback (most recent call last) Cell In[13], line 4 1 from statsforecast.feature_engineering import mstl_decomposition 2 from statsforecast.models import AutoARIMA, MSTL ----> 4 last_date = y_cl4.select(pl.max('ds').alias('last_date')).get_row(0)[0] 6 freq = 1 7 season_length = 52
AttributeError: 'DataFrame' object has no attribute 'get_row'
- Assistant
I apologize for the misunderstanding in the usage of Polars. You're right,
get_rowis not a Polars method. Instead, you can access the value directly using indexing. Here's how to correct the code to properly get the last date in the Polars DataFrame:from statsforecast.models import MSTL from statsforecast.feature_engineering import mstl_decomposition import polars as pl # Assuming y_cl4 is a polars DataFrame with columns `unique_id`, `ds`, and `y`. # Find the last date in the dataset to start the horizon split last_date = y_cl4.select(pl.max('ds')).to_numpy()[0] # Calculate the horizon date for validation set horizon = 52 * 2 # Depending on your data type, either cast your date to datetime using pl.datetime or use it directly if isinstance(last_date, str): last_date = pl.datetime(last_date) validation_start_date = last_date - pl.duration(days=horizon*7) # Split into training and validation sets train = y_cl4.filter(pl.col('ds') < validation_start_date) valid = y_cl4.filter(pl.col('ds') >= validation_start_date) # Define the MSTL model (ensure `season_length` matches your seasonality, typically 52 for weekly data) model = MSTL(season_length=52) # Perform MSTL decomposition on the training dataset # Freq here refers to frequency string, use 'W' for weekly data transformed_df, X_df = mstl_decomposition(train, model=model, freq='W', h=horizon)Please note the following corrections:
- Instead of using
.get_row(), we use.to_numpy()[0]to retrieve the maximum date directly after converting it to a numpy array. - I corrected the line for calculating
validation_start_dateto usepl.duration(days=horizon*7)as it's more explicit and will work regardless of the weekday being the maximumds. Iflast_dateis a string, we need to convert it to datetime; otherwise, if it's already a datetime object, we can subtract directly.
Now you should be able to run the MSTL decomposition and use the returned data frames
transformed_dfandX_dfto proceed with your forecasting tasks. - Instead of using
- User
is tehre an easier way to do this feature engineering? read thru these 2 websites from nixtla https://github.com/Nixtla/tsfeatures, https://medium.com/@gosshhh9/exploring-nixtlas-feature-engineering-power-987cb42f125d, so we can use create_seasonal to do the same thing?
- Assistant
Certainly! The
tsfeatureslibrary from Nixtla provides a range of functions for time series feature engineering, including functions for extracting trend and seasonal features. If you're looking for a simpler way to achieve this without going through MSTL decomposition, you can make use of these features.The
tsfeatureslibrary can compute a variety of time series features including those capturing trend and seasonality. According to the documentation and the medium article,tsfeaturesprovides simple APIs to extract features from time series datasets.Here's a basic example of how
tsfeaturescan be used to extract seasonal features:import polars as pl import pandas as pd from tsfeatures import tsfeatures # Assuming y_cl4 is your polars DataFrame and you have already preprocessed it # Convert polars DataFrame to pandas if tsfeatures library expects pandas DataFrame y_cl4_pd = y_cl4.to_pandas() # Define a function to extract seasonal features # You can adjust the window parameter based on your time series data def get_seasonal_features(df): return tsfeatures(df, freq=52) # Apply the function to extract features for each time series # Assuming 'unique_id' is a column that helps identify each time series uniquely # The resulting DataFrame will contain features for each 'unique_id' all_features = y_cl4_pd.groupby('unique_id').apply(get_seasonal_features) # If needed, convert the features back to polars all_features_pl = pl.from_pandas(all_features)In the example above,
tsfeaturesis being used to calculate features for each unique time series identified byunique_id. Thetsfeaturesfunction accepts a pandas DataFrame containing the time series data, and afreqparameter that specifies the seasonal frequency of the data (52 for weekly data). The resulting features are saved inall_features.Note that the
tsfeaturesfunction returns a pandas DataFrame, so we convert back to Polars usingpl.from_pandasif that's the data format required for subsequent steps in your analysis.Before integrating
tsfeaturesinto your workflow, make sure to read the documentation fortsfeaturesto understand the various features it can calculate and how they might be beneficial for your specific time series tasks. The library's functions could streamline your feature engineering process significantly.Also, check and confirm whether the
tsfeatureslibrary installation is compatible with your environment and that you have the appropriate versions of each package before incorporating it into your workflow. - User
use polars, no pandas
- Assistant
Understood. If you strictly want to use Polars and avoid converting to Pandas, you can still utilize Nixtla's
tsfeatureswith some adjustments. Thetsfeatureslibrary may require time series data in a long format, where each row represents an observation with a corresponding timestamp. Sincetsfeaturesis designed to work with Pandas, you would need to first convert the Polars DataFrame to Pandas, applytsfeatures, and then convert back to Polars if you want to continue with Polars.Since you prefer not to use Pandas at all, let's stick with Polars. Unfortunately, as of my last update, Polars might not have a direct equivalent to
tsfeatures. However, you can calculate the seasonal and trend components manually using Polars or use any other libraries that are compatible with Polars directly.To extract a simple seasonal component from your weekly time series data, you can use the
create_seasonalfunction fromstatsforecastwith Polars, like this:import polars as pl from statsforecast.utils import create_seasonal # Assuming y_cl4 is a polars DataFrame with 'ds' as datestamp or timestamp column season_length = 52 # Create seasonal dummy features seasonal_features = create_seasonal(y_cl4, season_length=season_length, col_name='ds') y_cl4_with_seasonals = y_cl4.hstack(seasonal_features) # Now y_cl4_with_seasonals will include seasonal dummy features that you can use in modelingThe
create_seasonalutility function should create multiple new columns in your DataFrame corresponding to the seasonal dummy variables for your weekly frequency.Please note that generating trend and seasonality features could sometimes be more complicated than just adding columns of seasonal dummies. If this is the case and the features you require are not readily available in Polars, you may need to build your own functions to calculate them or consider using libraries that can interfacing with Polars. Mistral, for instance, is a library developed by the same team as
statsforecastandtsfeaturesand has functions for time series feature extraction directly compatible with Numpy and, potentially, Polars.Keep in mind, since
tsfeaturesprimarily supports Pandas DataFrames, using this library would require conversion from Polars to Pandas and back to Polars, which you prefer to avoid. In case you need more complex seasonality or trend analysis directly in Polars, it might be best to explore or implement a custom solution. - User
this function is from tsfeatures nixtla? def get_seasonal_features(df): return tsfeatures(df, freq=52)
- Assistant
My previous message was not accurate regarding the
tsfeatureslibrary. I apologize for the confusion. Thetsfeatureslibrary is designed to work with Pandas DataFrames, and the functiontsfeaturesfrom that library expects a pandas DataFrame as input. Theget_seasonal_featuresfunction I outlined isn't directly fromtsfeaturesbut rather a wrapper that you would use to apply the tsfeatures library to your data.If you strictly want to avoid using Pandas and stick with Polars, you won't be able to directly apply
tsfeatures, which is made for Pandas. You would need to look for alternative feature engineering methods that are compatible with Polars.For feature engineering directly in Polars, you would need to manually extract the features you are interested in. Polars provides various window functions and groupby methods that allow you to perform sophisticated transformations and calculations that can serve as proxies for some of the time series features you might extract using
tsfeatures. However, Polars does not have inbuilt seasonal decomposition or similar time series analysis tools that are comparable totsfeaturesor thestatsmodelslibrary in Python.As of now, if you need to perform feature engineering for time series analysis within the Polars ecosystem without converting to Pandas, you will need to manually implement the extraction of such features or use a library that operates directly on Polars DataFrames if one exists.
You may also consider feature extraction functions that operate on Numpy arrays directly and apply these to columns of your Polars DataFrame, which can be converted to and from Numpy using the
.to_numpy()andpl.from_numpy()methods.For any custom trend or seasonality feature extraction in Polars, you would perform window operations, use
.groupby()with aggregation on the time column, or do calculations using Polars's datetime functionality. These techniques would involve bespoke code tailored to the particular patterns of seasonality and trend in your data. - User
I just wanted o make sure because one of the links I sent you it says Seasonal Features: Nixtla includes a tool for creating seasonal features, which are features that capture periodic patterns in the data. Seasonal features can be created using the create_seasonal function and i didnt see we using crweate_seasonal feature so that's why i'm asking
- Assistant
I apologize for the confusion earlier. It seems there was a misunderstanding regarding the
create_seasonalfunction. From what you described,create_seasonalappears to be a feature engineering tool provided by Nixtla that you can utilize to capture periodic patterns in your data.This
create_seasonalfunction should create seasonal dummy variables that can help your forecasting models account for regular periodic fluctuations, which is common in time series data such as weekly, monthly, or yearly seasonality.Let's implement
create_seasonalin your Polars workflow:import polars as pl from statsforecast.utils import create_seasonal # Assuming y_cl4 is your Polars DataFrame which includes your time series data # with 'ds' as the date column, 'y' as the target variable, and 'unique_id' as the time series identifier season_length = 52 # assuming weekly data with yearly seasonality # Creating seasonal features using create_seasonal seasonal_features = create_seasonal(y_cl4, season_length=season_length, date_col='ds', freq='W') # Join the seasonal features with your original DataFrame y_cl4_with_seasonal = y_cl4.join(seasonal_features, on='ds') # Now you can use y_cl4_with_seasonal for further modeling and forecastingThis code uses
create_seasonalto generate seasonal dummy features that correspond to the specificseason_length. These features should represent the seasonality in your data well if theseason_lengthcorrectly captures the dominant seasonal pattern. If your seasonality is not a perfect 52-week cycle or if you have multiple seasonal patterns, you may need to adjust theseason_lengthvalue or use a more sophisticated method of seasonality detection and encoding.Always consult the official documentation to understand the exact behavior and requirements of each function you use, especially when using third-party libraries like those developed by Nixtla, as there may be updates or changes that are not covered in this assistance.
- User
from statsforecast import StatsForecast from statsforecast.models import AutoARIMA, AutoETS, DynamicOptimizedTheta from statsforecast.utils import ConformalIntervals import numpy as np import polars as pl
Polars option to display all rows
pl.Config.set_tbl_rows(None)
Initialize the models
models = [ AutoARIMA(season_length=12), AutoETS(season_length=12), DynamicOptimizedTheta(season_length=12) ]
Initialize the StatsForecast model
sf = StatsForecast(models=models, freq='1w', n_jobs=-1)
Perform cross-validation with a step size of 1 to mimic an expanding window
crossvalidation_df = sf.cross_validation(df=y_cl4_filtered, h=2, step_size=1, n_windows=8, sort_df=True)
Calculate the ensemble mean
ensemble = crossvalidation_df[['AutoARIMA', 'AutoETS', 'DynamicOptimizedTheta']].mean(axis=1)
Create a Series for the ensemble mean
ensemble_series = pl.Series('Ensemble', ensemble)
Add the ensemble mean as a new column to the DataFrame
crossvalidation_df = crossvalidation_df.with_columns(ensemble_series)
def wmape(y_true, y_pred): return np.abs(y_true - y_pred).sum() / np.abs(y_true).sum()
Calculate the WMAPE for the ensemble model
wmape_value = wmape(crossvalidation_df['y'], crossvalidation_df['Ensemble']) print('Average WMAPE for Ensemble: ', round(wmape_value, 4))
Calculate the errors for the ensemble model
errors = crossvalidation_df['y'] - crossvalidation_df['Ensemble']
For an individual forecast
individual_accuracy = 1 - (abs(crossvalidation_df['y'] - crossvalidation_df['Ensemble']) / crossvalidation_df['y']) individual_bias = (crossvalidation_df['Ensemble'] / crossvalidation_df['y']) - 1
Add these calculations as new columns to DataFrame
crossvalidation_df = crossvalidation_df.with_columns([ individual_accuracy.alias("individual_accuracy"), individual_bias.alias("individual_bias") ])
Print the individual accuracy and bias for each week
for row in crossvalidation_df.to_dicts(): id = row['unique_id'] date = row['ds'] accuracy = row['individual_accuracy'] bias = row['individual_bias'] print(f"{id}, {date}, Individual Accuracy: {accuracy:.4f}, Individual Bias: {bias:.4f}")
For groups of forecasts
group_accuracy = 1 - (errors.abs().sum() / crossvalidation_df['y'].sum()) group_bias = (crossvalidation_df['Ensemble'].sum() / crossvalidation_df['y'].sum()) - 1
Print the average group accuracy and group bias over all folds for the ensemble model
print('Average Group Accuracy: ', round(group_accuracy, 4)) print('Average Group Bias: ', round(group_bias, 4))
Fit the models on the entire dataset
sf.fit(y_cl4_fit_1)
Instantiate the ConformalIntervals class
prediction_intervals = ConformalIntervals()
Generate 24 months forecasts
forecasts_df = sf.forecast(h=52*2, prediction_intervals=prediction_intervals, level=[95], id_col='unique_id', sort_df=True)
Apply the non-negative constraint to the forecasts of individual models
forecasts_df = forecasts_df.with_columns([ pl.when(pl.col('AutoARIMA') < 0).then(0).otherwise(pl.col('AutoARIMA')).alias('AutoARIMA'), pl.when(pl.col('AutoETS') < 0).then(0).otherwise(pl.col('AutoETS')).alias('AutoETS'), pl.when(pl.col('DynamicOptimizedTheta') < 0).then(0).otherwise(pl.col('DynamicOptimizedTheta')).alias('DynamicOptimizedTheta'), ])
Calculate the ensemble forecast
ensemble_forecast = forecasts_df[['AutoARIMA', 'AutoETS', 'DynamicOptimizedTheta']].mean(axis=1)
Calculate the lower and upper prediction intervals for the ensemble forecast
ensemble_lo_95 = forecasts_df.select( [ pl.when(pl.col('AutoARIMA-lo-95') < 0).then(0).otherwise(pl.col('AutoARIMA-lo-95')).alias('AutoARIMA-lo-95'), pl.when(pl.col('AutoETS-lo-95') < 0).then(0).otherwise(pl.col('AutoETS-lo-95')).alias('AutoETS-lo-95'), pl.when(pl.col('DynamicOptimizedTheta-lo-95') < 0).then(0).otherwise(pl.col('DynamicOptimizedTheta-lo-95')).alias('DynamicOptimizedTheta-lo-95'), ] ).mean(axis=1) ensemble_hi_95 = forecasts_df[['AutoARIMA-hi-95', 'AutoETS-hi-95', 'DynamicOptimizedTheta-hi-95']].mean(axis=1)
Create Series for the ensemble forecast and its prediction intervals
ensemble_forecast_series = pl.Series('EnsembleForecast', ensemble_forecast) ensemble_lo_95_series = pl.Series('Ensemble-lo-95', ensemble_lo_95) ensemble_hi_95_series = pl.Series('Ensemble-hi-95', ensemble_hi_95)
Add the ensemble forecast and its prediction intervals as new columns to the DataFrame
forecasts_df = forecasts_df.with_columns([ensemble_forecast_series, ensemble_lo_95_series, ensemble_hi_95_series])
Round the ensemble forecast and prediction intervals and convert to integer
forecasts_df = forecasts_df.with_columns([ pl.col("EnsembleForecast").round().cast(pl.Int32), pl.col("Ensemble-lo-95").round().cast(pl.Int32), pl.col("Ensemble-hi-95").round().cast(pl.Int32) ])
Split the unique_id concat into the original columns
forecasts_df = ( forecasts_df .with_columns(pl.col('unique_id').str.split('_', expand=True)) .rename({'unique_id_0': 'MaterialID', 'unique_id_1': 'SalesOrg', 'unique_id_2': 'DistrChan', 'unique_id_3': 'CL4'}) .drop('unique_id') )
Rename ‘ds’ to ‘WeekDate’
forecasts_df = forecasts_df.rename({'ds': 'WeekDate'})
Reorder the columns
forecasts_df = forecasts_df.select([ "MaterialID", "SalesOrg", "DistrChan", "CL4", "WeekDate", "EnsembleForecast", "Ensemble-lo-95", "Ensemble-hi-95", "AutoARIMA", "AutoARIMA-lo-95", "AutoARIMA-hi-95", "AutoETS", "AutoETS-lo-95", "AutoETS-hi-95", "DynamicOptimizedTheta", "DynamicOptimizedTheta-lo-95", "DynamicOptimizedTheta-hi-95" ])
Create an empty list
forecasts_list = []
Append each row to the list
for row in forecasts_df.to_dicts(): forecasts_list.append(row)
Print the list
for forecast in forecasts_list: print(forecast) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[19], line 116 107 forecasts_df = forecasts_df.with_columns([ 108 pl.col("EnsembleForecast").round().cast(pl.Int32), 109 pl.col("Ensemble-lo-95").round().cast(pl.Int32), 110 pl.col("Ensemble-hi-95").round().cast(pl.Int32) 111 ]) 113 # Split the unique_id concat into the original columns 114 forecasts_df = ( 115 forecasts_df --> 116 .with_columns(pl.col('unique_id').str.split('_', expand=True)) 117 .rename({'unique_id_0': 'MaterialID', 'unique_id_1': 'SalesOrg', 'unique_id_2': 'DistrChan', 'unique_id_3': 'CL4'}) 118 .drop('unique_id') 119 ) 121 # Rename ‘ds’ to ‘WeekDate’ 122 forecasts_df = forecasts_df.rename({'ds': 'WeekDate'})
TypeError: ExprStringNameSpace.split() got an unexpected keyword argument 'expand'
- Assistant
The error message you're encountering indicates that there is a mismatch in the expected parameters for the
splitfunction in the Polars library. In Polars, thesplitmethod does not have anexpandargument like in pandas. Instead, to split a string into multiple columns in Polars, you need to use thestr.split_bymethod and then explode the resulting list into separate columns.Here's how you correct your code for splitting the
unique_idcolumn into its original components in Polars:# Split the 'unique_id' column into the original columns using `str.split_by` and explode into columns split_columns = forecasts_df.select(pl.col('unique_id').str.split_by("_", inclusive=True).alias("split_id")) split_columns = split_columns.with_column(pl.col("split_id").arr.eval(pl.element().alias("MaterialID"), pl.element(1).alias("SalesOrg"), pl.element(2).alias("DistrChan"), pl.element(3).alias("CL4"))) forecasts_df = forecasts_df.hstack(split_columns.drop("split_id")) # Drop the original 'unique_id' column forecasts_df = forecasts_df.drop('unique_id') # Rename ‘ds’ to ‘WeekDate’ forecasts_df = forecasts_df.rename({'ds': 'WeekDate'}) # Reorder the columns forecasts_df = forecasts_df.select([ "MaterialID", "SalesOrg", "DistrChan", "CL4", "WeekDate", "EnsembleForecast", "Ensemble-lo-95", "Ensemble-hi-95", "AutoARIMA", "AutoARIMA-lo-95", "AutoARIMA-hi-95", "AutoETS", "AutoETS-lo-95", "AutoETS-hi-95", "DynamicOptimizedTheta", "DynamicOptimizedTheta-lo-95", "DynamicOptimizedTheta-hi-95" ])In the code above:
str.split_by()is used to split theunique_idstring into a list of components.- We use
arr.eval()to map each element of the list to new columns withpl.element().alias()for each part of the split. Sincearr.eval()is a relatively recent addition to Polars, make sure you're using an updated version of Polars that includes this method. - We then use
hstackto add these split columns back to the original DataFrame. - The
unique_idcolumn is then dropped as it's no longer needed. - The rest of the code should work as expected, and you can continue with your modeling and analytics workflow.
Make sure your Polars library is up to date to access these features. If you encounter any issues, consult the Polars documentation for the latest information on handling string operations and column manipulations.