Skip to content
TheChatArchiveThe Chat Archive
Mistralmistral-medium-2505· Mistral AIHugging Face LMArena Human Preference 140K

How to measure accuracy of a predicting model for sales time series. What is considered…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.5-flash-lite-preview-06-17-thinking’s answer.
  1. User

    How to measure accuracy of a predicting model for sales time series. What is considered a good accuracy?

  2. Mistral Assistant

    Measuring the accuracy of a sales time series forecasting model involves evaluating how well the model's predictions align with actual observed values. Here are the key steps and metrics to consider:

    1. Common Accuracy Metrics for Time Series Forecasting

    The choice of metric depends on the business context and the nature of the data. Commonly used metrics include:

    A. Scale-Dependent Metrics (Useful for comparing models on the same dataset)

    • Mean Absolute Error (MAE): [ MAE = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i| ]

      • Measures the average absolute difference between actual and predicted values.
      • Easy to interpret (same units as the data).
    • Mean Squared Error (MSE): [ MSE = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 ]

      • Penalizes larger errors more heavily (due to squaring).
    • Root Mean Squared Error (RMSE): [ RMSE = \sqrt{MSE} ]

      • More interpretable than MSE (same units as the data).

    B. Percentage-Based Metrics (Useful for comparing across different datasets)

    • Mean Absolute Percentage Error (MAPE): [ MAPE = \frac{100%}{n} \sum_{i=1}^{n} \left| \frac{y_i - \hat{y}_i}{y_i} \right| ]

      • Expressed as a percentage, making it easy to understand.
      • Problem: Can be misleading if actual values are close to zero.
    • Symmetric Mean Absolute Percentage Error (sMAPE): [ sMAPE = \frac{100%}{n} \sum_{i=1}^{n} \frac{|y_i - \hat{y}_i|}{(|y_i| + |\hat{y}_i|)/2} ]

      • Handles zero values better than MAPE.
    • Mean Absolute Scaled Error (MASE): [ MASE = \frac{MAE}{\text{MAE of a naive forecast (e.g., random walk)}} ]

      • Compares the model against a simple benchmark (values < 1 indicate better performance).

    C. Relative Metrics (For comparing models)

    • R² (R-squared): [ R^2 = 1 - \frac{\sum (y_i - \hat{y}_i)^2}{\sum (y_i - \bar{y})^2} ]
      • Measures how well the model explains variance (0 to 1, higher is better).
      • Problem: Can be misleading for time series data.

    2. What is Considered a "Good" Accuracy?

    The definition of "good" accuracy depends on:

    • Industry standards (e.g., retail vs. manufacturing).
    • Business impact (e.g., small errors may be acceptable if they don’t affect decisions).
    • Data volatility (highly volatile sales are harder to predict).

    General Benchmarks:

    MetricGoodExcellent
    MAELow relative to average salesVery low (e.g., <5% of mean sales)
    RMSELower than MAE (since it penalizes large errors)Close to zero
    MAPE<10%<5%
    sMAPE<15%<10%
    MASE<0.8<0.5
    R²>0.7>0.9

    3. Additional Considerations

    • Cross-validation: Use time-series cross-validation (e.g., rolling window) to avoid lookahead bias.
    • Benchmarking: Compare against a naive model (e.g., last observed value or moving average).
    • Business Context: A model with 10% MAPE may be excellent for some industries but poor for others.

    Conclusion

    A good sales forecasting model should have:

    • Low MAE/RMSE relative to sales volume.
    • MAPE/sMAPE below 10% (ideally <5%).
    • MASE < 1 (better than a naive forecast).
    • R² > 0.7 (but be cautious with time series data).

    Would you like help selecting the best metric for your specific use case?

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.