Assess Predictive Performance

If you plan to use a fitted model for forecasting, a good practice is to assess the predictive ability of the model. Models that fit well in-sample are not guaranteed to forecast well. For example, overfitting can lead to good in-sample fit, but poor predictive performance.

When checking predictive performance, it is important to not use your data twice. That is, the data you use to fit your model should be different than the data you use to assess forecasts. You can use cross validation to evaluate out-of-sample forecasting ability:

Prediction mean square error (PMSE) measures the discrepancy between model forecasts and observed data. Suppose you have a time series of length N, and you set aside M validation points, denoted $$y_1^v,y_2^v,\dots ,y_M^v.$$. After fitting your model to the first N – M data points (the training set), generate forecasts $${\widehat{y}}_1^v,{\widehat{y}}_2^v,\dots ,{\widehat{y}}_M^v.$$

The model PMSE is calculated as

You can calculate PMSE for various choices of M to verify the robustness of your results.

Related Examples

• Check Fit of Multiplicative ARIMA Model

More About

• Goodness of Fit
• Residual Diagnostics
• MMSE Forecasting of Conditional Mean Models
• MMSE Forecasting of Conditional Variance Models