forecasting - What type of analysis is appropriate for assessing the performance time-series forecasts?

When using time-series analysis to forecast some type of value, what types of error analysis are worth considering when trying to determine which models are appropriate.

One of the big issues that may arise is that successive residuals between the 'forecast' and the 'realized' value of the variable may not be properly independent of one another as large amounts of data will be reused from one data point to its successive one.

To give an example, if you fit a GARCH model to forecast volatility for a given time horizon, the fit will use a certain amount of data, and the forecast is produced and then compared to whatever the realized volatility was observed for the given period of time, and it is then possible to find some kind of 'loss' value for that forecast.

Once everything moves forward a time period, assuming we refit (but even if we reuse the data parameters), there will be a very large overlap in all the data for this second forecast and realized volatility.

Since it is common to desire a model that minimises these 'losses' in some sense, how do you deal with the residuals produced in this way? Is there a way to remove the dependency? Are successive residuals dependent, and how could this dependency be measured? What tools exist to analyse these residuals?

Answer

I think you're looking for some way to test for autocorrelation in your residuals. If your model is good -- let's say you have an ARMA(1, 1) model for your forecast -- then the residuals from this model will be white noise. Which is to say that the difference between your forecast and the realization can not be predicted any better. The residual is some zero mean normally-distributed error.

Let's pick an extreme example. If your residual (the diff between forecast and realized) were always 1, then the residuals would be autocorrelated. Clearly if your model is always off by 1, then you can do better. So if the residuals in your model are autocorrelated, then you can do better.

The standard test for this these days in Ljung-Box, but in the past Box-Pierce and Durbin-Watson were also used.