How to evaluate an early warning system?

This paper provides a statistical-toolkit to evaluate an early-warning-system (EWS). A EWS is a unique approach compared to traditional forecasting methods, since traditional methods usually predict levels or growth rates of one or more variables of interest. In the case of the EWS, a researcher is more interested in predicting specific states, which are structurally different from one another. For instance, in the present case, an ordered probit approach is utilized to predict probabilities of inflationary pressure, deflationary pressure and stable prices, and these three inflation states are structurally different from each other.

Given the nature of the EWS output, which are three probabilities representing three different states, traditional forecast evaluation measures such as root mean squared error (RMSE) cannot be utilized as forecast evaluation measures.

We utilize several different measures including, Kuiper Score, Bayesian Error Rate and quadratic probability score (QPS), to evaluate the ordered probit model’s performance. In addition, we evaluate the relative accuracy of the EWS as well. That is, an ordered probit model produces signals for three different states, simultaneously, and in reality, only one state can occur. Therefore, it is vital to determine whether the system has a higher accuracy to predict state-1 (inflationary pressure, for instance) compared to state-2 (deflationary pressure) and state-3(stable prices) or vice versa.

Another contribution of our work is to determine an optimal forecast horizon. Typically, the longer the forecast horizon, the lesser the forecast accuracy. On the other hand, decision makers would prefer to receive a warning sooner rather than later so they can prepare an appropriate policy stance. In an ideal world, a forecast horizon should not be either too-long (less accuracy, more false signals) or too-short (not enough time for policy reaction). We determine the optimal forecast horizon which gives enough time to decision makers with reliable early warning signals.

We utilize the 1960-2014 period to build an ordered probit model for the U.S. economy and generate out-of-sample probabilities of three prices scenarios for the 1980-2014 period. Six different forecast horizons are considered to determine the optimal forecast horizon, which are 3-month, 6-month, 9-month, 12-month, 18-month and 24-month out forecasts of inflationary pressure, deflationary pressure and stable prices.

Key Words: EWS; Evaluation; Optimal Forecast Horizon; Ordered Probit; Probability.