Please use this identifier to cite or link to this item: http://hdl.handle.net/10419/153942
Authors: 
Sarlin, Peter
Year of Publication: 
2013
Series/Report no.: 
ECB Working Paper 1509
Abstract: 
This paper introduces a new loss function and Usefulness measure for evaluating early warning systems (EWSs) that incorporate policymakers' preferences between issuing false alarms and missing crises, as well as individual observations. The novelty derives from three enhancements: i) accounting for unconditional probabilities of the classes, ii) computing the proportion of available Usefulness that the model captures, and iii) weighting observations by their importance for the policymaker. The proposed measures are model free such that they can be used to assess signals issued by any type of EWS, such as logit and probit analysis and the signaling approach, and flexible for any type of crisis EWSs, such as banking, debt and currency crises. Applications to two renowned EWSs, and comparisons to two commonly used evaluation measures, illustrate three key implications of the new measures: i) further highlights the importance of an objective criterion for choosing a final specification and threshold value, and for models to be useful ii) the need to be more concerned about the rare class and iii) the importance of correctly classifying observations of the most relevant entities. Beyond financial stability surveillance, this paper also opens the door for cost-sensitive evaluations of predictive models in other tasks.
Subjects: 
Early warning systems
misclassification costs
policymakers' preferences
JEL: 
E44
E58
F01
F37
G01
Document Type: 
Working Paper

Files in This Item:
File
Size
676.15 kB





Items in EconStor are protected by copyright, with all rights reserved, unless otherwise indicated.