Probabilistic forecasts, calibration and sharpness

• Published in: Journal of the Royal Statistical Society. Series B, Statistical methodology Vol. 69; no. 2; pp. 243 - 268
• Main Authors: Gneiting, Tilmann, Balabdaoui, Fadoua, Raftery, Adrian E
• Format: Journal Article
• Language: English
• Published: Oxford, UK Oxford, UK : Blackwell Publishing Ltd 01.04.2007; Blackwell Publishing Ltd; Blackwell Publishers; Blackwell; Royal Statistical Society; Oxford University Press
• Series: Journal of the Royal Statistical Society Series B
• Subjects: Analytical forecasting; Autocorrelation; Calibration; Cross-validation; cumulative distribution; Density forecast; diagnostic techniques; Distribution; Distribution theory; Ensemble prediction system; Evaluation; Ex post evaluation; Exact sciences and technology; Forecast verification; Forecasting; Forecasts; Histograms; Inference from stochastic processes; time series analysis; Mathematical methods; Mathematics; Model diagnostics; Model testing; Nonparametric inference; Posterior predictive assessment; prediction; Predictive distribution; Prequential principle; Probabilities; Probability; Probability and statistics; Probability forecasts; Probability integral transform; Probability theory and stochastic processes; Proper scoring rule; Sciences and techniques of general use; Statistical analysis; Statistical methods; Statistics; Stochastic processes; Studies; Time series; time series analysis; Time series forecasting; United States; Weather forecasting; Wind power; wind speed; Wind velocity; United States; Performance evaluation; Rank statistic; Density estimation; Histogram; Predictive distribution; Non parametric estimation; Cross validation; Forecast model; Hypothesis test; Probabilistic assessment; Posterior predictive assessment; Model matching; Cross-validation; Forecast verification; Ex post evaluation; Integral transformation; Posterior distribution; Density forecast; Model selection; Time series; Ensemble prediction system; Prequential principle; Proper scoring rule; Statistical method; Model diagnostics; Probability integral transform
• ISSN: 1369-7412; 1467-9868
• DOI: 10.1111/j.1467-9868.2007.00587.x

Probabilistic forecasts of continuous variables take the form of predictive densities or predictive cumulative distribution functions. We propose a diagnostic approach to the evaluation of predictive performance that is based on the paradigm of maximizing the sharpness of the predictive distributions subject to calibration. Calibration refers to the statistical consistency between the distributional forecasts and the observations and is a joint property of the predictions and the events that materialize. Sharpness refers to the concentration of the predictive distributions and is a property of the forecasts only. A simple theoretical framework allows us to distinguish between probabilistic calibration, exceedance calibration and marginal calibration. We propose and study tools for checking calibration and sharpness, among them the probability integral transform histogram, marginal calibration plots, the sharpness diagram and proper scoring rules. The diagnostic approach is illustrated by an assessment and ranking of probabilistic forecasts of wind speed at the Stateline wind energy centre in the US Pacific Northwest. In combination with cross-validation or in the time series context, our proposal provides very general, nonparametric alternatives to the use of information criteria for model diagnostics and model selection.