WeatherBench Probability: Medium-range weather forecasts with probabilistic machine learning methods.

<p>Because the atmosphere is inherently chaotic, probabilistic weather forecasts are crucial to provide reliable information. In this work, we present an extension to the WeatherBench, a benchmark dataset for medium-range, data-driven weather prediction, which was originally designed for deterministic forecasts. We add a set of commonly used probabilistic verification metrics: the spread-skill ratio, the continuous ranked probability score (CRPS) and rank histograms. Further, we compute baseline scores from the operational IFS ensemble forecast. </p><p>Then, we compare three different methods of creating probabilistic neural network forecasts: first, using Monte-Carlo dropout during inference with a range of dropout rates; second, parametric forecasts, which optimize for the CRPS; and third, categorical forecasts, in which the probability of occurrence for specific bins is predicted. We show that plain Monto-Carlo dropout does not provide enough spread. The parametric and categorical networks, on the other hand, provide reliable forecasts, with the categorical method being more versatile.</p>

Progress in ensemble forecasting and verification methodologies at ECMWF

<p>This talk focusses on progress in ensemble forecasting methodology (Part I) and ensemble verification methodology (Part II).</p><p>Operational ECMWF ensemble forecasts are global predictions from days to months ahead. At all forecast ranges, model uncertainties are represented stochastically with the Stochastically Perturbed Parametrization Tendency scheme (SPPT). Recently, considerable progress has been made in developing the Stochastically Perturbed Parametrization scheme (SPP). The SPP scheme offers improved physical consistency by naturally preserving the local conservation properties for energy and moisture of the unperturbed version of the corresponding parametrization. In contrast, the SPPT scheme lacks such local conservation properties, mainly because the scheme does not perturb fluxes at the surface and at the top of the atmosphere consistently with the tendency perturbations in the column.</p><p>NWP research and development relies on scoring rules to judge whether or not a change to the forecast systems results in better ensemble forecasts. A new tool will be presented that can improve the understanding of score differences between sets of forecasts for a widely used proper score, the Continuous Ranked Probability Score (CRPS). An analytical expression has been derived for the CRPS when a homogeneous Gaussian (hoG) forecast-observation distribution is considered. This leads to an approximation of the CRPS when actual verification data are considered, which deviate from a homogeneous Gaussian distribution. The hoG approximation of the CRPS permits a useful decomposition of score differences. The methodology will be illustrated with verification data for medium-range weather forecasts.</p>

Evaluating epidemic forecasts in an interval format

For practical reasons, many forecasts of case, hospitalization, and death counts in the context of the current Coronavirus Disease 2019 (COVID-19) pandemic are issued in the form of central predictive intervals at various levels. This is also the case for the forecasts collected in the COVID-19 Forecast Hub (https://covid19forecasthub.org/). Forecast evaluation metrics like the logarithmic score, which has been applied in several infectious disease forecasting challenges, are then not available as they require full predictive distributions. This article provides an overview of how established methods for the evaluation of quantile and interval forecasts can be applied to epidemic forecasts in this format. Specifically, we discuss the computation and interpretation of the weighted interval score, which is a proper score that approximates the continuous ranked probability score. It can be interpreted as a generalization of the absolute error to probabilistic forecasts and allows for a decomposition into a measure of sharpness and penalties for over- and underprediction.

Spatio-temporal prediction of missing temperature with stochastic Poisson equations

This paper presents our winning entry for the EVA 2019 data competition, the aim of which is to predict Red Sea surface temperature extremes over space and time. To achieve this, we used a stochastic partial differential equation (Poisson equation) based method, improved through a regularization to penalize large magnitudes of solutions. This approach is shown to be successful according to the competition’s evaluation criterion, i.e. a threshold-weighted continuous ranked probability score. Our stochastic Poisson equation and its boundary conditions resolve the data’s non-stationarity naturally and effectively. Meanwhile, our numerical method is computationally efficient at dealing with the data’s high dimensionality, without any parameter estimation. It demonstrates the usefulness of stochastic differential equations on spatio-temporal predictions, including the extremes of the process.

Verification of post-processed seasonal predictions

<p>Traditionally, verification of (ensemble) model predictions is done by comparing them to deterministic observations, e.g. with scores like the Continuous Ranked Probability Score (CRPS). While these approaches allow uncertain predictions basing on ensemble forecasts, it is open how to verify them against observations with non-parametric uncertainties.</p><p>This contribution focuses on statistically post-processed seasonal predictions of the Winter North Atlantic Oscillation (WNAO). The post-processing procedure creates in a first step for a dynamical ensemble prediction and for a statistical prediction basing on predictors two separate probability density functions (pdf). Afterwards these two distributions are combined to create a new statistical-dynamical prediction, which has been proven to be advantageous compared to the purely dynamical prediction. It will be demonstrated how this combination and with it the improvement of the prediction can be achieved before the focus will be set on the evaluation of those predictions at the hand of uncertain observations. Two new scores basing on the Earth Mover's Distance (EMD) and the Integrated Quadratic Distance (IQD) will be introduced and compared before it is shown how they can be used to effectively evaluate probabilistic predictions with uncertain observations. </p><p>Furthermore, a common approach (e.g. for correlation measures) is to compare predictions with observations over a longer time period. In this contribution a paradigm shift away from this approach towards comparing predictions for each single time step (like years) will be presented. This view give new insights into the performance of the predictions and allows to come to new understandings of the reasons for advantages or disadvantages of specific predictions. </p>

Calibration of direct normal irradiance (DNI) forecasts with quantile regression

<p>Direct normal irradiance (DNI) forecasts from two ensemble models, the global ECMWF-ENS and the limited area multimodel gSREPS, have been calibrated using the quantile regression method, taking DNI as the only input parameter to better understand the inner workings of the method. Forecasts for the southern part of Spain, with lead times up to 72 hours for ECMWF-ENS and 24 hours for gSREPS over a two-year period (from June 2017 to May 2019), have been used.</p><p>This study has focused on two particular aspects of the postprocess:</p><ul><li>The effect of quantile regression on the spread of the models. The results show that the spread of ECMWF-ENS greatly increases after the postprocess, which has a positive effect on the accuracy of the model, with an improvement of 20% in the continuous ranked probability score (CRPS) after the calibration. However, this increase is uniform over the whole period, affecting equally to situations with low or high spread, hence the postprocessed forecasts are not able to detect changes in predictability. On the other hand raw gSREPS forecasts behave better during episodes of both low or high predictability. The postprocess does not significantly change the spread and accuracy of gSREPS.</li> <li>The influence of the training sample. It has been found that DNI is a variable which can experience periods of low variability, particularly in regions like southern Spain, where long spells of sunny days are common. This has a sizeable impact on the performance of the quantile regression on certain days. Two study cases will be shown to illustrate this problem. Two possible solutions are proposed: use longer training periods (not always possible) or place restrictions on the value of the regression coefficients.</li> </ul>

Estimation Methods for Nonhomogeneous Regression Models: Minimum Continuous Ranked Probability Score versus Maximum Likelihood

Nonhomogeneous regression models are widely used to statistically postprocess numerical ensemble weather prediction models. Such regression models are capable of forecasting full probability distributions and correcting for ensemble errors in the mean and variance. To estimate the corresponding regression coefficients, minimization of the continuous ranked probability score (CRPS) has widely been used in meteorological postprocessing studies and has often been found to yield more calibrated forecasts compared to maximum likelihood estimation. From a theoretical perspective, both estimators are consistent and should lead to similar results, provided the correct distribution assumption about empirical data. Differences between the estimated values indicate a wrong specification of the regression model. This study compares the two estimators for probabilistic temperature forecasting with nonhomogeneous regression, where results show discrepancies for the classical Gaussian assumption. The heavy-tailed logistic and Student’s t distributions can improve forecast performance in terms of sharpness and calibration, and lead to only minor differences between the estimators employed. Finally, a simulation study confirms the importance of appropriate distribution assumptions and shows that for a correctly specified model the maximum likelihood estimator is slightly more efficient than the CRPS estimator.