A diagram for evaluating multiple aspects of model performance in simulating vector fields

Abstract. Vector quantities, e.g., vector winds, play an extremely important role in climate systems. The energy and water exchanges between different regions are strongly dominated by wind, which in turn shapes the regional climate. Thus, how well climate models can simulate vector fields directly affects model performance in reproducing the nature of a regional climate. This paper devises a new diagram, termed the vector field evaluation (VFE) diagram, which is a generalized Taylor diagram and able to provide a concise evaluation of model performance in simulating vector fields. The diagram can measure how well two vector fields match each other in terms of three statistical variables, i.e., the vector similarity coefficient, root mean square length (RMSL), and root mean square vector difference (RMSVD). Similar to the Taylor diagram, the VFE diagram is especially useful for evaluating climate models. The pattern similarity of two vector fields is measured by a vector similarity coefficient (VSC) that is defined by the arithmetic mean of the inner product of normalized vector pairs. Examples are provided, showing that VSC can identify how close one vector field resembles another. Note that VSC can only describe the pattern similarity, and it does not reflect the systematic difference in the mean vector length between two vector fields. To measure the vector length, RMSL is included in the diagram. The third variable, RMSVD, is used to identify the magnitude of the overall difference between two vector fields. Examples show that the VFE diagram can clearly illustrate the extent to which the overall RMSVD is attributed to the systematic difference in RMSL and how much is due to the poor pattern similarity.

Download Full-text

A diagram for evaluating multiple aspects of model performance in simulating vector fields

10.5194/gmd-2016-172 ◽

2016 ◽

Cited By ~ 1

Author(s):

Zhongfeng Xu ◽

Zhaolu Hou ◽

Ying Han ◽

Weidong Guo

Keyword(s):

Vector Field ◽

Vector Fields ◽

Climate Models ◽

Regional Climate ◽

Model Performance ◽

Systematic Difference ◽

Similarity Coefficient ◽

Vector Length ◽

Taylor Diagram ◽

Pattern Similarity

Abstract. Vector quantities, e.g. vector winds, play an extremely important role in climate system. Energy and water exchanges between different regions are strongly dominated by wind, which in turn shapes regional climate. Thus, how well climate models can simulate vector fields directly affect model performance in reproducing the nature of regional climate. The paper devises a new diagram, termed vector field evaluation (VFE) diagram, which is very similar to Taylor diagram but to provide a concise evaluation of model performance in simulating vector fields. The diagram can measure how well of two vector fields match each other in terms of three statistical variables, i.e. vector similarity coefficient, root-mean-square (RMS) length (RMSL), and RMS vector difference (RMSVD). As the Taylor diagram, the VFE diagram is especially useful in evaluating climate models. The pattern similarity of two vector fields is measured by a vector similarity coefficient (VSC) that is defined by the arithmetic mean of inner product of normalized vector pairs. Examples are given showing that VSC can well identify how close one vector field resemble another. Note that VSC can only describe the pattern similarity and do not reflect the systematic difference in the mean vector length between two vector fields. To measure the vector length, RMSL is included in the diagram. The third variable, RMSVD, is used to identify the magnitude of overall difference between two vector fields. Examples show that the new diagram can clearly illustrate how much the overall RMSVD is attributed to the systematic difference in RMSL and how much is due to the poor pattern similarity.

Download Full-text

Are we using the right fuel to drive hydrological models? A climate impact study in the Upper Blue Nile

Hydrology and Earth System Sciences ◽

10.5194/hess-22-2163-2018 ◽

2018 ◽

Vol 22 (4) ◽

pp. 2163-2185 ◽

Cited By ~ 9

Author(s):

Stefan Liersch ◽

Julia Tecklenburg ◽

Henning Rust ◽

Andreas Dobler ◽

Madlen Fischer ◽

...

Keyword(s):

Climate Change ◽

Bias Correction ◽

Climate Models ◽

Regional Climate ◽

Model Performance ◽

Climate Impact ◽

Hydrological Models ◽

Blue Nile ◽

Projected Change ◽

Model Ensembles

Abstract. Climate simulations are the fuel to drive hydrological models that are used to assess the impacts of climate change and variability on hydrological parameters, such as river discharges, soil moisture, and evapotranspiration. Unlike with cars, where we know which fuel the engine requires, we never know in advance what unexpected side effects might be caused by the fuel we feed our models with. Sometimes we increase the fuel's octane number (bias correction) to achieve better performance and find out that the model behaves differently but not always as was expected or desired. This study investigates the impacts of projected climate change on the hydrology of the Upper Blue Nile catchment using two model ensembles consisting of five global CMIP5 Earth system models and 10 regional climate models (CORDEX Africa). WATCH forcing data were used to calibrate an eco-hydrological model and to bias-correct both model ensembles using slightly differing approaches. On the one hand it was found that the bias correction methods considerably improved the performance of average rainfall characteristics in the reference period (1970–1999) in most of the cases. This also holds true for non-extreme discharge conditions between Q20 and Q80. On the other hand, bias-corrected simulations tend to overemphasize magnitudes of projected change signals and extremes. A general weakness of both uncorrected and bias-corrected simulations is the rather poor representation of high and low flows and their extremes, which were often deteriorated by bias correction. This inaccuracy is a crucial deficiency for regional impact studies dealing with water management issues and it is therefore important to analyse model performance and characteristics and the effect of bias correction, and eventually to exclude some climate models from the ensemble. However, the multi-model means of all ensembles project increasing average annual discharges in the Upper Blue Nile catchment and a shift in seasonal patterns, with decreasing discharges in June and July and increasing discharges from August to November.

Download Full-text

Evaluating reanalysis-driven CORDEX regional climate models over Australia: model performance and errors

Climate Dynamics ◽

10.1007/s00382-019-04672-w ◽

2019 ◽

Vol 53 (5-6) ◽

pp. 2985-3005 ◽

Cited By ~ 20

Author(s):

Giovanni Di Virgilio ◽

Jason P. Evans ◽

Alejandro Di Luca ◽

Roman Olson ◽

Daniel Argüeso ◽

...

Keyword(s):

Climate Models ◽

Regional Climate ◽

Model Performance ◽

Regional Climate Models

Download Full-text

On the potential use of glacier and permafrost observations for verification of climate models

Annals of Glaciology ◽

10.1017/s026030550001435x ◽

1997 ◽

Vol 25 ◽

pp. 400-406 ◽

Cited By ~ 2

Author(s):

Martin Beniston ◽

Wilfried Haeberli ◽

Martin Hoelzle ◽

Alan Taylor

Keyword(s):

General Circulation ◽

Climate Models ◽

Climate Model ◽

Global Climate ◽

Regional Climate ◽

Model Performance ◽

Regional Climate Models ◽

General Circulation Models ◽

Model Verification ◽

Dominant Feature

While the capability of global and regional climate models in reproducing current climate has significantly improved over the past few years, the confidence in model results for remote regions, or those where complex orography is a dominant feature, is still relatively low. This is, in part, linked to the lack of observational data for model verification and intercomparison purposes.Glacier and permafrost observations are directly related to past and present energy flux patterns at the Earth-atmosphere interface and could be used as a proxy for air temperature and precipitation, particularly of value in remote mountain regions and boreal and Arctic zones where instrumental climate records are sparse or non-existent. It is particularly important to verify climate-model performance in these regions, as this is where most general circulation models (GCMs) predict the greatest changes in air temperatures in a warmer global climate.Existing datasets from glacier and permafrost monitoring sites in remote and high altitudes are described in this paper; the data could be used in model-verification studies, as a means to improving model performance in these regions.

Download Full-text

Are we using the right fuel to drive hydrological models? A climate impact study in the Upper Blue Nile

10.5194/hess-2016-422 ◽

2016 ◽

Cited By ~ 4

Author(s):

Stefan Liersch ◽

Julia Tecklenburg ◽

Henning Rust ◽

Andreas Dobler ◽

Madlen Fischer ◽

...

Keyword(s):

Climate Change ◽

Bias Correction ◽

Climate Models ◽

Regional Climate ◽

Model Performance ◽

Climate Impact ◽

Hydrological Models ◽

Blue Nile ◽

Projected Change ◽

Model Ensembles

Abstract. Climate simulations are the fuel to drive hydrological models that are used to assess the impacts of climate change and variability on hydrological parameters, such as river discharges, soil moisture, and evapotranspiration. Unlike with cars, where we know which fuel the engine requires, we never know in advance what unexpected side-effects might be caused by the fuel we feed our models with. Sometimes we increase the fuel's octane number (bias-correction) to achieve better performance and find out that the model behaves differently but not always as was expected or desired. This study investigates the impacts of projected climate change on the hydrology of the Upper Blue Nile catchment using two model ensembles consisting of five global CMIP5 Earth System Models and ten Regional Climate Models (CORDEX Africa). WATCH forcing data were used to calibrate an eco-hydrological model and to bias-correct both model ensembles using slightly differing approaches. On the one hand it was found that the bias-correction methods considerably improved the performance of average rainfall characteristics in the reference period (1970–1999) in most of the cases. This also holds true for non-extreme discharge conditions between Q20 and Q80. On the other hand, bias-corrected simulations tend to overemphasise magnitudes of projected change signals and extremes. A general weakness of both uncorrected and bias-corrected simulations is the rather poor representation of high and low flows and their extremes, which were often deteriorated by bias-correction. This inaccuracy is a crucial deficiency for regional impact studies dealing with water management issues and it is therefore important to analyse model performance and characteristics, the effect of bias-correction, and eventually to exclude some climate models from the ensemble. However, the multi-model means of all ensembles project increasing average annual discharges in the Upper Blue Nile catchment and a shift in seasonal patterns, with decreasing discharges in June and July and increasing discharges from August to November.

Download Full-text

Multivariable Integrated Evaluation of Model Performance with the Vector Field Evaluation Diagram

10.5194/gmd-2017-95 ◽

2017 ◽

Cited By ~ 1

Author(s):

Zhongfeng Xu ◽

Ying Han ◽

Congbin Fu

Keyword(s):

Vector Field ◽

Root Mean Square ◽

Climate Model ◽

Model Performance ◽

Field Evaluation ◽

Scalar Fields ◽

General Idea ◽

Evaluation Methodology ◽

Mean Square ◽

Pattern Similarity

Abstract. This paper develops a multivariable integrated evaluation (MVIE) method to measure the overall performance of climate model in simulating multiple fields. The general idea of MVIE is to group various scalar fields into a vector field and compare the constructed vector field against the observed one using the vector field evaluation (VFE) diagram. The VFE diagram was devised based on the cosine relationship between three statistical quantities: root mean square length (RMSL) of a vector field, vector field similarity coefficient, and root mean square vector deviation (RMSVD). The three statistical quantities can reasonably represent the corresponding statistics between two multidimensional vector fields. Therefore, one can summarize the three statistics of multiple scalar fields using VFE diagram and facilitate the intercomparison of model performances. The VFE diagram can illustrate how much the overall root mean square deviation of various fields is attributable to the differences in the root mean square value and how much is due to the poor pattern similarity. The MVIE method can be flexibly applied to full fields (including both the mean and anomaly) or anomaly fields depending on the application. We also propose a multivariable integrated evaluation index (MIEI) which takes the amplitude and pattern similarity of multiple scalar fields into account. The MIEI is expected to provide a more accurate evaluation of model performance in simulating multiple fields. The MIEI, VFE diagram, and commonly used statistical metrics for individual variables constitute a hierarchical evaluation methodology, which can provide a more comprehensive evaluation on model performance.

Download Full-text

Sensitivity studies with the Regional Climate Model COSMO-CLM 5.0 over the CORDEX Central Asia Domain

10.5194/gmd-2019-22 ◽

2019 ◽

Author(s):

Emmanuele Russo ◽

Ingo Kirchner ◽

Stephan Pfahl ◽

Martijn Schaap ◽

Ulrich Cubasch

Keyword(s):

Regional Climate Model ◽

Central Asia ◽

Climate Models ◽

Climate Model ◽

Reference Model ◽

Developing Economies ◽

Regional Climate ◽

Model Performance ◽

Mitigation Strategies ◽

Climate Projections

Abstract. Due to its extension, geography and the presence of several under-developed or developing economies, the Central Asia domain of the Coordinated Regional climate Downscaling Experiment (CORDEX) is one of the most vulnerable regions on Earth to the effects of climate changes. Reliable information on potential future changes with high spatial resolution acquire significant importance for the development of effective adaptation and mitigation strategies for the region. In this context, Regional Climate Models (RCMs) play a fundamental role. In this paper, the results of a set of sensitivity experiments with the regional climate model COSMO-CLM version 5.0, for the Central Asia CORDEX domain, are presented. Starting from a reference model setup, general model performance is evaluated for present-days, testing the effects of a set of singular physical parameterizations and their mutual interaction on the simulation of monthly and seasonal values of three variables that are important for impact studies: 2-meter temperature, precipitation and diurnal temperature range. The final goal of this study is two-fold: having a general overview of model performance and its uncertainties for the considered region and determining at the same time an optimal model configuration. Results show that the model presents remarkable deficiencies over different areas of the domain. The combined change of the albedo taking into consideration the ratio of forest fractions and the soil conductivity taking into account the ratio of liquid water and ice in the soil, allows to achieve the best improvements in model performance in terms of climatological means. Importantly, the model seems to be particularly sensitive to those parameterizations that deal with soil and surface features, and that could positively affect the repartition of incoming radiation. The results for the mean climate appear to be independent of the observational dataset used for evaluation and of the boundary data employed to force the simulations. On the other hand, due to the large uncertainties in the variability estimates from observations, the use of different boundaries and the model internal variability, it has not been possible to rank the different simulations according to their representation of the monthly variability. This work is the first ever sensitivity study of an RCM for the CORDEX Central Asia domain and its results are of fundamental importance for further model development and for future climate projections over the area.

Download Full-text

Hydrological Responses of Headwater Basins to Monthly Perturbed Climate in the North American Cordillera

Journal of Hydrometeorology ◽

10.1175/jhm-d-18-0166.1 ◽

2019 ◽

Vol 20 (5) ◽

pp. 863-882 ◽

Cited By ~ 8

Author(s):

Kabir Rasouli ◽

John W. Pomeroy ◽

Paul H. Whitfield

Keyword(s):

Water Resources ◽

North American ◽

Climate Models ◽

Regional Climate ◽

Model Performance ◽

Regional Climate Models ◽

Annual Runoff ◽

Snow Accumulation ◽

Blowing Snow ◽

Considerable Uncertainty

Abstract How mountain hydrology at different elevations will respond to climate change is a challenging question of great importance to assessing changing water resources. Here, three North American Cordilleran snow-dominated basins—Wolf Creek, Yukon; Marmot Creek, Alberta; and Reynolds Mountain East, Idaho—each with good meteorological and hydrological records, were modeled using the physically based, spatially distributed Cold Regions Hydrological Model. Model performance was verified using field observations and found adequate for diagnostic analysis. To diagnose the effects of future climate, the monthly temperature and precipitation changes projected for the future by 11 regional climate models for the mid-twenty-first century were added to the observed meteorological time series. The modeled future was warmer and wetter, increasing the rainfall fraction of precipitation and shifting all three basins toward rainfall–runoff hydrology. This shift was largest at lower elevations and in the relatively warmer Reynolds Mountain East. In the warmer future, there was decreased blowing snow transport, snow interception and sublimation, peak snow accumulation, and melt rates, and increased evapotranspiration and the duration of the snow-free season. Annual runoff in these basins did not change despite precipitation increases, warming, and an increased prominence of rainfall over snowfall. Reduced snow sublimation offset reduced snowfall amounts, and increased evapotranspiration offset increased rainfall amounts. The hydrological uncertainty due to variation among climate models was greater than the predicted hydrological changes. While the results of this study can be used to assess the vulnerability and resiliency of water resources that are dependent on mountain snow, stakeholders and water managers must make decisions under considerable uncertainty, which this paper illustrates.

Download Full-text

On the importance of observational data properties when assessing regional climate model performance of extreme precipitation

Hydrology and Earth System Sciences ◽

10.5194/hess-17-4323-2013 ◽

2013 ◽

Vol 17 (11) ◽

pp. 4323-4337 ◽

Cited By ~ 30

Author(s):

M. A. Sunyer ◽

H. J. D. Sørup ◽

O. B. Christensen ◽

H. Madsen ◽

D. Rosbjerg ◽

...

Keyword(s):

Spatial Pattern ◽

Observational Data ◽

Extreme Precipitation ◽

Climate Models ◽

Climate Model ◽

Regional Climate ◽

Model Performance ◽

Regional Climate Models ◽

Data Sets ◽

Data Set

Abstract. In recent years, there has been an increase in the number of climate studies addressing changes in extreme precipitation. A common step in these studies involves the assessment of the climate model performance. This is often measured by comparing climate model output with observational data. In the majority of such studies the characteristics and uncertainties of the observational data are neglected. This study addresses the influence of using different observational data sets to assess the climate model performance. Four different data sets covering Denmark using different gauge systems and comprising both networks of point measurements and gridded data sets are considered. Additionally, the influence of using different performance indices and metrics is addressed. A set of indices ranging from mean to extreme precipitation properties is calculated for all the data sets. For each of the observational data sets, the regional climate models (RCMs) are ranked according to their performance using two different metrics. These are based on the error in representing the indices and the spatial pattern. In comparison to the mean, extreme precipitation indices are highly dependent on the spatial resolution of the observations. The spatial pattern also shows differences between the observational data sets. These differences have a clear impact on the ranking of the climate models, which is highly dependent on the observational data set, the index and the metric used. The results highlight the need to be aware of the properties of observational data chosen in order to avoid overconfident and misleading conclusions with respect to climate model performance.

Download Full-text