Jolliffe, Ian T.
Forecast Verification : A Practitioner's Guide in Atmospheric Science. - 2nd ed. - 1 online resource (304 pages)
Intro -- Forecast Verification -- Contents -- List of Contributors -- Preface -- Preface to the First Edition -- 1 Introduction -- 1.1 A brief history and current practice -- 1.1.1 History -- 1.1.2 Current practice -- 1.2 Reasons for forecast verification and its benefits -- 1.3 Types of forecast and verification data -- 1.4 Scores, skill and value -- 1.4.1 Skill scores -- 1.4.2 Artificial skill -- 1.4.3 Statistical significance -- 1.4.4 Value added -- 1.5 Data quality and other practical considerations -- 1.6 Summary -- 2 Basic concepts -- 2.1 Introduction -- 2.2 Types of predictand -- 2.3 Exploratory methods -- 2.4 Numerical descriptive measures -- 2.5 Probability, random variables and expectations -- 2.6 Joint, marginal and conditional distributions -- 2.7 Accuracy, association and skill -- 2.8 Properties of verification measures -- 2.9 Verification as a regression problem -- 2.10 The Murphy-Winkler framework -- 2.11 Dimensionality of the verification problem -- 3 Deterministic forecasts of binary events -- 3.1 Introduction -- 3.2 Theoretical considerations -- 3.2.1 Some basic descriptive statistics -- 3.2.2 A general framework for verification: the distributions-oriented approach -- 3.2.3 Performance measures in terms of factorizations of the joint distribution -- 3.2.4 Diagrams for visualizing performance measures -- 3.2.5 Case study: verification of cloud-fraction forecasts -- 3.3 Signal detection theory and the ROC -- 3.3.1 The signal detection model -- 3.3.2 The relative operating characteristic (ROC) -- 3.4 Metaverification: criteria for assessing performance measures -- 3.4.1 Desirable properties -- 3.4.2 Other properties -- 3.5 Performance measures -- 3.5.1 Overview of performance measures -- 3.5.2 Sampling uncertainty and confidence intervals for performance measures -- 3.5.3 Optimal threshold probabilities -- Acknowledgements. 4 Deterministic forecasts of multi-category events -- 4.1 Introduction -- 4.2 The contingency table: notation, definitions, and measures of accuracy -- 4.2.1 Notation and definitions -- 4.2.2 Measures of accuracy -- 4.3 Skill scores -- 4.3.1 Desirable attributes -- 4.3.2 Gandin and Murphy equitable scores -- 4.3.3 Gerrity equitable scores -- 4.3.4 LEPSCAT -- 4.3.5 SEEPS -- 4.3.6 Summary remarks on scores -- 4.4 Sampling variability of the contingency table and skill scores -- 5 Deterministic forecasts of continuous variables -- 5.1 Introduction -- 5.2 Forecast examples -- 5.3 First-order moments -- 5.3.1 Bias -- 5.3.2 Mean Absolute Error -- 5.3.3 Bias correction and artificial skill -- 5.3.4 Mean absolute error and skill -- 5.4 Second- and higher-order moments -- 5.4.1 Mean Squared Error -- 5.4.2 MSE skill score -- 5.4.3 MSE of scaled forecasts -- 5.4.4 Correlation -- 5.4.5 An example: testing the 'limit of predictability' -- 5.4.6 Rank correlations -- 5.4.7 Comparison of moments of the marginal distributions -- 5.4.8 Graphical summaries -- 5.5 Scores based on cumulative frequency -- 5.5.1 Linear Error in Probability Space (LEPS) -- 5.5.2 Quantile-quantile plots -- 5.5.3 Conditional quantile plots -- 5.6 Summary and concluding remarks -- 6 Forecasts of spatial fields -- 6.1 Introduction -- 6.2 Matching methods -- 6.3 Traditional verification methods -- 6.3.1 Standard continuous and categorical approaches -- 6.3.2 S1 and anomaly correlation -- 6.3.3 Distributional methods -- 6.4 Motivation for alternative approaches -- 6.5 Neighbourhood methods -- 6.5.1 Comparing neighbourhoods of forecasts and observations -- 6.5.2 Comparing spatial forecasts with point observations -- 6.6 Scale separation methods -- 6.7 Feature-based methods -- 6.7.1 Feature-matching techniques -- 6.7.2 Structure-Amplitude-Location (SAL) technique -- 6.8 Field deformation methods. 6.8.1 Location metrics -- 6.8.2 Field deformation -- 6.9 Comparison of approaches -- 6.10 New approaches and applications: the future -- 6.11 Summary -- 7 Probability forecasts -- 7.1 Introduction -- 7.2 Probability theory -- 7.2.1 Basic concepts from probability theory -- 7.2.2 Probability forecasts, reliability and sufficiency -- 7.3 Probabilistic scoring rules -- 7.3.1 Definition and properties of scoring rules -- 7.3.2 Commonly used scoring rules -- 7.3.3 Decomposition of scoring rules -- 7.4 The relative operating characteristic (ROC) -- 7.5 Evaluation of probabilistic forecasting systems from data -- 7.5.1 Three examples -- 7.5.2 The empirical ROC -- 7.5.3 The empirical score as a measure of performance -- 7.5.4 Decomposition of the empirical score -- 7.5.5 Binning forecasts and the leave-one-out error -- 7.6 Testing reliability -- 7.6.1 Reliability analysis for forecast A: the reliability diagram -- 7.6.2 Reliability analysis for forecast B: the chi-squared test -- 7.6.3 Reliability analysis for forecast C: the PIT -- Acknowledgements -- 8 Ensemble forecasts -- 8.1 Introduction -- 8.2 Example data -- 8.3 Ensembles interpreted as discrete samples -- 8.3.1 Reliability of ensemble forecasts -- 8.3.2 Multidimensional reliability -- 8.3.3 Discrimination -- 8.4 Ensembles interpreted as probabilistic forecasts -- 8.4.1 Probabilistic interpretation of ensembles -- 8.4.2 Probabilistic skill metrics applied to ensembles -- 8.4.3 Effect of ensemble size on skill -- 8.5 Summary -- 9 Economic value and skill -- 9.1 Introduction -- 9.2 The cost/loss ratio decision model -- 9.2.1 Value of a deterministic binary forecast system -- 9.2.2 Probability forecasts -- 9.2.3 Comparison of deterministic and probabilistic binary forecasts -- 9.3 The relationship between value and the ROC -- 9.4 Overall value and the Brier Skill Score. 9.5 Skill, value and ensemble size -- 9.6 Applications: value and forecast users -- 9.7 Summary -- 10 Deterministic forecasts of extreme events and warnings -- 10.1 Introduction -- 10.2 Forecasts of extreme events -- 10.2.1 Challenges -- 10.2.2 Previous studies -- 10.2.3 Verification measures for extreme events -- 10.2.4 Modelling performance for extreme events -- 10.2.5 Extreme events: summary -- 10.3 Warnings -- 10.3.1 Background -- 10.3.2 Format of warnings and observations for verification -- 10.3.3 Verification of warnings -- 10.3.4 Warnings: summary -- Acknowledgements -- 11 Seasonal and longer-range forecasts -- 11.1 Introduction -- 11.2 Forecast formats -- 11.2.1 Deterministic and probabilistic formats -- 11.2.2 Defining the predictand -- 11.2.3 Inclusion of climatological forecasts -- 11.3 Measuring attributes of forecast quality -- 11.3.1 Skill -- 11.3.2 Other attributes -- 11.3.3 Statistical significance and uncertainty estimates -- 11.4 Measuring the quality of individual forecasts -- 11.5 Decadal and longer-range forecast verification -- 11.6 Summary -- 12 Epilogue: new directions in forecast verification -- 12.1 Introduction -- 12.2 Review of key concepts -- 12.3 Forecast evaluation in other disciplines -- 12.3.1 Statistics -- 12.3.2 Finance and economics -- 12.3.3 Medical and clinical studies -- 12.4 Current research and future directions -- Acknowledgements -- Appendix: Verification Software -- A.1 What is good software? -- A.1.1 Correctness -- A.1.2 Documentation -- A.1.3 Open source/closed source/commercial -- A.1.4 Large user base -- A.2 Types of verification users -- A.2.1 Students -- A.2.2 Researchers -- A.2.3 Operational forecasters -- A.2.4 Institutional use -- A.3 Types of software and programming languages -- A.3.1 Spreadsheets -- A.3.2 Statistical programming languages -- A.4 Institutional supported software. A.4.1 Model Evaluation Tool (MET) -- A.4.2 Ensemble Verification System (EVS) -- A.4.3 EUMETCAL Forecast Verification Training Module -- A.5 Displays of verification information -- A.5.1 National Weather Service Performance Management -- A.5.2 Forecast Evaluation Tool -- Glossary -- References -- Index -- Color Plate.
Forecast Verification: A Practioner's Guide in Atmospheric Science, 2nd Edition provides an indispensible guide to this area of active research by combining depth of information with a range of topics to appeal both to professional practitioners and researchers and postgraduates. The editors have succeeded in presenting chapters by a variety of the leading experts in the field while still retaining a cohesive and highly accessible style. The book balances explanations of concepts with clear and useful discussion of the main application areas. Reviews of first edition: "This book will provide a good reference, and I recommend it especially for developers and evaluators of statistical forecast systems." (Bulletin of the American Meteorological Society; April 2004) "...a good mixture of theory and practical applications...well organized and clearly written..." (Royal Statistical Society, Vol.168, No.1, January 2005) NEW to the second edition: Completely updated chapter on the Verification of Spatial Forecasts taking account of the wealth of new research in the area New separate chapters on Probability Forecasts and Ensemble Forecasts Includes new chapter on Forecasts of Extreme Events and Warnings Includes new chapter on Seasonal and Climate Forecasts Includes new Appendix on Verification Software Cover image credit: The triangle of barplots shows a novel use of colour for visualizing probability forecasts of ternary categories - see Fig 6b of Jupp et al. 2011, On the visualisation, verification and recalibration of ternary probabilistic forecasts, Phil. Trans. Roy. Soc. (in press)..
9781119960010
Weather forecasting -- Statistical methods -- Evaluation.
Electronic books.
QC996.5 -- .F67 2011eb
551.63
Forecast Verification : A Practitioner's Guide in Atmospheric Science. - 2nd ed. - 1 online resource (304 pages)
Intro -- Forecast Verification -- Contents -- List of Contributors -- Preface -- Preface to the First Edition -- 1 Introduction -- 1.1 A brief history and current practice -- 1.1.1 History -- 1.1.2 Current practice -- 1.2 Reasons for forecast verification and its benefits -- 1.3 Types of forecast and verification data -- 1.4 Scores, skill and value -- 1.4.1 Skill scores -- 1.4.2 Artificial skill -- 1.4.3 Statistical significance -- 1.4.4 Value added -- 1.5 Data quality and other practical considerations -- 1.6 Summary -- 2 Basic concepts -- 2.1 Introduction -- 2.2 Types of predictand -- 2.3 Exploratory methods -- 2.4 Numerical descriptive measures -- 2.5 Probability, random variables and expectations -- 2.6 Joint, marginal and conditional distributions -- 2.7 Accuracy, association and skill -- 2.8 Properties of verification measures -- 2.9 Verification as a regression problem -- 2.10 The Murphy-Winkler framework -- 2.11 Dimensionality of the verification problem -- 3 Deterministic forecasts of binary events -- 3.1 Introduction -- 3.2 Theoretical considerations -- 3.2.1 Some basic descriptive statistics -- 3.2.2 A general framework for verification: the distributions-oriented approach -- 3.2.3 Performance measures in terms of factorizations of the joint distribution -- 3.2.4 Diagrams for visualizing performance measures -- 3.2.5 Case study: verification of cloud-fraction forecasts -- 3.3 Signal detection theory and the ROC -- 3.3.1 The signal detection model -- 3.3.2 The relative operating characteristic (ROC) -- 3.4 Metaverification: criteria for assessing performance measures -- 3.4.1 Desirable properties -- 3.4.2 Other properties -- 3.5 Performance measures -- 3.5.1 Overview of performance measures -- 3.5.2 Sampling uncertainty and confidence intervals for performance measures -- 3.5.3 Optimal threshold probabilities -- Acknowledgements. 4 Deterministic forecasts of multi-category events -- 4.1 Introduction -- 4.2 The contingency table: notation, definitions, and measures of accuracy -- 4.2.1 Notation and definitions -- 4.2.2 Measures of accuracy -- 4.3 Skill scores -- 4.3.1 Desirable attributes -- 4.3.2 Gandin and Murphy equitable scores -- 4.3.3 Gerrity equitable scores -- 4.3.4 LEPSCAT -- 4.3.5 SEEPS -- 4.3.6 Summary remarks on scores -- 4.4 Sampling variability of the contingency table and skill scores -- 5 Deterministic forecasts of continuous variables -- 5.1 Introduction -- 5.2 Forecast examples -- 5.3 First-order moments -- 5.3.1 Bias -- 5.3.2 Mean Absolute Error -- 5.3.3 Bias correction and artificial skill -- 5.3.4 Mean absolute error and skill -- 5.4 Second- and higher-order moments -- 5.4.1 Mean Squared Error -- 5.4.2 MSE skill score -- 5.4.3 MSE of scaled forecasts -- 5.4.4 Correlation -- 5.4.5 An example: testing the 'limit of predictability' -- 5.4.6 Rank correlations -- 5.4.7 Comparison of moments of the marginal distributions -- 5.4.8 Graphical summaries -- 5.5 Scores based on cumulative frequency -- 5.5.1 Linear Error in Probability Space (LEPS) -- 5.5.2 Quantile-quantile plots -- 5.5.3 Conditional quantile plots -- 5.6 Summary and concluding remarks -- 6 Forecasts of spatial fields -- 6.1 Introduction -- 6.2 Matching methods -- 6.3 Traditional verification methods -- 6.3.1 Standard continuous and categorical approaches -- 6.3.2 S1 and anomaly correlation -- 6.3.3 Distributional methods -- 6.4 Motivation for alternative approaches -- 6.5 Neighbourhood methods -- 6.5.1 Comparing neighbourhoods of forecasts and observations -- 6.5.2 Comparing spatial forecasts with point observations -- 6.6 Scale separation methods -- 6.7 Feature-based methods -- 6.7.1 Feature-matching techniques -- 6.7.2 Structure-Amplitude-Location (SAL) technique -- 6.8 Field deformation methods. 6.8.1 Location metrics -- 6.8.2 Field deformation -- 6.9 Comparison of approaches -- 6.10 New approaches and applications: the future -- 6.11 Summary -- 7 Probability forecasts -- 7.1 Introduction -- 7.2 Probability theory -- 7.2.1 Basic concepts from probability theory -- 7.2.2 Probability forecasts, reliability and sufficiency -- 7.3 Probabilistic scoring rules -- 7.3.1 Definition and properties of scoring rules -- 7.3.2 Commonly used scoring rules -- 7.3.3 Decomposition of scoring rules -- 7.4 The relative operating characteristic (ROC) -- 7.5 Evaluation of probabilistic forecasting systems from data -- 7.5.1 Three examples -- 7.5.2 The empirical ROC -- 7.5.3 The empirical score as a measure of performance -- 7.5.4 Decomposition of the empirical score -- 7.5.5 Binning forecasts and the leave-one-out error -- 7.6 Testing reliability -- 7.6.1 Reliability analysis for forecast A: the reliability diagram -- 7.6.2 Reliability analysis for forecast B: the chi-squared test -- 7.6.3 Reliability analysis for forecast C: the PIT -- Acknowledgements -- 8 Ensemble forecasts -- 8.1 Introduction -- 8.2 Example data -- 8.3 Ensembles interpreted as discrete samples -- 8.3.1 Reliability of ensemble forecasts -- 8.3.2 Multidimensional reliability -- 8.3.3 Discrimination -- 8.4 Ensembles interpreted as probabilistic forecasts -- 8.4.1 Probabilistic interpretation of ensembles -- 8.4.2 Probabilistic skill metrics applied to ensembles -- 8.4.3 Effect of ensemble size on skill -- 8.5 Summary -- 9 Economic value and skill -- 9.1 Introduction -- 9.2 The cost/loss ratio decision model -- 9.2.1 Value of a deterministic binary forecast system -- 9.2.2 Probability forecasts -- 9.2.3 Comparison of deterministic and probabilistic binary forecasts -- 9.3 The relationship between value and the ROC -- 9.4 Overall value and the Brier Skill Score. 9.5 Skill, value and ensemble size -- 9.6 Applications: value and forecast users -- 9.7 Summary -- 10 Deterministic forecasts of extreme events and warnings -- 10.1 Introduction -- 10.2 Forecasts of extreme events -- 10.2.1 Challenges -- 10.2.2 Previous studies -- 10.2.3 Verification measures for extreme events -- 10.2.4 Modelling performance for extreme events -- 10.2.5 Extreme events: summary -- 10.3 Warnings -- 10.3.1 Background -- 10.3.2 Format of warnings and observations for verification -- 10.3.3 Verification of warnings -- 10.3.4 Warnings: summary -- Acknowledgements -- 11 Seasonal and longer-range forecasts -- 11.1 Introduction -- 11.2 Forecast formats -- 11.2.1 Deterministic and probabilistic formats -- 11.2.2 Defining the predictand -- 11.2.3 Inclusion of climatological forecasts -- 11.3 Measuring attributes of forecast quality -- 11.3.1 Skill -- 11.3.2 Other attributes -- 11.3.3 Statistical significance and uncertainty estimates -- 11.4 Measuring the quality of individual forecasts -- 11.5 Decadal and longer-range forecast verification -- 11.6 Summary -- 12 Epilogue: new directions in forecast verification -- 12.1 Introduction -- 12.2 Review of key concepts -- 12.3 Forecast evaluation in other disciplines -- 12.3.1 Statistics -- 12.3.2 Finance and economics -- 12.3.3 Medical and clinical studies -- 12.4 Current research and future directions -- Acknowledgements -- Appendix: Verification Software -- A.1 What is good software? -- A.1.1 Correctness -- A.1.2 Documentation -- A.1.3 Open source/closed source/commercial -- A.1.4 Large user base -- A.2 Types of verification users -- A.2.1 Students -- A.2.2 Researchers -- A.2.3 Operational forecasters -- A.2.4 Institutional use -- A.3 Types of software and programming languages -- A.3.1 Spreadsheets -- A.3.2 Statistical programming languages -- A.4 Institutional supported software. A.4.1 Model Evaluation Tool (MET) -- A.4.2 Ensemble Verification System (EVS) -- A.4.3 EUMETCAL Forecast Verification Training Module -- A.5 Displays of verification information -- A.5.1 National Weather Service Performance Management -- A.5.2 Forecast Evaluation Tool -- Glossary -- References -- Index -- Color Plate.
Forecast Verification: A Practioner's Guide in Atmospheric Science, 2nd Edition provides an indispensible guide to this area of active research by combining depth of information with a range of topics to appeal both to professional practitioners and researchers and postgraduates. The editors have succeeded in presenting chapters by a variety of the leading experts in the field while still retaining a cohesive and highly accessible style. The book balances explanations of concepts with clear and useful discussion of the main application areas. Reviews of first edition: "This book will provide a good reference, and I recommend it especially for developers and evaluators of statistical forecast systems." (Bulletin of the American Meteorological Society; April 2004) "...a good mixture of theory and practical applications...well organized and clearly written..." (Royal Statistical Society, Vol.168, No.1, January 2005) NEW to the second edition: Completely updated chapter on the Verification of Spatial Forecasts taking account of the wealth of new research in the area New separate chapters on Probability Forecasts and Ensemble Forecasts Includes new chapter on Forecasts of Extreme Events and Warnings Includes new chapter on Seasonal and Climate Forecasts Includes new Appendix on Verification Software Cover image credit: The triangle of barplots shows a novel use of colour for visualizing probability forecasts of ternary categories - see Fig 6b of Jupp et al. 2011, On the visualisation, verification and recalibration of ternary probabilistic forecasts, Phil. Trans. Roy. Soc. (in press)..
9781119960010
Weather forecasting -- Statistical methods -- Evaluation.
Electronic books.
QC996.5 -- .F67 2011eb
551.63