Sfc Temperature Forecasting Research

Bias and Absolute Error in Model Output Statistics (MOS) Surface Temperature Forecasts for Continental U.S. Locations
Andrew A. Taylor* and Lance M. Leslie
Center for Analysis and Prediction of Storms* / School of Meteorology, University of Oklahoma, Norman, OK

Introduction:

Operational numerical weather prediction (NWP) models provide forecasters with output on a regular basis giving them guidance in predicting the evolution of the atmosphere. Forecasters also use NWP model forecasts to assist them in predicting future surface weather conditions in their area of responsibility. Surface weather forecasts are of great interest to the general public, as well as to those involved in "weather-sensitive" operations. One method intended to generate more accurate guidance assisting the forecaster in composing forecasts of surface weather is the Model Output Statistics (MOS) technique. MOS systems are developed by deriving statistical relationships between individual predictands and various predictors, which typically include variables forecast by an NWP model as well as observed variables. The statistical relationships are usually found using multiple linear regression. MOS forecasts are used on a regular basis in operational weather forecasting, both as a tool to assist in the preparation of official forecasts and as a benchmark against which to compare those forecasts. In this study, bias and absolute error statistics were calculated for Nested Grid Model (NGM) and Aviation model (AVN) MOS surface temperature forecasts at 22 observing locations in the continental U.S. By examining these statistics on a site-by-site basis rather than on a regional basis, error characteristics unique to individual sites may more easily be seen. It is hoped that the results provided by the use of this method will help determine under what meteorological conditions MOS techniques perform well or poorly.

Methodology:

Twice daily surface temperature forecasts from calendar year 2001 NGM and AVN MOS bulletins were compared with the corresponding surface observations at several locations throughout the United States (shown in Figure 1). The bias and the absolute error (in degrees Fahrenheit) were calculated for maximum temperature and minimum temperature forecasts, as well as hourly temperature forecasts for 00Z and 12Z. Means of the biases and absolute errors in the forecasts were determined for each site over the entire year. 95% confidence intervals constructed around the yearly means were used to determine whether or not the differences in the means of the two models (as well as the differences in the mean bias of each model from zero) are statistically significant. Mean values of bias and absolute error for one year of MOS forecasts are useful, but for a more comprehensive assessment of MOS performance at each site the variability of the bias and absolute error statistics at each site was also examined. The individual bias and absolute error calculations from the whole year were categorized and the relative frequencies of the errors falling into each category were calculated.

Figure 1.

The sites were chosen to represent a wide variety of locations. Significant variability exists among these sites in:

yearly average temperature
terrain
vegetation
orientation with respect to large bodies of water
climatic regimes

However, most of the sites chosen are located near metropolitan areas (with the exception of GGW) and this aspect of variability has not been represented well. For this study, it was decided to focus on population centers and on busy airports.

Results:
Samples of the results are depicted in the figures below.

Figure 2. Mean absolute error (MAE) in MOS 00Z temperature forecasts for Denver Int'l Airport, CO (DEN) in 2001. MAE is shown as a function of lead time. Error bars around the data points indicate a 95% confidence interval around the mean values. The MAE calculations at DEN for 00Z temperature forecasts are larger than those for any other location studied here. Forecasting challenges at DEN include downslope wind events and shallow cold air masses moving down from the north. Figure 3. Same as Figure 2, but for 12Z temperature forecasts for Reagan National Airport (DCA). This MAE profile is typical of 12Z temperature forecasts among sites studied in the eastern U.S. Forecasts with a 12 hr lead time have a mean absolute error of around 2 deg F, and the MAE slowly increases with increasing lead time. Figure 6. A comparison of the absolute error relative frequency calculations from (a) NGM and (b) AVN MOS 00Z temperature forecasts for Denver Int'l Airport in 2001. Each individual absolute error calculation was assigned to one of the categories listed at the bottom of the above plots. Units for the categories are degrees Fahrenheit. Relative frequencies of the absolute error falling into each category are shown for different lead times. In addition to recording the largest values of MAE found at any location in this study, DEN also had a higher relative frequency of absolute errors >= 10 degrees than any other station examined. At the 72 hour lead time (forecasts were available for this lead time only from the AVN MOS), the individual absolute errors were approximately evenly distributed among the four categories.

Figure 4. Mean bias in MOS maximum temperature forecasts for Los Angeles Int'l Airport, CA (LAX) in 2001, shown as a function of "lead time" (in quotes since maximum temperatures can occur at various times during the day). Again, the error bars represent 95% confidence intervals. The significant warm biases apparent at LAX in both the NGM and AVN MOS forecasts could possibly be explained by variations in the timing of the sea breeze in the afternoon and by variations in the extent of low cloudiness. Both can be difficult to predict. Figure 5. Same as Figure 4, but for minimum temperature forecasts for McCarran Int'l Airport , NV (LAS). In this case, statistically significant cold biases are present in both the NGM and AVN MOS forecasts. Possibly contributing to the biases could be the rapid urban growth in the Las Vegas area during the past several years, enhancing the "urban heat island" effect. On a related note, the mean biases at Phoenix Sky Harbor Airport, AZ (PHX) were also significantly less than zero. Figure 7. A comparison of the bias relative frequency calculations from (a) NGM and (b) AVN MOS maximum temperature forecasts for Los Angeles Int'l Airport in 2001. Classification of the biases into categories was done in the same manner as classification of the absolute errors. Units for the categories are identical. The relative frequencies of individual biases >= 10 degrees and <= -10 degrees never exceed .05 and rarely exceed .02. The comparatively larger warm biases at LAX seem to have resulted from the cumulative effects of many more forecasts with a positive bias of 5 degrees or less than those with a negative bias of 5 degrees or less., with more positive biases of >5 degrees occurring at longer "lead times".

Conclusion:

Bias and absolute error statistics for NGM and AVN MOS forecasts vary widely depending on location. Although perhaps not surprising, these results are valuable as they have been quantified here. Future work will involve the formulation of observations-only approaches to forecasting surface temperature at specific locations, for comparison with MOS forecasts.

References:

Glahn, H. R., and D. A. Lowry, 1972. The use of Model Output Statistics (MOS) in objective weather forecasting. J. Appl. Meteor., 11, 1203-1211.

Jacks, E., J. B. Bower, V. J. Dagostaro, J. P. Dallavalle, M. C. Erickson, and J. C. Su, 1990. New NGM-based MOS guidance for maximum/minimum temperature, probability of precipitation, cloud amount, and surface wind. Wea. Forecasting, 5, 128-138.

NOAA's MOS verification web site at http://isl715.nws.noaa.gov/tdl/verif

Acknowledgements:

NGM and AVN MOS bulletins were obtained from the web at http://isl715.nws.noaa.gov/mos/archives. Thanks to Mr. Steve Leyton for obtaining the necessary surface observations used in this study, and for providing the files containing the observations in an easy to use format. Thanks also to Dr. Kelvin Droegemeier and Mr. Matt Haugland for their helpful suggestions.


Figure 2. Mean absolute error (MAE) in MOS 00Z temperature forecasts for Denver Int'l Airport, CO (DEN) in 2001. MAE is shown as a function of lead time. Error bars around the data points indicate a 95% confidence interval around the mean values. The MAE calculations at DEN for 00Z temperature forecasts are larger than those for any other location studied here. Forecasting challenges at DEN include downslope wind events and shallow cold air masses moving down from the north.	Figure 3. Same as Figure 2, but for 12Z temperature forecasts for Reagan National Airport (DCA). This MAE profile is typical of 12Z temperature forecasts among sites studied in the eastern U.S. Forecasts with a 12 hr lead time have a mean absolute error of around 2 deg F, and the MAE slowly increases with increasing lead time.	Figure 6. A comparison of the absolute error relative frequency calculations from (a) NGM and (b) AVN MOS 00Z temperature forecasts for Denver Int'l Airport in 2001. Each individual absolute error calculation was assigned to one of the categories listed at the bottom of the above plots. Units for the categories are degrees Fahrenheit. Relative frequencies of the absolute error falling into each category are shown for different lead times. In addition to recording the largest values of MAE found at any location in this study, DEN also had a higher relative frequency of absolute errors >= 10 degrees than any other station examined. At the 72 hour lead time (forecasts were available for this lead time only from the AVN MOS), the individual absolute errors were approximately evenly distributed among the four categories.


Figure 4. Mean bias in MOS maximum temperature forecasts for Los Angeles Int'l Airport, CA (LAX) in 2001, shown as a function of "lead time" (in quotes since maximum temperatures can occur at various times during the day). Again, the error bars represent 95% confidence intervals. The significant warm biases apparent at LAX in both the NGM and AVN MOS forecasts could possibly be explained by variations in the timing of the sea breeze in the afternoon and by variations in the extent of low cloudiness. Both can be difficult to predict.	Figure 5. Same as Figure 4, but for minimum temperature forecasts for McCarran Int'l Airport , NV (LAS). In this case, statistically significant cold biases are present in both the NGM and AVN MOS forecasts. Possibly contributing to the biases could be the rapid urban growth in the Las Vegas area during the past several years, enhancing the "urban heat island" effect. On a related note, the mean biases at Phoenix Sky Harbor Airport, AZ (PHX) were also significantly less than zero.	Figure 7. A comparison of the bias relative frequency calculations from (a) NGM and (b) AVN MOS maximum temperature forecasts for Los Angeles Int'l Airport in 2001. Classification of the biases into categories was done in the same manner as classification of the absolute errors. Units for the categories are identical. The relative frequencies of individual biases >= 10 degrees and <= -10 degrees never exceed .05 and rarely exceed .02. The comparatively larger warm biases at LAX seem to have resulted from the cumulative effects of many more forecasts with a positive bias of 5 degrees or less than those with a negative bias of 5 degrees or less., with more positive biases of >5 degrees occurring at longer "lead times".