WESTERN REGION TECHNICAL ATTACHMENT
NO. 02-05
FEBRUARY 5, 2002

GFE FORECAST VS RAWS OBSERVATIONS

Mark Struthwolf, Weather Forecast Office, Salt Lake City, UT

Introduction

The Interactive Forecast Preparation System (IFPS ) is the cornerstone to a fundamentally different way of creating and presenting National Weather Service (NWS) forecast products.  A key component of IFPS is the National Oceanic and Atmospheric Administration's (NOAA) Forecast Systems Laboratory's Graphical Forecast Editor(GFE). Using the GFE, users at the NWS Forecast Office in Salt Lake City (SLC), Utah, generate a once-daily Clearing Index grid for use in fire and smoke management throughout the State of Utah and twice daily during the summer season the Southern Utah Graphical Flash Flood Potential. In this initial project stage, grids used for calculating basin clearing indices are initialized using the Eta model. Users are able to modify these grids using the GFE spatial editor prior to publishing.

To better understand the accuracy of these initialized GFE grids, a verification study of GFE temperature and relative humidity forecast grids was initiated in late May 2001. This ongoing study compares daily maximum and minimum temperature and relative humidity at six remote automatic weather stations (RAWS) in Utah, with unedited Eta model GFE forecast grid points nearest each station. The stations selected provided a varied range of elevations and spatial distribution within the SLC Fire Weather forecast area of responsibility. The 00Z and 12Z Eta model GFE temperature and relative humidity (maximum and minimum) forecasts were evaluated at 12 h time increments through 24 and 36 h, respectively. Results from identical forecast periods have been combined due to strong similarities. Links have been provided to access individual results from 00Z and 12Z forecasts.

Methodology

The GFE (Rapid Prototype Project (RPP) version 11) grids (5-km resolution) were initialized using the 80-km (AWIPS grid 211) Eta model grids. The 00Z and 12Z maximum and minimum GFE forecasts of temperature and relative humidity were verified by manually comparing actual RAWS observations to grid point values closest to the RAWS location. Maximum temperature (maxT) and minimum relative humidity (minRH) grids were evaluated in the period between 10Z and 04Z while minimum temperature (minT) and maximum relative humidity (maxRH) grids were evaluated in the period between 22Z and 16Z. A GFE "smart tool" was used to derive relative humidity from temperature and dew point temperature, since relative humidity is not directly computed by GFE. The RAWS locations were located on the GFE grid by using the Samples "Show Lat/Lon tool" under the Maps menu (Fig. 1, GFE background). Once the Lat/Lon position of the RAWS was located, the actual position of the RAWS was repositioned to the nearest pixel in any direction which best represented the actual elevation of the RAWS. The largest deviance was approximately 200 ft, while the majority of selected RAWS repositioned elevations were within 100 ft of actual.

Data collection was dictated by work schedule during a 4-month period from the end of May through mid-September, which can be considered a summer season data set. A total of 30 to 33 cases were collected for the Eta 12Z 12 h forecasts, while 76 to 81 cases were collected for both Eta 12Z 24 and 36 h (daytime) forecasts, and Eta 00Z 12 and 24-h (nighttime) forecasts. Mean absolute errors (MAE) and mean errors (BIAS) were calculated according to Meier and Barker (1993).

Results

The results from the GFE Eta forecasts and RAWS observations are displayed according to lowest to highest RAWS elevation, with Eta 00Z and 12Z results combined (Table 1). (Though not shown here, the results from the individual forecasts (Table 2, 00Z and 12Z) were nearly identical for both temperature and relative humidity. The 00Z MAE and BIAS scores are slightly better than the 12Z scores as a result of the shorter forecast lead-time, i.e.,the comparable forecast periods for 00Z are 0-12 h and 12-24 h and for 12Z are 12-24 h and 24-36 h).

In general, the forecasts are very good, with both the maxT and minRH GFE forecasts verifying better than minT and maxRH forecasts. As seen in Table 1, the average 12-, 24-, and 36-h maxT forecast MAE errors were nearly 4.0°, while the minRH forecast MAE during the same periods were approximately 4.5 percent. The individual RAWS temperatures show a consistency in the sign of the BIAS and five-of-six of the RAWS indicate less than a 0.5° change in the MAE between the two daytime periods. The magnitude of biases is within 1° of the MAE score for all but two of the daytime forecasts, with an overall average value of 0.9°. This close agreement between the MAE and BIAS indicates a consistent and direct relationship between forecast and observed temperatures. Greater differences occurred with relative humidity but overall, the relationship (with the exception of Brimstone) was direct. The indirect relationship at Brimstone was likely attributed to a cloudy moist period that occurred across that portion of Utah during the middle of August. Consequently, the Eta model consistently under forecast maxRH. This moist period with warmer than average nights also influences the minT BIAS, shifting what would have been a definitive cool bias to one that was marginal.

Nighttime forecasts of minT and maxRH did not fair as well. The MAE for 12- and 24-h temperature and relative humidity forecasts are 7.9° and 12.8%, respectively. This further translated into a greater variation between the magnitude of the BIAS and the MAE. While the MAE and the absolute value of the BIAS never differed by more than 1.4° for maxT, the difference increased to 4.2° for minT. The largest biases occurred at two of the three mountain locations, probably caused by radiational cooling effects. That likely was a result of their similar environmental surroundings, i.e., exposure, vegetation, and aspect.

As expected, similar patterns in MAE and sign of BIAS were found between RAWS with similar environments. The most notable was the warm minT and dry maxRH BIAS at both Norway and Tom Best Springs, both located in mountain valleys. The third mountain RAWS, Signal Peak, is located near ridge top level, unlike the other two high elevation RAWS. This site does not experience the micro-climate of cool air pooling common to mountain valleys, as experienced by Norway and Tom Best Spring. Rather, Signal Peak revealed a nighttime cool bias, similar to the three western Utah valley sites that are more exposed, have little surrounding vegetation, and are situated on open slopes. The maxRH moist BIAS of Signal Peak also more closely resembles that of the valley sites.

The second most notable patterns in maxT MAE and sign of BIAS was at Brimstone and Aragonite, nearly mirroring each other in magnitude and sign. These were the only two RAWS that reflected a maxT cool bias. The environmental surroundings of these two RAWS are once again very similar. Both are located on an open slope 500 to 1000-ft above the adjacent valley floor, and with little vegetation. The prevailing arid conditions could be a cool bias factor. However, Badger Spring has similar environmental surroundings, yet it showed a warm bias.

The collection period was relatively dry with some monsoonal surges interspersed. These wet periods produced only 5 or 6 days with wetting rains (defined by the SLC Fire Weather program as rains greater than 0.1 in) at the mountain sites, and only 3 to 5 days at the western Utah valley sites. By removing these wet periods, which affected both the temperature and relative humidity, the MAE and BIAS reduced on average nearly 1° for temperature and 2 percent for relative humidity. The largest improvement occurred during the nighttime period, when relative humidity varied most. Strong radiational cooling resulted in a GFE forecast that was too warm, producing a dry bias, while weak radiational cooling resulted in a GFE forecast that was too cool, producing a moist bias. During weak radiational cooling—nights typically with clouds or showers—sites with a moist bias (Badger Spring, Aragonite, and Signal Peak) reduced their maxRH MAE by as much as 2.5 percent, and increased their BIAS by as much as 3.2 percent.

Summary and Conclusion

The verification study between GFE Eta model forecasts and observed conditions at several RAWS locations shows promise for model performance.  The average MAE for maxT and minRH were 3.9° and 4.5 percent, respectively, while the minT and maxRH were 7.9° and 12.8 percent, respectively. The better scores obtained during the daytime period reflects the Eta's ability to account for a well mixed atmosphere. However, at nighttime during boundary layer decoupling local effects take over. These micro scale terrain and climate features cannot be resolved by the Eta's 80-km grid, resulting in the poorer nighttime performance. In addition, days with precipitation showed greater temperature and relative humidity errors. By removing these days from the data set, the MAE was reduced and the BIAS increased, narrowing the difference between the MAE and BIAS. A close relationship between the MAE and BIAS indicates a consistent (positive or negative) error between forecasts and the observed. Consequently, this gives the forecaster greater confidence in determining the amount of adjustment to the GFE forecast necessary to generate a more accurate forecast.

Relationships of MAE and BIAS between the RAWS sites and various forecast periods indicate that elevation is not a primary factor in determining the magnitude of the MAE, or magnitude and sign of the BIAS. Exposure, vegetation, and perhaps surrounding terrain, play equal, if not more important, roles that may be captured by initializing GFE grids with higher resolution models.

It is possible to improve MAE and BIAS without additional improvement to the GFE. RAWS collect instantaneous temperatures and 10-min average relative humidity once per hour (Finklin and Fischer, 1990). Consequently, temperature and relative humidity spikes are missed between hours. In Utah's climate, this could amount to errors of 3 to 5° in minT and 4 to 8 percent in maxRH. An increase in GFE grid resolution could reduce the error in elevation difference between actual and current best-fit plotting technique. The elevation sightings of the RAWS were up to 200 ft in error in this study which, when converted to the standard atmosphere change of 5° F /1000 ft, results in a 1° error. In addition, this data set included outliers that were typically associated with precipitation; that when removed, resulted in 1 to 1.5° and 2.5 to 3.2 percent lower MAE scores in temperature and relative humidity, respectively. Combining these maximum potential error improvements may translate to forecast temperature errors as low as less than 2°.

Although this was not an elaborate GFE verification study, it reveals important issues that must be considered during forecast grid editing, or for that matter, implications to the forecast production paradigm shift.  These verification results point out the non-uniformity of model error and bias across a particular grid, especially one used in such mountainous terrain. It is safe to say, a user cannot simply apply biases across an entire grid (e.g. GFE action "Assign Value Up or Down") without actually incorrectly adjusting values in certain areas. More elaborate verification studies (especially grid-based) are needed to possibly provide an automatic objective BIAS correction prior to manual grid editing. This correction, in addition to a climatological correction, eliminates, or reduces, the possible over-correction by forecasters.

Current work at the SLC is incorporating ASOS sites that automatically furnish maxT and minT spikes in their daily summaries, as well as enabling a comparison between MOS forecasts and Eta GFE forecasts. MesoEta GFE forecasts are also being validated.

References

Finklin, Arnold I., and Fischer, William C., 1990: Weather Station Handbook - an Interagency Guide for Wildland Managers: A publication of the National Wildfire Coordinating Group; Sponsored by the U.S. Department of Agriculture, U.S. Department of the Interior, and the National Association of State Foresters, 159 pp.

Meier, K. W., and T. W. Barker, 1993: AEV Local Verification for Aviation, Precipitation and Temperature Programs: AV, REL, TEM. NOAA Western Region Computer Programs and Problems NWS WRCP-No. 42. National Oceanic and Atmospheric Administration, U. S. Department of Commerce, 20 pp.

Thanks to Chris Gibson and Mark Jackson, SLC NWS Forecast Office, for sharing their knowledge of GFE and for editing this manuscript.