Verification Study of NWSFO SLC NFDRS Forecasts for 2005

Chris Gibson
NOAA National Weather Service
Salt Lake City, Utah
Contact author through the NWS SLC Webmaster

A verification study of the summer 2005 National Weather Service (NWS) Salt Lake City Office forecasts for the National Fire Danger Rating System (NFDRS) was completed through an analysis of mean absolute error (MAE) and mean error (BIAS). Forecasts were verified for temperature, relative humidity, wind speed and fuel moisture. Results are compared to a similar study from 2004. The forecasts are found to have a warm and dry bias mainly at higher elevations. As in 2004, the NWS NFDRS forecasts were found to provide an improvement over a persistence forecast for all elements except wind speed.

Introduction

A verification study was conducted on the National Weather Service (NWS) forecasts from the Salt Lake City NWS office for the National Fire Danger Rating System (NFDRS) during the summer of 2005. The NWS NFDRS forecasts are made each afternoon with a 24 hour lead time for the next afternoon. Forecasts of 4 weather parameters were verified - Temperature, Relative Humidity, Wind Speed and Fuel Moisture. Forecasts and verifying observations were collected for June 1, 2005 to August 30, 2005. Forecasts were created if an observation was received at the WFO for a NFDRS station. Observations are received at the WFO as part of the NFDRS system. For a forecast to be verified a sequence of three things must occur. First, an observation is received for a station through NFDRS for today. Second, a forecast is generated for the station valid tomorrow and the forecast must be successfully ingested into NFDRS. Finally, an observation must be received through NFDRS at the valid time of the NWS forecast.

Nineteen NFDRS forecast points had enough forecasts for a sample, averaging 71 forecasts each. The number of forecasts varied from 41 at St. George to 82 forecasts for Bear River RAWS, Otter Creek RAWS and Pleasant Grove RAWS. The remainder of stations for which we made an NFDRS forecast had limited number of observations, or one or more elements observations were corrupt (bad sensor data entered into NFDRS). The Black Cedar RAWS was removed from the study for 2005 due to bad relative humidity for a large part of the season. Table 2 lists the NFDRS stations used in the analysis in alphabetical order. All stations are RAWS except for St. George, which was a manual station in 2005.

Analysis

A mean absolute error (MAE) and an average error (BIAS) was calculated for each element. MAE and BIAS are defined in appendix A. Graphs of MAE and BIAS for the four verified elements have also been generated for each NFDRS forecast point. The MAE indicates the relative magnitude of the error in the NWS NFDRS forecasts, while the BIAS shows whether the forecasts are on average too dry, too windy, etc. A comparison with a forecast of persistence is also provided, as well as a comparison with stations at different elevations.

Results

Results are presented by diagrams of MAE and BIAS for each station or for groups of stations. Each diagram has 4 columns which represent the error for temperature (T), relative humidity (RH), wind speed (WS) and fuel moisture (FM) respectively. The vertical scale is multidimensional - in each diagram it represents temperature (degrees F), percent relative humidity (%), miles per hour (mph), and percent fuel moisture (%). Table 1 presents the averaged results for MAE and BIAS of all 19 stations used in the study.

For all stations (Table 1), MAE averaged 3.9 degrees for temperature, 6.8% for relative humidity, 4.2 mph for wind speed and 1.4% for fuel moisture. Station average BIAS was .71 degrees and -.19% RH, 1.16 mph and -.61% FM. This indicates a slight warm and dry BIAS (for T and RH), which is related to the negative FM BIAS (a relationship that occurs since fuel moisture is a function of temperature and humidity) for the NWS NFDRS forecasts. A comparison with the results from 2004 is completed below.

Table 1. MAE and BIAS averaged for all stations.

MAE and BIAS By Station

Table 2 below has links to view MAE and BIAS for 2005 for each element by each station. Results are viewed by running the mouse over each MAE and BIAS link to show the appropriate histogram in the right hand cell of the table.

Table 2. Diagrams of MAE and BIAS for each of the 19 stations used in the study. Stations are organized alphabetically. From left to right, elements in each diagram are T, RH, WS and FM.
STATION
ELEVATION (ft)
FIRE ZONE
MAE
BIAS
Aragonite
5030'
420
MAE
BIAS
Bear River
8536'
427
MAE
BIAS
Bues Canyon
5100'
422
MAE
BIAS
Cedar Mountain
4650'
420
MAE
BIAS
Clifton Flat
6384'
420
MAE
BIAS
Hewinta
9186'
427
MAE
BIAS
Horse Hollow
6010'
434
MAE
BIAS
Lost Creek
7490'
436
MAE
BIAS
Mud Springs
5902'
434
MAE
BIAS
Norway
8280'
427
MAE
BIAS
Otter Creek
7160'
426
MAE
BIAS
Pleasant Grove
5200'
424
MAE
BIAS
Ray's Valley
7300'
427
MAE
BIAS
Rosebud
4987'
420
MAE
BIAS
Sevier Reservoir
5369'
433
MAE
BIAS
Signal Peak
8792'
436
MAE
BIAS
St George
2650'
439
MAE
BIAS
Tule Valley
5200'
434
MAE
BIAS
Vernon
5639'
420
MAE
BIAS

Comparison Between Stations Above and Below 6000 Feet

A similar study from the summer of 2004 (Gibson 2004) showed a systematic forecast error by elevation. Forecasts for the higher elevations had a pronounced warm and dry BIAS compared to the lower elevation forecast points. For the 2005 data, verification of the 10 NFDRS stations below 6000 ft (Table 3) were compared to the results of 9 stations located above 6000 ft (Table 4).

Table 3. Element MAE and BIAS for NFDRS stations below 6000 ft

Table 4. Element MAE and BIAS for NFDRS Stations Above 6000 Feet

NWS Forecasts vs. Persistence

MAE was also calculated for a persistence forecast. In this approach, today's observation of each element is considered the forecast for tomorrow. This was done throughout the season. The persistence forecast MAE for all stations in 2005 is presented in Table 5.

Table 5. NWS Forecast vs. Persistence Forecast for All Stations for 2005

Discussion including comparison with the 2004 verification results

All Stations

For all stations (Table 1), MAE averaged approximately 3.9 degrees for temperature, 6.8% for relative humidity, 4.2 mph for wind speed and 1.4% for fuel moisture. Remember that relative humidity has the highest natural variability of these four scalar quantities. RH could easily vary between about 5% and 100% over 70 summer days at observation time. However, the other elements would logically have less variability. Temperature normally will range between 50 and 100 degrees for an average station at NFDRS observation time, wind speed may range from 0-30 mph and fuel moisture has a normal range of about 2% to 30%.

From the 2004 study, MAE averaged approximately 3.9 degrees for temperature, 6.7% for relative humidity, 3.6 mph for wind speed and 1.5% for fuel moisture. This represents an improvement of the 2005 forecasts vs. 2004 of 0% for temperature, -1.5% for relative humidity, -17% for wind speed and 7% for fuel moisture. The improvement was calculated as (MAE 2004 - MAE 2005) / MAE 2004. The most significant change from 2004 to 2005 was the increase in wind speed MAE from 3.6 mph to 4.2 mph.

The 2005 BIAS results for all stations (Table 1) indicate a small warm (.71 degrees) and dry (-.19%) bias in the NWS NFDRS forecasts. This also leads to a small dry bias in fuel moisture (-.61%). The dry bias with respect to fuel moisture makes sense since the forecast data for temperature and relative humidity are closely related to the fuel moisture forecasts. For wind speed, the 2005 NWS forecasts are too high by an average of about 1 mph. Compared to the 2004 results, the BIAS was significantly decreased for 2005. The 2004 study yielded a BIAS of 1.5 degrees for temperature, -1.7% for relative humidity, 1.2 mph for wind speed and -.7% for fuel moisture.

Above and Below 6000 Feet

Separating the forecasts above and below 6000 feet reveals an interesting signal in the data for the summer of 2005. The NWS forecasts were significantly warmer and drier than the verifying observations at the higher elevations. In general, forecasts for the NFDRS stations below 6000 feet were more accurate. MAE was improved below 6000 feet vs. above 6000 feet by 28% for temperature, 39% for relative humidity, 23% for wind speed and 49% for fuel moisture. The improvement was calculated as (MAE above 6000 feet - MAE Below 6000 feet) / MAE above 6000 feet.

The BIAS for 2005 was also smaller for all elements at the stations below 6000 feet. BIAS is virtually zero below 6000 feet for all elements while above 6000 feet a warm, dry and breezy BIAS is evident.

Compared to the 2004 verification results, the 2005 BIAS below 6000 feet was decreased to near zero for temperature and wind speed while the 2004 results had biases of +1 degree and -1%, respectively. For below 6000 feet wind speed BIAS increased from the near zero value in 2004 to about .3 mph in 2005. The below 6000 feet BIAS for fuel moisture was unchanged for 2005, about -.25%.

Above 6000 feet BIAS remained very similar for 2005. A warm, dry and breezy BIAS to the NFDRS forecasts remains evident in the 2005 data. However, the 2005 temperature and relative humidity BIAS is significantly smaller than in 2004.

Largest and Smallest MAE

MAE was averaged for all elements for each station to measure the overall relative accuracy. Aragonite RAWS and Cedar Mountain RAWS were the most accurate forecast points (both below 6000 feet). All of the stations with the most accurate forecasts were in the west deserts of Utah (Zones 420 and 434). Hewinta RAWS and Norway RAWS were the stations receiving the least accurate forecasts overall...both above 8000 feet in the Uinta Mountains.

Comparison to a Persistence Forecast

The NWS forecasts were compared against a persistence forecast with results presented in table 5. For MAE, NWS forecasts provide an improvement vs. the persistence forecasts for all elements except wind speed (WS). This is the same result as in 2004. The improvement (Appendix A, equation 3) of the NWS forecasts vs. persistence for 2005 was 39% for temperature, 29% for relative humidity, -14% for wind speed and 21% for fuel moisture. All of these values represent better forecasts than 2004 where improvement over persistence was measured as 23%, 21%, -16% and 9% respectively (Gibson 2004).

As in 2004, the BIAS for the 2005 persistence forecast is near zero (diagram not shown). This is expected since the 24 hour weather trends should be random over more than 1000 individual NFDRS forecasts.

Conclusions

This analysis of 2005 NWS NFDRS forecasts indicates there continues to be room for improvement. A systematic warm and dry BIAS identified in 2004 has been reduced overall, but most notably below 6000 feet. Mean absolute error was nearly identical to the 2004 results except wind speed MAE increased 1/2 mph for 2005.

Wind speed remains the most difficult individual element to forecast for NFDRS as persistence remains a better forecast for wind speed than the NWS forecasts. Overall, the higher elevations continue to pose more of a forecast challenge than the lower elevations for all elements. As noted in 2004 (Gibson 2004), the degree of sheltering of higher elevation weather stations is considered a significant part of the wind speed errors in the NFDRS forecasts.

Software has been developed to make this analysis much easier. With the new software and databasing approach these statistics can be easily computed during the fire season on a more limited dataset, rather than waiting until the end of the season. We plan to incorporate these analysis into forecast operations using the "real-time" access to verification results.

Appendix A.

    The verification study compared NWS NFDRS forecasts through the analysis of Mean Absolute Error (equation 1) and BIAS (equation 2). These statistics were calculated in a similar manner to those used in the National Weather Service verification program (Meier and Barker 1993).

Equation 1

 

 

 

    In equation 1, Fi and Oi are the ith forecast and observation. N is the total number of observations. In the case of MAE for persistence, today's NFDRS observation becomes the persistence forecasts for tomorrow and verified against the observation from tomorrow

Equation 2

 

Equation 2 is similar to equation 1 and shows that BIAS is just the average error over all forecast.

Equation 3

 

Equation 3 provides improvement of unique forecasts compared to a persistence forecast where MAEr and MAEp are Mean Absolute Error for the NWS NFDRS forecasts and persistence, respectively. An improvement score is 100% is the best possible improvement over persistence, 0% for forecasts that make no improvement over persistence, and negative for forecasts that are less accurate than persistence.

References

Gibson, C. V. 2004: Verification Study of NWSFO SLC NFDRS Forecasts for 2005. http://www.wrh.noaa.gov/slc/projects/ifp/NFDRS2004/nfdrsVer2004.htm

Meier, K. W., and T. W. Barker, 1993: AEV Local Verification for Aviation, Precipitation and Temperature Programs: AV, REL, TEM. NOAA Western Region Computer Programs and Problems NWS WRCP-No. 42. National Oceanic and Atmospheric Administration, U. S. Department of Commerce, 20 pp.

Acknowledgments

I'd like to thank Mr. Randall Graham, Science Operations Officer at NWS SLC for thoughtfully reviewing and suggesting improvements for this paper. I'd also like to thank Student Volunteer Sheldon Baumgartner for help with organizing the data used during this study.