Verification Study of NWSFO SLC NFDRS Forecasts for 2006

Chris Gibson
NOAA National Weather Service
Salt Lake City, Utah
Contact author through the NWS SLC Webmaster

A verification study of the summer 2006 National Weather Service Salt Lake City Office (NWS-SLC) forecasts for the National Fire Danger Rating System (NFDRS) was completed through an analysis of mean absolute error (MAE) and mean error (BIAS). Forecasts were verified for temperature, relative humidity, wind speed and fuel moisture. Results are compared to similar studies from 2004 and 2005.

Introduction

A verification study was conducted on the National Weather Service (NWS) forecasts from the Salt Lake City NWS weather forecast office (NWS-SLC) for the National Fire Danger Rating System during the summer of 2006. The NWS NFDRS forecasts are made each afternoon with a 24 hour lead time for the next afternoon. Forecasts of 4 weather parameters were verified - Temperature, Relative Humidity, Wind Speed and Fuel Moisture. As in previous years, forecasts and verifying observations were collected for June 1 to August 31. Forecasts were created if an observation was received at the WFO for a NFDRS station. Observations are received at the WFO as part of the NFDRS system. For a forecast to be verified, a sequence of three things must occur. First, an observation is received for a station through NFDRS for today. Second, a forecast is generated for the station valid tomorrow and the forecast must be successfully ingested into NFDRS. Finally, an observation must be received through NFDRS at the valid time of the NWS forecast.

Stations with less than 30 forecasts verified were not included in the study for 2006. Three days are missing from the verification dataset for all stations: June 28, August 2 and August 9. Brimstone reservoir RAWS was removed from the analysis due to low wind speed observations. The sensor appears to be damaged.

For 2006, 29 stations had enough forecasts for a sample, averaging 79 forecasts each. The number of forecasts varied from 46 at Sevier Bridge Reservoir to 88 at Bues Canyon and Bear River. A total of 2223 forecasts where verified for this analysis, up from 1350 observations for 19 stations verified in 2005. Table 2 lists the NFDRS stations used in the analysis in alphabetical order. All stations are RAWS.

Analysis

A mean absolute error (MAE) and an average error (BIAS) was calculated for each element. MAE and BIAS are defined in appendix A. Graphs of MAE and BIAS for the four verified elements have also been generated for each NFDRS forecast point. The MAE indicates the relative magnitude of the error in the NWS NFDRS forecasts, while the BIAS shows whether the forecasts are on average too dry, too windy, etc. A comparison with a forecast of persistence is also provided, as well as a comparison with stations at different elevations.

Results

Results are presented by diagrams of MAE and BIAS for each station or for groups of stations. Each diagram has 4 columns which represent the error for temperature (T), relative humidity (RH), wind speed (WS) and fuel moisture (FM) respectively. The vertical scale is multidimensional - in each diagram it represents temperature (degrees F), percent relative humidity (%), miles per hour (mph), and percent fuel moisture (%). Table 1 presents the averaged results for MAE and BIAS of all 19 stations used in the study.

Table 1. MAE and BIAS averaged for all stations.

MAE and BIAS By Station

Table 2 below has links to view MAE and BIAS for 2006 for each element by each station. Results are viewed by running the mouse over each MAE and BIAS link to show the appropriate histogram in the right hand cell of the table.

Table 2. Diagrams of MAE and BIAS for each of the 29 stations used in the study. Stations are organized alphabetically. From left to right, elements in each diagram are T, RH, WS and FM.
STATION
ELEVATION (ft)
FIRE ZONE
MAE
BIAS
Agua
8890'
437
MAE
BIAS
Aragonite
5030'
420
MAE
BIAS
Asay
8100'
437
MAE
BIAS
Badger Spring
3990'
437
MAE
BIAS
Bear River
8536'
427
MAE
BIAS
Black Cedar
6490'
434
MAE
BIAS
Bryce Canyon
7855'
437
MAE
BIAS
Buck Flat
8000'
437
MAE
BIAS
Bues Canyon
5100'
422
MAE
BIAS
Cedar Mountain
4650'
420
MAE
BIAS
Clifton Flat
6384'
420
MAE
BIAS
Enterprise
5340'
437
MAE
BIAS
Hewinta
9186'
427
MAE
BIAS
Horse Hollow
6010'
434
MAE
BIAS
Larb Hollow
8450'
437
MAE
BIAS
Lost Creek
7490'
436
MAE
BIAS
Mud Springs
5902'
434
MAE
BIAS
Norway
8280'
427
MAE
BIAS
Otter Creek
7160'
426
MAE
BIAS
Pleasant Grove
5200'
424
MAE
BIAS
Ray's Valley
7300'
427
MAE
BIAS
Rosebud
4987'
420
MAE
BIAS
Sevier Reservoir
5369'
433
MAE
BIAS
Signal Peak
8792'
436
MAE
BIAS
Telegraph Flat
5460'
440
MAE
BIAS
Tom Best Spring
7500'
437
MAE
BIAS
Tule Valley
5200'
434
MAE
BIAS
Vernon
5639'
420
MAE
BIAS
White Reef
3440'
439
MAE
BIAS
  

Comparison Between Stations Above and Below 6000 Feet

Similar studies from 2004 (Gibson 2004) and 2005 (Gibson 2005) showed a systematic forecast error by elevation. Forecasts for the higher elevations had a pronounced warm and dry BIAS compared to the lower elevation forecast points. For the 2006 study, verification of the 13 NFDRS stations below 6000 ft (Table 3) were compared to the results of 16 stations located above 6000 ft (Table 4).

Table 3. Element MAE and BIAS for NFDRS stations below 6000 ft

Table 4. Element MAE and BIAS for NFDRS Stations Above 6000 Feet

NWS Forecasts vs. Persistence

MAE was also calculated for a persistence forecast. In this approach, today's observation of each element is considered the forecast for tomorrow. This was done throughout the season. The persistence forecast MAE for all stations in 2006 is presented in Table 5.

Table 5. NWS Forecast vs. Persistence Forecast for All Stations for 2006

Comparison for 2004, 2005 and 2006

This is the third year this study has been completed for the NWS-SLC NFDRS forecasts. MAE and BIAS for all stations was compared for the three years and 4 elements. Results are shown in table 6.

Table 6. Element MAE and BIAS for 2004, 2005 and 2006
3 Year BIAS
3 Year MAE

Discussion including comparison with the 2004 and 2005 verification results

All Stations

For all stations (Table 1), MAE averaged approximately 3.9 degrees for temperature, 6.4% for relative humidity, 3.5 mph for wind speed and 1.5% for fuel moisture. Remember that relative humidity has the highest natural variability of these four scalar quantities. RH could easily vary between about 5% and 100% over 70 summer days at observation time. However, the other elements would logically have less variability. Temperature normally will range between 50 and 100 degrees for an average station at NFDRS observation time, sustained wind speed will typically range from 0-30 mph and fuel moisture has a normal range of about 2% to 30%.

The 2006 BIAS results for all stations (Table 1) indicates a small warm (.78 degrees) and dry (-1.4%) BIAS in the NWS NFDRS forecasts. For wind speed, the 2006 NWS forecasts are too high by an average of about .6 mph.

The BIAS for fuel moisture is very small at -.1%. A bug was found in the tool used to derive fuel moisture from the forecast relative humidity and temperature. The bug was fixed for the 2006 season and the fuel moisture bias improved from -.61% to -.1%, or an improvement of 84% from 2005 to 2006.

Above and Below 6000 Feet

As found in 2004 and 2005, separating the forecasts above and below 6000 feet reveals an interesting signal in the data for the summer of 2006. With the exception of wind speed, forecasts were more accurate, with smaller MAE for stations below 6000 feet than at the higher elevations (Tables 3 and 4). MAE was improved below 6000 feet vs. above 6000 feet by 29% for temperature, 40% for relative humidity, -5% for wind speed and 29% for fuel moisture. The improvement was calculated as (MAE above 6000 feet - MAE Below 6000 feet) / MAE above 6000 feet.

This was the first year that any element (wind speed) exhibited higher MAE (worse forecasts) below 6000 feet vs. above.

The BIAS for 2006 was also smaller for all elements at the stations below 6000 feet. A slight warm and dry BIAS is indicated below 6000 feet, while a warm, dry and breezy BIAS is evident above 6000 feet.

Comparison to a Persistence Forecast

The NWS forecasts were compared against a persistence forecast with results presented in table 5. For MAE, NWS forecasts provide an improvement vs. the persistence forecasts for all elements except wind speed (WS). This is the same result as in 2004 and 2005.

BIAS for the 2006 persistence forecast is near zero (diagram not shown). This is expected since the 24 hour weather trends should be random over more than 2000 individual NFDRS forecasts.

Three Year Comparison

Table 6 presents 2 diagrams which compare MAE and BIAS for the three years this study has been conducted. For 2004, 2005 and 2006 nearly 4500 individual forecasts have been verified. Over this period, MAE has changed little for each element. The calculated BIAS for each element has more variability, but not a clear signal whether there has been a reduction or increase in BIAS in these three years.

Conclusions

This analysis of 2006 NWS NFDRS forecasts indicates very similar pattern in forecast error to the two previous summer forecast periods. Overall, the forecasts remain slightly warm, dry and breezy with larger errors at the higher elevations.

Wind speed remains the most difficult individual element to forecast for NFDRS as persistence remains a better forecast for wind speed than the NWS forecasts. Overall, the higher elevations continue to pose more of a forecast challenge than the lower elevations for all elements. As noted in 2004 (Gibson 2004), the degree of sheltering of higher elevation weather stations is considered a significant part of the wind speed errors in the NFDRS forecasts.

This analysis was greatly accelerated for 2006 due to a comprehensive databasing of the NFDRS forecasts and verifying observations at the WFO SLC. Training was also completed by forecasters during May and June of 2006 to expose them to the systematic error and BIAS evident in the forecasts from the 2004 and 2005 studies, especially the larger errors noted at higher elevations.

Ideas for reducing the NFDRS MAE and BIAS forecast in the future include applying a constantly updated BIAS correction technique to the latest forecasts. Methodology and software for applying such a correction are under development.

Appendix A.

    The verification study compared NWS NFDRS forecasts through the analysis of Mean Absolute Error (equation 1) and BIAS (equation 2). These statistics were calculated in a similar manner to those used in the National Weather Service verification program (Meier and Barker 1993).

Equation 1

 

 

 

    In equation 1, Fi and Oi are the ith forecast and observation. N is the total number of observations. In the case of MAE for persistence, today's NFDRS observation becomes the persistence forecasts for tomorrow and verified against the observation from tomorrow

Equation 2

 

Equation 2 is similar to equation 1 and shows that BIAS is just the average error over all forecast.

Equation 3

 

Equation 3 provides improvement of unique forecasts compared to a persistence forecast where MAEr and MAEp are Mean Absolute Error for the NWS NFDRS forecasts and persistence, respectively. An improvement score of 100% is the best possible improvement over persistence, 0% for forecasts that make no improvement over persistence, and negative for forecasts that are less accurate than persistence.

References

Gibson, C. V. 2004: Verification Study of NWSFO SLC NFDRS Forecasts for 2004. http://www.wrh.noaa.gov/slc/projects/ifp/NFDRS2004/nfdrsVer2004.htm

Gibson, C. V. 2005: Verification Study of NWSFO SLC NFDRS Forecasts for 2005. http://www.wrh.noaa.gov/slc/projects/ifp/NFDRS2005/nfdrsVer2005.htm

Meier, K. W., and T. W. Barker, 1993: AEV Local Verification for Aviation, Precipitation and Temperature Programs: AV, REL, TEM. NOAA Western Region Computer Programs and Problems NWS WRCP-No. 42. National Oceanic and Atmospheric Administration, U. S. Department of Commerce, 20 pp.

Acknowledgments

I'd like to thank Mr. Randall Graham, Science Operations Officer at NWS SLC and Mark Struthwolf, Fire Weather Focal Point for reviewing and suggesting improvements for this paper.