Table 1. MAE and BIAS averaged for all stations.
Verification Study of NWSFO SLC NFDRS Forecasts for 2004
Chris Gibson
NOAA National Weather Service
Salt Lake City, Utah
Contact author through the NWS SLC
Webmaster
A verification study of the summer 2004 National Weather Service (NWS) Salt Lake City Office forecasts for the National Fire Danger Rating System (NFDRS) was completed through an analysis of mean absolute error (MAE) and mean error (BIAS). Forecast were verified for temperature, relative humidity, wind speed and fuel moisture. The forecasts are found to have a warm and dry bias especially at higher elevations. The NWS NFDRS forecasts were found to provide an improvement over a persistence forecast for all elements except wind speed.
Introduction
A verification study was conducted on the National Weather Service (NWS) forecasts from the Salt Lake City NWS office for the National Fire Danger Rating System (NFDRS) during the summer of 2004. The NWS NFDRS forecasts are made each afternoon with a 24 hour lead time for the next afternoon. Forecasts of 4 weather parameters were verified - Temperature, Relative Humidity, Wind Speed and Fuel Moisture. Forecasts and verifying observations were collected for June 1, 2004 to August 30, 2004. Forecasts were created if an observation was received at the WFO for a NFDRS station. Observations are received at the WFO as part of the NFDRS system. For a forecast to be verified a sequence of three things must occur. First, an observation is received for a station through NFDRS for today. Second, a forecast is generated for the station valid tomorrow and the forecast must be successfully ingested into NFDRS. Finally, an observation must be received through NFDRS at the valid time of the NWS forecast.
Nineteen NFDRS forecast points had enough forecasts for a sample, having approximately 70 forecasts each. The remainder of stations for which we made an NFDRS forecast had limited number of observations, or one or more elements observations were corrupt (bad sensor data entered into NFDRS). Two stations were not analyzed due to bad sensor data. Five Mile RAWS had bad relative humidity data and Ray's Valley RAWS had bad temperature data entered into NFDRS. Table 2 lists the NFDRS stations used in the analysis in alphabetical order. All stations are RAWS except for St George, which was a manual station in 2004.
Analysis
A mean absolute error (MAE) and an average error (BIAS) was calculated for each element. MAE and BIAS are defined in appendix A. Graphs of MAE and BIAS for the four verified elements have also been generated for each NFDRS forecast point. The MAE indicates the relative magnitude of the error in the NWS NFDRS forecasts, while the BIAS shows whether the forecasts are on average too dry, too windy, etc. A comparison with a forecast of persistence is also provided, as well as a comparison with stations at different elevations.
Results
Results are presented by diagrams of MAE and BIAS for each station or groups of stations. Each diagram has 4 columns which represent the error for temperature (T), relative humidity (RH), wind speed (WS) and fuel moisture (FM) respectively. The vertical scale is multidimensional - in each diagram it represents temperature (degrees F), percent relative humidity (%), miles per hour (mph), and percent fuel moisture (%). Table 1 presents the averaged results for MAE and BIAS of all 19 station used in the study.
For all stations (Table 1), MAE averaged about 4 degrees for temperature, 6.7% for relative humidity, 3.6 mph for wind speed and 1.5% for fuel moisture. Station average BIAS was about 1.5 degrees and -1.7% RH, 1 mph and -.7 FM. This indicates a slight warm and dry BIAS (for T and RH), which is related to the negative FM BIAS for the NWS NFDRS forecasts - a relationship that fuel moisture is a function of temperature and humidity.
|
Table 1. MAE and BIAS averaged for all stations.
|
|
|
|
|
MAE and BIAS By Station
Table 2 below has links to view MAE and BIAS for each element by each station. Results are viewed by running the mouse over each MAE and BIAS link to show the appropriate histogram in the right hand cell of the table.
|
Table 2. Diagrams of MAE and BIAS for each of the
19 stations used in the study. Stations are organized alphabetically.
From left to right, elements in each diagram are T, RH, WS and FM.
|
 |
||||
|
STATION
|
ELEVATION (ft)
|
FIRE ZONE
|
MAE
|
BIAS
|
|
|
Aragonite
|
5030'
|
420
|
|||
|
Bear River
|
8536'
|
427
|
|||
|
Black Cedar
|
6480'
|
434
|
|||
|
Cedar Mountain
|
4650'
|
420
|
|||
|
Clifton Flat
|
6384'
|
420
|
|||
|
Hewinta
|
9186'
|
427
|
|||
|
Horse Hollow
|
6010'
|
434
|
|||
|
Horse Ridge
|
8480'
|
429
|
|||
|
Lost Creek
|
7490'
|
436
|
|||
|
Mud Springs
|
5902'
|
434
|
|||
|
Norway
|
8280'
|
427
|
|||
|
Pleasant Grove
|
5200'
|
424
|
|||
|
Rosebud
|
4987'
|
420
|
|||
|
Sevier Reservoir
|
5369'
|
433
|
|||
|
Signal Peak
|
8792'
|
436
|
|||
|
St George
|
2650'
|
439
|
|||
|
Tule Valley
|
5200'
|
434
|
|||
|
Vernon
|
5639'
|
420
|
|||
|
Yellowstone
|
7800'
|
428
|
|||
Comparison Between Stations Above and Below 6000 Feet
In an attempt to show possible systematic forecast errors by elevation, verification of the nine NFDRS stations below 6000 ft (Table 3) were compared to the results of 10 stations located above 6000 ft (Table 4).
|
Table 3. Element MAE and BIAS for NFDRS stations
below 6000 ft
|
|
 |
 |
|
Table 4. Element MAE and BIAS for NFDRS Stations
Above 6000 Feet
|
|
 |
 |
Overall Station Accuracy Analysis
An analysis was made to determine which stations were the most accurate and least accurate overall, and to look at any spatial patterns in the error. MAE for the four elements in the study (T,RH,WS,FM) were averaged for each station to produce a Station Average MAE or SA-MAE. The stations with the smallest element SA-MAE had the best overall accuracy. Table 5 lists the stations from lowest to highest SA-MAE. The graphic indicates the geographic distribution of the smallest and largest mean absolute error. (NOTE: BIAS is not shown since errors can be inversely related and thus results less meaningful. In other words, alarge positive temperature bias is canceled by a large negative relative humidity bias, resulting in a combined bias near zero.)
|
Table 5. Stations organized by SA-MAE. Small circles
represent more accurate forecasts while larger circles are least accurate.
|
||||
|
AVG MAE
|
ELEVATION (ft)
|
FIRE ZONE
|
||
|
Aragonite
|
2.375
|
5030
|
420
|
 |
|
Cedar Mountain
|
2.835
|
4650
|
420
|
|
|
Sevier Reservoir
|
2.885
|
5369
|
433
|
|
|
Rosebud
|
3.0275
|
4987
|
420
|
|
|
Black Cedar
|
3.1625
|
6480
|
434
|
|
|
Pleasant Grove
|
3.2075
|
5200
|
424
|
|
|
Tule Valley
|
3.255
|
5200
|
434
|
|
|
Vernon
|
3.4175
|
5639
|
420
|
|
|
Horse Hollow
|
3.4225
|
6010
|
434
|
|
|
St George
|
3.5875
|
2650
|
439
|
|
|
Clifton
|
3.9275
|
6384
|
420
|
|
|
Mud Spring
|
4.0375
|
5902
|
434
|
|
|
Horse Ridge
|
4.0625
|
8480
|
429
|
|
|
Norway
|
4.76
|
8280
|
427
|
|
|
Lost Creek
|
4.815
|
7490
|
436
|
|
|
Signal Peak
|
4.8275
|
8792
|
436
|
|
|
Yellowstone
|
5.29
|
7800
|
428
|
|
|
Bear River
|
5.4275
|
8536
|
427
|
|
|
Hewinta
|
6.6975
|
9186
|
427
|
|
NWS Forecasts vs. Persistence
MAE was also calculated for a persistence forecast. In this approach, today's observation of each element is considered the forecast for tomorrow. This was done throughout the season. The persistence forecast MAE for all stations averaged together is presented in Table 6.
|
Table 6. NWS Forecast vs. Persistence Forecast for
All Stations
|
 |
Discussion
All Stations
For all stations (Table 1), MAE averaged approximately 3.9 degrees for temperature, 6.7% for relative humidity, 3.6 mph for wind speed and 1.5% for fuel moisture. Remember that relative humidity has the highest natural variability of these four scalar quantities. RH could easily vary between about 5% and 100% over 70 summer days at observation time. However, the other elements would logically have less variability. Temperature normally will range between 50 and 100 degrees for an average station at NFDRS observation time, wind speed may range from 0-30 and fuel moisture has a normal range of about 2% to 30%.
The BIAS results for all stations (Table 1) indicate a small warm and dry bias in the NWS NFDRS forecasts. This also leads to a small dry bias in fuel moisture. This makes sense since the forecast data for temperature and relative humidity are closely related to the fuel moisture forecasts. For wind speed, the NWS forecasts are too high by an average of 1 mph.
Above and Below 6000 Feet
Separating the forecasts above and below 6000 feet reveals an interesting signal in the data. The NWS forecasts were significantly warmer and drier at the higher elevations. In general, forecasts for the NFDRS stations below 6000 feet were more accurate. MAE was improved below 6000 feet vs. above 6000 feet by 29% for temperature, 38% for relative humidity, 11% for wind speed and 50% for fuel moisture. The improvement was calculated as (MAE above 6000 feet - MAE Below 6000 feet) / MAE above 6000 feet.
The BIAS was also smaller for all elements at the stations below 6000 feet. Most notable is the BIAS near 0 mph for Wind Speed below 6000 feet. The spatial pattern of average MAE (Table 5 Graphic) is dramatic with a smaller MAE (more accuracy) at lower elevations of western Utah and a larger MAE (less accuracy) at higher elevations of central Utah and the Uinta Mountains.
Largest and Smallest MAE
Overall, Aragonite RAWS was the most accurate forecast point. Aragonite had the smallest MAE of all stations for RH, WS and FM. Hewinta RAWS was the least accurate forecast point. Hewinta had the largest MAE of all stations for RH and WS. Table 5 lists the stations from lowest to highest SA-MAE.
Comparison to a Persistence Forecast
The NWS forecasts were compared against a persistence forecast with results presented in table 6. For MAE (Table 6 left hand diagram), NWS forecasts provide an improvement vs. the persistence forecasts for all elements except wind speed (WS). The improvement (Appendix A, equation 3) of the NWS forecasts vs. persistence was 23% for temperature, 21% for relative humidity, -16% for wind speed and 9% for fuel moisture.
The BIAS for the persistence forecast is near zero (diagram not shown). This is expected since the 24 hour weather trends should be random over ~1100 NFDRS forecasts.
Conclusions
This analysis of NWS NFDRS forecasts indicates there is room for improvement. A systematic warm and dry BIAS has been identified as well as a positive BIAS in wind speed. Our forecast methodology will be critically examined as we proceed into the 2005 fire season with the hope of reducing the systematic BIAS in temperature and relative humidity (and the related dry BIAS in fuel moisture).
Through the analysis of MAE, wind speed from the NWS forecasts was slightly less accurate than a forecast of persistence. The NFDRS wind speed forecast is a particularly difficult. This is because the observation is one 10-minute average observation per day. An average of wind speed throughout the afternoon, or for a number of hours, will likely be more representative of the character of the wind speed for an afternoon than one observation. Also, sheltering of RAWS by vegetation is a problem at higher elevations. For example, in fire weather zone 427, Bear River and Hewinta RAWS are quite sheltered by nearby vegetation. Even on a windy day at high elevations, these two RAWS rarely report sustained wind speeds above 5 mph for the single afternoon NFDRS observation. For Hewinta, 63 forecasts were verified for June through July of 2004. Three observations had a wind speed higher than 5 mph (6 mph July 7th, 7 mph August 5th, 6 mph August 21st) or 4.7% of the observations. This can be compared with Aragonite RAWS, a relatively exposed RAWS at much lower elevation where of 65 verified observations 41 (63%) had wind speeds observed at greater than 5 mph, with a maximum of 17 mph observed on both July 9th and August 25th.
These results provide a baseline of NWS NFDRS forecast accuracy and targets for improvement. We plan to repeat this study on a seasonal basis to evaluate NWS NFDRS forecasts. We also hope to develop web based software which will allow these type of data to be generated anytime during the season for a single station, or groups of stations.
Appendix A.
The verification study compared NWS NFDRS forecasts through the analysis of Mean Absolute Error (equation 1) and BIAS (equation 2). These statistics were calculated in a similar manner to those used in the National Weather Service verification program (Meier and Barker 1993).
Equation 1
In equation 1, Fi and Oi are the ith forecast and observation. N is the total number of observations. In the case of MAE for persistence, today's NFDRS observation becomes the persistence forecasts for tomorrow and verified against the observation from tomorrow
Equation 2
 |
Equation 2 is similar to equation 1 and shows that BIAS is just the average error over all forecast.
Equation 3
 |
Equation 3 provides improvement of unique forecasts compared to a persistence forecast where MAEr and MAEp are Mean Absolute Error for the NWS NFDRS forecasts and persistence, respectively. An improvement score is 100% is the best possible improvement over persistence, 0% for forecasts that make no improvement over persistence, and negative for forecasts that are less accurate than persistence.
References
Meier, K. W., and T. W. Barker, 1993: AEV Local Verification for Aviation, Precipitation and Temperature Programs: AV, REL, TEM. NOAA Western Region Computer Programs and Problems NWS WRCP-No. 42. National Oceanic and Atmospheric Administration, U. S. Department of Commerce, 20 pp.
Acknowledgments
I'd like to thank Mr. Mark Jackson, Science Operations Officer at NWS SLC for thoughtfully reviewing and suggesting improvements for this paper.
