Introduction

Results of a comprehensive verification of TAF ceiling and visibility forecasts are presented. The period of verification was 20 November 1997 to 28 February 1998 for Boise, Twin Falls, and Lewiston, in Idaho. All TAFs were prepared by forecasters at NWSFO Boise and verified against 24 hour ASOS METAR observations. This Technical Attachment briefly explains the verification method and then discusses the results. The verification program itself is available as an AFOS program which can be run at any AFOS site.

Verification Method

Verification is done separately for ceiling and visibility. All ceiling and visibility forecasts (both prevailing and conditional) are compared against corresponding MTR observations. The method is designed to measure the meteorological accuracy of the forecast elements without regard to the non-meteorological boundaries VFR, MVFR, IFR, LIFR. Weighted raw scores are given for prevailing forecasts (7 or 0 points), TEMPO (+4 or -4 points), PROB40 (+2 or -2 points),and PROB30 forecasts (+1 or -1 point).

In particular, 7 point prevailing credit is given for all hours in a TAF period when more than half the period is supported by MTR observations that are within half to twice the forecast value. Otherwise, zero prevailing score is given for those hours. For the conditional terms (TEMPO, PROB40, and PROB30), it is only necessary to have one MTR observation within half to twice the forecast value to earn the credit. But points will be deducted (down to a minimum net score of zero) if no observation is within half to twice the forecast value. When prevailing credit has been earned, negative conditional scores will reduce it, down to a minimum total score of zero. On the other hand, when prevailing score is zero, positive conditional scores give partial compensation. Positive conditional scores do not increase positive prevailing scores, and negative conditional scores do not decrease zero prevailing scores. In other words, conditional scores act as insurance for the prevailing forecast. Conditional forecasts are discouraged unless the prevailing forecast needs protection against a possible zero prevailing score.

Conditional scores can also affect each other. The calculation order is as follows: For each hour in the upper matrix of Fig. 1, reading from the left, find the first non-zero score. If it is negative, the Total Raw Score is zero. If positive, the score is reduced by all negative scores further to the right. This results in Total Raw Scores that always range between 0 and 7. For example:

Prevailing	Tempo	PROB40	PROB30	Total Raw Score	(Reason)
7	-4			3	7-4
7	4			7	exclude the 4
7		2	-1	6	7-1 (exclude the 2)
0	4	-2		2	4-2
0	4	2		4	exclude the 2
0	-4			0	minimum score is zero
0	-4	2		0	first score was negative

The complete verification system also features difficulty factors which address how difficult a given forecast is in terms of how low the ceiling or visibility conditions are (which narrows the half to twice windows) and how much they vary from one to the next. Difficulty factors allow TAF verifications to be compared for different weather situations or different sites, i.e., they help normalize the scores. In this article, however, only the raw scores (without difficulty factors included) are examined. Figure 1 shows a complete verification of one TAF.

Results

Figure 2 shows hour by hour deterioration of total ceiling raw scores as the TAF progresses. As explained above, these scores can range from zero to 7. The 24th hour score of 3.5 is about 75% as good as the initial hour score.

Figure 3 shows scoring for prevailing ceiling alone. Comparing this to Fig. 1, it is seen that prevailing ceiling scores comprise almost all of the total ceiling raw score.

Figures 4, 5, and 6and 9, 10, and 119-11 show scoring for conditional terms before the calculations of the Total Raw Scores (which may have excluded some of them, as explained in the previous section).

Figure 4 shows the steep erosion of skill in forecasting TEMPO ceiling. Beyond 12 hours, TEMPO ceiling forecasts show no skill. Note that the range on the vertical axis is only -.2 to 1.0.

Figures 5 and 6 show PROB40 and PROB30 ceiling scores, respectively. (These terms are not allowed in the first 6 hours of the TAF). Slight skill is shown in these scores (note that the axis range on the vertical axis is only -.02 to +.10). Note also that there is little erosion in skill in the later hours.

Figure 7 shows total visibility raw scores. Idaho sites do not have frequent visibility restrictions, and scores are uniformly high.

Figure 8 shows scoring for prevailing visibility alone. These are uniformly high and comprise nearly all of the total visibility raw scores. But comparing Fig. 8 to Fig. 7, it is also seen that prevailing scores average slightly higher than the total scores, which means that in most cases, conditional visibility forecasts hinder rather than help the TAFs.

Figure 9 shows scoring for TEMPO visibility. Benefit to the TAFs is questionable at best.

Note the small range on the vertical-axis.

Figures 10 and 11 show PROB40 and PROB30 visibility scores, respectively. Slight skill is shown in both figures (again note the small range on the vertical axis). Comparing these two figures to Fig. 9 suggests that TEMPO visibility forecasts for fog are quite poor. The PROB groups can only be used for precipitation and thunderstorms, but TEMPO can be used for fog. Since the Tempo visibility scores are generally negative while the PROB40 and PROB30 visibility scores are positive, the problem is apparently due to poor TEMPO fog forecasts.

Summary and Recommendations

The above results can be summarized as follows:

1. There is no significant skill in the use of conditional terms beyond 12 hours.
2. Forecasters do show skill with TEMPO ceiling in the first 12 hours but skill erodes rapidly with time.
3. Forecasting any conditional visibility is of doubtful value. Perhaps this can be explained by a relative lack of guidance for visibility vs guidance for ceiling forecasts. With ceiling forecasts, cloud layers and coverage can be gauged from levels where high RH is forecast. There is no corresponding guidance for visibility.
4. Forecasters should remember that at least one supporting observation is needed to earn credit for a conditional forecast. They should make the forecast period long enough to allow for one MTR to verify it. On the other hand, if forecast periods are too long, it is possible to have no prevailing weather at all, e.g., it might happen that a third of the period is clear, the next third has ceiling 5000 feet, and the last third has ceiling 500 feet. Then there is no prevailing weather (weather that lasts for more than half of the forecast period), and prevailing credit can never be earned. The solution here is to reduce to length of the forecast period. If they are short enough prevailing conditions will always occur.
5. Minimize the use of conditional forecasts. Use them only to protect the prevailing forecast, or to show a specific alternative to prevailing conditions.

References

Colin, L.R., 1993: Comprehensive verification of terminal forecast ceiling and visibility, 5th AMS Conference on Aviation Weather Systems, Vienna, VA., pp. 245-248.

Colin, L.R., 1995: Numerical verification of conditional terms in NWS terminal forecasts, 6th AMS Conference on Aviation Weather Systems, Dallas, TX, pp. 225-229.

Colin, L.R., 1995: FTFIND (now known as TAFIND). NOAA Western Region Computer Program, WRCP 63, National Oceanic and Atmospheric Administration, U.S. Department of Commerce, 11 pp.