WESTERN REGION TECHNICAL ATTACHMENT
NO. 99-20
September 28, 1999

A COMPOSITE LOOK AT WEATHER SURVEYS:
USING SEVERAL WEATHER SURVEYS TO GET AN ESTIMATE
OF PUBLIC OPINION

Larry Van Bussum, Jr., - NWSO Sacramento, CA

Introduction

The science of forecasting has improved by leaps and bounds in the last ten years. Weather forecasts are more accurate and timely than in any other time in history. However, how to get information across to the general public in terms that they understand has always been a subject for debate. There have been several studies and surveys done over the last 20 years covering many aspects of forecast terminology and presentation. Unfortunately, in every case this author has studied (Sink (1995), Murphy and Curtis (1985), Saviers and Van Bussum (1997), Racy (1998), Krenz and Evans (1993), Shea (1996), Murphy et al (1980),) basic survey techniques were not followed, which resulted in a true random sample not being surveyed. This introduces biases in the data, which, however small, prevent one from using the results from any of these surveys to hypothesize how the public, either at one particular city or the nation as a whole, thinks. For example, one could say, "According to the results of these surveys, of the people surveyed, they prefer X." One could not say however, "According to the results of these surveys, 90% of the public prefer X." This is an important difference. Many of the studies have used similar questions and have produced similar results. The results of the surveys are not useless, as they do serve an important purpose in getting a general idea what the public wants or understands, and the surveys serve as great test vehicles for a properly conducted survey. The sheer number of responses, and the fact that the results of each of these surveys were gathered differently, will approach the closest thing to a properly done random sample survey.

The twelve questions in the following section appeared in two or more of the seven surveys done over the last twenty years that were studied for this paper. The wording might have been slightly different in each separate survey, but were close enough to each other that the differences were deemed trivial. One must keep in mind, though, that these tiny differences also introduce error to the overall results of this report as well. The results cannot be used to say, "NN% of the public thinks X." With the number of responses and the spread of cities in which these questions were asked, one could say with some confidence, "Of the people surveyed, NN% thought X, which is probably a good estimate of the opinion of the public at large." The total responses to each question were tallied and the percentage of the total responses were calculated for each answer. A discussion of these results follows. Appendix A contains a list of the surveys studied, the date of the paper written, a brief description of where and how the survey was done, and the number of respondents

General Information

Two questions were asked in two surveys that dealt with how people receive their weather forecasts, and how often they pay attention to the forecasts.

1. What is your primary source of weather information?

863 total responses from Sink (1995) and Saviers and Van Bussum (1997)

Some interesting things can be seen in this first question. First of all, nearly 65% get their weather from local media sources, namely radio and TV. In the survey, both radio and TV received nearly equal percentage of responses. Sink's survey showed a larger bias towards TV while Saviers' and Van Bussum's survey showed a bias towards radio. This is most likely due to the fact that the community that Sink did her survey, Raleigh, North Carolina, had several local TV stations, while the community that Saviers and Van Bussum did their survey, Juneau, Alaska, did not have any local TV stations. Many people who commute to work in larger cities most likely listen to the radio on their way to and from work, and many stations have frequent weather reports on their programs. Sink's results would probably be more representative of larger cities. Even though many respondents say they get their primary weather information from local TV, they most likely hear the weather report more often over the radio.

Another interesting thing to note is that more respondents chose "Looking out the window" more often than the last 5 responses combined. Also, the number of respondents that see or hear National Weather Service products exactly as they are written (the "Local Weather" section of The Weather Channel, NOAA Weather Radio, and possibly the Internet) is 17.6%. Only 3.4% (NOAA Weather Radio) is from a source known to include National Weather Service products exclusively, as some respondents might not pay attention to the "Local Forecast" portion of The Weather Channel, and some respondents get their Internet weather from other commercial weather vendors.

When these surveys were done, the Internet was just catching on. And was not as well known as it is today. Therefore, one would expect this number to go up as time goes by.

2. How often during the week to you pay attention to the weather forecasts?

796 total responses from Sink (1995) and Saviers and Van Bussum (1997)

From these results, most of the people surveyed paid attention to the forecast some time during the week. The top choice was 7 days a week, second highest being 1-3 days, and very few respondents never pay attention to the forecast. The high number of responses in the "1-3 days" category could be explained if respondents were paying attention to a weekend forecast.

Probability of Precipitation (POPs)

Most of the questions similar to the several surveys dealt with Probability of Precipitation forecasts (POPs). POPs are probably among the most debated elements in National Weather Service forecasts. What does a POP mean? How should it be applied? Should it be included in public forecasts? If so, how should it be done, verbally or numerically? The following questions and results may shed more light on these questions.

3. At what percent chance of precipitation would you consider altering plans or adjusting to accommodate precipitation? (Note: the responses are broken down into categories for discussion, but only a percentage, 0% through 100%, was offered as possible answers.)

1018 total responses from Sink (1995), Saviers and Van Bussum (1997) and Hayes (1997) and Racy (1998)

The results of this question are encouraging in that the people surveyed took more action as the chance of precipitation increased. The responses peaked at the "chance" and "likely" category, with a slight drop off for categorical POPs. These respondents seemed to take action at the appropriate percentage, without much sensitivity on the low end, a heightened sensitivity in the middle, and a drop off at the high end showing that most respondents probably took a precipitation forecast seriously in their planning.

The next set of questions asked respondents to define POPs.

4. On the morning weather forecast, the meteorologist says, "Rain likely today." You understand this to mean:

A. Rain is likely to occur most of the day. (7.9%)
B. At a specific point in the forecast area (for example, your house), rain is likely. (9.5%)
C. Rain is likely somewhere in the forecast area during the day. (52.0%)
D. Rain is likely to occur over most of the forecast area during the day. (30.6%)

844 total responses from Murphy et al. (1980), Sink (1995), and Saviers and Van Bussum (1997)

5. Today the meteorologist says, "...chance of rain 60%." You understand this to mean:

A. Rain will occur 60% of the day. (1.2%)
B. At a specific point in the forecast area, (for example, your house), there is a 60% chance of rain occurring. (15.0%)
C. There is a 60% chance that rain will occur somewhere in the forecast area during the day. (80.9%)
D. 60% of the forecast area will receive rain and 40% will not. (2.9%)

833 total responses from Murphy et al. (1980), Sink (1995) and Saviers and Van Bussum (1997)

The first response is a temporal forecast, the second is a point forecast (also the correct response), the third is an area probability forecast and the fourth is an areal coverage forecast. The reason "C" is not a correct answer is that it is not a point forecast but an area probability forecast. When using POPs in zone forecasts, an average point probability is used, not an area probability.

The results of this question were some cause for alarm when Murphy did his study in 1980, and since then the results have only gotten worse. Each survey done since, using the same question, has had less response to answer "B" than the previous one. When cross referencing both questions (in other words, only considering revealing the individuals who responded "B" for both questions, therefore eliminating people who guessed the true amount of people surveyed who actually knew the answer), the results were dismal. Sink's cross reference produced only 7% out of about 473 respondents who knew the answer, and in Saviers' and Van Bussum's survey, only 1 respondent out of about 285 people knew the answer. It is safe to assume that of those surveyed, practically no one knew the correct answer to what a POP is. In defense of those surveyed, when personally asking fellow meteorologists, the results do not seem to be much better.

The encouraging thing about these results, though, is the fact that the answer closest to the correct one, "C" (area probability), was the most popular response by a wide margin. The National Weather Service officially determines, at least on paper, a POP over a wide area by taking the average point probability for a certain location. For example, in Sacramento there are at least 3 observing sites, one downtown, one at Executive Airport, and one at International Airport. If the POP for downtown was 30%, the one for Executive Airport was 40% and the one for International Airport was 50%, then the POP for the Sacramento area, based on these three individual POPs, should be 40%. So, although the strict definition of a POP, according to the Weather Service Operations Manual chapter C-11, is answer "B", the average point probability used to determine a POP for a wide area is similar (although not exactly like) answer "C".

Saviers and Van Bussum (1997) pointed out that one reason for the possibility that few people picked answer "B" was the fact that "(for example, your house)" was in parenthesis, which may have somehow tricked people into not picking that response since it was the only answer available with parenthesis in it. Although a valid point, one questions why people in Murphy et al (1980), who used the same exact punctuation in his study, did better, with 39% of the respondents answering "B" to this question, while those in Sink (1995) and Saviers and Van Bussum (1997) did so poorly. A better written answer "B" may be needed. This is covered in Appendix 1 and 2 of Saviers and Van Bussum (1997.)

The next four questions asked respondent's reaction to various POP forecasts. These were paired with a verbal and numerical question for each category.

6. The forecaster said there was a "...chance of rain," but it did not rain at your house. What is your opinion of the forecast?

A. Excellent (8.0%)
B. Good (52.7%)
C. Fair (35.4%)
D. Poor (3.9%)

714 total responses from Sink (1995) and Saviers and Van Bussum (1997)

7. The forecaster said, "...30% chance of rain today," but it did not rain at your house. What is your opinion of the forecast?

A. Excellent (10.7%)
B. Good (64.8%)
C. Fair (20.8%)
D. Poor (3.6%)

745 total responses from Sink (1995) and Saviers and Van Bussum (1997)

8. What would be your opinion of a forecast that stated, "Rain likely," and it did not rain at your house?

A. Excellent (2.4%)
B. Good (28.1%)
C. Fair (55.6%)
D. Poor (13.9%)

750 total responses from Sink (1995) and Saviers and Van Bussum (1997)

9. What would be your opinion of a forecast that predicted a "70% chance of rain," and it did not rain at your house?

A. Excellent (2.7%)
B. Good (14.9%)
C. Fair (46.4%)
D. Poor (36.1%)

740 total responses from Sink (1995) and Saviers and Van Bussum (1997)

Questions 6 and 7 dealt with a "chance" event, while questions 8 and 9 dealt with a "likely" event. In both sets, the percentage of respondents answering "Excellent" changed little between a verbal and numerical POP. However, there was some notable differences in B, C, and D when going from a verbal to a numerical POP. Those with a favorable response went up in the "chance" event when an actual number was given (from C to B). In the "likely" event, though, opinion dropped dramatically when a number was used. The difference in the "likely" event was much larger and crossed more categories than the response to the "chance" event. This would seem to imply that the term "likely" is not as well understood by the public than the term "chance".

The obvious question would be, "What percentage do people associate with all these terms?" This question was asked by several surveys.

10. Circle the percent probability of precipitation you associate with the following terms.

.............................................................................................................................................................................................

Approximately 738 total responses from Sink (1995), Saviers and Van Bussum (1997), Racy (1998) and Krenz and Evans (1993)

There are some interesting results from this question. When comparing expressions of uncertainty with those in the table in Weather Service Operations Manual (Chapter C-11, 8.3.5) the order is similar in both. The interesting thing to note, though, is that the mean for "Widely Scattered" is the same as the mean for "Scattered". One reason that the mean for "Widely Scattered" is higher than the figure given in the table of the Weather Service Operations Manual (Chapter C-11, 8.3.5), especially considering the results in questions 4 and 5, is that the respondents may be interpreting "Widely Scattered" to mean "spread out over a wide area," instead of the intended meaning, "widely separated." Using proper rounding techniques on the means, "Widely Scattered" and "Few" were the only terms covered in the table in the Weather Service Operations Manual (Chapter C-11, 8.3.5) that were not placed in the proper percentage category, as defined by the table, by the respondents.

Also note that none of the categorical qualifiers, "possible", "occasional", or "periods of," had means above 60%, much less than the 80% needed for categorical terminology. This suggests that when temporal qualifiers are added to a forecast, that the POP is then associated with the temporal qualifier, and not the event itself. An example of this would be a forecast that had "Occasional rain" and another forecast that just had "Rain". The POPs for both these events would be categorical. Now assume the respondent hears that the POPs with the "Rain" forecast is 90%. When you add the term "Occasional" to this event, the mean POP that the respondent thinks of drops nearly in half (48.3%), even though it is still a categorical event. This was the conclusion of Saviers and Van Bussum (1997). The more adjectives you add to a forecast, the more ambiguous the meaning becomes, especially in terms of POPs.

The most ambiguous terms, as determined by standard deviation from the mean, were "Widely Scattered" and "Periods of". Following closely behind was "Occasional" and "Likely". Also, "Likely" had the highest mean score out of all the terms available.

With the results of the previous questions, what, then, does the public want to hear as far as POPs are concerned? That question was posed by several surveys.

11. In which way would you prefer chance of precipitation forecasts to be expressed?

A. Verbally (ex. "Rain likely") (17.7%)
B. Numerically (ex. "Chance of rain 60%) (61.1%)
C. Both (21.2%)

1206 total responses from Sink (1995), Saviers and Van Bussum (1997), Racy (1998), Krenz and Evans (1993) and Shea (1996)

The overwhelming response is numerically. Even though it has been shown that the exact definition of POP was not known by the respondents, their interpretation was close enough to be functional. Also, most people realize that a 60% chance of precipitation is greater than a 40% chance, thus expressing POPs in numerical terms serves an important purpose. The use of both, with care, can serve to reinforce each other, but again one must be aware that the meaning of the POP may be wrongly applied to the qualifier or adjective rather than the event itself.

Sky Cover

The final question covered in this study pertained to sky cover.

12. What percentage of the sky do you expect to be covered by clouds upon hearing the following terms?

............................................................................................................

Approximately 2175 respondents from Curtis and Murphy (1985), Racy (1998), Krenz and Evans (1993)

Notice that the respondents didn't think the sky was completely clear with the terms "Clear" or "Sunny", nor thought the sky was completely cloudy with "Cloudy". Also note that the respondents indicated a 23% sky cover with the term "Fair", while many meteorologists use the term "Fair" to describe a high, thin overcast.

Conclusion

By using a "composite" of several surveys from the last 20 years, several questions about forecast terminology, format and use can be studied and the results from the individual surveys combined to give a national "pseudo-random" sample. Although by definition these results do not represent the public at large, they do give a forecaster an idea on how the public might possibly interpret a forecast, and how they may use it. The results of this paper are functional for use by any forecaster or office across the country as the sample is large enough and diverse enough to give meaningful results. However, this does not replace the need for a properly done survey, both at the local and national level. A properly done national survey would give a proper baseline to establish some standardization of products across the country. Ideally, a person from Florida should be able to read a product from Washington and not be confused about what they are seeing. Properly done local surveys would point out subtle differences in public opinion as a result of local climate and terrain that can then be used to better tailor forecasts for that area.

Acknowledgment

Special thanks to Aimee Saviers and Scott Cunningham for reviewing this paper.

References

Sink, S. A., 1995: Determining the Public's Understanding of Precipitation Forecasts; Results of a Survey. Nat. Wea. Dig., 19-3, 9-15.

Weather Service Operation Manual (WSOM), 95-3, 1995: Part C – Basic/Public Weather Services, Zone and Local Forecasts. C–11.

Murphy, A. H., S. Lichtenstein, B. Fischoff, and R.L. Winkler, 1980: Misinterpretations of Precipitation Probability Forecasts. Bulletin of the American Meteorological Society, 61, 695-701.

Murphy, A. H. and J. C. Curtis, 1985: Public Interpretation and Understanding of Forecast Terminology: Some Results of a Newspaper Survey in Seattle, Washington. Bulletin of the American Meteorological Society, 66, 810–819.

Saviers, A. M. and L. J. Van Bussum, 1997: Juneau Public Questionnaire: Results, Analyses and Conclusions. NOAA Technical Memorandum, NWS AR–44.

Racy, J. P., 1998: How Northeast Indiana and Northwest Ohio Residents Interpret Meteorological Terminology and Services Through NOAA Weather Radio. NOAA Technical Service Publications. NWS CR–05.

Krenz, S. H. and J. S. Evans, 1993: Weather Terms Used in National Weather Service
Forecasts – Does the Public Understand These Terms? A User's Survey. Central Region Highlights. DOC, NOAA, NWS Central Region Headquarters, Kansas City, MO.

Shea, T., 1996: ARX Open House Survey. Personal Correspondence.