WESTERN REGION TECHNICAL ATTACHMENT
NO. 96-33
December 24, 1996
USE OF THE LDM FOR MAINTAINING A NWS WEBPAGE
David Zaff - NWSFO Salt Lake City, Utah
The Salt Lake City (SLC) forecast office, along with many other NWS offices, has joined the Internet community through a webpage on the World Wide Web (WWW). What differentiates the SLC webpage from most other NWS pages is that SLC has total control over most products available on its webpage. This is because we are not simply linking to other locations that have SLC weather information; we supply the data. Webpage maintenance is accomplished through the UNIDATA Local Data Manager (LDM), which runs c-shell scripts and C programs to work with incoming AFOS files, and in the end uses File Transfer Protocol (FTP) to update the site at WRH. This configuration serves several important functions: 1.) Real-time NWS data and forecasts generated by the SLC office are placed directly at the website location. These products are not dependent on the reliability of other distant and often heavily used private/university weather servers. 2.) SLC can make available local products that are not broadly disseminated, and are thus not available on other servers. 3) SLC can put any type of UNIX based product on the webpage, including real-time graphics, such as images of our mesonet, weather video camera sites, and satellite data. 4.) The page is dynamic; for example, watches and warnings issued will cause the webpage to be rewritten, highlighting statements as blinking links until their expiration time. 5.) Many SLC forecasters now use the page as a forecasting tool within the office. This is due to the "point and click" nature of web browsers, making data more accessible than through conventional systems. The above configuration has also allowed SLC to experiment with combining satellite, radar, and video camera imagery, as well as a time series of automated weather observations (ASOS/AWOS) and nowcasts. These experimental products are being used to develop a real-time "nowcast" page and the "total observation concept". SLC has also developed a METAR decoder to better serve both the public and forecaster with easy-to-read column-type observations.
When SLC began designing a webpage, it became immediately apparent that all Internet links would primarily be under the control of whoever was providing the data. This was not desirable since these locations could not be depended on for a continuous real-time data feed. In general, access to these private/university sites is reliable, but during newsworthy weather events, they are often quite slow, or possibly not even available. Hurricane Bertha (July, 1996) caused the University of Purdue to completely shut down its Internet weather server, leaving some NWS webpage sites helpless until another Internet weather provider could be incorporated into the page. The same situation happened with the University of Illinois weather server early in 1996. To avoid this complication, SLC set out to find a way to feed its own page. The solution is to use the Local Data Manager (LDM), the same piece of software Western Region (WR) offices now use to receive gridded model data. The LDM, running much like the AFOS watchdog program, can be configured to run various programs as AFOS files come in to the UNIX system. These programs make HTML compatible versions of AFOS files and send them via FTP to WRH, where the SLC and most other WR office webpages reside. Several SLC web programs designed to be transportable from office to office are accessible from the SLC webpage. In order to use these programs at other sites, some reprogramming, and ideally some C language experience is necessary. However, once some minor modifications are made, the webpage can become an extremely useful tool to both the end user and forecaster. Detailed information on SLC's webpage programs (including installation procedures) can be found in the READ.ME file and within program comments. This TA will talk about some of the pros and cons associated with the LDM-webpage configuration.
The most immediate benefit to having a forecast office feeding a NWS webpage is at the user end. Rather than setting up a page filled with links to other sites, the page can become interactive. SLC has set up several clickable graphics of the state by which the user can get a time-series of decoded surface METAR and RAWS observations, as well as single zone forecasts (rather than having to sift through a large zone forecast product). Information on how to create a clickable image can be found in TA 96-14 (Brewerton and Hallsten).
SLC is proud to be one of the first forecast offices to offer mixed media products, combining both text and graphics. One link, locally called the nowcast page, provides detailed RAMSDIS 4km infrared and 1km visible imagery, as well as the latest nowcast (Fig. 1). Since the nowcast often refers to satellite imagery during active weather, the nowcast page gives the user a better feel for what the forecaster might be trying to say in words. SLC even has the ability to annotate images, circling or highlighting areas of interest if necessary. This would bring new meaning to a nowcast, but the idea has not been implemented. In the event that radar graphics could be disseminated by the NWS, annotation would further enhance the usefulness of this nowcast concept (Fig. 2). As is, the nowcast page has already received rave reviews from users of the SLC website, and typically gets more "hits" than any other link during active weather (automated hits excluded).
Another set of mixed media links can be found in a section of the webpage called the "Total Observation Concept" (Fig. 3). This idea was thought up by SLC Lead Forecaster Larry Burch, the focal point of an experimental video camera project. With the Internet, we can now combine satellite, radar, video camera imagery, as well as a time series of automated weather observations into one location. (Radar graphics cannot legally be disseminated through the Internet by forecast offices, although it is possible. This concept has also been successful, to the point where the NWS is expanding the video camera project to the national level.
Since SLC has control of the files being sent to the webpage, we can manipulate them; that is cut and paste. This has been done for surface observations (METARS/RAWS) and the ZFP. For surface observations, we have set up a program that saves roughly 24 hours of the most recent observations. Thus the user will get a time series of observations, rather than only the most recent one. We have also decoded portions of the METAR, since few within the general public would know how to handle such an observation. The decoded METARs, with decoded data displayed in uniform columns, are also useful to both the public and aviation forecaster. The public forecaster now has a quick means of comparing observed Fahrenheit temperatures to forecast values, while the aviation forecaster can quickly glance through the latest wind observations, as well as ceiling heights and visibilities.
The ZFP has been broken up into individual zone files (Fig. 4). This was done after it was realized that the general public would not want to scan through the entire ZFP product to find the zone forecast pertaining to them. Allowing users to choose the zone appropriate to their location has also been a big plus on our webpage.
In addition to the WRH webpage server designed to be accessed by the general public, SLC has set up its own webpage server at the office to be used by forecasters. This prevents the forecast office from wasting Internet bandwidth space when connecting to the WRH and doesn't contaminate "hit" statistics. Thus, SLC has an "office" page and a "real" page. (Among Internet enthusiasts, the "office" page is called an Intranet, since links aren't really going through the Internet.) The main difference between the two is the observation image (Fig. 5a and Fig. 5b). Here, the "office" page has a detailed graphic including observations from surrounding states. The forecaster can choose between METARS, decoded METARs, and RAWS observations, all linking to files containing at least 24 hours of the most recent set of observations. METAR links show data in a format similar to ALL:CCCMTRXXX in AFOS, but now data can be viewed with a simple click of the mouse. Since this was implemented, a growing number of forecasters have been using the "office" page as a forecasting tool. It is assumed the general public has little interest in METARs or surrounding state information, so an easy to read observation image has been set up for the "real" page. Links from this image provide decoded METARs, although the option to link to METARs is still available.
Because warnings are the most important suite of products issued by the NWS, highlighting these as blinking links seemed a worthy thing to do, since webpages cannot send an alarm to the user. (Actually this IS possible, but beyond the scope of this TA.) Warning links are not shown on the SLC webpage unless one is current. This prevents the user from reading dated information. In addition, up to five current products from the same AFOS pil can appear on the page at one time (the most recently issued version at the top). This was done to accommodate multiple short-lived warnings, where two may be out for different sections of the state. Otherwise, simply having a link to a warning will only provide the user with the latest warning, when more than one may be current. Currently, when a warning is issued, information about which counties or zones is also displayed, as well as the expiration time. SLC has future plans to incorporate a graphic to display the warning area.
One of the problems with feeding the webpage is with UNIX computer loading. If the LDM is overworked, it may not receive files, and subsequently will not send them to the webpage. Computer loading may also cause the LDM to jumble a product, resulting in some very unusual text products. This type of problem may occur if the computer running the LDM is also running GEMPAK scripts and several other computer memory intensive background processes. SLC has been running the program since February 1996 with the above configuration. After streamlining the main webpage program and spreading some of the more intensive background computer tasks out to other UNIX workstations, the missing/garbled data problem is now a rare occurrence.
After SLC's webpage implementation, it was quickly realized that short-fused warnings were not making it into the LDM. It turns out that this was due to old version of the AFOS Product Transfer (APT), the software in-between AFOS and the LDM. Once the APT was upgraded, the problem was solved.
Occasionally, web programs run by the LDM crash, leaving a core file in the LDM directory. This is not a serious problem, but it is worthy of noting. In the event of a crash, the file being sent over from the LDM will not make it to the webpage.
The warning feature, while generally a big success with SLC's webpage, still has several bugs that need to be addressed. A special program has been written to let the forecaster remove a blinking warning link from the page if there is a problem, but this procedure usually falls on the webmaster's shoulders. The warning feature is also site specific; that is the program needs to know how the webpage warning section should look, thus requiring special html coding. It also needs names of counties and zones. Therefore, it is difficult to configure this section of programming to other offices, particularly since each NWS webpage is different.
The last problem is unfortunately inevitable: UNIX systems occasionally crash. For example, if the system the LDM runs on has a problem, the webpage will not be updated, leaving both the "office" and "real" page out of service. Similarly, if the webserver at WRH is down, the public will not have access to the webpage. In this case however, the "office" page will still be up and running.
Having control over webpage files has allowed the SLC forecast office to create an extremely useful weather page for both the general public and the forecaster. We have been able to create easy-to-use graphical interfaces for data retrieval. This is particularly useful for the forecaster, since it places most forms of real-time data into one location. It has enabled us to experiment with graphic products, and allowed us to disseminate data otherwise unavailable from the NWS to anyone with Internet access. Using the LDM, we have also been able to create a dynamic page; one that alerts the user of watches and warnings.
SLC's LDM webpage configuration is run with little intervention, running virtually non-stop since February. It is quite stable; only failing when there are network or local UNIX problems. The Internet may never be considered an official way of disseminating NWS data, nevertheless it can still be an extremely useful service. SLC hopes to continue experimenting with this new way of data dissemination.