Overview
Forecast offices within the Western Region have been receiving gridded model data, digital satellite imagery, and other forms of model/observed data via the Western Region's Wide Area Network (WAN) for nearly two years. The data are being distributed using the Local Data Manager (LDM) software package developed by Unidata several years ago.
Although this data feed is very dependable, there are occasions when problems at a local site or the regional server contribute to missing data. Rather than accept missing data, a series of procedures was developed to allow the forecaster to proactively alleviate problems created by missing data. These procedures were developed so that the forecaster would not have to remember how to fix particular problems or possess substantial knowledge of Unix, but simply consult a flow chart and run the indicated choice from a pop-up menu by answering a few simple questions.
The series of procedures detailed below do not require the forecaster to know separate passwords or manually edit files, but simply run particular scripts to complete the desired action to fix missing data problems. In addition, these procedures also provide safeguards against local system problems that may contribute to data ingest problems.
System Problems
On occasion, local system problems may kill processes that are critical for the ingest of data (LDM) and the generation of graphics for use with Ntrans (NAG). To alleviate this, two separate safeguards have been put in place: 1) Start LDM and NAG after reboot, and 2) periodically check for LDM/NAG processes and restart if necessary.
Starting LDM and NAG after a system reboot is handled by entries in the localrc file Attachment 1 contained in the /usr/local/bin directory, as documented by Croft (1996). The entries in the localrc file replicate the interactive process of reinitializing and starting the LDM and NAG processes. Although this procedure works very well and in most instances, occasionally the LDM and NAG do not get restarted after a system reboot. An "INCLUDE" statement to call the localrc file must be included in the rc file (located in /usr/bin under HPUX 9.x and /usr/sbin under HPUX 10.x). To ensure that LDM and NAG are restarted, a script Attachment 2 is run in the cron every 30 minutes (as user Gempak) to determine if these processes are running and to restart them if they are not running. This ensures that the LDM and NAG processes do not remain inactive for a prolonged period.
Determining Necessary Action
A flowchart Figure 1 was developed to assist the forecasters in determining the appropriate action for various situations and is accessible through the office intranet. For instance, if the data may be on the local system but Ntrans graphics are not available, the forecaster would choose the pop-up menu to "Generate Graphics." Or, if the Eta model raw data were missing for the zero through 18 hour forecasts, the forecaster would first go to the WR Data Status homepage and check to see if the data was on the WR Server. If so, then the forecaster would choose the pop-up menu to "Retrieve Data and Generate Graphics". However, if the data was not on the WR Server the forecaster could also check to see if the data was on the OSO Server via the WR Data Status Homepage.
Missing Data and Data Processing Problems
With the large volume of gridded model data obtained via the WR WAN and AFOS data via the AFOS Protocol Translator (APT), it is not surprising that some data are missed or not properly processed. However, this is not necessarily a satisfying reason for the operational forecaster. In order to ensure the forecaster has some control over the data received, several procedures have been implemented to assist the forecaster in monitoring the local data feed and to actively solve data problems on the workstations.
Monitoring Gridded Data Ingest and Processing
The forecaster must be able to quickly determine which raw gridded data has been received and resides on the local system, as well as determine which data has been processed into graphics for Ntrans. This is accomplished through the use of a script which determines which raw gridded model files reside on the local system (Barker, 1996) and also identifies the number of files waiting to be processed. This script Attachment 3 is run from a pop-up menu and refreshes every two minutes until terminated by the user.
An example of the output of this script appears as Attachment 4 . The listing provides the model, cycle and forecast times for which raw gridded files exist on the local system, which allows the forecaster to quickly determine if any files are missing from the data set. The listing also provides the model, cycle and forecast times that have been processed into graphics, which allows the forecaster to determine the progress of the processing. The complete listing will refresh itself every two minutes, if left running.
Generating Graphics
In the event the graphics for a specified model(s) were not generated from the raw data or were generated out of order (i.e. the raw data are on the local system), a script Attachment 5 may be run from a pop-up menu to delete any existing meta files and regenerate meta files for the specified model(s). This is accomplished through a series of questions that the user answers from which the script spawns a series of separate processes. The initial scriptAttachment 5 executes a C program Attachment 6 which runs a C-shell script Attachment 7 as a specified user, in this case as Gempak. The advantage of this method is the username and password of user Gempak is compiled within the C program and thus does not require the user to know the password or pose a security problem. The initial script Attachment 5 gathers input variables, such as the cycle and model(s) for which graphics should be generated. The C program Attachment 6 simply initiates a second script with the privileges of user Gempak. This second scriptAttachment 7 stops NAG, rebuilds the qf.q file with the requested model/cycle GRIB filenames, and restarts NAG which begins processing the appropriate GRIB files.
Get Raw Data and Generate Graphics
Occasionally, local problems may prevent some gridded model files from being received. Rather than live without this data, the forecaster can retrieve the raw gridded data files and generate the graphics for use with Ntrans, through the use of an interactive script Attachment 8 that prompts the forecaster for the data of interest. Based on the input from the forecaster, this script will initiate the retrieval of the raw gridded data files from the WR Server and generate the model graphics for Ntrans. This is accomplished by a similar process as described above for Generating Graphics Attachment 9, Attachment 10 .
Conclusion
The steps outlined above allow the forecaster to ensure viewable data (either through Ntrans or GARP) in situations when a data problem can be fixed locally. These local problems can be things like: data missed when a system was shutdown/rebooted; out of order graphics; LDM/NAG stopped for various reasons. The major advantages of this methodology is that the forecasters can fix many data problems very simply and all of these procedures are accomplished by users with a variety of computer skills (from little to many) by answering a few simple questions through scripts accessed from pop-up menus. This is all accomplished without having the forecasters log-in as the Gempak user (or local applications supervisor), which provides some security against inadvertently deleting important files.