Directory Structure

Much of the buoy processing is handled via shell scripts that refer to an environment variable called ${DATA_ROOT} . This determines a base directory from which other necessary subdirectories are found. For the buoy processing, this is set to /data/gomoos/buoy .

The following subdirectories under ${DATA_ROOT} are pertinent to buoy processing and are as follows:

• The bin directory contains all the shell scripts and perl scripts necessary to launch the processing. These scripts are generally called via cron jobs.

• The archive directory contains subdirectories for all the deployments and their associated NetCDF files.

• The cdl directory is roughly analogous to the buoy directory, except that it contains the CDL files from which the NetCDF files are generated. There should be a subdirectory with CDL files for each buoy instrument, as well as a subdirectory where CDL files specific to each deployment are created.

  • sensor-raw. Templates for NetCDF files that house the telemetered data as-is via cell phone, with no algorithms run or quality computed.

  • goes-sensor-raw. Templates for NetCDF files that house the telemetered data as-is via GOES satellite, with no algorithms run or quality computed. There are some minor differences with the cell phone files.

  • realtime. Templates for NetCDF files that house data upon which algorithms have been run and quality has been computed. This data stream originates from the cell phone data stream.

  • goes-realtime. Analogous to the realtime. The biggest difference is that the doppler data typically has fewer bins reported.

  • raw. Deprecated. Old datastream that encompasses both of what now is sensor-raw and realtime. Don't use it, it's only still present for backwards compatibility with GOMOOS.

  • goes-raw. Deprecated. Old datastream that encompasses both of what now is goes-sensor-raw and goes-realtime. Don't use it, it's only still present for backwards compatibility with GOMOOS.

  • post-recovery-sensor-raw. Templates for NetCDF files that house the data recovered as-is directly from individual instruments, with no algorithms run or quality computed.

  • post-recovery. Templates for NetCDF files that house the data processed from the post-recovery-sensor-raw data stream.

  • data-logger-post-recovery-sensor-raw. Templates for NetCDF files that house the data recovered as-is directly from the data logger. If the telemetered cell phone data stream was perfect, then this data stream should be identical.

  • data-logger-post-recovery. Templates for NetCDF files that house the data processed from the data-logger-post-recovery-sensor-raw data stream.

  • hourly. Templates for NetCDF files that house the data processed from the realtime data stream. This is not a GOMOOS requirement.

  • filtered. Templates for NetCDF files that house the data processed from the hourly data stream. This is not a GOMOOS requirement.

• The doc directory contains all pertinent documentation for the GOMOOS buoy processing, including this document. Also included are subdirectories for referenced documents.

• The incoming directory contains subdirectories where incoming data may be deposited for later processing. The /data/gomoos/incoming/buoy/DATA subdirectory is where the Campbell logger writes the incoming buoy data files. The /data/gomoos/incoming/goes subdirectory houses downloaded GOES binary files.

• The etc directory may contain sed scripts necessary for certain preprocessing operations.

• The matlab directory is the root for all m-files particular to buoy processing.

  • buoy-mfiles. These are for the most part home brewed m-files for GOMOOS specific processing.

    • process-control. Contains entry points for starting MATLAB processing and taking the handoff from the shell scripts.

    • buffer-io. These are mostly reader routines for all the bazillion different file formats I have to deal with, including post recovery log files. Some output routines as well.

    • bigelow-optics. Code imported from Andrew Barnard to process the optics. It is kept here until it is better integrated with the rest of the code.

    • core_processing. Routines for handling all of the processing for a particular instrument. buoy_core_processing.m is the crucial routine here, as it takes the entire set of inputs for all instruments and routes them into the desired set of processing routines. Anything like process_raw_*.m is deprecated.

    • daily_processing. Routines for handling anything that need only run once per day.

    • misc_hourly_processing. Routines for handling anything that should run approximately hourly, except for the core data processing itself. For example, creating plots would fall under this category.

    • m. Miscellaneous routines that are important, but do not really fall into any one category.

    • matlab_webserver. Avoid it as you would the plague. No, really.

    • mysql-netcdf-io. Routines that query the MySQL database or NetCDF files in various ways.

    • parameter_algorithms. Processing algorithms that for the most part are given only geophysical parameters as input and give only geophysical parameters as output. I'm not dogmatic about this, though.

    • plots. Guess.

    • quality-control. Routines for handling data quality issues.

  • ancillary. Static binary MATLAB files, such as coastlines.

  • cf-netcdf. M-files that operate on the assumption that the NetCDF input files are CF compliant.

  • ncmex_wrapper. M-files that act as a wrapper around the lower level MEXCDF and ncmex routines for NetCDF I/O.

  • gomoos-timeplt. Derived from timeplt package written by Rich Signell. Minor improvements. Mostly used in plotting.

  • m_map. Mapping package written by Rich Pawlowicz.

  • mysql. MEX interface between MySQL and MATLAB.

  • netcdf. MEX interface between NetCDF and MATLAB.

  • omviz. Bastardized from the OMVIZ package from WHOI. In serious need of a cleanup.

• The html directory contains HTML documents that are refreshed via PHP to provide a semblance of dynamic content. By refreshing these HTML files at set times rather than dynamically generating them every time a request is made, the download time is dramatically decreased.

• The var directory contains things of the kind that one might find under the /var directory in Linux.

  1. The log directory contains working log files from the current and previous three days of operation. After each cron job finishes, its particular log file is flushed into this directory.

  2. The cron directory contains a copy of the current crontab file for gomoos.

  3. The mysql directory contains dumps of the databases.

• The man directory contains UNIX man pages for any C executables.

• The php directory contains PHP code for the web site.

• The run directory contains PID files for each data ingest process. After a process finishes, this PID file (which doubles as a log file) is flushed to the log directory.

• The src directory may contain source code for any C executables.

All text files (program source files, makefiles, PHP scripts, etc.) are managed through [CVS]. The CVSROOT is set to /assorted/cvsstuff.