Buoy System Handbook
<<< Previous	Processing Overview	Next >>>

Pre-Deployment Testing

Once the criticial information about the configuration of the mooring and instruments has been entered into the mySQL tables, the next step is to set up ekman for processing the test buoy.

Any reference to ${BUOY_ROOT} below refers to the official GoMOOS disk layout. Right now, this is ekman:/data/gomoos/buoy

In order to successfully prepare for a buoy deployment, one must create the NetCDF files that describe the instruments and parameters being measured by the buoy. The files should be created from CDL files (see the man page for ncgen ( 1 ) and ncdump ( 1 ) ). Rather than create these from scratch, one should copy existing CDL templates from the appropriate ${BUOY_ROOT}/cdl directories by using /data/gomoos/buoy/bin/make_cdlfiles.sh to create CDL files from a similar buoy. Generally, a previous deployment's CDL files will serve as a good guide to what the pending deployment's CDL files should look like.

Due to the high overhead of getting the setup done correctly, it is a good idea to start the deployment while in the test phase. For example, assuming that buoy A0107 is to be deployed. The deployment plan called for the following instruments:

RM Young (met)
RDI Workhorse (doppler)
1. SBE37 SM at 1m
2. SBE37 IM at 20m
3. SBE37 IM at 50m
Aanderaa Current Meter
Accelerometer
Diagnostics
Above and surface optics.

All instruments will be accounted for via GOES as well.

Make the new buoy directory (eg, A0118), make the necessary sub-directories (eg, sensor-raw, realtime, goes-sensor-raw, goes-realtime) and copy all of the necessary CDL files into the sub-directories by using /data/gomoos/buoy/cdl/make_cdlfiles.sh A0117 (where A0117 is the previous deployment that you want to use as an example). The CDL files can be found in ${BUOY_ROOT}/cdl. As of recent deployments, the CDL files are logically separated by processing stream and physically separated into subdirectories. This is done to make the NetCDF files easier to visually inspect. Since the A0107 deployment will have data telemetered back via both cell phone and GOES, there will be a sensor-raw and realtime datastream, as well as a goes-sensor-raw and goes-realtime datastream. For backwards compatibility with GOMOOS, the raw and goes-raw datastreams are provided as well. The raw datastream is an amalgamation of sensor-raw and realtime products (and is also END-OF-LIFEd). PHOG requirements also give rise to an hourly and filtered datastream deriving from the arising out of the sensor-raw datastream. This gives eight total logical datastreams (and a lot of NetCDF files), so separating them into subdirectories makes sense.

jevans@micmac: > ls /data/gomoos/buoy/archive/A0117
goes-realtime/  goes-sensor-raw/  realtime/  sensor-raw/

jevans@micmac: > ls /data/gomoos/buoy/archive/A0117/*/*.nc

/data/gomoos/buoy/archive/A0117/goes-realtime:
A0117.aanderaa.goes-realtime.nc
A0117.accelerometer.goes-realtime.nc
A0117.doppler.goes-realtime.nc
A0117.met.goes-realtime.nc
A0117.sbe37.goes-realtime.1m.nc
A0117.sbe37.goes-realtime.20m.nc
A0117.sbe37.goes-realtime.50m.nc
A0117.optics_s.std_flntus.goes-realtime.nc

/data/gomoos/buoy/archive/A0117/goes-sensor-raw:
A0117.aanderaa.goes-sensor-raw.nc
A0117.accelerometer.goes-sensor-raw.nc
A0117.diagnostics.goes-sensor-raw.nc
A0117.doppler.goes-sensor-raw.nc
A0117.met.goes-sensor-raw.nc
A0117.optics_s.std_flntus.goes-sensor-raw.nc
A0117.sbe37.goes-sensor-raw.1m.nc
A0117.sbe37.goes-sensor-raw.20m.nc
A0117.sbe37.goes-sensor-raw.50m.nc

/data/gomoos/buoy/archive/A0117/realtime:
A0117.aanderaa.realtime.nc
A0117.accelerometer.realtime.nc
A0117.doppler.realtime.nc
A0117.met.realtime.nc
A0117.optics_s.std_flntus.realtime.nc
A0117.sbe37.realtime.1m.nc
A0117.sbe37.realtime.20m.nc
A0117.sbe37.realtime.50m.nc

/data/gomoos/buoy/archive/A0117/sensor-raw:
A0117.aanderaa.sensor-raw.nc
A0117.accelerometer_fft.sensor-raw.nc
A0117.accelerometer.sensor-raw.nc
A0117.diagnostics.sensor-raw.nc
A0117.doppler.sensor-raw.nc
A0117.goes_tx.sensor-raw.nc
A0117.met.sensor-raw.nc
A0117.monitor.sensor-raw.nc
A0117.optics_s.std_flntus.sensor-raw.nc
A0117.sbe37.sensor-raw.1m.nc
A0117.sbe37.sensor-raw.20m.nc
A0117.sbe37.sensor-raw.50m.nc

The template files must be edited before turning them into actual NetCDF files. For the most part, the necessary edits are global attributes. The required global attributes are

buffer_type: specifies the datastream for this NetCDF file. In this example, we have an accelerometer for wave data.
instrument_number: a numeric ID that can discriminate one of possibly multiple identical instruments (such as Seacats at 1 meter, 10 meters, 20 meters, or 50 meters). Usually (but not always), if there is just a single instrument of a specific type, this value is zero.
processing_type: for the realtime data coming in from the cell phone, this is [goes-]sensor-raw ( [goes-]raw for backwards compatibility ) for the data telemetered back, and [goes-]realtime for the processed products (also [goes-]raw in backwards compatibility case ).
history: any non-routine operation on the NetCDF file should append to this attribute, detailing exactly what is being done.
water_depth: required for certain doppler operations, general metadata
mooring_site_id: in this example, it is A0117
mooring_site_desc: general text description of where the buoy will be
breakout_id: specifies a numeric id that identifies it in the Campbell input file
magnetic_variation: specific to each buoy, used to modify the directions computed for wind and currents due to the offset of the magnetic north pole.
last_mysql_update_index: currently used to keep track of the last record for which the MySQL gomoos_data database has been updated. May be deprecated.

It would be a pain to have to edit this information for each CDL file (there can by typically be around 30 to 40 per deployment), so much of it can be all at once with a sed script. Look for a file called script.sed and edit it properly for the current mooring. When invoked with sed, certain tokens in the template CDL files get replaced with the desired attributes. Run

$ runsed *.cdl

The result of all this should be proper CDL files waiting to be turned into real NetCDF files.

When finished, create the NetCDF files:

$ /data/gomoos/buoy/bin/make_ncfiles.sh A0117

If this command fails for any reason, you will need to edit the CDL files until they are acceptable.

Make sure that each file is owned by the gomoos user and users group. This is crucial. Just about every problem you will encounter after setting the data processing in motion can be traced back to a permissions problem.

Invoke MATLAB and place them into the MySQL database:

>> options = get_options ( 'project=gomoos', 'mooring_id=A0117'  );
>> init_deployment ( options, 'relative/subdir/of/of/root_dir' );
Example:  options=get_options('A0117'); init_deployment (options,'archive/A0117')

This MATLAB script will descend into the installation directory and properly load the NetCDF files found there into the MySQL database. init_deployment.m will call init_directory.m for each subdirectory found, which in turn calls init_file.m for each NetCDF file found in each subdirectory. This makes it easy to add additional NetCDF files to the deployment after the fact by just invoking the proper entry point, whether it be init_deployment.m, init_directory.m, or init_file.m. The relative/subdir/of/of/root_dir is a directory relative to options.buoy_root_dir.

The Campbell data file needs to be pointed in the proper directory. This is done in the Setup Connections window. Click on the proper buoy, then hit the Browse button. Point to the gomoos share. On gyre, the data file will show up as gyre:/data/gomoos/buoy/incoming/DATA/A0107.DAT. Change to the above directory and execute the command "ln -s DATA/A0117.DAT ./A0117.dat". Note the lowercase "dat".

A good way to test the deployment is to use matlab and try to read the input file as follows:

>> f = '/data/gomoos/buoy/incoming/A0117.dat'

f =

/data/gomoos/buoy/incoming/A0117.dat

>> [b,s] = read_campbell_buffer ( options, f );
Reading Campbell data...
Discriminating met data...
Discriminating doppler data...
Discriminating sbe37 data...
Discriminating aanderaa data...
Discriminating accelerometer data...
Discriminating diagnostics data...
Discriminating fft data...
Discriminating monitor data...
Discriminating optics_s data...
>> b

b =

  met           : {[1x1 struct]}
  doppler       : {[1x1 struct]}
  sbe37         : {[1x1 struct]  [1x1 struct]  [1x1 struct]}
  aanderaa      : {[1x1 struct]}
  accelerometer : {[1x1 struct]}
  diagnostics   : {[1x1 struct]}
  fft           : {[1x1 struct]}
  monitor       : {[1x1 struct]}
  optics_s      : {[1x1 struct]}

This indicates a successful read operation.

You can just run the matlab processing by starting up MATLAB from the command line and typing something like

>> process_realtime('A0117);

This essentially runs thru the entire MATLAB processing as it would generally be invoked from the shell script.

Assuming you are ok at this point, the last thing you need to do is to add an entry for A0117 in the crontab file. The following entry times the processing to start 10 minutes after the hour. You should make sure that whatever time you choose is a few minutes after the buoy is scheduled to call in.

10 * * * *     /data/gomoos/buoy/bin/process_cellphone.sh A0117 
29 * * * *     /data/gomoos/buoy/bin/process_goes.sh A0117

Now watch the setup as it processes the buoy data each hour. The system should catch any particularly egregious errors and report them via email, and the /data/gomoos/buoy/var/log/buoy.log file may prove helpful in further determining any problems.

When satisfied with the setup, soft-link the file correctly.

$> cd ${BUOY_ROOT}/buoy/incoming
$> ln -s DATA/A0117.DAT ./A0117.dat".  You are now setup to

run the data in realtime.

<<< Previous	Home	Next >>>
Pre-Deployment Setup	Up	Post Deployment