| Buoy System Handbook | ||
|---|---|---|
| <<< Previous | Database Layout | Next >>> |
This is the primary database consulted in the processing.
mooring
datatypes
processing_datatypes
processing_files
| User | Host | Required Permissions | Comments |
|---|---|---|---|
| operator | whatever machine is doing the processing | Select, Insert, Update | This is the account under which the processing actually occurs. The host permissions must be set for any host running this software. Right now it is only micmac.umeoce.maine.edu |
| webuser | whatever machine is running the website (or just generating the web pages) | Select, Create | This is the account under which web access (generally thru PHP) occurs. Create privileges are required because there are a few instances where temporary tables are necessary. Drop privileges are NOT required because the temporary tables vanish after the PHP/MySQL session is over. |
| matlab_webuser | whatever machine is running the MATLAB webserver | Select | This account could probably be folded into the webuser account. |
| gm_mysql | localhost | Select | This account is setup for the benefit of gomoos.org. It is only to be used under PHPMyAdmin, therefore the host access is only localhost. |
| umeoce_user | %.umeoce.maine.edu | Select | This is a generic account for anyone within the School of Marine Sciences. Generally speaking, it would be done thru the command line. Currently, only Steve Cousins makes use of it. |
This is a partial duplicate of gomoos_control:mooring that was placed into gomoos_processing in order to make some portions of the code more efficient.
| Column Name | Type | Attributes | Comments |
|---|---|---|---|
| mooring_id | varchar(10) | NOT NULL, PRIMARY KEY | An example would be "F0101". The first character species a region. The second and third specify a location within that region. The fourth and fifth characters specify a deployment at that particular site. |
| active | char(1) | Either "Y" or "N" | |
| latitude | float(10,4) | Latitude where deployed. | |
| longitude | float(10,4) | Longitude where deployed. | |
| water_depth | float(10,1) | Depth of water column at deployment site. | |
| site_name | varchar(40) | Short 2-3 word description of mooring location. | |
| add_magnetic_variation | float(10,1) | Needs explanation. | |
| goes_platform_id | char | ID string to use when processing a buoy via the GOES satellite. | |
| time_entry | timestamp(14) | Identifies when a change to the record was made. |
An example would be given by
| mooring_id | F0101 |
| active | N |
| type | Coastal |
| latitude | 44.0552 |
| longitude | -68.9980 |
| water_depth | 110.1 |
| scope | 1.320 |
| time_dep | 2000-07-10 19:30:00 |
| time_rec | 2001-07-10 00:00:00 |
| light_list | G |
| buoy_sn | X |
| wirerope_length | 80 |
| chain_length | 60 |
| anchor_weight | 1800.0 |
| solar_watts | 20.0 |
| solar_angle | 65.0 |
| campbell_id | 0 |
| location_comment | W. Penobscot Bay |
| add_magnetic_variation | 0 |
| location_description | 5 nm SE of Rockland |
| buoy_schematic_image_url | http://gyre.umeoce.maine.edu/gomoos/buoy/K0101.gif |
| buoy_schematic_thumbnail_image_url | http://gyre.umeoce.maine.edu/gomoos/buoy/K0101_sm.gif |
| goes_platform_id | NULL |
| time_entry | 20020906115755 |
| status_comment | Full speed ahead. |
In this case, buoy F0101 had no GOES transmitter, so there has no GOES platform id.
This is a table with general information about all NetCDF file variables, but not specific to any one particular NetCDF file. The only time it is written to is during buoy deployment, and only then when a never-before-seen parameter is being introduced.
| Column Name | Type | Attributes | Comments |
|---|---|---|---|
| datatype | varchar(50) | NOT NULL, PRIMARY KEY | A general name by which a datatype is usually referenced. |
| infile_type | char(30) | How the datatype is stored within its host file. For example, "air_temperature" is stored as "float", while "air_temperature_qc" is "byte", and "time" is stored as "double". | |
| longname | char(100) | This is a standard NetCDF attribute. It is a string by which perhaps a plot title could be labeled. For example, "Air Temperature". | |
| units | char(20) | This is a standard NetCDF/HDF attribute. "meters" or "miles per hour", for example. | |
| shortname | char(8) | This may give a shortened (just a few characters) name by which perhaps a plot label might be given. For example, "u" for speed_east. | |
| Comment | text | Each datatype should have a full-text description of what it is, perhaps how it measured. | |
| diagnostic | char(1) | default "Y" | If "Y", then it is not suitable for public display. |
| epic_code | int(11) | default NULL | For compatibility with the EPIC System. It also provided a useful benchmark as to what units, longnames, shortnames, etc. that we should be using. |
| time_entry | timestamp(14) | Identifies when a change to the record was made. |
The entry for salinity looks like
| data_type | salinity |
| infile_type | float |
| longname | Salinity |
| units | psu |
| shortname | S |
| Comment | The Practical Salinity Scale defines salinity in terms of the conductivity ratio of a sample to that of a solution of 32.4356 g of KCl at 15°C in a 1 kg solution. A sample of seawater at 15°C with a conductivity equal to this KCl solution has a salinity of exactly 35 practical salinity units (psu). A typical measurement far out in the ocean might be around 34.7, but near shore measurements might be expected to be 31-32 due to fresh water discharge. |
| diagnostic | N |
| epic_code | 41 |
| time_entry | 20020906115755 |
This is a table linking a NetCDF file with all of its variables. It is not used during the actual processing, but helps to drive plot production as well as storing the most recent observations for each parameter, which enables the web site to quickly present them.
| Field | Type | Attributes | Comments |
|---|---|---|---|
| id | smallint(5) unsigned | NOT NULL, PRIMARY KEY, AUTO INCREMENT | |
| mooring_id | char(5) | matlab_update_index(UNIQUE KEY) | An example would be "F0101". The first character species a region. The second and third specify a location within that region. The fourth and fifth characters specify a deployment at that particular site. |
| buffer_name | char(20) | matlab_update_index(UNIQUE KEY) | See detailed listing. |
| processing_type | char(20) | matlab_update_index(UNIQUE KEY) | Most of the time will be one of "sensor-raw", "goes-sensor-raw", "realtime" or "goes-realtime". "raw" and "goes-raw" are still supported, although deprecated, "hourly" and "filtered" are used for non-GOMOOS purposes. A better name for "hourly" would be "gap-filled", but "hourly" has stuck for historical reasons. |
| instrument_number | smallint(5) | matlab_update_index(UNIQUE KEY) | It is possible to have several buffers of the same type on a buoy. For example, F0102 has three sbe37 buffers. It is necessary to distinguish between them. |
| datatype | char(255) | matlab_update_index(UNIQUE KEY) | This is a variable which must be present in the datatypes table. |
| last_update | datetime | Time of the last data recorded for this value, so this changes every hour. This field and last_value are what's reported on the "Latest Buoy Data" web page. | |
| last_value | double | Last reported value, changes every hour. This field and last_update are what's reported on the "Latest Buoy Data" web page. | |
| last_qc | tinyint | Last reported quality flag, may change every hour. "0" is good, anything else is bad. | |
| sensor_depth | float(8,2) | matlab_update_index(UNIQUE KEY) | The depth at which a sensor is fixed. Depths are measure positive downwards, so anything above the surface has a negative water depth. It is possible to have several temperature readings, for instance, coming in at separate sensor depths on a single mooring. |
| historical_plot_file | char(255) | Location of an image file (probably PNG format) on the local file system. The MATLAB processing software writes to this file a time series plot of the data from a particular mooring location that may span several deployments. | |
| historical_plot_file_url | char(255) | URL of the historical_plot_file | |
| file | varchar(255) | Relates the parameter back to the NetCDF file from whence it comes. Essentially this is a foreign key, although MySQL does not support such things. | |
| file_id | smallint(6) | Relates the parameter back to the NetCDF file from whence it comes. Essentially this is a foreign key, although MySQL does not support such things. In the processing_files table, this is the same as the id field. | |
| recent_plot_file | char(255) | Location of an image file (probably PNG format) of most recent data (maybe 4 days or so) on the local file system. | |
| recent_plot_file_url | char(255) | URL of the monthly_plot_file | |
| deployment_plot_file | char(255) | Location of an image file (probably PNG format) on the local file system. The MATLAB processing software writes to this file a time series plot of the data from a particular mooring location that spans the full deployment. | |
| deployment_plot_file_url | char(255) | URL of the deployment_plot_file |
The matlab_update_index is a MySQL index that MATLAB uses during the processing to speed up UPDATE operations. The speed up factor is considerable, often resulting in a 90-95% time savings.
Each buffer_name stands for a particular datastream coming back from the buoy. This can consist of a single instrument such as doppler or a whole collection of instruments, such as one sees in met.
met: Meteorological data, such as wind speed and air temperature.
doppler
fsi: Short for Falmouth Scientific Instruments. This is a current meter.
aanderaa: A type of current meter.
sbe37: These are Conductivity/Temperature recorders from which we derive salinity and density.
sbe16: A SEACAT that measures conductivity, temperature, and transmissivity.
diagnostics
An example entry might look like
| id | 81767 |
| mooring_id | E0104 |
| buffer_name | doppler |
| processing_type | raw |
| instrument_number | 0 |
| datatype | current_v |
| last_update | 2002-12-18 16:00:00` |
| last_value | -4106.761465 |
| last_qc | 1 |
| sensor_depth | 26 |
| historical_plot_file | /home/webpages/gomoos/buoy/realtime/E01/E01:current_v:doppler:raw:0:26:historical.png |
| historical_plot_file_url | http://gyre.umeoce.maine.edu/gomoos/buoy/realtime/E01/E01:current_v:doppler:raw:0:26:historical.png |
| file | /data/gomoos/buoy/E0104/E0104.doppler.raw.nc |
| file_id | 2659 |
| recent_plot_file | /home/webpages/gomoos/buoy/realtime/E01/E01:current_v:doppler:raw:0:26:recent.png |
| recent_plot_file_url | http://gyre.umeoce.maine.edu/gomoos/buoy/realtime/E01/E01:current_v:doppler:raw:0:26:recent.png |
| deployment_plot_file | /home/webpages/gomoos/buoy/realtime/E01/E01:current_v:doppler:raw:0:26:deployment.png |
| deployment_plot_file_url | http://gyre.umeoce.maine.edu/gomoos/buoy/realtime/E01/E01:current_v:doppler:raw:0:26:deployment.png |
This is a table linking a buoy with all of its NetCDF files. In effect, it tells the processing system where to archive the data in NetCDF format.
| Field | Type | Attributes | Comments |
|---|---|---|---|
| id | smallint(5) unsigned | NOT NULL, PRIMARY KEY, AUTO INCREMENT | Particular to MySQL and not necessary for individual queries. |
| mooring_id | char(5) | An example would be "F0101". The first character species a region. The second and third specify a location within that region. The fourth and fifth characters specify a deployment at that particular site. | |
| buffer_name | char(20) | Specifies the name of a datastream. Usually this is just the name of the instrument from which the data is originating. | |
| processing_type | char(20) | One of "raw", "goes-raw", "hourly", or "filtered". A better name for "hourly" would be "gap-filled", but "hourly" has stuck for historical reasons. Only "raw" and "goes-raw" are used for GOMOOS. | |
| instrument_number | smallint(5) | It is possible to have several buffers of the same type on a buoy. For example, F0102 has three sbe37 buffers. It is necessary to distinguish between them. | |
| num_depths | smallint(5) | This will always be one except for one case, that of doppler data. Doppler data usually comes in thirty bins at thirty depths, usually ranging from 10 meters to 126 meters. | |
| breakout_id | smallint(5) | This is a numeric id tagged to a particular buffer. All the data streams are logged in the campbell and transmitted back to the receiving PC in a single data burst. It is necessary to be able to differentiate between the various buffers, hence the need for this item. | |
| delta_t | smallint(5) unsigned | Number of minutes between recordings. Since F0101 this has always been 60. | |
| file | varchar(200) | Location of the NetCDF file on the local filesystem. | |
| file_url | varchar(255) | URL of the file | |
| dods_url | varchar(255) | DODS URL of the file | |
| time_entry | timestamp(14) | Denotes when the last change to this record occured. |
An example entry for processing_datatypes might look like
| Field | Value |
|---|---|
| id | 2743 |
| mooring_id | I0101 |
| buffer_name | sbe37 |
| processing_type | raw |
| instrument_number | 0 |
| num_depths | 1 |
| breakout_id | 4 |
| delta_t | 60 |
| file | /data/gomoos/buoy/I0101/I0101.sbe37.raw.1m.nc |
| file_url | http://gyre.umeoce.maine.edu/data/gomoos/buoy/I0101/I0101.sbe37.raw.1m.nc |
| dods_url | http://gyre.umeoce.maine.edu/cgi-bin/nph-dods/data/gomoos/buoy/I0101/I0101.sbe37.raw.1m.nc |
| timestamp | 20020910141753 |
| <<< Previous | Home | Next >>> |
| Database Layout | Up | gomoos_data |