| Buoy System Handbook | ||
|---|---|---|
| <<< Previous | Processing Overview | Next >>> |
After a buoy has been recovered, data that has been recorded on-board must be processed and compared to the telemetered data. Data in the telemetered stream does not have to be thrown away in this step, we only need to verify that it was correct. All that the post processing does at this point is to put the data into NetCDF format so that it can be manually analyzed. M-files for doing post-recovery processing can be found in /data/gomoos/buoy/matlab/buoy-mfiles/post-recovery .
Until the post recovery operations are complete, there is something of a gap between the current deployment and the blessedhistorical data. The best candidate between goes-realtime and realtime should be chosen and provisionally added to the historical file immediately after a deployment is over. Use /data/gomoos/buoy/bin/ncradd_history.sh A0117 realtime to add A0117 realtime to A01 historical. Note - doppler and optics need to be added manually. Either the NCO utility ncrcat or MATLAB's nc_archive_buffer.m can be used for the purpose. Example: in MATLAB, fn='/data/gomoos/buoy/archive/A0117/realtime/A0117.doppler.realtime.nc';[data]=get_ncbuffer(fn); [new]=nc_archive_buffer(data,'/data/gomoos/buoy/archive/A01/A01.doppler.historical.nc');
The :history global attribute of the historical NetCDF file should be modified to reflect the fact that provisional data is present.
Plot all the datasets. plot_ncfile.m is well-suited to this task. Flag any bad datums and possibly tighten up valid ranges if necessary.
Fix any attributes that are obviously not correct.
Record the time of the quality check in the history attribute.
Record the sensor serial number if it is not already there. For many instruments it is not necessary to have this in place before deployment.
Concatenate the observations onto the master instrument NetCDF file. For example, an aanderaa for the imaginary ZA104 deployment would probably be named something like /data/gomoos/buoy/archive/ZA104/realtime/ZA104.aanderaa.realtime.nc and would be concatenated onto the end of the equally imaginary file /data/gomoos/buoy/archive/ZA1/ZA1.aanderaa.historical.nc. Be sure to give proper attention to the history attribute. If provisional data is present (you did add the provisional data immediately after' the deployment ended, did you not?), you can remove it with ncrcat.
Modify the global source attribute to leave the proper trail of breadcrumbs back to the original source file. For example, consider that of mooring location A01 after four deployments...
// global attributes:
:title = "Gulf of Maine Ocean Observing System: Realtime Buoy Observations" ;
:institution = "Department of Physical Oceanography, School of Marine Sciences, University of Maine" ;
:institution_url = "http://gyre.umeoce.maine.edu/" ;
:source = "Aggregation from following files:\n",
"A0101.met.raw.nc: time indices 0: 5787\n",
"A0102.met.raw.nc: time indices 5788:10736\n",
"A0103.met.raw.nc: time indices 10737:12391\n",
"A0107.met.raw.nc: time indices 12392:12954\n",
"" ;
|
Specific instructions for specific instruments (buffers) are provided for the following buffers
Accelerometer. The accelerometer data does not get stored in any instrument but the data logger itself. After recovery, the entire data logger file is downloaded. It is not really necessary to have to process the data logger post recovery files unless there were significant gaps in communication when the buoy was in operation. Note that the processing attribute of the NetCDF file is either data-logger-post-recovery-sensor-raw or data-logger-post-recovery to distinguish this data origin from those NetCDF files whose heritage is coming directly from a specific instrument download (e.g. NOT the data logger).
Construct the data-logger-post-recovery-sensor-raw and data-logger-post-recovery NetCDF files.
Read the raw sensor data with read_campbell_buffer.m. The input file should be the data logger file, which has the same format as the realtime input data file. However, it may have more data in it. If data file spans the end of the year, you will need to split the file in two. That is a consequence of the campbell file format not having a year in it. You just have to live with that. Also, you will need to take a look at discriminate_accelerometer.m, particularly the year_override option. Example: in MATLAB: global year_override; year_override=2006; options=get_options('A0117'); [buffer,status]=read_campbell_buffer(options,'/data/gomoos/buoy/archive/A0117/downloads/A0117SM.DAT');
Archive the raw sensor data with nc_archive_buffer.m. Example: fout='/data/gomoos/buoy/archive/A0117/post-recovery-sensor-raw/A0117.accelerometer.data-logger-post-recovery-sensor-raw.nc'; [bubffer1]=nc_archive_buffer(buffer.accelerometer{1},fout);
If desired, run sanitize_buffer.m with the not_in_realtime option in order to clean up duplicated data points among other things. Example: [buffer2]=sanitize_buffer(buffer1,1);
If desired, massage the data onto the hour with gomoos_massage_to_hourly.m. Example: [buffer3]=gomoos_massage_to_hourly(buffer2);
Process into geophysical parameters with process_realtime_accelerometer.m. This also archives it into the data-logger-post-recovery NetCDF file. Example: fout='/data/gomoos/buoy/archive/A0117/post-recovery/A0117.accelerometer.data-logger-post-recovery.nc'; [new,status]=process_realtime_accelerometer(fout,buffer3);
Compare the results with what was recorded in realtime operation and manually add any comments or quality information.
Add the NetCDF files to the data system with init_file.m.
Add the appropriate deployment data to the historical file using either ncrcat or nc_archive_buffer.m. Ncrcat will probably be faster.
It is not really necessary to have to process the data logger post recovery files unless there were significant gaps in communication when the buoy was in operation.
Aanderaa. Aanderaa post recovery data is often of lower quality than the telemetered data stream. It is usually only useful when the telemetered data stream has quit early.
Construct the post-recovery-sensor-raw and post-recovery NetCDF files.
Read the raw sensor data with read_post_aanderaa_new.m. Example: [buffer,status]=read_post_aanderaa_new('/data/gomoos/buoy/archive/A0117/downloads/AANCM711_20070712.Asc');
Archive the raw sensor data with nc_archive_buffer.m. Example: [buffer1]=nc_archive_buffer(buffer,'/data/gomoos/buoy/archive/A0117/post-recovery-sensor-raw/A0117.aanderaa.post-recovery-sensor-raw.nc');
If desired, run sanitize_buffer.m with the not_in_realtime option in order to clean up duplicated data points among other things. Example: [buffer2]=sanitize_buffer(buffer1,1);
If desired, massage the data onto the hour with gomoos_massage_to_hourly.m. Example: [buffer3]=gomoos_massage_to_hourly(buffer2);
Process into geophysical parameters with process_realtime_aanderaa.m. This also archives it into the post-recovery NetCDF file. Example: fout='/data/gomoos/buoy/archive/A0117/post-recovery/A0117.aanderaa.post-recovery.nc'; [new,status]=process_realtime_aanderaa(fout,buffer3);
Compare the results with what was recorded in realtime operation and manually add any comments or quality information.
Add the NetCDF files to the data system with init_file.m.
Add the appropriate deployment data to the historical file using either ncrcat or nc_archive_buffer.m.
Doppler. The post recovery file is in a binary format. It can be read by a set of MATLAB tools made available by the god of MATLAB oceanographic programming, Rich Pawlowicz (Rich Signell is relegated to the role of minor deity). These tools expose a multitude of data variables that are not transmitted in realtime, such as temperature and echo intensity. It is usually the case that the post recovery data is preferable to either the campbell or GOES data.
Construct the post-recovery-sensor-raw and post-recovery NetCDF files.
Read the raw sensor data with rdradcp.m. Example:fname='/data/gomoos/buoy/archive/A0117/downloads/A0117000.000'; [adcp,cfg]=rdradcp(fname,1,-1);
Archive the raw sensor data with marchive_post_adcp.m. Example:fout='/data/gomoos/buoy/archive/A0117/post-recovery-sensor-raw/A0117.doppler.post-recovery-sensor-raw.nc'; [buffer,status]=marchive_post_adcp(fout,adcp,cfg);
Massage data to hourly with gomoos_massage_to_hourly.m. Example: [buffer1]=gomoos_massage_to_hourly(buffer);
Process into geophysical parameters with process_realtime_doppler.m. This also archives it into the post-recovery NetCDF file. Example: options=get_options('A0117'); fout='/data/gomoos/buoy/archive/A0117/post-recovery/A0117.doppler.post-recovery.nc'; [new,status]=process_realtime_doppler(options,fout,buffer1);
Find starting/ending dates for the deployment, then: ncrcat -d time,jd1,jd2 A0117.doppler.post-recovery.nc new.nc; In MATLAB: ncattput('new.nc','global','starting_julian_day_number',dop.time.data(1)); ncattput('new.nc','global','ending_julian_day_number',dop.time.data(end)); ncattput('new.nc','global','starting_julian_day_string','2007-04-01 01:00:00'); ncattput('new.nc','global','ending_julian_day_string','2007-09-10 14:00'); mv new.nc A0117.doppler.post-recovery.nc
Compare the results with what was recorded in realtime operation and manually add any comments or quality information.
Add the NetCDF files to the data system with init_file.m.
Add the appropriate deployment data to the historical file using either nc_archive_buffer.m.
FSI.
Construct the post-recovery-sensor-raw and post-recovery NetCDF files.
Read the raw sensor data with read_post_fsi.m
Archive the raw sensor data with nc_archive_buffer.m
If desired, run sanitize_buffer.m with the not_in_realtime option in order to clean up duplicated data points among other things.
If desired, massage the data onto the hour with gomoos_massage_to_hourly.m.
Process into geophysical parameters with process_realtime_fsi.m. This also archives it into the post-recovery NetCDF file.
Compare the results with what was recorded in realtime operation and manually add any comments or quality information. Note - check to make sure that units are cm/sec and that the ranges agree with the realtime data.
Add the NetCDF files to the data system with init_file.m.
Add the appropriate deployment data to the historical file using either ncrcat or nc_archive_buffer.m. ncrcat usually works fastest.
Met. The met data does not get stored in any instrument but the data logger itself. After recovery, the entire data logger file is downloaded. It is not really necessary to have to process the data logger post recovery files unless there were significant gaps in communication when the buoy was in operation. Note that the processing attribute of the NetCDF file is either data-logger-post-recovery-sensor-raw or data-logger-post-recovery to distinguish this data origin from those NetCDF files whose heritage is coming directly from a specific instrument download (e.g. NOT the data logger).
Construct the data-logger-post-recovery-sensor-raw and data-logger-post-recovery NetCDF files.
Read the raw sensor data with read_campbell_buffer.m. The input file should be the data logger file, which has the same format as the realtime input data file. However, it may have more data in it. If data file spans the end of the year, you will need to split the file in two. That is a consequence of the campbell file format not having a year in it. You just have to live with that. Also, you will need to take a look at discriminate_met.m, particularly the year_override option. Example: global year_override; year_override=2007; [buffer]=read_campbell_buffer(fname); met=buffer.met{1};
Archive the raw sensor data with nc_archive_buffer.m Example: [bufferm1]=nc_archive_buffer(buffer.met{1},'F0116.met.data-loggger-post-recovery-sensor-raw.nc');
If desired, run sanitize_buffer.m with the not_in_realtime option in order to clean up duplicated data points among other things. Example: [bufferm2=sanitize_buffer(bufferm1,1);
If desired, massage the data onto the hour with gomoos_massage_to_hourly.m. Example: [jd]=massage_to_interval(bufferm2.time,10);bufferm2.time=jd;
Process into geophysical parameters with process_realtime_met.m. This also archives it into the data-logger-post-recovery NetCDF file. Example: [new]=process_realtime_met(fout,bufferm3);
Compare the results with what was recorded in realtime operation and manually add any comments or quality information.
Add the NetCDF files to the data system with init_file.m.
Add the appropriate deployment data to the historical file using either ncrcat or nc_archive_buffer.m. Ncrcat will probably be faster.
SBE16.
Construct the post-recovery-sensor-raw and post-recovery NetCDF files. There are two kinds of post-recovery-sensor-raw CDL files, one for data recovered from HEX files, and the other for data recovered for ASCII files. The HEX files reveal more information, such as conductivity_frequency, conductivity_counts, and temperature_counts. There are also two kinds of post-recovery CDL files, one for transmissivity, the other for dissolved oxygen.
Read the raw sensor data with read_post_sbe16plus.m Example: [buffer,std16p,status]=read_post_sbe16plus(fname,'sbe16','raw hex',50,1); where 50=depth and 1 is a hex file (0=ascii).
Archive the raw sensor data with nc_archive_buffer.m Example: [buffer1]=nc_archive_buffer(buffer,fout);
If desired, run sanitize_buffer.m with the not_in_realtime option in order to clean up duplicated data points among other things.
If desired, massage the data onto the hour with gomoos_massage_to_hourly.m. Example: [buffer2]=gomoos_massage_to_hourly(buffer1);
Conductivity is recorded in siemens/m, but must be converted into msiemens/cm in order for the salinity and density algorithms to work. Just multiply it by 10. Example: buffer2.conductivity=buffer2.conductivity*10:);
Process into geophysical parameters with process_realtime_sbe16.m. This also archives it into the post-recovery NetCDF file. Example: [new]=process_realtime_sbe16(fout,buffer2);
Compare the results with what was recorded in realtime operation and manually add any comments or quality information.
Add the NetCDF files to the data system with init_file.m.
Add the appropriate deployment data to the historical file using either ncrcat or nc_archive_buffer.m. ncrcat usually works fastest.
SBE37. The post recovery SBE37 data is usually superior to the telemetered data stream.
Construct the post-recovery-sensor-raw and post-recovery NetCDF files.
Read the raw sensor data with read_post_sbe37im.m or read_post_sbe37sm.m. Example: [buffer,header,status]=read_post_sbe37sm(fname);
Archive the raw sensor data with nc_archive_buffer.m Example: [buffer1]=nc_archive_buffer(buffer,fout);
If desired, massage the data to interval with massage_to_interval.m. Example: [jd]=massage_to_interval(buffer1.time,3);buffer1.time=jd;
Process into geophysical parameters with process_realtime_sbe37.m. This also archives it into the post-recovery NetCDF file. Example: [new]=process_realtime_sbe37(fnew,buffer1);
Compare the results with what was recorded in realtime operation and manually add any comments or quality information.
Add the NetCDF files to the data system with init_file.m.
Add the appropriate deployment data to the historical file using either ncrcat or nc_archive_buffer.m. ncrcat usually works fastest. You may have to subsample the post-recovery data to get it to 30min (or 60min) sampling interval. Use subsample_interval to do this. Example: [oldbuf,newbuf,dum]=subsample_interval(fname,delta_t,int);
SEACAT. There has never been any realtime seacat data, so the post recovery is all you get.
Construct the post-recovery-sensor-raw and post-recovery NetCDF files.
Read the raw sensor data with read_post_ascii_seacat.m.
Archive the raw sensor data with nc_archive_buffer.m
If desired, run sanitize_buffer.m with the not_in_realtime option in order to clean up duplicated data points among other things.
If desired, massage the data onto the hour with gomoos_massage_to_hourly.m.
Compute the quality with qc_archive_seacat.m. This also archives it into the post-recovery NetCDF file.
Add the NetCDF files to the data system with init_file.m.
Add the appropriate deployment data to the historical file using either ncrcat or nc_archive_buffer.m. ncrcat usually works fastest.
Optics A. This discussion may be deprecated if Bigelow starts sending us post recovery data on their own.
Construct the data-logger-post-recovery-sensor-raw and post-recovery NetCDF files.
Read the raw sensor data with read_campbell_buffer.m. No special editing is required since Andrew Barnard had the foresight to include a full time stamp in the data, for which I am eternally grateful.
Archive the raw sensor data with nc_archive_buffer.m
If desired, run sanitize_buffer.m with the not_in_realtime option in order to clean up duplicated data points among other things. Probably a good idea.
If desired, massage the data onto the hour with gomoos_massage_to_hourly.m. Probably a good idea.
Run either process_optics_std_above or process_optics_oc_above, depending on the type of optics package.
Move them into their correct places in the directory tree.
Add the NetCDF files to the data system with init_file.m.
Add the appropriate deployment data to the historical file using either ncrcat or nc_archive_buffer.m. ncrcat usually works fastest.
Optics Persun. The percent sun and sun icon data sets are probably the least important of all and really do not need any post processing unless the algorithm is changed. If that is the case, then re-create an empty optics_persun NetCDF file and rerun the m-file optics_persun_processing_stream.m.
| <<< Previous | Home | Next >>> |
| Post Deployment | Up | Base Software |