| Buoy System Handbook | ||
|---|---|---|
| <<< Previous | Disaster Recovery | Next >>> |
Micmac is the compute and database engine for all of GoMOOS processing, but other workstations in PhoG would be able to take over this responsibility. All the software is mirrored to gyre , so gyre could actually do the job itself. Tryfan would be another good candidate, in fact it would be better than gyre since gyre really should remain just a file server.
Make sure that the software requirements are met.
Start the VNC server and make sure that the gomoos user can connect to it. This should be started from the /etc/rc.d/rc.local script upon system startup. The pertinent section in /etc/rc.d/rc.local should read
# # Add a virtual display for gomoos su -c "vncserver :2" gomoos |
The MySQL gomoos databases are dumped nightly to the ${DATA_ROOT}/var/mysql directory. Use these to recreate the databases on whichever workstation is to take over processing duties. Make sure that MySQL will start upon boot time with
sudo chkconfig --level 35 mysqld on service mysqld start |
$> cat gomoos_processing.sql | mysql gomoos_processing |
Edit the shell scripts
${DATA_ROOT}/bin/setup_gomoos_environment.sh
${DATA_ROOT}/bin/process_cellphone.sh
${DATA_ROOT}/bin/process_goes.sh
Edit the startup m-file get_options.m appropriately. Pay special attention to the database host, webserver_root buoy_root_dir fields. The buoy_root_dir should be exactly the same as ${DATA_ROOT} in setup_gomoos_environment.sh.
Set up the new processing NetCDF archive on the new host machine. You do not have to use the exact same ${DATA_ROOT} as previously outlined, but the directory structure underneath needs to be the same. Take the time to map it out on paper before proceeding. Assuming that the NetCDF archive is still available, rsync it over with ${RUNTIME_ROOT}/bin/rsync_emergency.sh. The rsync configuration file /etc/rsync/rsyncd.conf on gyre may have to be modified to allow the script to run. Make sure that permissions are set to user gomoos and group users. If the NetCDF archive is NOT available, you may need to start nearly from scratch. The daily dump of the CDL descriptions in ${DATA_ROOT}/cdl/active_buoy_master_cdl.cdl.gz may be helpful to help you to recreate the NetCDF files. This will be a slow process, as the dimension values will not be present and will have to be resupplied. It would be best to finish up with one buoy at a time to make sure it works. Alternatively it may just be easier to start completely from scratch using the the new deployment initialization method.
If the ${DATA_ROOT} is NOT the same as before, edit the ~gomoos/matlab/startup.m as needed. This file is backed up to ${DATA_ROOT}/tmp nightly.
If the ${DATA_ROOT} is NOT the same as before, reinitialize the MySQL database, using "init_deployment.m" on each buoy.
Try running the GOES scripts by hand first, ${DATA_ROOT}/bin/process_goes.sh. Edit out the logging portion so that you can actuall see the output.
Setting up the SAMBA connections may take a bit of time. This will be needed in order to facilitate cellphone processing. You will need to modify the /etc/samba/smb.conf according to what you choose as ${DATA_ROOT}. Then, make sure SAMBA will start upon system boot with
chkconfig --level 35 smb on service smb start |
Assuming that you've had reasonable success so far (not absolutely everything needs to work in order to have a functioning system, e.g. web server stuff), you can begin to reconstruct the crontab file that was save to ${DATA_ROOT}/var/cron It would be advisable to run each command in the crontab file by hand first, just to be sure.
Lastly, if the problem on micmac is fixed at some point, before just switching the processing back on from micmac, you need to re-sync the NetCDF archive on micmac with gyre. Use the ${RUNTIME_ROOT}/bin/rsync_emergency.sh script to do this, but modify it appropriately first.
| <<< Previous | Home | Next >>> |
| Gorry2 Dies | Up | Critical Issues |