Database Management - David Court
Summary of My Position
Current Tasks
News & Updates:
OUTLINE OF THE NEW GLOBAL DATABASE: HOW IS IT BUILT?
I. Import ALL radiometric, pigment, and absorption data from SeaBASS
II. Use PERL script to extract the data from its SeaBASS format.
- Save as hyperspectral values
- Lu412.2 = Lu412
- Es682.6 = Es683
- Values are then averaged into 2 nm bins (evenly numbered)
- Lu412 = average of Lu411, Lu412, Lu413
- Es684 = average of Es683, Es684, Es685
- Map data to ensure the measurements are from appropriate locations
- Eliminate data where water depth and/or secchi depth are too shallow
- Eliminate data from estuaries, bays, etc
- Prioritize data that include a 'surface' (0-) value
- Create a separate file for profiles that do not include surface values
- Perform integration methods
- Plot each profile - but separate by appropriate channels (UV, Visible, NIR) to save time
- Perform integration methods
02.01.2012: Absorption Figures for Our Global Dataset
01.30.2012: The Global Database has been updated Global_Data_2012_01_28.xls
1) Plumes & Blooms chlorophyll values corrected (cruises pb168 and pb213)
2) New data added - NSF-BWZ (chlorophyll, reflectance and absorption)
3) Recommend omitting data from AMT19. Reflectance values appear to be too high for wavelengths gt 490.
12.16.2011: BBOP data processed. See notes below in BBOP section
12.14.2011: The global database has been updated. /home/dcourt/SeaBASS_Data/Global_Data/Global_Data_2011_12_14.xls
PLEASE NOTE that I will be making some major improvements over the winter break. I will be re-doing the database so that it includes hyper-spectral measurements for each parameter. This will allow the user to choose a much wider range of options and it will also allow for better updates in the future.
11.30.2011: I've reprocessed the BBOP MicroPro data with the calibrations listed below - as specified in the timeline.
I then compared the newly reprocessed profiles (where different calibration files were used) with the former data.
BBOP RE-PROCESSING RESULTS: a comparison of the data - REPROCESSED/ORIGINAL (2009 - 2010)
UV channels - ED [306, 325, 340, 380]
Visible channels - ED [412, 443, 490, 510]
NIR channels - ED [555, 664, 683]
UV channels - LU [306, 325, 340, 380]
Visible channels - LU [412, 443, 490, 510]
NIR channels - LU [555, 664, 683]
BBOP CALIBRATION HISTORY: a closer look at the changes that have occurred with the MicroPro (2008 - 2011)
- Click on the links below:
- Calibration History Figures: MicroPro-113 Irradiance
- Calibration History Figures: MicroPro-113 Radiance
- Calibration History: Reference Irradiance
- Calibration History Spreadsheet
The figure below illustrates how each calibration was done in correlation with the cruise schedule:
10.01.2011: I will be in Switzerland from Dec 20th - Jan 7th.
I will be working some days while I am away and I will almost always be available via email
09.14.2011: DATA EDITS AND SUBMISSIONS
- The sample times for all the MicroPro data have been edited. All times are GMT.
- All MicroPro data have been RE-submitted to SeaBASS to ensure correct sample times. Jermey has been notified
- PAR values have been calculated from all in-water SPMR & MicroPro irradiance profile data 18.10.2012
- I've asked for a Microsoft operating system to be installed - along with VMware Fusion 02.10.2012
- I've found the "underway" irradiance data associated with the MER.
- I've extracted (and formatted) these Es data and have calculated daily PAR values based on the sampling interval of 1.4 seconds.
- These daily PAR values are now available on-line: ftp://ftp.eri.ucsb.edu/pub/org/oceancolor/BBOP/BBOP_PAR/BBOP_daily_PAR_1992_2011.txt
Ongoing:
Global Database of In-Situ Matchups
- Adding new data as it becomes available from SeaBASS
- Applying QC/QA measures to updates
- Most recent update: September 01, 2012
- Need to redevelop MySQL tools for on-line access
BBOP
- WEBPAGE: http://www.icess.ucsb.edu/bbop/
- Processing BBOP MicroPro data
- Formatting data for SeaBASS submissions
- Latest cruise: B276
- Processed: YES:
- Latest data submitted to SeaBASS: B276 & BVAL46 (01.08.2012)
- Data available on-line: YES
Plumes & Blooms
- WEBPAGE: http://www.icess.ucsb.edu/PnB/PnB.html
- Formatting processed CTD for SeaBASS submissions
- Processing and formatting PRR data for SeaBASS submissions
- Latest cruise: PB234
- Latest data submitted to SeaBASS:
CTD: PB234
PRR: PB225 *waiting for new processing scripts
- Data available on-line: YES
CLIVAR
- Process MicroPro radiometric data as it becomes available
- Matchup data products (AOPs, IOPs & pigments)
- Last cruise: P6 - April 2010
- Submitted to SeaBASS: YES
- Additional cruises: A20 A22 AMMA I08S I09N P16S P16N P18
Global Matchups (In-Situ Data + Remotely Sensed Data Products)
- WEBPAGE: http://www.icess.ucsb.edu/~dcourt/MATCHUPS/
MEaSUREs
- Plan to build some web-tools per Stéphane's needs
Current Projects:
Re-organization of the BBOP database
Purpose:
- To increase the performance of the (in-situ) matchups
Methods:
- Imported all pigment measurements from SeaBASS
- Organized all radiometric data (MER, SPMR, & MicrPro)
- Organized all absorption data (ag, ad, ap & aph)
- Cross-referencing all measurements with BATS log sheets
Issues:
- The chlorophyll values in the original database were incorrect. The chlorophyll values were taken from the IOP bottle-files that had been submitted to SeaBASS. These were labeled as "chl_a", which is defined by SeaBASS as an HPLC product. However, these are Flourometric measurements. At some point, a script needs to be written to re-format these files and a they must then be re-submitted to SeaBASS.
- Many of the sample times are incorrect. This creates inaccuracies when trying to match-up the different measurements. Most of the problems are a result of submitting the data in local (Bermuda) time rather than GMT - as required by SeaBASS. Other issues exist as a result of the technician's computer clock being incorrectly set - sometimes by an order of years. Again, these data will ultimately have to edited and re-submitted to SeaBASS.
