Data Management - David Court

Current Tasks

WORK-WEEK 16. - 20. MARCH 2015
I. Improve chlorophyll - AOP - IOP matchups

1. In an effort to make the matchup process easier for future users, I changed the scripts to PERL - from IDL
2. These scripts are located at the OCEANCOLOR/OCisD/incoming_data/PERL directory and provide the following steps:
   1. Extract Ed and Lu from SeaBASS profiles
   2. Calculate RRS (from Ed and Lu)
   3. Extract pigments from SeaBASS files
   4. Extract IOPs from SeaBASS files
   5. Match files based on time and distance
3. Data are located in the outgoing_data directory

WORK-WEEK 02. - 06. MARCH 2015
time reduced to 50% - will be in Europe from 04. February - 04. April

I. Improve chlorophyll - AOP - IOP matchups

1. Broaden matchup parameters (< 2KM and <5 hours between samples)
2. Provide Stephane with updated matchup-file

WORK-WEEK 23. - 27. FEBRUARY 2015
time reduced to 50% - will be in Europe from 04. February - 04. April

I. Improve chlorophyll - AOP - IOP matchups

1. Search SeaBASS database for any additionally available files
2. Broaden matchup parameters (< 2KM and <5 hours between samples)

WORK-WEEK 16. - 20. FEBRUARY 2015
time reduced to 50% - will be in Europe from 04. February - 04. April

I. Continue assisting Eric with COPS QA/QC and SeaBASS submissions

1. Working with recently re-processed files
2. SeaBASS formatting script needed to be edited due to some changes in the processing steps - mostly headers and date formats
3. Will submit QA/QC plots for review. Some edits may need to be made before submission

QA/QC plots for pb265

1. C-OPS RRS

QA/QC plots for pb266

1. C-OPS RRS

WORK-WEEK 02. - 06. FEBRUARY 2015
time reduced to 50% - will be in Europe from 04. February - 04. April

I. Continue assisting Eric with COPS QA/QC and SeaBASS submissions

1. We discovered some issues with the processed files that require RE-processing
2. Once re-processing is complete, we will resume QA/QC and submit final products to SeaBASS

II. Continue assisting Nathalie with IOP submissions

III. BBOP Uploads

1. B307-B311 - IOPs + Pigments
2. Pigments for B308, B309 and B310 were submitted to SeaBASS *note that latitudes were incorrect and edited
3. All IOPs for 2014 were resubmitted 03. February 2015)

IV. Plumes and Blooms Data Inventory

1. Spreadsheet of available P&B data: http://www.oceancolor.ucsb.edu/plumes_and_blooms/
2. Choose "Data Inventory" from DATA tab (pull-down menu)

IV. Global Database

1. Search for additional CDOM and Rrs data
2. include UV measurements
3. Widen match parameters (1km + 3 hours)

WORK-WEEK 26. - 30. JANUARY 2015

I. Continue assisting Eric with COPS QA/QC and SeaBASS submissions

1. Plot COPS profiles and post statistics
2. If QA/QC measures pass, send data to SeaBASS *** ERRORS FOUND
3. Status: in progress

II. Additional edits for SeaBASS submissions
We are still awaiting the final protocol updates that are being written based on the NASA Particulate Absorption Workshop, but below are the current recommendations from SeaBASS. There could be some changes after the protocol updates are finished. For example, submitters might be asked to provide information about the blanks, and there will likely be a template or check-list created that would be suggested for use as documentation accompanying absorption submissions.
A list of all minimum required fields containing ag, ap and ad:

1. wavelength
2. abs_ag <-- average of the absorbance replicates (with baseline corrections applied, blank values subtracted, but before applying any null-correction)
3. abs_ag_sd <-- standard deviation of the averaged replicates
4. ag
5. abs_ap
6. abs_ap_sd
7. ap
8. abs_ad
9. abs_ad_sd
10. ad

Information about volume filtered should be preserved, although not as a "/field".
The method of recording the volume filtered is being refined, as people in the past have recorded that info as comments or else in separate log files.
We must now use the new "/volfilt" header (with units of L) described below:

/volfilt=PARAMETER_NAME:VALUE
For example: /volfilt=ap:0.1

In the event a single file needs to contain volumes filtered for different or multiple measurements, additional volumes filtered can be added, separated by commas.  In the event that there were ever multiple different volumes filtered for a single product (as a hypothetical example, 3 ap measurements of 100, 100 and 80 ml all averaged together) simply list them all ( /volfilt=ap:0.1,ap:0.1,ap:0.08 )

WORK-WEEK 20. - 23. JANUARY 2015

I. Assist Eric Stassinos with COPS QA/QC and SeaBASS submissions

1. Plot COPS profiles and post statistics
2. If QA/QC measures pass, send data to SeaBASS
3. Make sure SeaBASS formatting script is working properly: complete
4. Check submission inventory at SeaBASS (for ALL Plumes and Blooms instruments): complete
5. FCHECK and submit any missing files
6. Status: in progress

II. Assist Stephane with finding additional AOP measurements

1. Search the SeaBASS database for AOPs with UV spectra
2. Sort our hyperspectral collection (on the ERI server)
3. Status: complete

WORK-WEEK 12. - 16. JANUARY 2015

I. Assist Nathalie with new SeaBASS submission requirements

1. SeaBASS now requires that IOP submissions include additional absorption calculations
2. New script (iop2sb-with-abs-calculations) was completed on Friday 16. January
3. Script can be found in /home/oceancolor/Plumes_and_Blooms/processing_scripts
4. Status: complete

II. Schedule time to discuss new tasks

########

OUTLINE OF THE NEW GLOBAL DATABASE: HOW IS IT BUILT?

I. Import ALL radiometric, pigment, and absorption data from SeaBASS

1. Use the Bio-Optical Search Tool
2. Search by product: Ed, Lu, CHL, ap, ag, ...

For Radiometric Data (AOPs):

II. Extract the data from its SeaBASS format (PERL).

1. Save as hyperspectral values - but round to nearest nm
   1. Lu412.2 = Lu412
   2. Es682.6 = Es683
   3. Separate values into evenly spaced averaged bins
      1. Lu412 = average of Lu411, Lu412, Lu413
      2. Es684 = average of Es683, Es684, Es685

III. Map data to ensure the measurements are from appropriate locations

1. Eliminate data where water depth and/or secchi depth are too shallow
2. Eliminate data from estuaries, bays, etc

IV. Prioritize data that include a 'surface' (0-) value

V. Create a separate file for profiles that do not include surface values

1. Perform regression to calculate surface values (IDL)
   1. Plot each profile - but separate by appropriate channels (UV, Visible, NIR) to save time
   2. Verify maximum depth for the regression input
   3. Calculate below surface (0-) values with regression script - using ROBUST REGRESSION to compensate for any outliers
2. Calculate above surface (0+) values (example: Ed(0+) = Ed(0-)/0.98 Lw = Lu(0-) * 0.543)
3. Calculate reflectance ratios (RRS) (example: Rrs = Lw/Ed)

VI. QC results

1. Compare surface ED values with surface irradiance (ES) (example: ED(0-) < ES)
2. Compare relationship between Rrs and Chlorophyll

VII. Transfer data to relational database

---

02.01.2012: Absorption Figures for Our Global Dataset

---

News & Updates - January 2012:
01.30.2012: The Global Database has been updated Global_Data_2012_01_28.xls
1) Plumes & Blooms chlorophyll values corrected (cruises pb168 and pb213)
2) New data added - NSF-BWZ (chlorophyll, reflectance and absorption)
3) Recommend omitting data from AMT19. Reflectance values appear to be too high for wavelengths gt 490.

---

News & Updates - December 2011:
12.16.2011: BBOP data processed. See notes below in BBOP section

12.14.2011: The global database has been updated. /home/dcourt/SeaBASS_Data/Global_Data/Global_Data_2011_12_14.xls

PLEASE NOTE that I will be making some major improvements over the winter break. I will be re-doing the database so that it includes hyper-spectral measurements for each parameter. This will allow the user to choose a much wider range of options and it will also allow for better updates in the future.

---

News & Updates - November 2011:

11.30.2011: I've reprocessed the BBOP MicroPro data with the calibrations listed below - as specified in the timeline.
I then compared the newly reprocessed profiles (where different calibration files were used) with the former data.

The figure below illustrates how each calibration was done in correlation with the cruise schedule:

BBOP RE-PROCESSING RESULTS: a comparison of the data - REPROCESSED/ORIGINAL (2009 - 2010)

- RE-PROCESSING HISTOGRAMS

BBOP CALIBRATION HISTORY: a closer look at the changes that have occurred with the MicroPro (2008 - 2011)

- Calibration History Figures: MicroPro-113 Irradiance

- Calibration History Figures: MicroPro-113 Radiance

- Calibration History: Reference Irradiance

- Calibration History Spreadsheet

---

News & Updates - October 2011:

10.01.2011: I will be in Switzerland from Dec 20th - Jan 7th.
I will be working some days while I am away and I will almost always be available via email

09.14.2011: DATA EDITS AND SUBMISSIONS
- The sample times for all the MicroPro data have been edited. All times are GMT.
- All MicroPro data have been RE-submitted to SeaBASS to ensure correct sample times. Jeremy has been notified
- PAR values have been calculated from all in-water SPMR & MicroPro irradiance profile data 18.10.2012
- I've asked for a Microsoft operating system to be installed - along with VMware Fusion 02.10.2012
- I've found the "underway" irradiance data associated with the MER.
- I've extracted (and formatted) these Es data and have calculated daily PAR values based on the sampling interval of 1.4 seconds.
- These daily PAR values are now available on-line: ftp://ftp.eri.ucsb.edu/pub/org/oceancolor/BBOP/BBOP_PAR/BBOP_daily_PAR_1992_2011.txt

---

Ongoing:

Global Database of In-Situ Matchups
- Adding new data as it becomes available from SeaBASS
- Applying QC/QA measures to updates
- Most recent update: March 20, 2012
- Need to redevelop MySQL tools for on-line access

BBOP
- WEBPAGE: http://www.icess.ucsb.edu/bbop/
- Processing BBOP MicroPro data
- Formatting data for SeaBASS submissions
- Latest cruise: B280
- Processed: YES:
- Latest data submitted to SeaBASS: B280 (05.03.2012)
- Data available on-line: YES

Plumes & Blooms
- WEBPAGE: http://www.icess.ucsb.edu/PnB/PnB.html
- Formatting processed CTD for SeaBASS submissions
- Processing and formatting PRR data for SeaBASS submissions
- Latest cruise: PB236
- Latest data submitted to SeaBASS:

 CTD: PB234

 PRR: PB234

- Data available on-line: YES

CLIVAR
- Process MicroPro radiometric data as it becomes available
- Matchup data products (AOPs, IOPs & pigments)
- Last cruise: P6 - April 2010
- Submitted to SeaBASS: YES
- Additional cruises: A20 A22 AMMA I08S I09N P16S P16N P18

Global Matchups (In-Situ Data + Remotely Sensed Data Products)
- WEBPAGE: http://www.icess.ucsb.edu/~dcourt/MATCHUPS/

MEaSUREs
- Plan to build some web-tools per Stéphane's needs

Current Projects:

Re-organization of the BBOP database
Purpose:
- To increase the performance of the (in-situ) matchups
Methods:
- Imported all pigment measurements from SeaBASS
- Organized all radiometric data (MER, SPMR, & MicrPro)
- Organized all absorption data (ag, ad, ap & aph)
- Cross-referencing all measurements with BATS log sheets
Issues:
- The chlorophyll values in the original database were incorrect. The chlorophyll values were taken from the IOP bottle-files that had been submitted to SeaBASS. These were labeled as "chl_a", which is defined by SeaBASS as an HPLC product. However, these are Flourometric measurements. At some point, a script needs to be written to re-format these files and a they must then be re-submitted to SeaBASS.
- Many of the sample times are incorrect. This creates inaccuracies when trying to match-up the different measurements. Most of the problems are a result of submitting the data in local (Bermuda) time rather than GMT - as required by SeaBASS. Other issues exist as a result of the technician's computer clock being incorrectly set - sometimes by an order of years. Again, these data will ultimately have to edited and re-submitted to SeaBASS.