View source for Data Management - David Court

== Current Tasks ==
'''WORK-WEEK 12. - 16. JANUARY 2015'''

I. Assist Nathalie with new SeaBASS submission requirements
#SeaBASS now requires that IOP submissions include additional absorption calculations
#New script (iop2sb-with-abs-calculations) was completed on Friday 16. January
#Script can be found in /home/oceancolor/Plumes_and_Blooms/processing_scripts
#Status: complete
II. Schedule time to discuss new tasks
<br/>
<br/>
'''OUTLINE OF THE NEW GLOBAL DATABASE: HOW IS IT BUILT?'''

I. Import '''ALL''' radiometric, pigment, and absorption data from SeaBASS
# Use the Bio-Optical Search Tool
# Search by product: Ed, Lu, CHL, ap, ag, ...

For Radiometric Data (AOPs):

II. Extract the data from its SeaBASS format (PERL).
# Save as hyperspectral values - but round to nearest nm
## Lu412.2 = Lu412
## Es682.6 = Es683
## Separate values into evenly spaced averaged bins
### Lu412 = average of Lu411, Lu412, Lu413 
### Es684 = average of Es683, Es684, Es685

III. Map data to ensure the measurements are from appropriate locations 
# Eliminate data where water depth and/or secchi depth are too shallow
# Eliminate data from estuaries, bays, etc

IV. Prioritize data that include a 'surface' (0-) value

V. Create a separate file for profiles that do not include surface values
# Perform regression to calculate surface values (IDL)
## Plot each profile - but separate by appropriate channels (UV, Visible, NIR) to save time
## Verify maximum depth for the regression input
## Calculate below surface (0-) values with regression script - using '''ROBUST REGRESSION''' to compensate for any outliers
# Calculate above surface (0+) values (example: Ed(0+) = Ed(0-)/0.98  Lw = Lu(0-) * 0.543)
# Calculate reflectance ratios (RRS) (example: Rrs = Lw/Ed)
VI. QC results
# Compare surface ED values with surface irradiance (ES) (example: ED(0-) < ES)
# Compare relationship between Rrs and Chlorophyll
VII. Transfer data to relational database


----

<br/>
02.01.2012: Absorption Figures for Our Global Dataset <br/>

[[File:Absorption_Figures_for_Global_Data.jpg]]<br/>

----
'''News & Updates - January 2012:'''
<br/>
01.30.2012:  The Global Database has been updated '''Global_Data_2012_01_28.xls'''<br/>
1) Plumes & Blooms chlorophyll values corrected (cruises pb168 and pb213)<br/>
2) New data added - NSF-BWZ (chlorophyll, reflectance and absorption)<br/>
3) Recommend omitting data from AMT19. Reflectance values appear to be too high for wavelengths gt 490.<br/>
<br/>
----
'''News & Updates - December 2011:'''
<br/>
12.16.2011: BBOP data processed.  See notes below in BBOP section<br/>
<br/>
12.14.2011: The global database has been updated.  '''/home/dcourt/SeaBASS_Data/Global_Data/Global_Data_2011_12_14.xls'''<br/>
<br/>
'''PLEASE NOTE''' that I will be making some major improvements over the winter break.  I will be re-doing the database so that it includes '''hyper-spectral measurements''' for each parameter.  This will allow the user to choose a much wider range of options and it will also allow for better updates in the future.<br/>
<br/>

----
'''News & Updates - November 2011:'''

11.30.2011: I've reprocessed the BBOP MicroPro data with the calibrations listed below - as specified in the timeline.<br/>
I then compared the newly reprocessed profiles (where different calibration files were used) with the former data.<br/>

The figure below illustrates how each calibration was done in correlation with the cruise schedule:

[[File:Cruise_History_vs_Calibration_History.jpg|700px]]

<br/>
'''BBOP RE-PROCESSING RESULTS''': a comparison of the data - REPROCESSED/ORIGINAL (2009 - 2010)<br/>

- [[RE-PROCESSING HISTOGRAMS]]<br/>


'''BBOP CALIBRATION HISTORY''': a closer look at the changes that have occurred with the MicroPro (2008 - 2011)<br/>

- [[Calibration History Figures: MicroPro-113 Irradiance]]

- [[Calibration History Figures: MicroPro-113 Radiance]]

- [[Calibration History: Reference Irradiance]]

- [[Calibration History Spreadsheet]]

----
'''News & Updates - October 2011:'''

10.01.2011:  '''I will be in Switzerland from Dec 20th - Jan 7th.'''<br/>
I will be working some days while I am away and I will almost always be available via email<br/>
<br/>
09.14.2011: '''DATA EDITS AND SUBMISSIONS'''<br/>
- The sample times for all the MicroPro data have been edited.  All times are GMT.<br/>
- All MicroPro data have been RE-submitted to SeaBASS to ensure correct sample times. Jeremy has been notified<br/>
- PAR values have been calculated from all in-water SPMR & MicroPro irradiance profile data 18.10.2012<br/>
- I've asked for a Microsoft operating system to be installed - along with VMware Fusion 02.10.2012<br/>
- I've found the "underway" irradiance data associated with the MER.<br/>
- I've extracted (and formatted) these Es data and have calculated daily PAR values based on the sampling interval of 1.4 seconds.<br/>
- These daily PAR values are now available on-line: ftp://ftp.eri.ucsb.edu/pub/org/oceancolor/BBOP/BBOP_PAR/BBOP_daily_PAR_1992_2011.txt <br/>
<br/>

----

'''Ongoing:'''
<br/>
<br/>
'''Global Database of In-Situ Matchups'''<br/>
- Adding new data as it becomes available from SeaBASS<br/>
- Applying QC/QA measures to updates<br/>
- Most recent update: March 20, 2012<br/>
- Need to redevelop MySQL tools for on-line access<br/>
<br/>
'''BBOP'''<br/>
- WEBPAGE: http://www.icess.ucsb.edu/bbop/<br/>
- Processing BBOP MicroPro data<br/>
- Formatting data for SeaBASS submissions<br/>
- Latest cruise: B280<br/>
- Processed: YES:<br/>
- Latest data submitted to SeaBASS: B280 (05.03.2012)<br/>
- Data available on-line: YES<br/>
<br/>
'''Plumes & Blooms'''<br/>
- WEBPAGE: http://www.icess.ucsb.edu/PnB/PnB.html<br/>
- Formatting processed CTD for SeaBASS submissions<br/>
- Processing and formatting PRR data for SeaBASS submissions<br/>
- Latest cruise: PB236<br/>
- Latest data submitted to SeaBASS:<br/>
  CTD: PB234<br/>
  PRR: PB234<br/>
- Data available on-line: YES<br/>
<br/>
'''CLIVAR'''<br/>
- Process MicroPro radiometric data as it becomes available<br/>
- Matchup data products (AOPs, IOPs & pigments)<br/>
- Last cruise: P6 - April 2010<br/>
- Submitted to SeaBASS: YES<br/>
- Additional cruises: A20 A22 AMMA I08S I09N P16S P16N P18<br/>
<br/>
'''Global Matchups (In-Situ Data + Remotely Sensed Data Products)'''<br/>
- WEBPAGE: http://www.icess.ucsb.edu/~dcourt/MATCHUPS/<br/>
<br/>
'''MEaSUREs'''<br/>
- Plan to build some web-tools per Stéphane's needs<br/>
<br/>

'''Current Projects:'''<br/>
<br/>
'''Re-organization of the BBOP database'''<br/>
Purpose:<br/>
- To increase the performance of the (in-situ) matchups<br/>
Methods:<br/>
- Imported all pigment measurements from SeaBASS<br/>
- Organized all radiometric data (MER, SPMR, & MicrPro)<br/>
- Organized all absorption data (ag, ad, ap & aph)<br/>
- Cross-referencing all measurements with BATS log sheets<br/>
Issues:<br/>
- The chlorophyll values in the original database were incorrect.  The chlorophyll values were taken from the IOP bottle-files that had been submitted to SeaBASS.  These were labeled as "chl_a", which is defined by SeaBASS as an HPLC product.  However, these are Flourometric measurements.  At some point, a script needs to be written to re-format these files and a they must then be re-submitted to SeaBASS.<br/>
- Many of the sample times are incorrect.  This creates inaccuracies when trying to match-up the different measurements.  Most of the problems are a result of submitting the data in local (Bermuda) time rather than GMT - as required by SeaBASS.  Other issues exist as a result of the technician's computer clock being incorrectly set - sometimes by an order of years.  Again, these data will ultimately have to edited and re-submitted to SeaBASS.<br/>