Difference between revisions of "C-OPS processing"
| (25 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
| − | |||
| − | Second column is noise limits for each channel picked from pb240stn4 averaged values at 60m over 200 data points. In the future more casts should be looked at. | + | '''Processing Steps:''' |
| + | |||
| + | '''1) Place relevant processing files in your workspace directory.''' | ||
| + | |||
| + | - *data*.csv * "C" indicates "cast" | ||
| + | |||
| + | - *gps*.csv | ||
| + | |||
| + | - calibrations_C-OPS#[serial_number].txt (*To determine system serial number, open BSI-generated CFG file, or file containing serial numbers column) | ||
| + | |||
| + | - COPSbscalc_window.txt | ||
| + | |||
| + | - Copsrqclist.txt | ||
| + | |||
| + | 2) '''Convert C-OPS .CSV files to LCD format.''' | ||
| + | |||
| + | - cops[serial_number]_to_lcd.pl EXAMPLE: cops#23_to_lcd.pl | ||
| + | |||
| + | It's important that the filename is formatted so we can get the date and time. | ||
| + | |||
| + | EXAMPLE: PB[cruise_number]_YYMMDD_HHMM_station_data_cast.csv -> PB278_20160519_2124_C_data_001.csv | ||
| + | |||
| + | To run a batch, do the following (in TCSH): | ||
| + | |||
| + | ----------- | ||
| + | for i in $(ls *_data_*.csv) | ||
| + | |||
| + | perl /home/oceancolor/processing_scripts/C-OPS/cops#[serial_number]_to_lcd.pl $i | ||
| + | |||
| + | end | ||
| + | |||
| + | ----------- | ||
| + | |||
| + | '''3) Run MATLAB processing script''' | ||
| + | |||
| + | '''NOTE''' | ||
| + | |||
| + | Before running the MATLAB script, open LCD files and check for errors, C-OPS processing expects a "FrameCount" or "LINECOUNT" and "LuZDepth" column. "LINECOUNT" is inserted by the perl script and has no real data bearing. If "FrameCount" is specified in the auto_bt function input, the function will default to a simple integer counter if LINECOUNT is what's used in the lcd file. | ||
| + | |||
| + | Also make sure you've set the path(s) that contain the scripts needed for the processing. | ||
| + | |||
| + | EXAMPLE: addpath /home/oceancolor/Plumes_and_Blooms/processing_scripts/PnB_C-OPS_Matlab/ | ||
| + | |||
| + | EXAMPLE: addpath /home/oceancolor/Plumes_and_Blooms/processing_scripts/PnB_PRR_Matlab/ | ||
| + | |||
| + | - Start MATLAB in your working directory | ||
| + | |||
| + | - Type "process_COPS('**')" where ** are the two prefix letters of the file to process | ||
| + | |||
| + | EXAMPLE: process_COPS('PB') | ||
| + | |||
| + | - Automatic processing will start. | ||
| + | |||
| + | '''NOTE''' | ||
| + | |||
| + | Auto_bt script will write two cast ID matrices if you attempt to call the script twice. The easiest way to re-process is to start with a clean directory with only .lcd file extensions. | ||
| + | |||
| + | '''4) Convert final file to SeaBASS format''' | ||
| + | |||
| + | - use file with 'cmbsbanKrqcm' prefix | ||
| + | |||
| + | ----------- | ||
| + | |||
| + | Second column is noise limits for each channel(radQC) picked from pb240stn4 averaged values at 60m over 200 data points. In the future more casts should be looked at. | ||
Depth values can either be taken from Eu(CERBERUS) or Lu sensors. Careful as these values are not exactly the same. LCD files generated by DBC scripts take LuZ depth. | Depth values can either be taken from Eu(CERBERUS) or Lu sensors. Careful as these values are not exactly the same. LCD files generated by DBC scripts take LuZ depth. | ||
| + | To review the profiles use plotCOPS_final.m | ||
| + | |||
| + | [[File:C-OPSprocessing_instructions.pdf]] | ||
| + | |||
| + | '''For system #23''' | ||
| + | |||
| + | [[calibrations.txt]] | ||
| + | |||
| + | [[COPS_bscalc_window.txt]] | ||
| + | |||
| + | [[Copsrcqlist.txt]] | ||
| + | [[plotCOPS_final]] | ||
COPSradQC params | COPSradQC params | ||
| Line 143: | Line 217: | ||
EuZ780 0.000154474 1.53149 -0.581 -0.136 -0.138 | EuZ780 0.000154474 1.53149 -0.581 -0.136 -0.138 | ||
</pre> | </pre> | ||
| + | |||
| + | |||
| + | |||
| + | '''C-OPS Software version notes(Also seen in oceancolor/CALIBRATION/Biospherical)''' | ||
| + | |||
| + | 0.6.23 Original Version 8/3/2011 | ||
| + | |||
| + | 1.5.3: <1.5 had a problem where gps data were not being reported for UTC < 10. The installer for version 1.5.3 addresses this problem. 10/22/2014 | ||
| + | |||
| + | 1.6.17: This version is different from versions < 1.5, tabs have been rearranged. Multiple file formats can be written at once. 6/10/2014 | ||
| + | |||
| + | 1.7.20: Has the fix for the GPS problem. At some point I will need to update BuoyProfiler as well for this error. 4/21/2015 | ||
| + | |||
| + | 1.8.5: In the Misc tab select “None”. I will create a new version 1.8.6 without the Misc tab, which should be available next week and will send a follow up email then 8/14/2015 | ||
Latest revision as of 14:27, 16 August 2016
Processing Steps:
1) Place relevant processing files in your workspace directory.
- *data*.csv * "C" indicates "cast"
- *gps*.csv
- calibrations_C-OPS#[serial_number].txt (*To determine system serial number, open BSI-generated CFG file, or file containing serial numbers column)
- COPSbscalc_window.txt
- Copsrqclist.txt
2) Convert C-OPS .CSV files to LCD format.
- cops[serial_number]_to_lcd.pl EXAMPLE: cops#23_to_lcd.pl
It's important that the filename is formatted so we can get the date and time.
EXAMPLE: PB[cruise_number]_YYMMDD_HHMM_station_data_cast.csv -> PB278_20160519_2124_C_data_001.csv
To run a batch, do the following (in TCSH):
for i in $(ls *_data_*.csv)
perl /home/oceancolor/processing_scripts/C-OPS/cops#[serial_number]_to_lcd.pl $i
end
3) Run MATLAB processing script
NOTE
Before running the MATLAB script, open LCD files and check for errors, C-OPS processing expects a "FrameCount" or "LINECOUNT" and "LuZDepth" column. "LINECOUNT" is inserted by the perl script and has no real data bearing. If "FrameCount" is specified in the auto_bt function input, the function will default to a simple integer counter if LINECOUNT is what's used in the lcd file.
Also make sure you've set the path(s) that contain the scripts needed for the processing.
EXAMPLE: addpath /home/oceancolor/Plumes_and_Blooms/processing_scripts/PnB_C-OPS_Matlab/
EXAMPLE: addpath /home/oceancolor/Plumes_and_Blooms/processing_scripts/PnB_PRR_Matlab/
- Start MATLAB in your working directory
- Type "process_COPS('**')" where ** are the two prefix letters of the file to process
EXAMPLE: process_COPS('PB')
- Automatic processing will start.
NOTE
Auto_bt script will write two cast ID matrices if you attempt to call the script twice. The easiest way to re-process is to start with a clean directory with only .lcd file extensions.
4) Convert final file to SeaBASS format
- use file with 'cmbsbanKrqcm' prefix
Second column is noise limits for each channel(radQC) picked from pb240stn4 averaged values at 60m over 200 data points. In the future more casts should be looked at.
Depth values can either be taken from Eu(CERBERUS) or Lu sensors. Careful as these values are not exactly the same. LCD files generated by DBC scripts take LuZ depth.
To review the profiles use plotCOPS_final.m
File:C-OPSprocessing instructions.pdf
For system #23
COPSradQC params
EdZ320 0.00063993 0.1057925 -0.289 -0.15 -0.15 EdZ340 0.00040786 0.09229812 0.037 0.136 0.136 EdZ380 0.00214926 0.03190586 -0.232 -0.104 -0.105 EdZ395 0.00040039 0.1271224 -0.293 -0.164 -0.164 EdZ412 0.00055682 0.1383356 -0.272 -0.138 -0.138 EdZ443 0.00056002 0.1830219 0.135 0.233 0.233 EdZ465 0.00222684 0.1535409 -0.159 -0.056 -0.056 EdZ490 0.01217142 0.4508201 -0.316 -0.199 -0.199 EdZ510 0.00001690 0.3828456 -0.025 0.081 0.08 EdZ532 0.01527306 0.4843704 -0.083 0.025 0.024 EdZ555 0.01683447 0.6248441 0.023 0.119 0.118 EdZ560 0.01812439 0.5728049 -0.041 0.059 0.058 EdZ625 0.00010200 0.8248582 -0.185 -0.077 -0.078 EdZ665 0.00003937 1.423046 0.038 0.133 0.134 EdZ670 0.00020907 0.5043991 -0.05 0.074 0.073 EdZ683 0.00006593 1.001773 -0.194 -0.063 -0.065 EdZ710 0.00006305 1.158732 -0.037 0.112 0.111 EdZ780 0.00003961 1.495014 -0.3 -0.169 -0.17 LuZ320 0.00001181 4.112603 -0.117 -0.019 -0.02 LuZ340 0.00007107 2.528736 0.091 0.266 0.265 LuZ380 0.00029265 0.7300571 -0.31 -0.103 -1.04 LuZ395 0.00012805 2.24874 -0.446 -0.164 -0.166 LuZ412 0.00008520 2.439732 -0.139 0.066 0.064 LuZ443 0.00004279 3.507107 -0.296 -1.25 -0.126 LuZ465 0.00007125 2.577199 -0.054 0.104 0.103 LuZ490 0.00009057 6.130876 -0.057 0.053 0.052 LuZ510 0.00011126 5.432522 -0.339 -0.147 -0.148 LuZ532 0.00007556 5.947128 -0.177 -0.021 -0.021 LuZ555 0.00005829 7.142782 -0.233 -0.091 -0.091 LuZ560 0.00006101 6.520918 -0.226 -0.065 -0.064 LuZ625 0.00001042 10.92597 -0.565 -0.452 -0.452 LuZ665 0.00001206 14.42576 0.031 0.211 0.21 LuZ670 0.00003364 4.578427 -0.305 -0.148 -0.148 LuZ683 0.00001260 10.67856 -0.125 0.047 0.048 LuZ710 0.00001034 11.18393 -0.002 0.122 0.12 LuZ780 0.00002035 13.74776 -0.511 -0.189 -0.191 Ed0320 0.00000000 0.1468176 -0.177 -0.06 -0.061 Ed0340 0.00000000 0.1296039 -0.076 0.054 0.053 Ed0380 0.00000000 0.04120935 -0.392 -0.236 -0.236 Ed0395 0.00000000 0.1586371 -0.172 -0.026 -0.027 Ed0412 0.00000000 0.1568148 -0.117 -0.014 -0.015 Ed0443 0.00000000 0.2382892 -0.031 0.087 0.087 Ed0465 0.00000000 0.2087017 -0.412 -0.25 -0.251 Ed0490 0.00000000 0.594159 -0.174 -0.062 -0.062 Ed0510 0.00000000 0.5814521 -0.121 -0.004 -0.005 Ed0532 0.00000000 0.6468537 -0.033 0.088 0.087 Ed0555 0.00000000 0.8434793 -0.145 -0.015 -0.015 Ed0560 0.00000000 0.7490816 -0.236 -0.124 -0.125 Ed0625 0.00000000 1.122741 -0.044 0.064 0.063 Ed0665 0.00000000 1.70267 0.008 0.126 0.125 Ed0670 0.00000000 0.8276352 0.003 0.119 0.118 Ed0683 0.00000000 1.374109 -0.066 0.05 0.049 Ed0710 0.00000000 1.570383 -0.333 -0.204 -0.205 Ed0780 0.00000000 2.187715 -0.174 -0.052 -0.053
============================================================
CERBERUS radQC params. Same but with EU
EdZ320 0.00063993 0.1057925 -0.289 -0.15 -0.15 EdZ340 0.00040786 0.09229812 0.037 0.136 0.136 EdZ380 0.00214926 0.03190586 -0.232 -0.104 -0.105 EdZ395 0.00040039 0.1271224 -0.293 -0.164 -0.164 EdZ412 0.00055682 0.1383356 -0.272 -0.138 -0.138 EdZ443 0.00056002 0.1830219 0.135 0.233 0.233 EdZ465 0.00222684 0.1535409 -0.159 -0.056 -0.056 EdZ490 0.01217142 0.4508201 -0.316 -0.199 -0.199 EdZ510 0.00001690 0.3828456 -0.025 0.081 0.08 EdZ532 0.01527306 0.4843704 -0.083 0.025 0.024 EdZ555 0.01683447 0.6248441 0.023 0.119 0.118 EdZ560 0.01812439 0.5728049 -0.041 0.059 0.058 EdZ625 0.00010200 0.8248582 -0.185 -0.077 -0.078 EdZ665 0.00003937 1.423046 0.038 0.133 0.134 EdZ670 0.00020907 0.5043991 -0.05 0.074 0.073 EdZ683 0.00006593 1.001773 -0.194 -0.063 -0.065 EdZ710 0.00006305 1.158732 -0.037 0.112 0.111 EdZ780 0.00003961 1.495014 -0.3 -0.169 -0.17 LuZ320 0.00001181 4.112603 -0.117 -0.019 -0.02 LuZ340 0.00007107 2.528736 0.091 0.266 0.265 LuZ380 0.00029265 0.7300571 -0.31 -0.103 -1.04 LuZ395 0.00012805 2.24874 -0.446 -0.164 -0.166 LuZ412 0.00008520 2.439732 -0.139 0.066 0.064 LuZ443 0.00004279 3.507107 -0.296 -1.25 -0.126 LuZ465 0.00007125 2.577199 -0.054 0.104 0.103 LuZ490 0.00009057 6.130876 -0.057 0.053 0.052 LuZ510 0.00011126 5.432522 -0.339 -0.147 -0.148 LuZ532 0.00007556 5.947128 -0.177 -0.021 -0.021 LuZ555 0.00005829 7.142782 -0.233 -0.091 -0.091 LuZ560 0.00006101 6.520918 -0.226 -0.065 -0.064 LuZ625 0.00001042 10.92597 -0.565 -0.452 -0.452 LuZ665 0.00001206 14.42576 0.031 0.211 0.21 LuZ670 0.00003364 4.578427 -0.305 -0.148 -0.148 LuZ683 0.00001260 10.67856 -0.125 0.047 0.048 LuZ710 0.00001034 11.18393 -0.002 0.122 0.12 LuZ780 0.00002035 13.74776 -0.511 -0.189 -0.191 Ed0320 0.00000000 0.1468176 -0.177 -0.06 -0.061 Ed0340 0.00000000 0.1296039 -0.076 0.054 0.053 Ed0380 0.00000000 0.04120935 -0.392 -0.236 -0.236 Ed0395 0.00000000 0.1586371 -0.172 -0.026 -0.027 Ed0412 0.00000000 0.1568148 -0.117 -0.014 -0.015 Ed0443 0.00000000 0.2382892 -0.031 0.087 0.087 Ed0465 0.00000000 0.2087017 -0.412 -0.25 -0.251 Ed0490 0.00000000 0.594159 -0.174 -0.062 -0.062 Ed0510 0.00000000 0.5814521 -0.121 -0.004 -0.005 Ed0532 0.00000000 0.6468537 -0.033 0.088 0.087 Ed0555 0.00000000 0.8434793 -0.145 -0.015 -0.015 Ed0560 0.00000000 0.7490816 -0.236 -0.124 -0.125 Ed0625 0.00000000 1.122741 -0.044 0.064 0.063 Ed0665 0.00000000 1.70267 0.008 0.126 0.125 Ed0670 0.00000000 0.8276352 0.003 0.119 0.118 Ed0683 0.00000000 1.374109 -0.066 0.05 0.049 Ed0710 0.00000000 1.570383 -0.333 -0.204 -0.205 Ed0780 0.00000000 2.187715 -0.174 -0.052 -0.053 EuZ320 0.00143408 0.09791 0.033 0.157 0.156 EuZ340 0.013987571 0.08936 0.072 0.172 0.172 EuZ380 0.122181794 0.02819 -0.187 -0.034 -0.035 EuZ395 0.19546726 0.09209 0.068 0.176 0.175 EuZ412 0.337328837 0.10665 -0.098 0.019 0.019 EuZ443 0.555936739 0.16495 -0.014 0.1 0.1 EuZ465 0.883383569 0.17083 0.054 0.142 0.142 EuZ490 1.13230505 0.41978 0.071 0.171 0.17 EuZ510 0.935274001 0.41944 -0.149 -0.023 -0.024 EuZ532 0.783264663 0.45267 -0.464 -0.112 -0.113 EuZ555 0.516853004 0.60115 -0.152 0.007 0.007 EuZ560 0.461559671 0.55471 -0.165 0.002 -0.002 EuZ625 0.00594545 0.91131 -0.078 0.084 0.083 EuZ665 0.011046655 1.33934 -0.17 -0.006 -0.006 EuZ670 0.018466335 0.51449 -0.827 -0.121 -0.123 EuZ683 0.028129248 0.94644 -0.126 -0.001 -0.001 EuZ710 0.003167644 1.15732 -0.077 0.095 0.094 EuZ780 0.000154474 1.53149 -0.581 -0.136 -0.138
C-OPS Software version notes(Also seen in oceancolor/CALIBRATION/Biospherical)
0.6.23 Original Version 8/3/2011
1.5.3: <1.5 had a problem where gps data were not being reported for UTC < 10. The installer for version 1.5.3 addresses this problem. 10/22/2014
1.6.17: This version is different from versions < 1.5, tabs have been rearranged. Multiple file formats can be written at once. 6/10/2014
1.7.20: Has the fix for the GPS problem. At some point I will need to update BuoyProfiler as well for this error. 4/21/2015
1.8.5: In the Misc tab select “None”. I will create a new version 1.8.6 without the Misc tab, which should be available next week and will send a follow up email then 8/14/2015
