Difference between revisions of "C-OPS processing"

From Pnb
Jump to navigation Jump to search
 
(11 intermediate revisions by the same user not shown)
Line 2: Line 2:
 
'''Processing Steps:'''
 
'''Processing Steps:'''
  
−
1) Place relevant processing files in your workspace directory.
+
'''1) Place relevant processing files in your workspace directory.'''
  
−
- calibrations_C-OPS#__.txt (*To determine system serial number, open BSI-generated CFG file, or file containing serial numbers column)
+
- *data*.csv * "C" indicates "cast"
 +
 
 +
- *gps*.csv
 +
 
 +
- calibrations_C-OPS#[serial_number].txt (*To determine system serial number, open BSI-generated CFG file, or file containing serial numbers column)
  
 
- COPSbscalc_window.txt
 
- COPSbscalc_window.txt
Line 10: Line 14:
 
- Copsrqclist.txt
 
- Copsrqclist.txt
  
−
2) Convert C-OPS .CSV files to LCD format:
+
2) '''Convert C-OPS .CSV files to LCD format.'''
  
 
- cops[serial_number]_to_lcd.pl EXAMPLE: cops#23_to_lcd.pl
 
- cops[serial_number]_to_lcd.pl EXAMPLE: cops#23_to_lcd.pl
Line 21: Line 25:
  
 
-----------
 
-----------
−
for i in $(ls *P.csv) ; do perl /home/oceancolor/processing_scripts/cops[serial_number]_to_lcd.pl $i; done
+
for i in $(ls *_data_*.csv)
 +
 
 +
perl /home/oceancolor/processing_scripts/C-OPS/cops#[serial_number]_to_lcd.pl $i
 +
 
 +
end
 +
 
 
-----------
 
-----------
 +
 +
'''3) Run MATLAB processing script'''
 +
 
'''NOTE'''
 
'''NOTE'''
−
Before running matlab script, open LCD files and check for errors, C-OPS processing expects a "FrameCount" or "LINECOUNT" and "LuZDepth" column. "LINECOUNT" is inserted by the perl script and has no real data bearing.  If "FrameCount" is specified in the auto_bt function input, the function will default to a simple integer counter if LINECOUNT is what's used in the lcd file.
 
  
−
------------
+
Before running the MATLAB script, open LCD files and check for errors, C-OPS processing expects a "FrameCount" or "LINECOUNT" and "LuZDepth" column. "LINECOUNT" is inserted by the perl script and has no real data bearing.  If "FrameCount" is specified in the auto_bt function input, the function will default to a simple integer counter if LINECOUNT is what's used in the lcd file.
−
Start matlab in your file directory, then type "process_COPS('**')" where ** are the two prefix letters of the file to process
+
 
 +
Also make sure you've set the path(s) that contain the scripts needed for the processing.
  
−
Automatic processing will start.
+
EXAMPLE: addpath /home/oceancolor/Plumes_and_Blooms/processing_scripts/PnB_C-OPS_Matlab/
  
−
NOTE: Auto_bt script will write two cast ID matrices if you attempt to call the script twice.  The easiest way to re-process is to start with a clean
+
EXAMPLE: addpath /home/oceancolor/Plumes_and_Blooms/processing_scripts/PnB_PRR_Matlab/
−
directory with only .lcd file extensions.
 
  
−
similar scripts to prr
+
- Start MATLAB in your working directory
 +
 
 +
- Type "process_COPS('**')" where ** are the two prefix letters of the file to process
 +
 
 +
EXAMPLE: process_COPS('PB')
 +
 
 +
- Automatic processing will start.
 +
 
 +
'''NOTE'''
 +
 
 +
Auto_bt script will write two cast ID matrices if you attempt to call the script twice.  The easiest way to re-process is to start with a clean directory with only .lcd file extensions.
 +
 
 +
'''4) Convert final file to SeaBASS format'''
 +
 
 +
- use file with 'cmbsbanKrqcm' prefix
  
−
Converting CERBERUS files to lcd format in unix command line.
 
−
-----------
 
−
for i in $(ls *P.csv) ; do perl cerberus-csv2lcd.pl $i; done
 
 
-----------
 
-----------
  

Latest revision as of 14:27, 16 August 2016

Processing Steps:

1) Place relevant processing files in your workspace directory.

- *data*.csv * "C" indicates "cast"

- *gps*.csv

- calibrations_C-OPS#[serial_number].txt (*To determine system serial number, open BSI-generated CFG file, or file containing serial numbers column)

- COPSbscalc_window.txt

- Copsrqclist.txt

2) Convert C-OPS .CSV files to LCD format.

- cops[serial_number]_to_lcd.pl EXAMPLE: cops#23_to_lcd.pl

It's important that the filename is formatted so we can get the date and time.

EXAMPLE: PB[cruise_number]_YYMMDD_HHMM_station_data_cast.csv -> PB278_20160519_2124_C_data_001.csv

To run a batch, do the following (in TCSH):

for i in $(ls *_data_*.csv)

perl /home/oceancolor/processing_scripts/C-OPS/cops#[serial_number]_to_lcd.pl $i

end

3) Run MATLAB processing script

NOTE

Before running the MATLAB script, open LCD files and check for errors, C-OPS processing expects a "FrameCount" or "LINECOUNT" and "LuZDepth" column. "LINECOUNT" is inserted by the perl script and has no real data bearing. If "FrameCount" is specified in the auto_bt function input, the function will default to a simple integer counter if LINECOUNT is what's used in the lcd file.

Also make sure you've set the path(s) that contain the scripts needed for the processing.

EXAMPLE: addpath /home/oceancolor/Plumes_and_Blooms/processing_scripts/PnB_C-OPS_Matlab/

EXAMPLE: addpath /home/oceancolor/Plumes_and_Blooms/processing_scripts/PnB_PRR_Matlab/

- Start MATLAB in your working directory

- Type "process_COPS('**')" where ** are the two prefix letters of the file to process

EXAMPLE: process_COPS('PB')

- Automatic processing will start.

NOTE

Auto_bt script will write two cast ID matrices if you attempt to call the script twice. The easiest way to re-process is to start with a clean directory with only .lcd file extensions.

4) Convert final file to SeaBASS format

- use file with 'cmbsbanKrqcm' prefix

Second column is noise limits for each channel(radQC) picked from pb240stn4 averaged values at 60m over 200 data points. In the future more casts should be looked at.

Depth values can either be taken from Eu(CERBERUS) or Lu sensors. Careful as these values are not exactly the same. LCD files generated by DBC scripts take LuZ depth.

To review the profiles use plotCOPS_final.m

File:C-OPSprocessing instructions.pdf

For system #23

calibrations.txt

COPS_bscalc_window.txt

Copsrcqlist.txt

plotCOPS_final

COPSradQC params 

EdZ320	0.00063993	0.1057925	-0.289	-0.15	-0.15	
EdZ340	0.00040786	0.09229812	0.037	0.136	0.136	
EdZ380	0.00214926	0.03190586	-0.232	-0.104	-0.105	
EdZ395	0.00040039	0.1271224	-0.293	-0.164	-0.164	
EdZ412	0.00055682	0.1383356	-0.272	-0.138	-0.138	
EdZ443	0.00056002	0.1830219	0.135	0.233	0.233	
EdZ465	0.00222684	0.1535409	-0.159	-0.056	-0.056	
EdZ490	0.01217142	0.4508201	-0.316	-0.199	-0.199	
EdZ510	0.00001690	0.3828456	-0.025	0.081	0.08	
EdZ532	0.01527306	0.4843704	-0.083	0.025	0.024	
EdZ555	0.01683447	0.6248441	0.023	0.119	0.118	
EdZ560	0.01812439	0.5728049	-0.041	0.059	0.058	
EdZ625	0.00010200	0.8248582	-0.185	-0.077	-0.078	
EdZ665	0.00003937	1.423046	0.038	0.133	0.134	
EdZ670	0.00020907	0.5043991	-0.05	0.074	0.073	
EdZ683	0.00006593	1.001773	-0.194	-0.063	-0.065	
EdZ710	0.00006305	1.158732	-0.037	0.112	0.111	
EdZ780	0.00003961	1.495014	-0.3	-0.169	-0.17	
LuZ320	0.00001181	4.112603	-0.117	-0.019	-0.02	
LuZ340	0.00007107	2.528736	0.091	0.266	0.265	
LuZ380	0.00029265	0.7300571	-0.31	-0.103	-1.04	
LuZ395	0.00012805	2.24874		-0.446	-0.164	-0.166	
LuZ412	0.00008520	2.439732	-0.139	0.066	0.064	
LuZ443	0.00004279	3.507107	-0.296	-1.25	-0.126	
LuZ465	0.00007125	2.577199	-0.054	0.104	0.103	
LuZ490	0.00009057	6.130876	-0.057	0.053	0.052	
LuZ510	0.00011126	5.432522	-0.339	-0.147	-0.148	
LuZ532	0.00007556	5.947128	-0.177	-0.021	-0.021	
LuZ555	0.00005829	7.142782	-0.233	-0.091	-0.091	
LuZ560	0.00006101	6.520918	-0.226	-0.065	-0.064	
LuZ625	0.00001042	10.92597	-0.565	-0.452	-0.452	
LuZ665	0.00001206	14.42576	0.031	0.211	0.21	
LuZ670	0.00003364	4.578427	-0.305	-0.148	-0.148	
LuZ683	0.00001260	10.67856	-0.125	0.047	0.048	
LuZ710	0.00001034	11.18393	-0.002	0.122	0.12	
LuZ780	0.00002035	13.74776	-0.511	-0.189	-0.191	
Ed0320	0.00000000	0.1468176	-0.177	-0.06	-0.061	
Ed0340	0.00000000	0.1296039	-0.076	0.054	0.053	
Ed0380	0.00000000	0.04120935	-0.392	-0.236	-0.236	
Ed0395	0.00000000	0.1586371	-0.172	-0.026	-0.027	
Ed0412	0.00000000	0.1568148	-0.117	-0.014	-0.015	
Ed0443	0.00000000	0.2382892	-0.031	0.087	0.087	
Ed0465	0.00000000	0.2087017	-0.412	-0.25	-0.251	
Ed0490	0.00000000	0.594159	-0.174	-0.062	-0.062	
Ed0510	0.00000000	0.5814521	-0.121	-0.004	-0.005	
Ed0532	0.00000000	0.6468537	-0.033	0.088	0.087	
Ed0555	0.00000000	0.8434793	-0.145	-0.015	-0.015	
Ed0560	0.00000000	0.7490816	-0.236	-0.124	-0.125	
Ed0625	0.00000000	1.122741	-0.044	0.064	0.063	
Ed0665	0.00000000	1.70267	 	0.008	0.126    0.125	
Ed0670	0.00000000	0.8276352	0.003	0.119	0.118	
Ed0683	0.00000000	1.374109	-0.066	0.05	0.049	
Ed0710	0.00000000	1.570383	-0.333	-0.204	-0.205	
Ed0780	0.00000000	2.187715	-0.174	-0.052	-0.053


============================================================

CERBERUS radQC params. Same but with EU 

EdZ320	0.00063993	0.1057925	-0.289	-0.15	-0.15	
EdZ340	0.00040786	0.09229812	0.037	0.136	0.136	
EdZ380	0.00214926	0.03190586	-0.232	-0.104	-0.105	
EdZ395	0.00040039	0.1271224	-0.293	-0.164	-0.164	
EdZ412	0.00055682	0.1383356	-0.272	-0.138	-0.138	
EdZ443	0.00056002	0.1830219	0.135	0.233	0.233	
EdZ465	0.00222684	0.1535409	-0.159	-0.056	-0.056	
EdZ490	0.01217142	0.4508201	-0.316	-0.199	-0.199	
EdZ510	0.00001690	0.3828456	-0.025	0.081	0.08	
EdZ532	0.01527306	0.4843704	-0.083	0.025	0.024	
EdZ555	0.01683447	0.6248441	0.023	0.119	0.118	
EdZ560	0.01812439	0.5728049	-0.041	0.059	0.058	
EdZ625	0.00010200	0.8248582	-0.185	-0.077	-0.078	
EdZ665	0.00003937	1.423046	0.038	0.133	0.134	
EdZ670	0.00020907	0.5043991	-0.05	0.074	0.073	
EdZ683	0.00006593	1.001773	-0.194	-0.063	-0.065	
EdZ710	0.00006305	1.158732	-0.037	0.112	0.111	
EdZ780	0.00003961	1.495014	-0.3	-0.169	-0.17	
LuZ320	0.00001181	4.112603	-0.117	-0.019	-0.02	
LuZ340	0.00007107	2.528736	0.091	0.266	0.265	
LuZ380	0.00029265	0.7300571	-0.31	-0.103	-1.04	
LuZ395	0.00012805	2.24874		-0.446	-0.164	-0.166	
LuZ412	0.00008520	2.439732	-0.139	0.066	0.064	
LuZ443	0.00004279	3.507107	-0.296	-1.25	-0.126	
LuZ465	0.00007125	2.577199	-0.054	0.104	0.103	
LuZ490	0.00009057	6.130876	-0.057	0.053	0.052	
LuZ510	0.00011126	5.432522	-0.339	-0.147	-0.148	
LuZ532	0.00007556	5.947128	-0.177	-0.021	-0.021	
LuZ555	0.00005829	7.142782	-0.233	-0.091	-0.091	
LuZ560	0.00006101	6.520918	-0.226	-0.065	-0.064	
LuZ625	0.00001042	10.92597	-0.565	-0.452	-0.452	
LuZ665	0.00001206	14.42576	0.031	0.211	0.21	
LuZ670	0.00003364	4.578427	-0.305	-0.148	-0.148	
LuZ683	0.00001260	10.67856	-0.125	0.047	0.048	
LuZ710	0.00001034	11.18393	-0.002	0.122	0.12	
LuZ780	0.00002035	13.74776	-0.511	-0.189	-0.191	
Ed0320	0.00000000	0.1468176	-0.177	-0.06	-0.061	
Ed0340	0.00000000	0.1296039	-0.076	0.054	0.053	
Ed0380	0.00000000	0.04120935	-0.392	-0.236	-0.236	
Ed0395	0.00000000	0.1586371	-0.172	-0.026	-0.027	
Ed0412	0.00000000	0.1568148	-0.117	-0.014	-0.015	
Ed0443	0.00000000	0.2382892	-0.031	0.087	0.087	
Ed0465	0.00000000	0.2087017	-0.412	-0.25	-0.251	
Ed0490	0.00000000	0.594159	-0.174	-0.062	-0.062	
Ed0510	0.00000000	0.5814521	-0.121	-0.004	-0.005	
Ed0532	0.00000000	0.6468537	-0.033	0.088	0.087	
Ed0555	0.00000000	0.8434793	-0.145	-0.015	-0.015	
Ed0560	0.00000000	0.7490816	-0.236	-0.124	-0.125	
Ed0625	0.00000000	1.122741	-0.044	0.064	0.063	
Ed0665	0.00000000	1.70267	 	0.008	0.126    0.125	
Ed0670	0.00000000	0.8276352	0.003	0.119	0.118	
Ed0683	0.00000000	1.374109	-0.066	0.05	0.049	
Ed0710	0.00000000	1.570383	-0.333	-0.204	-0.205	
Ed0780	0.00000000	2.187715	-0.174	-0.052	-0.053	
EuZ320	0.00143408	0.09791		0.033	0.157	0.156
EuZ340	0.013987571	0.08936		0.072	0.172	0.172
EuZ380	0.122181794	0.02819		-0.187	-0.034	-0.035
EuZ395	0.19546726	0.09209		0.068	0.176	0.175
EuZ412	0.337328837	0.10665		-0.098	0.019	0.019
EuZ443	0.555936739	0.16495		-0.014	0.1	0.1
EuZ465	0.883383569	0.17083		0.054	0.142	0.142
EuZ490	1.13230505	0.41978		0.071	0.171	0.17
EuZ510	0.935274001	0.41944		-0.149	-0.023	-0.024
EuZ532	0.783264663	0.45267		-0.464	-0.112	-0.113
EuZ555	0.516853004	0.60115		-0.152	0.007	0.007
EuZ560	0.461559671	0.55471		-0.165	0.002	-0.002
EuZ625	0.00594545	0.91131		-0.078	0.084	0.083
EuZ665	0.011046655	1.33934		-0.17	-0.006	-0.006
EuZ670	0.018466335	0.51449		-0.827	-0.121	-0.123
EuZ683	0.028129248	0.94644		-0.126	-0.001	-0.001
EuZ710	0.003167644	1.15732		-0.077	0.095	0.094
EuZ780	0.000154474	1.53149		-0.581	-0.136	-0.138


C-OPS Software version notes(Also seen in oceancolor/CALIBRATION/Biospherical)

0.6.23 Original Version 8/3/2011

1.5.3: <1.5 had a problem where gps data were not being reported for UTC < 10. The installer for version 1.5.3 addresses this problem. 10/22/2014

1.6.17: This version is different from versions < 1.5, tabs have been rearranged. Multiple file formats can be written at once. 6/10/2014

1.7.20: Has the fix for the GPS problem. At some point I will need to update BuoyProfiler as well for this error. 4/21/2015

1.8.5: In the Misc tab select “None”. I will create a new version 1.8.6 without the Misc tab, which should be available next week and will send a follow up email then 8/14/2015

Retrieved from "http://wiki-pnb.eri.ucsb.edu/index.php?title=C-OPS_processing&oldid=960"