processing bs3 2011->2012 CMP using Paula's code =============================================== This follows processing the DS 1->5 CMP's, for which I wrote extra code. However, the DS CMPs had FSI CTDs, whereas the BS3 recent moorings used RBRs. 2013/08/02: ran UnPacker-2.01.26.exe. Had to change from default firmware to firmware version 4.12, as prompted by the UnPacker software. Made two version, including a "human readable" one with dir name " _hr" from C*.MLHEX files: "instrumentModel": "RBR XRX-620", "instrumentFirmwareVersion": "6.60", "instrumentSerialNumber": "018025", copy mkdirs.src from denmarkstrait/cmp_recov2012/ds3, run it. - cp proc_paula from cmp_recov2012\ds2 to bs3_2011to12 - profiles range from 000 to 1429. However, last good profile is in 1426. - edit cmp_asc2mat_fb2.m to allow for reading of different RBR C*.TXT files. Used to have just three data columns; now has 7, and leading/trailing text. Save as cmp_asc2mat_fb3.m Then set input files and range (by profile number), direc = 'c:\projects\healy10_ta\cmp_process\bs3_2011to12\bs3_unpack'; % input dir outdir = 'c:\projects\healy10_ta\cmp_process\bs3_2011to12\raw'; % input dir then run cmp_asc2mat_fb3.m next: e_cutstops2.m finds and excludes the "sitting" portion at the start/end of profiles. It generates plots to see if it works correctly. In preparation: - edit e_cutstops2.m to set profiles to work on (small number to start, then the whole she-bang); also pplot_y, the y-axis plot - edit ds3_pgrid_input.m to set specific information for this CMP/mooring, then save as bs3_pgrid_input.m. - check nominal_ctd_calfile.mat. It contains: c_coefs 1x2 16 double p_coefs 1x2 16 double t_coefs 1x2 16 double tlag 1x1 8 double tlags_str 1x765 110152 struct the "_coefs" are all [1 0]. The lags are from DS5; right now, tlag=3. This all needs to be determined and modified. - for BS3 2011->12, used f_prfse2.m, which uses filtered pressure rather than averaged pressure to find traveling start/stops run e_cutstops2.m, files 0000 to 1426 check out: 17, 19, 45, 46, 1177, 1186 1223: got stuck for a while 1273, 1275, 1285:1288, 1325:1329, 1351, 1360, 1368:1371, 1384, 1392, 1402, 1410 around 730: smooth cycling changes to some difficult profiles, then goes back to smooth 826 has hard time going down around 1070: more difficult cycling again around 1162; 1163 is very short. "missing": 1401, 1403, increase filtering threshold slightly, then do manual rerun - now 1327 has no points ----- finished cutstops ---- next: raw editing. However, as I see from my previous notes, I also compare results to gridded data, to see how necessary raw editing would be. But: in order to run gridding, I need to find sensor lags. run: t_lags3p.m generates a bunch of .png files. Look through all of those (simple step through, using all three figures per profile), and determine lag. initially, it looks like even (0,2,4, ...) down-profiles: lag = 1 odd (1,3,5, ...) up-profile: lag = 0 17: 1 18: 0 19 1 20 1 ... see lags_bs3_v0.txt profile 86: things went bad 113 trends develop pretty solidly: switching back between lags of 0 and 1. This does not work with mmy previous scheme where I made a file " lags_ds3_v0.txt" that contains "prf# lag#" when lags change, or an empty line if lots of prfs kept the same lag. This file was then "filled" with lines for the unchanged lags using fill_lags.m Here, I write "h_laginput.m" to prompt me in matlab to provide a lag for the prof# listed. Starting around prf 1000, things get very noisy in density, in several series. At some point, I wonder if the whole CTD is broken. Since (relatively) smoother profiles return, it seems a combination of compensating T/S layering, with some bad (fouled) profiles in-between. prf 1286: classic "1" lag after looking at all 1426 of them: ffinished file lags_bs3_v0.txt It does contain a set of initial entries where the toggle between 0 and 1 was not so pronounced. On that portion (cut/pasted into lags_bs3_v00.txt), I run fill_lags.m Convert lags.txt into a matlab structure using r_lagstxt.m. Note that lags.txt could have contained additional text etc., as it did for BS3_2009_2010. Save -append tlags_str to nominal_ctd_calfile.mat: save nominal_ctd_calfile tlags_str -append ====== next to come: =========================== - run gridding: make sure input dir in bs3_pgrid_input is "rawcutstops", not just "raw" then run cmp_pgrid_mfile_after_rp: >> cmp_pgrid_mfile_after_rp Program mmp_pgrid version 1.2 - July 2002 Enter deployment (e.g. EGC,NOPP) aon Enter instrument name bs3 programm bombed: turns out variable edpdt is missing from raw####.mat files. The eng files used to have more columns, including that one, although it appeared to always have been 99.00. Further, they were set to "NaN" by cmp_asc2mat_fb2.m (the previous program version). I added a fake edpdt set to 99.00 in cmp_asc2mat_fb3.more however: to rerun from memcard, I would also have to rerun e_cutstops2.m That involved some manual decisions. Prefer not to do that again. So instead, I open the existing rawcutstops/raw####.mat files and add the variable edpdt: program "add_to_raw.m" rerun cmp_pgrid_mfile_after_rp - after run of gridding: copy grd to grd_pre_e, because the raw editing program also grids and saves again if changes were made. also: copy "rawcutstops" to "rawcutstops_pre_e". Both of these "_pre_e" directories are used as the "_restore_dir"s in e_rawcmp7.m - run e_rawcmp7.m (new since DS5, using polygon to determine bad points). found one minor cond spike around prf 120 (roughly). got up to 655, with no more raw spikes (spikes worthwhile to edit in raw) Consider modification: make overview plots first, then interactively look at "problem cases" only: program "p_rawcmp7.m" - finished looking through all the raw plots, without editing more raw (few point) spikes. However, found that starting roughly around profile 1000, density (salinity) appears remarkably "poorer". It gets a little better with prf 1200. It appears as a combination of two things: a more complex T/S stratification (more compensated T/S changes, more layering), but also a poorer CTD behavior. The further reduces by prf 1200, though small T/C changes still produce spikes and inversions lateron. Email Terry H. to put concerns on this RBR CTD ---------------- Aug 26, 2013 after finishing raw edits, rerun gridding again: run cmp_pgrid_mfile_after_rp next: either cal or edit gridded data; Paula did cal, I like to do grid editing first. In either case: the next steps need the microcat: for 2011->12 BS3, we have bs3_201011_2136_cal.mat, which has had the the SeaBird lab call applied. Copy it from c:\projects\healy10_ta\mc_process\bs3_201112_mcd. to cmp_process\bs3_2011to12\mc ==> However, it still needs the field mc.datenumm: load it; >> mc.datenumm = mc.datenum; then save it again. next program on the list: MP_extract_bs.m it loads the grd/*.mat files for the CMP, and looks into the mc directory for any .mat files, which it interprets as mc/sc files. Saves output to grd/[inst]_MP_bot.mat error encountered for BS3_2010to11: calculating a CTD time from engineering data by cmp_pgrid_mfile_after_rp.m failed for a "profile" (1296) that consisted of short up/down segments. Calculating the minimum difference between eng and CTD pressure for index range assignment got the same indeces for start and end. Deviding by their difference led to devide by zero, and "-inf" in the gridded data (e.g., check in grd_rep_e/grd1296.mat). I inserted a fix to use the whole eng pressure range rather than from min_diff to CTD_p to max_diff to CTD_p. Ideally, the whole timing gets revised, though it would include finding out first WHY Paula used this odd relating between eng and CTD pressure. in cmp_pgrid_mfile_after_rp.m look for "if ncp == mcp" 2013/08/15 rerun cmp_pgrid_mfile_after_rp.m again with that fix in place 1296 was the only profile where ncp == mcp try again: MP_extract_bs.m ... okay this time. run identinv_EGC get statistical overview of CMP density inversions select "raw" - which actually loads grd files - as the only one available now program makes plots of statistics about inversions also makes {inst}_flag.mat needed for inversion_look.m, likely others as well. prepare to run inversion_look.m next. run MP_fakecal_egc.m, making cal/{inst}_cal.mat When asked, answer "n" to whether bs3 stopped profiling partway through, even if it did (see denmark strait ds2 notes) ====> don't forget!!! copy grd/{inst}_flag.mat from grd to cal. (could) run inversion_look.m (For bs3, as for ds4 and ds2 earlier, I skip running inversion_look.m ) prepare to run inversion_EGC run inversion_EGC.m - select "calibrated files", with only the fake cal being done. The other option would be "edited" , which is not the case yet. - for 2011->12, with the non-optimal 1Hz raw data rate, I did not edit as rigorously the instances of "slight inversions". ## if needed: for an earlier dataset I had to undo: ## - 126 (Oct 2, 18:00) undo all cadj, sadj fields; since no cal yet, can replace with c,s ## after edits are done: ## load ../edit/bs3_edt.mat ## datestr(MP.time(64,30)) % matches the above time ## MP.sadj(64,:) = MP.s(64,:); ## save ../edit/bs3_edt.mat MP - after making it all through, check ../edit/bs3_edt.mat with: pcolor([1:1421], -MP.pgrid(1,:), MP.s'); shading flat; colorbar found some S instances where the whole profile clearly is an outlier: -- 88 13-Nov-2011 00:04:34 -- 114 19-Nov-2011 12:04:15 -- 1077 17-Jul-2012 06:02:57 -- 1078 17-Jul-2012 12:04:11 verify, then set to nan in ../edit/bs3_edt.mat: n=[88, 114, 1077, 1078]; MP.c(n,:)=nan; MP.t(n,:)=nan; MP.cadj(n,:)=nan; run once more to double-check first: need to rerun identinv_EGC.m selecting "edited" input now: find 19 profiles with inversions. second: run inversion_EGC.m. Fix those. Autoimatically save output to edit/bs3_edt.mat ------------------------- prepare to adjust calibration based on comparison of deep CMP bin to microcat. 1) generate cal: Under Paula's scheme, this was done prior to editing. It made more sense to me to exclude bad data (e.g. when cond was obviously fouled on the way down) before making the comparison. I had already generated bs3_MP_bot by running MP_extract_bs.m earlier. Now I simply substitute the deep bin selected there: program adj_MP_bot.m After running adj_MP_bot, copied bs3_MP_bot_edt.mat to bs3_MP_bot.mat ' Check temperature: load ../grd/bs3_MP_bot plot(MPbot.time, MPbot.t) hold on plot(SC.time, SC.t, 'r') grid datetick('x', 2) title('BS3 compare CMP deep bin temperature (b) to MC (r)', 'fontsize', 12) orient portrait; print -dpng cmp_mc_deept find that aside from the events of warm/salty water intrusions (?), T is right on. run MP_cal_w_SEACAT_EGC.m (note, I am not running MP_cal_w_SEACAT_EGC_fb.m) Depth difference between MP and SC is 15.9763 Calculate gradient over what depth? 16 Gradient will be calculated over 11 points Do you want to change the depth range ("y" to change)? n --------------------- done up to here ------------------------------------- ------------------------------- 2010-11 done earlier this month ---- the cal correction is very small. A comparison of proc_ini CMP data with MC cond indicate that at times of low variability, CMP and MC are very close. But it looks like the small correction still improves things. Doing a "before and after" comparison of proc_ini CMP and cal'ed MC and looking at two details early and late in the deployment, the small correction looks reasonable, and better than none. run MP_applycal_EGC; Did bs3 stop profiling partway through deployment (y/n)? n (did not stop, use "2" for edit) ------------------------------------------ make final files and plots 1) mkdir ../final don't use writeout_egc.m; instead make much simpler writeout_egc_ds3.m modified here as writeout_egc_bs3.m run writeout_egc_bs3.m takes cal/bs3_cal.m, generates final/bs3_edt.mat Note: check if transpose is needed (not here). We want final/[]_edt.mat to have MP(prof_no, bin_no): yes. 2) use f_rawprftimes_ds4.m, modified to f_rawprftimes_bs3.m, use the times from ascii file ds2_unpack/timetags.txt, which I parse with a code snipped from the Denmarkstrait routine proc_ini/r_cmpdata_dms2012.m: r_timetags_bs3.m run f_rawprftimes_bs3.m adds potential temp and density, inserts nans where there are gaps IMTRODUCED during editing into the initial profiler time series, as reflected in timetags.txt. ==> SO: <=== if timetags.txt, meaning the original profiler performance, has irregularities, they would still be in there!!! loads final/bs3_edt.mat, outputs final/bs3_edt_fb.mat make final plots: p_cmp_final.m =============================================================== SUNA =============================================================== Aug 28, 2013 CTD on CMP start down CMP stop CTD off 0000010 10 24 2011 17 58 26 10 24 2011 18 00 01 10 24 2011 18 05 15 10 24 2011 18 07 22 0 14 Profile 10 SUNA turned on at 10 24 2011 17 58 26 Profile 10 SUNA turned off at 10 24 2011 18 07 22 Profile 10 RBR turned on at 10 24 2011 17 58 26 Profile 10 RBR turned off at 10 24 2011 18 07 22 1:34 + 7:22 = 9:56 min ontime SATSDB0052 149.113 0.000000 10/24/2011 18:00:30 419.652 0.000000 10/24/2011 18:05:01 270.5 seconds 4:29min = 271 looking at C0000010.TXT: starting (second) time stamp: TIMESTAMP: TIM 11/10/24 17:58:25 ending time stamp (has only one) TIMESTAMP: STP 11/10/24 18:07:20 08:55 minutes: 535 seconds 537 data lines times from the TIMETAGS.TXT (shown above) work out a bit better: there, CTD off occurs 07:22 start occurs one second later, though. steps: - in c:\projects\healy10_ta\cmp_process\bs3_2011to12\suna, mkdir smatfiles % for .mat version of S*.TXT) mkdir smatfiles1 % for SUNA data matched with C pressure) - convert SUNA files from unpacked, not "human readable" S*.TXT files: run r_sunadata_1 note: only every other file is non-zero; => convert files 1:2:1430, where files is an index into the structure "D=dir('S0*.TXT') (need the "0" to exclude SNSRTIME.TXT) - derive SUNA pressure by matching SUNA dark readings with E stopgap lines. Note: this requires a parsed version of the C*.TXT files as typically generated in the proc_ini run onboard ship. currently, that (derive p) is done by t_match_se_1.m. Note: it sets the file name by specifying the file number, rather than the index into a D=dir structure. This makes a difference since our C,E,S files begin with number 0, not 1. - ran t_match_se_1.m for files 0:2:1426, or 714 files (SUNA appears to be collected on only every other profile; the other files are 0 size). - vertically bin the SUNA profiles: call_binsuna.m calls (a local copy of) depth_bin_cmp.m, set to match to match the 2-dbar bins of the final processed CMP, not the initial depth_bin_cmp.m settings used in proc_ini. Still, given the interrupted sampling (see fig of SUNA depth/time series) of the SUNA, I interpolate cross gaps within a SUNA profile. The final .mat file keeps the initial time series (SUNAraw), as well as the original binned and the interpolated binned data (structure SUNA with fields "_i" for interplated variables). - plot the data: p_cmpsuna_final.m; puts .png file into ../doc