Charlene - Automated Data Processor¶

REQUIRED:

Caris 10.3 or greater, licensed and configured
Caris support files installed and configured
POSPac 8.1 or greater, licensed and configured

Charlene is an automated night processing and data transfer tool developed by NOAA’s Office of Coast Survey in early 2017. Charlene allows the user to:

Perform verification of raw data
Build deliverable directory structure
Transfer and verify raw data
Process MBES and SSS data with Caris Batch Processor
Generate SBETs with POSPac Batch
Use NOAA tools like AutoQC, QCTools and TCARI

with only a few mouse clicks!

Brief Overview¶

Charlene is a HSTB developed software utility that automates hydrographic ‘night processing’. Night processing includes all of those tasks in between raw data collection and a final daily product that occur each night on our hydrographic vessels. Our goal with Charlene is to make night processing something that requires little to no user interaction.

While we have long been able to read and write certain proprietary data formats, the latest Caris and Applanix versions (10.2 and 8.1 respectively) open machine access to the processes in these packages; we now have the ability to run these software packages outside of the graphical interface. Charlene leverages this to become a one-stop shop for night processing. Just plug in your data drive, sit back, and wait for your processed data to arrive. Key to all this is that Charlene works across software platforms; it can transfer your raw data to the appropriate submission folders, process your SBET, convert survey lines into Caris, apply sound velocity profiles, water levels, and SBETs, run in-house QC reports, and generate logs.

How can we do this?¶

Charlene relies on individual modules to process. These are all callable within Python, allowing for robust and intelligent operation.

Caris Batch¶

Charlene utilizes the new Caris Batch, which is a command line driven tool new in Caris 10 for automating Caris processing. Charlene takes these commands and wraps them in Python, so they can be used from the Charlene GUI. This makes Caris Batch user friendly, as the native commands are often long and complicated, unsuitable for the average user. Below is an example for just the data conversion step for nine lines using Caris Batch.

“C:\Program Files\CARIS\HIPS and SIPS\10.3\bin\carisbatch.exe” –run ImportToHIPS –input-format KONGSBERG –input-crs EPSG:26910

–convert-navigation –gps-height-device GGK “E:\BatchHydroData\Test_&_Calibration\HSRR_2017\Data\MBES\FA_2805_EM2040\2017-142\0000_20170522_220650_FA2805.all”

“E:\BatchHydroData\Test_&_Calibration\HSRR_2017\Data\MBES\FA_2805_EM2040\2017-142\0001_20170522_221613_FA2805.all”

“E:\BatchHydroData\Test_&_Calibration\HSRR_2017\Data\MBES\FA_2805_EM2040\2017-142\0002_20170522_222027_FA2805.all”

“E:\BatchHydroData\Test_&_Calibration\HSRR_2017\Data\MBES\FA_2805_EM2040\2017-142\0003_20170522_224815_FA2805.all”

“E:\BatchHydroData\Test_&_Calibration\HSRR_2017\Data\MBES\FA_2805_EM2040\2017-142\0004_20170522_225233_FA2805.all”

“E:\BatchHydroData\Test_&_Calibration\HSRR_2017\Data\MBES\FA_2805_EM2040\2017-142\0005_20170522_225620_FA2805.all”

“E:\BatchHydroData\Test_&_Calibration\HSRR_2017\Data\MBES\FA_2805_EM2040\2017-142\0006_20170522_230220_FA2805.all”

“E:\BatchHydroData\Test_&_Calibration\HSRR_2017\Data\MBES\FA_2805_EM2040\2017-142\0007_20170522_230553_FA2805.all”

“E:\BatchHydroData\Test_&_Calibration\HSRR_2017\Data\MBES\FA_2805_EM2040\2017-142\0008_20170522_231007_FA2805.all”

file:///E:\BatchHydroData\TO\Test_&_Calibration\HSRR_2017\Data\Processed\Sonar_Data\HDCS_Data\HSRR_2017\HSRR_2017.hips?Vessel=FA_2805_EM2040;Day=2017-142

And this is a simplified example. You can see that the command itself would be obnoxious to run directly. By wrapping it in Python, we can determine many of these options programmatically without direct input from the user.

POSPac Batch¶

With POSPac 8, we now have the use of the POSPac command line toolset, which allows the user to run POSBat files and a template file to create SBET files. Charlene uses specific POSBat files for each processing type (Smartbase, Marinestar, and RTX) and different export options for coordinate systems (NAD83 vs WGS84). Charlene is able to create an SBET in the middle of Caris processing and load the generated SBET during night processing, if the satellite data is available.

Pydro Tools¶

As much of Pydro is modular and all of it is written in Python, incorporating existing Pydro tools midstream in Charlene is relatively simple. Simple calls in Charlene allow for these tools to be used during night processing without running a separate GUI.

Charlene Tools¶

Charlene is currently configured to do several things on completion, including:

Automatically run Caris with project and lines open for review
Generate a Processing Log as an excel workbook (or add a tab for a new day number if one has been started. See Figure 7)
If ‘Benchmark’ is selected, display benchmarking results.
If a QCTools option is chosen, show the QCTools output directory for review

Changes List¶

v2.1.0¶

Added logic to handle different versions of psutil library
Charlene converts correctly from raw directory instead of transfer drive
Will prefer existing TCARI file for loading water levels
Added functionality to streamline POSMV file reading
Will create process log folder if it didnt exist prior to reprocessing
Added utmzone from pos file function
Added check for mixed up files (mbes in pos folder, etc.)
Added timestamp check (see if files are in correct day number folder)
Added ‘stubfile’ check, Caris will not convert MBES files under 500kB
Added Autoqc to tools - create sbet
Added optional tide/hvf paths to create sbet
Added option to not transfer HVF if pointed at HVF in Charlene structure
New: SHAM, automated feature attribution tool
Fix for SHAM AGEN and FIDS attribution
Removed waterline from SHAM traditional/tides workflow
Added popup dialog for HYPACK SSS frequency selection
Added day number to Charlene log file name
Added project and sheet to Charlene log
Fix bug in Charlene log for not logging tide loaded when using TCARI
Added minor version and Pydro rev number
Added antenna type AT1675 for SBET generation
Added logic to prevent hanging on loading settings from previous run
Made switching modes and updating UI more efficient

v2.0.4¶

Added ‘Are you sure’ confirmation prompt to reconversion
Process New Day will overwrite raw and proc data on prompt
Sheet name now adds _MB or _SSS if you don’t have it already (NOAA only)
Optimize MBES/POSMV checking
Fix Charlene log carriage returns
File transfer now transfers acq logs in ship mode
Acquisition Log button updates files to transfer window correctly
Updated SSS documentation
Files to transfer window correctly segregates between MBES and SSS

v2.0.3¶

Changes to create raw drive/filetransfer to support/document pyinstaller operation
Marinestar check includes vbs mode
Generate new xlsx offset report
Include reading caris vessel shape
Save all offsets as float type in rawdatachecks
Fix bug with svp being searched for by files to transfer window during sss
Fix bug with acq_log not being defined for SSS in filetransfer
Bug fix in create raw drive - if batchprocessor file doesnt exist
Added support for GA530 antennas in sbet creation
Added close for vdatum file handler when finding band name
Caris license check now displays output in message box
Caris license check catches environment issues
Only transfer no longer checks Charlene settings in prechecks
Added YYYY-DDD format descriptor for day number reproc box
Svp zone check now skips non svp files
Sbet generation now skips non .000 files
Filetransfer will only transfer appropriate mbes/sss files

v2.0.2¶

New - Chucky, Charlene’s less elegant brother. Chucky allows for running Charlene outside of the GUI using python functions. More info is available in the documentation.
New - Tools - Create Offset Report. Allows you to pull offsets from hvf, .all, and .000 files and write them out to csv
.all is the only MBES file format currently supported
.000 reader currently does not read out ref to imu mounting angles. I’m working on it...
New - Tools - SNM From Tiff. Get a number for SNM from a tiff.
Tools - Merge Geotiff. New Calculate SNM checkbox to get a number for SNM after merging tiffs
Moved to QCTools2 for Raster and VR grids
Rawdatachecks now only looks for marinestar status in a single line to continue
AutoQC now runs as subprocess, allows for separation from GUI
If you see a python crash here please let me know. I have seen intermittent python crashes after autoqc finishes. Even with the crash, it should complete and continue on with the rest of processing.
Fixed some bugs with the project/sheet/vessel/day boxes
Fixed lost handle for progress bar bug
Added support for NAD83_MLLW and WGS84_MLLW vdatum band names
Added nightly xlsx process log to cover for situation where master log is open and unable to access
xlsx process log now discriminates at vessel and day levels
Fix bug for svp file count
Added headers to daily benchmark spreadsheet file

v2.0.1¶

Include hypack sidescan processing
Include discover processing
Add new directory structure option. Points to ini file to determine layout of final directory structure. Allows for supporting multiple years’ Specs directory structures.
Add new ini file checker for mandatory options.
Add new ‘NOAA Support Files’. Uncheck to allow Charlene to run without the use of the NOAA Caris Support Files.
Updated recompute towfish nav to run without any boxes checked. (verified with edgetech acquisition)
Add new UTM zone reference image
Updated all internal workings to new ini file structure
Add leading zeros to create transfer drive calendar dialog
Modified create sbet tool - new field names for sbet and smrmsg that make more sense
Updated documentation

v2.0.0¶

Features:

General efficiency improvement and reorganization of codebase
New documentation available in PydroXL for Charlene
New References available from toolbar - julian day calendar and ref systems
HVF now separate from transfer drive. Each run, user must point to HVF locally/on network.
Transfer all additional files during processing (tides, vdatum) and use them from the destination
New Option - Raw Data Checks - Examines POS files and Kongsberg files for any issues that might cause processing problems
Raw Data Checks, POS - Solution status and FDIR warning messages. Marinestar processing will not occur if Marinestar solution is not present
Raw Data Checks, Kongsberg - Looks for GGK and SV profiles. If GGK not present (and gps height isn’t coming from an external source), will not compute GPS Tides.
Reprocess - Only merge/TPU if required on Reprocessing
Reprocess - Charlene now runs ‘Reprocess…’ from Destination instead of from Source (You no longer reprocess off of the transfer drive)
Reprocess - Now works successfully on half-converted projects. No need to delete directories to restart as long as transfer worked successfully.
Ship mode - process data without launch transfer drive, as long as you can see all data from computer!
Reprocess and Ship modes - use dropdown boxes for choosing Project/Sheet/Vessel/Day. Populate based off of PSVD in the TO directory
Just Transfer mode - Only transfer data without any processing
Prechecks - examines user entered options and data to see if there are any issues before start
Create Transfer Drive - No longer creates vesselconfig folder.
Create Transfer Drive - allows for additional vessel in run
Create Transfer Drive - Add calendar button for selecting day number
Open/Save Session - Saves path to config directory
ProgressBar - now dynamically creates processes in order of operations
ProgressBar - SBET and SSS processes now register in ProgressBar
GUI - Buttons now include text instead of generic select message
GUI - Now correctly freezes/refreshes after operations
GUI - Precheck messages now show as message boxes instead of exceptions
Processing log - Added messages for each process in log
Processing log - Includes SSS options
Processing log - includes all processing options and raw data check summary
TPU - now accepts values out to 3 decimal places
Backscatter/SSS - Mosaic options with resolution selection checkboxes
SVP Concat - Now creates master sheet-wide svp file as well
TCARI - TCARI now saves loaded waterlevels to tc file.
POS/SBET - New options for specifying antenna type for PCO/PCV values
POS/SBET - If SBET is not generated, Charlene will downgrade processing to loading only .000 delayed heave (not attempting to load from SBET/SMRMSG)
POS/SBET - New marinestar (height and heave, no height rms) and RTK (height and heave, all rms) options for loading POS files
POS/SBET - Added in Tools a separate program for generating SBET
HIBASE Database - Update database for processin options and time elapsed recording
HIBASE Database - Moved to Vincenty calculation for LNM to account for WGS84 ellipsoid

Bugs:

File transfer now correctly excludes other MBES formats
Create Transfer Drive - correctly saves options
Update s7k conversion to reflect Hassler option set
Fix bug with AutoQC and tidefiles not loading correctly
Create SBET now also correctly loads .000 File/delayed heave
SVP/projection message now prints only when new projection determined from svp file
SVP concat now supports SSM header
SVP concat tool gives metrics on duplicates/excluded casts
Widen progress bar window to prevent wraparound
Fix issue with autoqc hanging on loading waterlevels from TCARI
Fix save session overwriting existing options sometimes
Fix sidescan/mbes sometimes getting mixed together
Fix sidescan/mbes files showing up together on transfer window
Fix bug in carisapi mosaic call, would fail on path with space
Added overwrite prompt to save session to deal with accidental file overwrites by user
Opening session now clears fields that are not in the file
Fix bug with benchmark log overwriting itself

v1.4.1¶

Reported from the RA that there were issues reading from Sound Speed Manager exported .svp files. Charlene’s SVP concatenator will now read these just fine. I added some text at the end to give a status of the run.
Added an automatic print-to-pdf for the caris and applanix log.
Restart process is now more robust and will terminate processes individually if they fail to restart successfully

v1.4.0¶

ProgressBar will now track the status of each individual Caris process (conversion, for instance) and report metrics on that process. Here, we see that conversion has spawned 3 threads (1 master, 2 for the two lines i’m converting) and the CPU is at around 55.4% utilization. Conversion is reading from the network at about 6MB/s and writing to the network storage at 3MB/s.
ProgressBar has an Emergency Stop button that is the same as the old ‘Nuke Caris’ button with some additional streamlining in the function
The Restart Process button is intended to be a solution to the hung processes issue the FA is experiencing. When the button is clicked, the process (Conversion in this instance) will restart. If it is halfway done, it will reconvert the already-converted lines and then convert the unconverted lines. I believe these hung processes are likely due to network congestion and the NetApp being unresponsive and overloaded. I believe the restart might allow you to get past this issue.
The Idle Timer will tell you how long Read and Write are below 5MB/s. The idea here is to tell you if it has been idle for a long time, and might need a restart.
I’ve found that two instances of Caris converting into a new project will cause an error (‘ERROR: Unable to decode Auto UTM key’). I talked to Caris about it and they told me that ‘Basically the batch #2 can’t even get the HIPS file open to see if it can start processing because Windows is blocking access while batch #1 is busy writing to it.’ To deal with this issue, I’ve built a loop that, if the error appears, will stall processing until the HIPS file is available. You’ll know if it is happening, as the error will appear in Charlene and the progress bar will say ‘OnHold, Hips Unaccessable’ with a 30 sec timer. Timer runs out, process tries again, hopefully finding the HIPS file to be accessible this time.
No longer brings up excel log after process finishes (causes issues in multiuser tests). Will still save/create log.
Modified POSPac processing to work within 8.1. This includes changing Charlene’s posbat files, handling some path issues, some other minor issues. FH has used Charlene Marinestar SBET creation with success.
Issues with AutoQC within Charlene. Disabled for now until a solution is determined.
Fixed issues with determining Caris/POSPac version

v1.3.3¶

Some issues with ‘Raster/Finalize - compcoverage’ not creating the correct depth ranges. These have been resolved.
Charlene now uses checkboxes for QCTools items. QCTools calls will no longer halt the operation if there is an issue.
Failing to send benchmarking statistics would occasionally cause Charlene to halt. This has been resolved.
Some issues with Charlene not correctly prioritizing 10.3 locations for support file searching. This is fixed.

v1.3.2¶

VR surfacing is back in. You have to have Caris 10.3 to have the fix. Please don’t do VR in Charlene unless you are using 10.3. I’ll be restricting it to 10.3 and higher in the future.
Significantly reduced loading time from config file
Cleaned up the interface a bit, with explanations of fields
Auto-populate projection if you are using a SVP file
Segregate ‘stub files’ AKA ‘fat finger files’ - MBES files smaller than 500kB
Automatically captures data from night processing run. You can see the data from all the ships at https://hibase.noaa.gov/benchmark/default (Requires a Hibase login)
Fixed bug due to rounding errors with EPSG estimation tool

v1.3.1¶

Fixed check for Caris license. Should now produce error message if Caris isn’t properly licensed.
Added fix for program not initializing properly if GUI fields are set to nonexistent folders.
Data transfer will now segregate .wcd files to a separate directory before transfer/processing.
Added fix for progressbar not working properly when loading tides and processing HYPACK data
In addition, I wanted to provide an update on some Charlene related items (not related to normal Caris processing):
Sidescan in Charlene - Recompute Towfish Navigation does not work in the batch engine. Working to get this into 10.3.2 hopefully.
VR in Charlene - We’ve verified that the VR surface you get from the batch engine is nearly identical. There is an issue that might be related to the order of operations in how VR is generated in the batch engine. Working with Caris to resolve this.
Project Coordinate Reference System (CRS) in Charlene - I might not have communicated this very well. Charlene is unable to set the project CRS (defaults to WGS84). When you create a new project using Charlene and you want NAD83, you’ll need to change it from WGS84 to NAD83 the first time you open it. It’s a quick process, see below:

v1.3.0¶

Sidescan and VR have been removed from the GUI.
There are some differences in how the Batch VR and the normal Caris GUI VR behave. I think it is still related to how the cube parameters file is accessed. Until we figure it out, I’ve removed it from Charlene.
The Sidescan compute towfish issue still exists for Charlene.
New file transfer process utilizing Windows Robocopy and Python filecmp. Files are transferred and verified by name and other properties. Should produce a rock solid file copy mechanism.
Raster only processing now can include QCTools
Refined SVC concatenation to a single day number folder
Fixed bug with Load SBET/SMRMSG not correctly loading all files entered
Fixed bug with benchmarking not logging at exactly 1 sec intervals
Fixed bug with ComputeTPU sometimes not processing with delayed heave when applicable
Minor fixes with handling carriage returns and timestamps in log

v1.2.1¶

New Processing Log. Will open on completion, saved in the ProjectSheetDataSeparatesI_Acquisition_&_Processing_LogsProcessing_LogsVessel folder. It is an excel workbook that will automatically populate with each day run for that vessel.
Please verify the information in the log with the information you normally get from Caris! And let me know if there are any inconsistencies.
SV concatenator now correctly reports the number of profiles found.
Updated Charlene output process to correctly log the processes run in the caris_log file
TCARI now runs as thread and should not lock up Charlene during operation. There may be a brief period at the end of loading TCARI water level data where it will still lock up. This should only be for a few seconds.

v1.2.0¶

I’ve determined that there is a bug in the caris batch engine related to sidescan processing/towfish navigation. This only affects Charlene and those of you who might be using carisbatch for processing. The regular Sidescan processing through Caris appears to work fine.
Revamped distance metric developed. Distance reported is based off assumption that HDCS lines are straight. Curved lines will throw it off. Reported distance should mirror Caris line query to ~0.01 NM (This is reported in the Charlene output window after processing finishes running)
Added Process modes for TJ. This is to reflect the difference between the RP being the transducer for Kongsberg (FA and RA) and the RP being the IMU on the TJ.
New option for selecting multiple lines on ‘Load SBET’. Removed Create SBET and Import Aux Data. User should use the existing processing options for this once POSPac 8 becomes available.
Fixed a bug where SBET and RMS files were getting mixed up on occasion and applied incorrectly
Load SBET no longer applies Gyro Pitch and Roll
Removed pop up on computing GPS Tide with a static value of 0.0
Added delayed heave on merge options for processes that don’t load .000 directly (i.e. first Charlene run loads .000, second run loads SBET and VDatum w/ Delayed Heave. This will now merge with delayed heave.)

v1.1.1¶

Add 2m option. Broke out raster surfacing by object detection / complete coverage finalized depth ranges.
VR can now create specific resolution tifs. Select TIF and a resolution check box to create a TIF at that resolution.
User can scroll through the output window while process is running
Merge Geotiff now works with paths with spaces
Hides progress bar on completion (helps signify process is complete)
Runs SVP concatenator only on MBES processing

v1.1.0¶