Charlene - Automated Data Processor


  • Caris 10.3 or greater, licensed and configured
  • Caris support files installed and configured
  • POSPac 8.1 or greater, licensed and configured

Charlene is an automated night processing and data transfer tool developed by NOAA’s Office of Coast Survey in early 2017. Charlene allows the user to:

  1. Perform verification of raw data
  2. Build deliverable directory structure
  3. Transfer and verify raw data
  4. Process MBES and SSS data with Caris Batch Processor
  5. Generate SBETs with POSPac Batch
  6. Use NOAA tools like AutoQC, QCTools and TCARI

with only a few mouse clicks!

Brief Overview

Charlene is a HSTB developed software utility that automates hydrographic ‘night processing’. Night processing includes all of those tasks in between raw data collection and a final daily product that occur each night on our hydrographic vessels. Our goal with Charlene is to make night processing something that requires little to no user interaction.


While we have long been able to read and write certain proprietary data formats, the latest Caris and Applanix versions (10.2 and 8.1 respectively) open machine access to the processes in these packages; we now have the ability to run these software packages outside of the graphical interface. Charlene leverages this to become a one-stop shop for night processing. Just plug in your data drive, sit back, and wait for your processed data to arrive. Key to all this is that Charlene works across software platforms; it can transfer your raw data to the appropriate submission folders, process your SBET, convert survey lines into Caris, apply sound velocity profiles, water levels, and SBETs, run in-house QC reports, and generate logs.

How can we do this?

Charlene relies on individual modules to process. These are all callable within Python, allowing for robust and intelligent operation.

Caris Batch

Charlene utilizes the new Caris Batch, which is a command line driven tool new in Caris 10 for automating Caris processing. Charlene takes these commands and wraps them in Python, so they can be used from the Charlene GUI. This makes Caris Batch user friendly, as the native commands are often long and complicated, unsuitable for the average user. Below is an example for just the data conversion step for nine lines using Caris Batch.

“C:\Program Files\CARIS\HIPS and SIPS\10.3\bin\carisbatch.exe” –run ImportToHIPS –input-format KONGSBERG –input-crs EPSG:26910

–convert-navigation –gps-height-device GGK “E:\BatchHydroData\Test_&_Calibration\HSRR_2017\Data\MBES\FA_2805_EM2040\2017-142\0000_20170522_220650_FA2805.all”










And this is a simplified example. You can see that the command itself would be obnoxious to run directly. By wrapping it in Python, we can determine many of these options programmatically without direct input from the user.

POSPac Batch

With POSPac 8, we now have the use of the POSPac command line toolset, which allows the user to run POSBat files and a template file to create SBET files. Charlene uses specific POSBat files for each processing type (Smartbase, Marinestar, and RTX) and different export options for coordinate systems (NAD83 vs WGS84). Charlene is able to create an SBET in the middle of Caris processing and load the generated SBET during night processing, if the satellite data is available.

Pydro Tools

As much of Pydro is modular and all of it is written in Python, incorporating existing Pydro tools midstream in Charlene is relatively simple. Simple calls in Charlene allow for these tools to be used during night processing without running a separate GUI.

Charlene Tools

Charlene is currently configured to do several things on completion, including:

  1. Automatically run Caris with project and lines open for review
  2. Generate a Processing Log as an excel workbook (or add a tab for a new day number if one has been started. See Figure 7)
  3. If ‘Benchmark’ is selected, display benchmarking results.
  4. If a QCTools option is chosen, show the QCTools output directory for review

Changes List


  • Added logic to handle different versions of psutil library
  • Charlene converts correctly from raw directory instead of transfer drive
  • Will prefer existing TCARI file for loading water levels
  • Added functionality to streamline POSMV file reading
  • Will create process log folder if it didnt exist prior to reprocessing
  • Added utmzone from pos file function
  • Added check for mixed up files (mbes in pos folder, etc.)
  • Added timestamp check (see if files are in correct day number folder)
  • Added ‘stubfile’ check, Caris will not convert MBES files under 500kB
  • Added Autoqc to tools - create sbet
  • Added optional tide/hvf paths to create sbet
  • Added option to not transfer HVF if pointed at HVF in Charlene structure
  • New: SHAM, automated feature attribution tool
  • Fix for SHAM AGEN and FIDS attribution
  • Removed waterline from SHAM traditional/tides workflow
  • Added popup dialog for HYPACK SSS frequency selection
  • Added day number to Charlene log file name
  • Added project and sheet to Charlene log
  • Fix bug in Charlene log for not logging tide loaded when using TCARI
  • Added minor version and Pydro rev number
  • Added antenna type AT1675 for SBET generation
  • Added logic to prevent hanging on loading settings from previous run
  • Made switching modes and updating UI more efficient


  • Added ‘Are you sure’ confirmation prompt to reconversion
  • Process New Day will overwrite raw and proc data on prompt
  • Sheet name now adds _MB or _SSS if you don’t have it already (NOAA only)
  • Optimize MBES/POSMV checking
  • Fix Charlene log carriage returns
  • File transfer now transfers acq logs in ship mode
  • Acquisition Log button updates files to transfer window correctly
  • Updated SSS documentation
  • Files to transfer window correctly segregates between MBES and SSS


  • Changes to create raw drive/filetransfer to support/document pyinstaller operation
  • Marinestar check includes vbs mode
  • Generate new xlsx offset report
  • Include reading caris vessel shape
  • Save all offsets as float type in rawdatachecks
  • Fix bug with svp being searched for by files to transfer window during sss
  • Fix bug with acq_log not being defined for SSS in filetransfer
  • Bug fix in create raw drive - if batchprocessor file doesnt exist
  • Added support for GA530 antennas in sbet creation
  • Added close for vdatum file handler when finding band name
  • Caris license check now displays output in message box
  • Caris license check catches environment issues
  • Only transfer no longer checks Charlene settings in prechecks
  • Added YYYY-DDD format descriptor for day number reproc box
  • Svp zone check now skips non svp files
  • Sbet generation now skips non .000 files
  • Filetransfer will only transfer appropriate mbes/sss files


  • New - Chucky, Charlene’s less elegant brother. Chucky allows for running Charlene outside of the GUI using python functions. More info is available in the documentation.
  • New - Tools - Create Offset Report. Allows you to pull offsets from hvf, .all, and .000 files and write them out to csv
  • .all is the only MBES file format currently supported
  • .000 reader currently does not read out ref to imu mounting angles. I’m working on it...
  • New - Tools - SNM From Tiff. Get a number for SNM from a tiff.
  • Tools - Merge Geotiff. New Calculate SNM checkbox to get a number for SNM after merging tiffs
  • Moved to QCTools2 for Raster and VR grids
  • Rawdatachecks now only looks for marinestar status in a single line to continue
  • AutoQC now runs as subprocess, allows for separation from GUI
  • If you see a python crash here please let me know. I have seen intermittent python crashes after autoqc finishes. Even with the crash, it should complete and continue on with the rest of processing.
  • Fixed some bugs with the project/sheet/vessel/day boxes
  • Fixed lost handle for progress bar bug
  • Added support for NAD83_MLLW and WGS84_MLLW vdatum band names
  • Added nightly xlsx process log to cover for situation where master log is open and unable to access
  • xlsx process log now discriminates at vessel and day levels
  • Fix bug for svp file count
  • Added headers to daily benchmark spreadsheet file


  • Include hypack sidescan processing
  • Include discover processing
  • Add new directory structure option. Points to ini file to determine layout of final directory structure. Allows for supporting multiple years’ Specs directory structures.
  • Add new ini file checker for mandatory options.
  • Add new ‘NOAA Support Files’. Uncheck to allow Charlene to run without the use of the NOAA Caris Support Files.
  • Updated recompute towfish nav to run without any boxes checked. (verified with edgetech acquisition)
  • Add new UTM zone reference image
  • Updated all internal workings to new ini file structure
  • Add leading zeros to create transfer drive calendar dialog
  • Modified create sbet tool - new field names for sbet and smrmsg that make more sense
  • Updated documentation



  • General efficiency improvement and reorganization of codebase
  • New documentation available in PydroXL for Charlene
  • New References available from toolbar - julian day calendar and ref systems
  • HVF now separate from transfer drive. Each run, user must point to HVF locally/on network.
  • Transfer all additional files during processing (tides, vdatum) and use them from the destination
  • New Option - Raw Data Checks - Examines POS files and Kongsberg files for any issues that might cause processing problems
  • Raw Data Checks, POS - Solution status and FDIR warning messages. Marinestar processing will not occur if Marinestar solution is not present
  • Raw Data Checks, Kongsberg - Looks for GGK and SV profiles. If GGK not present (and gps height isn’t coming from an external source), will not compute GPS Tides.
  • Reprocess - Only merge/TPU if required on Reprocessing
  • Reprocess - Charlene now runs ‘Reprocess…’ from Destination instead of from Source (You no longer reprocess off of the transfer drive)
  • Reprocess - Now works successfully on half-converted projects. No need to delete directories to restart as long as transfer worked successfully.
  • Ship mode - process data without launch transfer drive, as long as you can see all data from computer!
  • Reprocess and Ship modes - use dropdown boxes for choosing Project/Sheet/Vessel/Day. Populate based off of PSVD in the TO directory
  • Just Transfer mode - Only transfer data without any processing
  • Prechecks - examines user entered options and data to see if there are any issues before start
  • Create Transfer Drive - No longer creates vesselconfig folder.
  • Create Transfer Drive - allows for additional vessel in run
  • Create Transfer Drive - Add calendar button for selecting day number
  • Open/Save Session - Saves path to config directory
  • ProgressBar - now dynamically creates processes in order of operations
  • ProgressBar - SBET and SSS processes now register in ProgressBar
  • GUI - Buttons now include text instead of generic select message
  • GUI - Now correctly freezes/refreshes after operations
  • GUI - Precheck messages now show as message boxes instead of exceptions
  • Processing log - Added messages for each process in log
  • Processing log - Includes SSS options
  • Processing log - includes all processing options and raw data check summary
  • TPU - now accepts values out to 3 decimal places
  • Backscatter/SSS - Mosaic options with resolution selection checkboxes
  • SVP Concat - Now creates master sheet-wide svp file as well
  • TCARI - TCARI now saves loaded waterlevels to tc file.
  • POS/SBET - New options for specifying antenna type for PCO/PCV values
  • POS/SBET - If SBET is not generated, Charlene will downgrade processing to loading only .000 delayed heave (not attempting to load from SBET/SMRMSG)
  • POS/SBET - New marinestar (height and heave, no height rms) and RTK (height and heave, all rms) options for loading POS files
  • POS/SBET - Added in Tools a separate program for generating SBET
  • HIBASE Database - Update database for processin options and time elapsed recording
  • HIBASE Database - Moved to Vincenty calculation for LNM to account for WGS84 ellipsoid


  • File transfer now correctly excludes other MBES formats
  • Create Transfer Drive - correctly saves options
  • Update s7k conversion to reflect Hassler option set
  • Fix bug with AutoQC and tidefiles not loading correctly
  • Create SBET now also correctly loads .000 File/delayed heave
  • SVP/projection message now prints only when new projection determined from svp file
  • SVP concat now supports SSM header
  • SVP concat tool gives metrics on duplicates/excluded casts
  • Widen progress bar window to prevent wraparound
  • Fix issue with autoqc hanging on loading waterlevels from TCARI
  • Fix save session overwriting existing options sometimes
  • Fix sidescan/mbes sometimes getting mixed together
  • Fix sidescan/mbes files showing up together on transfer window
  • Fix bug in carisapi mosaic call, would fail on path with space
  • Added overwrite prompt to save session to deal with accidental file overwrites by user
  • Opening session now clears fields that are not in the file
  • Fix bug with benchmark log overwriting itself


  • Reported from the RA that there were issues reading from Sound Speed Manager exported .svp files. Charlene’s SVP concatenator will now read these just fine. I added some text at the end to give a status of the run.
  • Added an automatic print-to-pdf for the caris and applanix log.
  • Restart process is now more robust and will terminate processes individually if they fail to restart successfully


  • ProgressBar will now track the status of each individual Caris process (conversion, for instance) and report metrics on that process. Here, we see that conversion has spawned 3 threads (1 master, 2 for the two lines i’m converting) and the CPU is at around 55.4% utilization. Conversion is reading from the network at about 6MB/s and writing to the network storage at 3MB/s.
  • ProgressBar has an Emergency Stop button that is the same as the old ‘Nuke Caris’ button with some additional streamlining in the function
  • The Restart Process button is intended to be a solution to the hung processes issue the FA is experiencing. When the button is clicked, the process (Conversion in this instance) will restart. If it is halfway done, it will reconvert the already-converted lines and then convert the unconverted lines. I believe these hung processes are likely due to network congestion and the NetApp being unresponsive and overloaded. I believe the restart might allow you to get past this issue.
  • The Idle Timer will tell you how long Read and Write are below 5MB/s. The idea here is to tell you if it has been idle for a long time, and might need a restart.
  • I’ve found that two instances of Caris converting into a new project will cause an error (‘ERROR: Unable to decode Auto UTM key’). I talked to Caris about it and they told me that ‘Basically the batch #2 can’t even get the HIPS file open to see if it can start processing because Windows is blocking access while batch #1 is busy writing to it.’ To deal with this issue, I’ve built a loop that, if the error appears, will stall processing until the HIPS file is available. You’ll know if it is happening, as the error will appear in Charlene and the progress bar will say ‘OnHold, Hips Unaccessable’ with a 30 sec timer. Timer runs out, process tries again, hopefully finding the HIPS file to be accessible this time.
  • No longer brings up excel log after process finishes (causes issues in multiuser tests). Will still save/create log.
  • Modified POSPac processing to work within 8.1. This includes changing Charlene’s posbat files, handling some path issues, some other minor issues. FH has used Charlene Marinestar SBET creation with success.
  • Issues with AutoQC within Charlene. Disabled for now until a solution is determined.
  • Fixed issues with determining Caris/POSPac version


  • Some issues with ‘Raster/Finalize - compcoverage’ not creating the correct depth ranges. These have been resolved.
  • Charlene now uses checkboxes for QCTools items. QCTools calls will no longer halt the operation if there is an issue.
  • Failing to send benchmarking statistics would occasionally cause Charlene to halt. This has been resolved.
  • Some issues with Charlene not correctly prioritizing 10.3 locations for support file searching. This is fixed.


  • VR surfacing is back in. You have to have Caris 10.3 to have the fix. Please don’t do VR in Charlene unless you are using 10.3. I’ll be restricting it to 10.3 and higher in the future.
  • Significantly reduced loading time from config file
  • Cleaned up the interface a bit, with explanations of fields
  • Auto-populate projection if you are using a SVP file
  • Segregate ‘stub files’ AKA ‘fat finger files’ - MBES files smaller than 500kB
  • Automatically captures data from night processing run. You can see the data from all the ships at (Requires a Hibase login)
  • Fixed bug due to rounding errors with EPSG estimation tool


  • Fixed check for Caris license. Should now produce error message if Caris isn’t properly licensed.
  • Added fix for program not initializing properly if GUI fields are set to nonexistent folders.
  • Data transfer will now segregate .wcd files to a separate directory before transfer/processing.
  • Added fix for progressbar not working properly when loading tides and processing HYPACK data
  • In addition, I wanted to provide an update on some Charlene related items (not related to normal Caris processing):
  • Sidescan in Charlene - Recompute Towfish Navigation does not work in the batch engine. Working to get this into 10.3.2 hopefully.
  • VR in Charlene - We’ve verified that the VR surface you get from the batch engine is nearly identical. There is an issue that might be related to the order of operations in how VR is generated in the batch engine. Working with Caris to resolve this.
  • Project Coordinate Reference System (CRS) in Charlene - I might not have communicated this very well. Charlene is unable to set the project CRS (defaults to WGS84). When you create a new project using Charlene and you want NAD83, you’ll need to change it from WGS84 to NAD83 the first time you open it. It’s a quick process, see below:


  • Sidescan and VR have been removed from the GUI.
  • There are some differences in how the Batch VR and the normal Caris GUI VR behave. I think it is still related to how the cube parameters file is accessed. Until we figure it out, I’ve removed it from Charlene.
  • The Sidescan compute towfish issue still exists for Charlene.
  • New file transfer process utilizing Windows Robocopy and Python filecmp. Files are transferred and verified by name and other properties. Should produce a rock solid file copy mechanism.
  • Raster only processing now can include QCTools
  • Refined SVC concatenation to a single day number folder
  • Fixed bug with Load SBET/SMRMSG not correctly loading all files entered
  • Fixed bug with benchmarking not logging at exactly 1 sec intervals
  • Fixed bug with ComputeTPU sometimes not processing with delayed heave when applicable
  • Minor fixes with handling carriage returns and timestamps in log


  • New Processing Log. Will open on completion, saved in the ProjectSheetDataSeparatesI_Acquisition_&_Processing_LogsProcessing_LogsVessel folder. It is an excel workbook that will automatically populate with each day run for that vessel.
  • Please verify the information in the log with the information you normally get from Caris! And let me know if there are any inconsistencies.
  • SV concatenator now correctly reports the number of profiles found.
  • Updated Charlene output process to correctly log the processes run in the caris_log file
  • TCARI now runs as thread and should not lock up Charlene during operation. There may be a brief period at the end of loading TCARI water level data where it will still lock up. This should only be for a few seconds.


  • I’ve determined that there is a bug in the caris batch engine related to sidescan processing/towfish navigation. This only affects Charlene and those of you who might be using carisbatch for processing. The regular Sidescan processing through Caris appears to work fine.
  • Revamped distance metric developed. Distance reported is based off assumption that HDCS lines are straight. Curved lines will throw it off. Reported distance should mirror Caris line query to ~0.01 NM (This is reported in the Charlene output window after processing finishes running)
  • Added Process modes for TJ. This is to reflect the difference between the RP being the transducer for Kongsberg (FA and RA) and the RP being the IMU on the TJ.
  • New option for selecting multiple lines on ‘Load SBET’. Removed Create SBET and Import Aux Data. User should use the existing processing options for this once POSPac 8 becomes available.
  • Fixed a bug where SBET and RMS files were getting mixed up on occasion and applied incorrectly
  • Load SBET no longer applies Gyro Pitch and Roll
  • Removed pop up on computing GPS Tide with a static value of 0.0
  • Added delayed heave on merge options for processes that don’t load .000 directly (i.e. first Charlene run loads .000, second run loads SBET and VDatum w/ Delayed Heave. This will now merge with delayed heave.)


  • Add 2m option. Broke out raster surfacing by object detection / complete coverage finalized depth ranges.
  • VR can now create specific resolution tifs. Select TIF and a resolution check box to create a TIF at that resolution.
  • User can scroll through the output window while process is running
  • Merge Geotiff now works with paths with spaces
  • Hides progress bar on completion (helps signify process is complete)
  • Runs SVP concatenator only on MBES processing



  • Charlene now automatically concatenates all .svp files and creates a master vessel .svp file. This file is checked to ensure all svp files have a valid position.
  • Charlene now creates nightly surfaces saved to the ‘ProjectSheetWorking_Surfaces_&_Mosaics’. The user can then vet the surface and perform nightly cleaning before adding the data to the master surface in the Specs appropriate directory. This update was designed to bypass issues with adding to existing surfaces and more closely follow existing night processing workflows. User will now get an error if attempting to create a surface which already exists.
  • Added ‘MergeGeoTiff’ feature (Tools - Merge GeoTiff). This will find all .tif/.tiff files in a directory (including/not including subdirectories) and combine them into a master geotiff using GDAL libraries. User can now go from Charlene generated GeoTiffs to a master geotiff for that day’s acquisition entirely through Charlene.
  • Create Raw Data Drive will now find and display existing directory names automatically. If the user wishes to enter names not found, he/she can type them into the dropdown boxes.
  • Velodyne csv to s57 tool added for VLP-16 users
  • New Default Window Layout option (View - Default Window Layout) will restore windows the user accidentally (or intentionally) deletes.
  • Added warning box if user selects reconversion to prevent accidental reconversion.


  • All pre-start checks occur before creation of any directories. Prechecks now include all user options and included files.
  • QCTools output folder will now match the sheet name.
  • Modified SSS mosaic creation to be inline with existing TJ SOPs.
  • All GUI options are hidden/shown appropriately depending on processing mode.
  • Transfer Data display now only shows .hvf being transferred.
  • TCARI window related to data out of grid will no longer pop up.

Indices and tables


Online Resources:


If you find errors in the documentation or want to contribute, you are encouraged to email the following personnel emails @NOAA.GOV

  • barry.gallagher
  • jack.riley
  • chen.zhang
  • eric.younkin