Steps#

Steps are the individual operations that make up a pipeline. They are executed in the order they appear in your configuration file. Each step can be customised with parameters to suit your specific dataset.

A step is defined in Python and configured through the pipeline config file. Examples include:

  • I/O steps — e.g. reading in data with Load Data.

  • Variable processing steps — e.g. adjusting existing salinity measurements with the salinity step.

Steps are not limited to one per file — a single module can define multiple steps — and any step can be called multiple times within a single config.

Some steps can filter out bad data before processing, using the QC handling mechanism. The pipeline preserves the original dataset dimensions, so filtered data is either replaced or reinserted once the step completes. For how to build a step with this behaviour, see Extending pelagos_py.

Non-specific Steps#

These steps are general utility tools used for data management, preparation, and export. They are not tied to a specific sensor or variable.

  • Apply QC: A container step used to run various quality control.

  • Derive CTD: Calculates derived physical properties such as density or potential temperature.

  • Export: Saves the final processed dataset to a specified format.

  • Find Profiles: Segments a time series into individual vertical profiles.

  • Generate Data: Creates synthetic data for testing and validation purposes.

  • Interpolate Data: Fills gaps in variables using mathematical interpolation.

  • Load Data: The entry point for importing data into the pelagos_py.

  • Find Profile Direction: Identifies whether data was collected during an ascent or descent.

Variable Specific Steps#

  • CTD: Specialised adjustments for Conductivity, Temperature, and Depth data.

  • Oxygen: Corrections and calibrations for dissolved oxygen sensors.

  • Chlorophyll: Processing steps for fluorescence and chlorophyll-a concentration.

  • Backscatter: Optical backscatter processing and scaling.

Quality Control (QC)#

Quality Control is handled differently from other steps. Rather than performing a single calculation, the Apply QC step acts as a manager for multiple sub-steps called QC.

When you add Apply QC to your configuration, you define a list of QC (such as range checks or spike tests) within its parameters. This step is responsible for:

  1. Running QC: Executing individual quality checks on your variables.

  2. Merging Flags: Combining results from multiple QC into a single QC column using standardised Argo flagging logic.

  3. Diagnostics: Generating visualisations to show which data points were flagged and why.

For a full list of available QC and how to configure them, please refer to the qc page.