pelagos_py.steps.input_output.load_data#

class pelagos_py.steps.input_output.load_data.LoadOG1(name, parameters=None, diagnostics=False, context=None)[source]#

Bases: pelagos_py.steps.base_step.BaseStep

Loads NetCDF files from file_path.

If filter_bad_times is set then measurements made outside of the time range specified by data_start and data_end are removed before further processing.

Time values are expected to be stored under the “TIME” variable name in the NetCDF file (corresponding to OG1 format). If “TIME” is not monotonically increasing, or has missing values (NaT) then this will raise an error.

Parameters:

  • filter_bad_time (bool, optional) – If True (default), removes all timestamps outside the expected time window.

  • data_start (str or np.datetime64, optional) – The minimum valid timestamp for the data. If not provided, the filter defaults to the DEPLOYMENT_TIME found in the dataset, or 1990-01-01T00:00:00 if no deployment time is found.

  • data_end (str or np.datetime64, optional) – The maximum valid timestamp for the data. If not provided, it defaults to the current system time when the pipeline is run.

Examples

Example usage in a pipeline configuration:

steps:
  - name: Load OG1
    parameters:
      file_path: "/path/to/your/dataset.nc"
      filter_bad_time: false
      data_start: "2023-05-01T00:00:00"
      data_end: "2024-05-01T00:00:00"
normalise_qc_flags()[source]#

Coerce QC flag variables to integers, mapping NaN to 9 (missing).

Raw OG1 files store QC flags as float with NaN (which lines up with missing measurements). Downstream steps use flags as integer array indices / dict keys, so normalise them once at load rather than having each step guard against float NaN flags.

generate_diagnostics()[source]#

Print a structural summary of the loaded dataset.

Called automatically at the end of run() when diagnostics is enabled. Delegates to pelagos_py.utils.diagnostics.generate_info(), which prints the dataset’s dimensions, variables and global attributes (via xarray.Dataset.info()) to stdout — a quick check that the data was loaded as expected.