src.toolbox.pipeline#

Pipeline class definition to handle configuration and step execution.

Classes#

Pipeline

Pipeline that manages a sequence of processing steps.

Module Contents#

class src.toolbox.pipeline.Pipeline(config_path=None)[source]#

Bases: toolbox.utils.config_mirror.ConfigMirrorMixin

Pipeline that manages a sequence of processing steps.

Config-aware pipeline that can:

  • Load config YAML into private self._parameters

  • Keep global_parameters mirrored to _parameters[‘pipeline’]

  • Build, run, and export steps as before

Parameters:

ConfigMirrorMixin (Class) – Class to handle configuration

steps = [][source]#
graph[source]#
global_parameters[source]#
build_steps(steps_config)[source]#

Build steps from configuration.

Individual steps, including parameters and diagnostics, are saved to self.steps using add_step() for other functions.

Parameters:

steps_config (list of dict) – List of step configurations.

add_step(step_name, parameters=None, diagnostics=False, run_immediately=False)[source]#

Dynamically adds a step and optionally runs it immediately.

Parameters:

  • step_name (str) – Name of the step to add.

  • parameters (dict, optional) – Parameters for the step.

  • diagnostics (bool, optional) – Whether to enable diagnostics for this step.

  • run_immediately (bool, optional) – Whether to run the step immediately after adding it.

Raises:

ValueError – If the step name is not recognized.

execute_step(step_config, _context)[source]#

Executes a single step.

Parameters:

  • step_config (dict) – Configuration for the step to execute.

  • _context (dict) – Current context to pass to the step.

run_last_step()[source]#

Runs only the most recently added step based on the index in self.steps.

run()[source]#

Runs the entire pipeline.

If visualisation is specified in the configuration parameters, a visualisation of the pipeline execution will be generated.

visualise_pipeline()[source]#

Generates a visualisation of the pipeline execution.

generate_config()[source]#

Generate a configuration dictionary from the current pipeline setup.

Returns:

Configuration dictionary of the current pipeline.

Return type:

dict

export_config(output_path='generated_pipeline.yaml')[source]#

Write current config to file (respects private _parameters)

Parameters:

output_path (str) – Path to save the exported configuration YAML file.

Returns:

Configuration dictionary of the current pipeline.

Return type:

dict

save_config(path='pipeline_config.yaml')[source]#

Save the canonical private config (same as manager.save_config).

Parameters:

path (str) – Path to save the exported configuration YAML file.

get_data()[source]#

Returns data from the current pipeline context.