From 4face14944f8de01b10243201881cebc82c23306 Mon Sep 17 00:00:00 2001 From: Tom Durrant Date: Thu, 16 Apr 2026 10:26:58 +1000 Subject: [PATCH 1/2] Align pipeline docs with current config format Replace the stale standalone migration guide and missing WW3-based example references with core rompy pipeline examples. Update the published CLI and backend docs to document the nested pipeline config structure and current override flags. --- docs/PIPELINE_MIGRATION.md | 299 ------------------ docs/backends.md | 7 +- docs/cli.md | 123 ++++--- docs/developer/backend_reference.md | 5 +- examples/configs/README.md | 138 ++++---- examples/configs/basic_modelrun.yml | 4 +- examples/configs/basic_pipeline.yml | 31 +- .../configs/basic_pipeline_with_includes.yml | 8 + 8 files changed, 183 insertions(+), 432 deletions(-) delete mode 100644 docs/PIPELINE_MIGRATION.md create mode 100644 examples/configs/basic_pipeline_with_includes.yml diff --git a/docs/PIPELINE_MIGRATION.md b/docs/PIPELINE_MIGRATION.md deleted file mode 100644 index cf892053..00000000 --- a/docs/PIPELINE_MIGRATION.md +++ /dev/null @@ -1,299 +0,0 @@ -# Pipeline Command Migration Guide - -## Overview - -The `rompy pipeline` command has been redesigned to support a cleaner, more consistent configuration structure with YAML include functionality. - -## What Changed - -### Old Structure (Deprecated) - -```bash -# Separate files required -rompy pipeline model.yaml \ - --run-backend local \ - --processor-config processor.yaml -``` - -**Problems with old approach:** -- `--run-backend` only accepted type string ("local", "docker") -- No way to configure backend timeout, commands, or environment variables -- Mandatory separate processor config file -- Inconsistent with `rompy run` command structure - -### New Structure - -```bash -# All-in-one pipeline config -rompy pipeline pipeline.yaml - -# Or with overrides -rompy pipeline pipeline.yaml \ - --backend-config custom_backend.yaml \ - --processor-config custom_processor.yaml -``` - -**Benefits:** -- Single file for complete workflows -- Full backend configuration support -- Optional inline or referenced configs -- YAML `!include` directives for composition -- Consistent nested structure - -## New Config Structure - -All pipeline configs must now use this three-section structure: - -```yaml -config: # ModelRun configuration - run_id: my_run - period: - start: "2023-01-01T00:00:00" - end: "2023-01-02T00:00:00" - config: - model_type: shel - # ... model-specific config - -backend: # Backend configuration - type: local # or docker, slurm - timeout: 7200 - command: ./run.sh - env_vars: - VAR: value - -postprocessor: # Postprocessor configuration - type: ww3_transfer - destinations: - - "file:///backup/" - timeout: 3600 -``` - -## YAML Include Support - -### Basic Include - -Compose configs from multiple files: - -```yaml -config: !include model_config.yaml -backend: !include backends/local.yaml -postprocessor: !include postprocessors/transfer.yaml -``` - -### Mixed Inline and Includes - -```yaml -config: !include shared/global_model.yaml - -backend: - type: local - timeout: 7200 - -postprocessor: !include postprocessors/ww3_transfer.yaml -``` - -### Include Features - -- **Relative paths**: Resolved from the including file's directory -- **Absolute paths**: Supported -- **Environment variables**: `!include $HOME/configs/backend.yaml` -- **Nested includes**: Includes can reference other includes -- **Circular detection**: Prevents infinite loops -- **Depth limit**: Maximum 10 levels of nesting - -## Migration Examples - -### Example 1: Basic Migration - -**Old:** -```bash -# model.yaml -run_id: test -period: {...} -config: {...} - -# Command -rompy pipeline model.yaml \ - --run-backend local \ - --processor-config proc.yaml -``` - -**New:** -```yaml -# pipeline.yaml -config: - run_id: test - period: {...} - config: {...} - -backend: - type: local - -postprocessor: !include proc.yaml -``` - -```bash -rompy pipeline pipeline.yaml -``` - -### Example 2: With Backend Options - -**Old (not possible):** -```bash -# Could NOT configure timeout or custom commands -rompy pipeline model.yaml --run-backend local --processor-config proc.yaml -``` - -**New:** -```yaml -config: !include model.yaml - -backend: - type: local - timeout: 7200 - command: ./custom_run.sh - env_vars: - OMP_NUM_THREADS: "4" - -postprocessor: !include proc.yaml -``` - -### Example 3: Modular Configs - -**Organize by environment:** -``` -configs/ -├── models/ -│ ├── global_3deg.yaml -│ └── regional_1deg.yaml -├── backends/ -│ ├── local_dev.yaml -│ ├── local_prod.yaml -│ └── docker.yaml -├── postprocessors/ -│ ├── transfer_s3.yaml -│ └── transfer_local.yaml -└── pipelines/ - ├── dev_pipeline.yaml - └── prod_pipeline.yaml -``` - -**dev_pipeline.yaml:** -```yaml -config: !include ../models/global_3deg.yaml -backend: !include ../backends/local_dev.yaml -postprocessor: !include ../postprocessors/transfer_local.yaml -``` - -**prod_pipeline.yaml:** -```yaml -config: !include ../models/global_3deg.yaml -backend: !include ../backends/local_prod.yaml -postprocessor: !include ../postprocessors/transfer_s3.yaml -``` - -## CLI Override Behavior - -CLI flags override inline configurations: - -```yaml -# pipeline.yaml -config: {...} -backend: - type: local - timeout: 3600 -postprocessor: - type: noop -``` - -```bash -# Override backend -rompy pipeline pipeline.yaml --backend-config docker_backend.yaml - -# Override postprocessor -rompy pipeline pipeline.yaml --processor-config custom_proc.yaml - -# Override both -rompy pipeline pipeline.yaml \ - --backend-config docker_backend.yaml \ - --processor-config custom_proc.yaml -``` - -## Breaking Changes - -### Removed - -1. **`--run-backend` flag** - Replaced with nested `backend` config or `--backend-config` -2. **Flat config structure** - All configs must now use nested `config`/`backend`/`postprocessor` sections - -### Required Changes - -| Old | New | -|-----|-----| -| `--run-backend local` | `backend: {type: local}` in config or `--backend-config` | -| `--processor-config FILE` (required) | `postprocessor:` in config or `--processor-config` (optional) | -| Top-level `run_id`, `period` | Nest under `config:` section | - -## Error Messages - -### Missing Sections - -``` -Error: Pipeline config must contain 'config' section with ModelRun configuration. -``` -**Fix:** Wrap your model config in a `config:` section. - -``` -Error: Backend configuration required. Provide either: - - 'backend' section in pipeline config, or - - --backend-config flag -``` -**Fix:** Add `backend:` section or use `--backend-config`. - -``` -Error: Postprocessor configuration required. Provide either: - - 'postprocessor' section in pipeline config, or - - --processor-config flag -``` -**Fix:** Add `postprocessor:` section or use `--processor-config`. - -### Include Errors - -``` -Error: Included file not found: configs/model.yaml -Referenced from: /path/to/pipeline.yaml -``` -**Fix:** Check the file path is correct relative to the including file. - -``` -Error: Circular include detected: /path/to/file1.yaml -Include chain: file1.yaml -> file2.yaml -> file1.yaml -``` -**Fix:** Remove circular references in your includes. - -## Backward Compatibility - -**None.** This is a breaking change. The old `--run-backend` string flag is no longer supported. - -All existing pipeline commands must be updated to use the new structure. - -## Best Practices - -1. **Use includes for reusability**: Keep backend and postprocessor configs in separate files for reuse -2. **Environment-specific configs**: Create separate pipeline files for dev/staging/prod -3. **Validate early**: Use `rompy pipeline --help` to see expected structure -4. **Keep model configs portable**: Model configs (`config:` section) should not contain backend/processor details -5. **Version control**: Commit all config files including referenced includes - -## Examples - -See the `examples/` directory for complete working examples: - -- `global_3deg_pipeline.yaml` - All inline configuration -- `global_3deg_pipeline_with_includes.yaml` - Using !include directives - -## Questions? - -- Run `rompy pipeline --help` for command documentation -- Check existing tests: `tests/test_yaml_loader.py` for include examples -- See `src/rompy/core/yaml_loader.py` for include implementation details diff --git a/docs/backends.md b/docs/backends.md index 8a7f60ee..beaa8332 100644 --- a/docs/backends.md +++ b/docs/backends.md @@ -242,7 +242,7 @@ rompy backends create --backend-type local --output template.yml rompy run model_config.yml --backend-config my_backend.yml # Run pipeline with configuration -rompy pipeline --config pipeline_config.yml +rompy pipeline examples/configs/basic_pipeline.yml ``` ## Configuration Examples @@ -679,8 +679,7 @@ Use postprocessor configurations with CLI commands: rompy postprocess model_config.yml --processor-config processor.yml # Run complete pipeline with postprocessor -rompy pipeline model_config.yml \ - --run-backend local \ +rompy pipeline examples/configs/basic_pipeline.yml \ --processor-config processor.yml # Validate postprocessor configuration @@ -889,4 +888,4 @@ For complete API documentation, see: * `rompy.postprocess` - Postprocessor implementations * [backend_reference](developer/backend_reference.md) - Comprehensive technical reference -The backend system provides a robust, type-safe foundation for model execution while maintaining flexibility for different deployment scenarios. From simple local development to complex containerized production environments, the backend system adapts to your needs while ensuring consistent, reproducible results. \ No newline at end of file +The backend system provides a robust, type-safe foundation for model execution while maintaining flexibility for different deployment scenarios. From simple local development to complex containerized production environments, the backend system adapts to your needs while ensuring consistent, reproducible results. diff --git a/docs/cli.md b/docs/cli.md index d9b00826..f04cb769 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -101,11 +101,11 @@ rompy pipeline [] [OPTIONS] **Options:** -`--run-backend TEXT` -: Execution backend for run stage (default: local) +`--backend-config PATH` +: YAML/JSON file with backend configuration. Optional if `backend:` is defined inline in the pipeline config. `--processor-config PATH` -: **Required.** YAML/JSON file with postprocessor configuration +: YAML/JSON file with postprocessor configuration. Optional if `postprocessor:` is defined inline in the pipeline config. `--cleanup-on-failure, --no-cleanup` : Clean up outputs on pipeline failure (default: False) @@ -119,13 +119,54 @@ rompy pipeline [] [OPTIONS] **Examples:** ```bash -# Run pipeline with postprocessor configuration -rompy pipeline config.yaml --processor-config processor.yml +# Run a complete pipeline from a single config file +rompy pipeline examples/configs/basic_pipeline.yml + +# Override the backend and postprocessor from separate files +rompy pipeline examples/configs/basic_pipeline.yml \ + --backend-config examples/configs/local_backend.yml \ + --processor-config examples/backends/postprocessor_configs/noop_basic.yml +``` + +#### Pipeline Configuration Migration + +`rompy pipeline` now expects a nested three-section configuration. This is the same structure shown in `rompy pipeline --help`. + +```yaml +config: + run_id: my_run + output_dir: ./outputs + period: + start: "2023-01-01T00:00:00" + end: "2023-01-02T00:00:00" + interval: "1H" + config: + model_type: base + +backend: + type: local + timeout: 3600 + +postprocessor: + type: noop + validate_outputs: true +``` + +The old `--run-backend local` pattern is no longer supported. Define `backend:` inline or provide `--backend-config` instead. -# Run pipeline with Docker backend and custom postprocessor config -rompy pipeline config.yaml --run-backend docker --processor-config noop.yml +YAML includes are supported for pipeline composition: + +```yaml +config: !include basic_modelrun.yml +backend: !include local_backend.yml +postprocessor: !include ../backends/postprocessor_configs/noop_basic.yml ``` +Published examples: + +- `examples/configs/basic_pipeline.yml`: inline pipeline configuration +- `examples/configs/basic_pipeline_with_includes.yml`: include-based pipeline configuration + ### postprocess Run postprocessing on existing model outputs. @@ -387,8 +428,8 @@ For complete postprocessor configuration options, see [Postprocessor Configurati Orchestrate complete workflows: - **local**: Execute all stages locally -- **hpc**: HPC-optimized pipeline execution -- **cloud**: Cloud-native pipeline execution + +Additional pipeline backends can be provided by plugins, but the built-in CLI pipeline command uses the local pipeline backend and accepts typed run backend and postprocessor configs. ## Examples @@ -403,9 +444,9 @@ rompy run swan_config.yaml --backend-config local_backend.yml Complete pipeline with analysis: ```bash -rompy pipeline ocean_model.yaml \ - --run-backend local \ - --processor-config analysis.yml \ +rompy pipeline examples/configs/basic_pipeline.yml \ + --backend-config examples/configs/local_backend.yml \ + --processor-config examples/backends/postprocessor_configs/noop_basic.yml \ --validate-stages ``` @@ -424,41 +465,41 @@ rompy run config.yaml --backend-config local.yml --dry-run ## Configuration Files -### Enhanced Configuration Structure +### Configuration Structure -The modern CLI supports enhanced configuration files with run and pipeline settings: +`rompy run` takes a `ModelRun` configuration plus a separate backend config file: ```yaml -# Basic model configuration run_id: my_ocean_model period: - start: 20230101T00 - end: 20230102T00 - interval: 3600 + start: "2023-01-01T00:00:00" + end: "2023-01-02T00:00:00" + interval: "1H" output_dir: ./outputs config: - model_type: schism - # ... model-specific configuration - -# Run configuration (optional) -run: - backend: local - local: - env_vars: - OMP_NUM_THREADS: "4" - timeout: 3600 - -# Pipeline configuration (optional) -pipeline: - backend: local - local: - run_backend: docker - processor_config: - type: analysis - validate_outputs: true - timeout: 3600 - cleanup_on_failure: false + model_type: base +``` + +`rompy pipeline` takes a nested workflow config containing `config`, `backend`, and `postprocessor` sections: + +```yaml +config: + run_id: my_pipeline_run + output_dir: ./outputs + period: + start: "2023-01-01T00:00:00" + end: "2023-01-02T00:00:00" + interval: "1H" + config: + model_type: base + +backend: + type: local + timeout: 3600 + +postprocessor: + type: noop ``` ## Environment Variables @@ -521,7 +562,7 @@ kubectl create configmap rompy-config --from-file=ROMPY_CONFIG=config.yml # CI/CD Pipeline export ROMPY_CONFIG="$(envsubst < config_template.yml)" -rompy pipeline --config-from-env --run-backend docker +rompy pipeline --config-from-env --backend-config docker_backend.yml --processor-config noop.yml ``` ## Monitoring and Debugging @@ -617,4 +658,4 @@ rompy --help rompy run --help rompy pipeline --help rompy backends --help -``` \ No newline at end of file +``` diff --git a/docs/developer/backend_reference.md b/docs/developer/backend_reference.md index 105dfe87..202c9cb9 100644 --- a/docs/developer/backend_reference.md +++ b/docs/developer/backend_reference.md @@ -493,8 +493,7 @@ results = model_run.postprocess(processor=config) rompy postprocess model.yml --processor-config processor.yml # Run complete pipeline with postprocessor -rompy pipeline model.yml \ - --run-backend local \ +rompy pipeline examples/configs/basic_pipeline.yml \ --processor-config processor.yml # Validate postprocessor configuration @@ -1013,4 +1012,4 @@ For complete API documentation, see: * `rompy.backends.postprocessors` - Postprocessor implementations * `rompy.backends` - Backend discovery and registry -This reference covers the key concepts and patterns for working with Rompy's backend system. For implementation details and complete parameter documentation, refer to the API documentation. \ No newline at end of file +This reference covers the key concepts and patterns for working with Rompy's backend system. For implementation details and complete parameter documentation, refer to the API documentation. diff --git a/examples/configs/README.md b/examples/configs/README.md index 0265b3f1..7d062b9b 100644 --- a/examples/configs/README.md +++ b/examples/configs/README.md @@ -1,6 +1,6 @@ -# ROMPY Backend Configuration Examples +# ROMPY Configuration Examples -This directory contains example configuration files for ROMPY backend systems. These YAML files demonstrate how to configure local and Docker backends for various use cases. +This directory contains example `ModelRun`, backend, and pipeline configuration files for core rompy workflows. ## Configuration Files @@ -10,7 +10,8 @@ This directory contains example configuration files for ROMPY backend systems. T - **`docker_backend.yml`** - Single-document Docker backend configuration - **`slurm_backend.yml`** - Single-document SLURM backend configuration - **`basic_modelrun.yml`** - Basic model run configuration for CLI testing -- **`basic_pipeline.yml`** - Basic pipeline configuration for CLI testing +- **`basic_pipeline.yml`** - Inline pipeline configuration using built-in core rompy types +- **`basic_pipeline_with_includes.yml`** - Pipeline configuration composed with `!include` - **`local_backend_examples.yml`** - Multi-document local backend examples - **`docker_backend_examples.yml`** - Multi-document Docker backend examples - **`slurm_backend_examples.yml`** - Multi-document SLURM backend examples @@ -19,13 +20,11 @@ This directory contains example configuration files for ROMPY backend systems. T ## Configuration Format -ROMPY uses a unified configuration format that works for all use cases: +Backend config files use a typed structure with a required `type` field: ```yaml -# Backend type -backend_type: local # or 'docker' +type: local -# Configuration parameters timeout: 3600 command: "python run_model.py" env_vars: @@ -33,10 +32,25 @@ env_vars: MODEL_CONFIG: "production" ``` -This format works for: -- CLI validation -- Pipeline usage -- Programmatic usage +Pipeline config files use a nested structure: + +```yaml +config: + run_id: example_pipeline + output_dir: ./output + period: + start: "2023-01-01T00:00:00" + end: "2023-01-02T00:00:00" + interval: "1H" + config: + model_type: base + +backend: + type: local + +postprocessor: + type: noop +``` ## Using Configuration Files @@ -62,10 +76,10 @@ rompy backends create --backend-type local --output my_config.yml ```bash # Run with backend configuration -rompy run --config model_config.yml --backend-config local_backend.yml +rompy run model_config.yml --backend-config local_backend.yml # Run complete pipeline -rompy pipeline --config pipeline_config.yml +rompy pipeline basic_pipeline.yml ``` ## Configuration Options @@ -124,7 +138,7 @@ rompy pipeline --config pipeline_config.yml ### Local Backend ```yaml -backend_type: local +type: local timeout: 7200 command: "python run_model.py" env_vars: @@ -135,7 +149,7 @@ env_vars: ### Docker Backend ```yaml -backend_type: docker +type: docker image: "python:3.9-slim" timeout: 7200 cpu: 4 @@ -149,21 +163,20 @@ env_vars: ### SLURM Backend ```yaml -backend_type: slurm -config: - queue: "general" - timeout: 7200 - nodes: 2 - ntasks: 8 - cpus_per_task: 4 - time_limit: "02:00:00" - account: "myproject" - additional_options: - - "--gres=gpu:v100:2" - job_name: "simulation_job" - env_vars: - OMP_NUM_THREADS: "4" - MODEL_CONFIG: "production" +type: slurm +queue: "general" +timeout: 7200 +nodes: 2 +ntasks: 8 +cpus_per_task: 4 +time_limit: "02:00:00" +account: "myproject" +additional_options: + - "--gres=gpu:v100:2" +job_name: "simulation_job" +env_vars: + OMP_NUM_THREADS: "4" + MODEL_CONFIG: "production" ``` ### Basic ModelRun Configuration @@ -176,14 +189,14 @@ period: interval: "1H" output_dir: "./output/cli_test" delete_existing: true +config: + model_type: base ``` ### Basic Pipeline Configuration ```yaml -pipeline_backend: local - -model_run: +config: run_id: "cli_test_backend_run" output_dir: "./output/cli_test" delete_existing: true @@ -191,42 +204,28 @@ model_run: start: "2023-01-01T00:00:00" end: "2023-01-02T00:00:00" interval: "1H" + config: + model_type: base -run_backend: - backend_type: local +backend: + type: local timeout: 3600 - command: "echo 'Running basic model test'" + command: "ls -l" env_vars: - MODEL_TYPE: "test" - ENVIRONMENT: "cli" + MODEL_CONFIG: "example" + LOG_LEVEL: "INFO" -postprocessing: - processor: "noop" +postprocessor: + type: noop + validate_outputs: true ``` -### Pipeline Configuration +### Pipeline Configuration With Includes ```yaml -pipeline_backend: local - -model_run: - run_id: "example_run" - output_dir: "./output" - period: - start: "2023-01-01T00:00:00" - end: "2023-01-02T00:00:00" - interval: "1H" - config: - model_type: "swan" - -run_backend: - backend_type: local - timeout: 3600 - env_vars: - OMP_NUM_THREADS: "4" - -postprocessing: - processor: "noop" +config: !include basic_modelrun.yml +backend: !include local_backend.yml +postprocessor: !include ../backends/postprocessor_configs/noop_basic.yml ``` ## Common Use Cases @@ -234,7 +233,7 @@ postprocessing: ### Development and Testing ```yaml -backend_type: local +type: local timeout: 1800 # 30 minutes env_vars: ENV: "development" @@ -244,7 +243,7 @@ env_vars: ### Production Runs ```yaml -backend_type: docker +type: docker image: "my-model:latest" timeout: 14400 # 4 hours cpu: 8 @@ -256,7 +255,7 @@ env_vars: ### High-Performance Computing ```yaml -backend_type: docker +type: docker image: "hpc-model:latest" timeout: 86400 # 24 hours cpu: 32 @@ -269,7 +268,7 @@ mpiexec: "mpirun -np 32" Configuration files support environment variable substitution: ```yaml -backend_type: local +type: local timeout: ${ROMPY_TIMEOUT:-3600} env_vars: MODEL_CONFIG: ${MODEL_CONFIG:-production} @@ -324,7 +323,7 @@ python validate_configs.py ### Quick Start Template ```yaml -backend_type: local # or 'docker' +type: local # or 'docker' timeout: 3600 command: "your_command_here" env_vars: @@ -335,13 +334,16 @@ env_vars: ``` rompy/examples/configs/ +├── basic_modelrun.yml # Minimal ModelRun example +├── basic_pipeline.yml # Inline pipeline example +├── basic_pipeline_with_includes.yml # Include-based pipeline example ├── local_backend.yml # Simple local configuration ├── docker_backend.yml # Simple Docker configuration ├── local_backend_examples.yml # Comprehensive local examples ├── docker_backend_examples.yml # Comprehensive Docker examples -├── pipeline_config.yml # Pipeline workflow examples +├── pipeline_config.yml # Extended pipeline examples ├── validate_configs.py # Validation script └── README.md # This documentation ``` -This clean, focused configuration system provides everything needed to configure ROMPY backends for any use case, from simple local execution to complex distributed workflows. \ No newline at end of file +This clean, focused configuration system provides everything needed to configure ROMPY backends for any use case, from simple local execution to complex distributed workflows. diff --git a/examples/configs/basic_modelrun.yml b/examples/configs/basic_modelrun.yml index 52e3d8ac..f347d9fa 100644 --- a/examples/configs/basic_modelrun.yml +++ b/examples/configs/basic_modelrun.yml @@ -7,4 +7,6 @@ period: end: "2023-01-02T00:00:00" interval: "1H" output_dir: "./output/cli_test" -delete_existing: true \ No newline at end of file +delete_existing: true +config: + model_type: base diff --git a/examples/configs/basic_pipeline.yml b/examples/configs/basic_pipeline.yml index ea7a32d2..0c15c4b4 100644 --- a/examples/configs/basic_pipeline.yml +++ b/examples/configs/basic_pipeline.yml @@ -1,13 +1,13 @@ -# Complete Pipeline Configuration for CLI Testing -# This demonstrates the new unified pipeline configuration structure +# Complete pipeline configuration using only core rompy types # # Usage: -# rompy pipeline --config basic_pipeline.yml +# rompy pipeline examples/configs/basic_pipeline.yml # -# Or with external configs: -# rompy pipeline --config basic_pipeline.yml --backend-config my_backend.yml --processor-config my_processor.yml +# Override either section from the CLI if needed: +# rompy pipeline examples/configs/basic_pipeline.yml \ +# --backend-config examples/configs/local_backend.yml -# Model run configuration (required) +# ModelRun configuration config: run_id: "cli_test_backend_run" output_dir: "./output/cli_test" @@ -16,20 +16,19 @@ config: start: "2023-01-01T00:00:00" end: "2023-01-02T00:00:00" interval: "1H" - # Add your model-specific configuration here - # config: - # model_type: "swan" - # ... + config: + model_type: base -# Backend configuration (optional - can be overridden with --backend-config) +# Run backend configuration backend: type: local timeout: 3600 - command: "echo 'Running basic model test'" + command: "ls -l" env_vars: - MODEL_TYPE: "test" - ENVIRONMENT: "cli" + MODEL_CONFIG: "example" + LOG_LEVEL: "INFO" -# Postprocessor configuration (optional - can be overridden with --processor-config) +# Postprocessor configuration postprocessor: - type: noop \ No newline at end of file + type: noop + validate_outputs: true diff --git a/examples/configs/basic_pipeline_with_includes.yml b/examples/configs/basic_pipeline_with_includes.yml new file mode 100644 index 00000000..37090856 --- /dev/null +++ b/examples/configs/basic_pipeline_with_includes.yml @@ -0,0 +1,8 @@ +# Complete pipeline configuration composed from reusable YAML files +# +# Usage: +# rompy pipeline examples/configs/basic_pipeline_with_includes.yml + +config: !include basic_modelrun.yml +backend: !include local_backend.yml +postprocessor: !include ../backends/postprocessor_configs/noop_basic.yml From 8cf16991f1cc4c3cfe3d21693285f2777e92fd98 Mon Sep 17 00:00:00 2001 From: Tom Durrant Date: Thu, 16 Apr 2026 10:35:34 +1000 Subject: [PATCH 2/2] Expand backend pipeline integration docs Add practical pipeline integration guidance to the backends page, including the nested config structure, CLI usage patterns, include-based composition, and a corrected programmatic example using the current pipeline API. --- docs/backends.md | 80 ++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 68 insertions(+), 12 deletions(-) diff --git a/docs/backends.md b/docs/backends.md index beaa8332..c4220489 100644 --- a/docs/backends.md +++ b/docs/backends.md @@ -832,42 +832,98 @@ else: ### Pipeline Integration +The pipeline workflow combines three pieces: + +* A `ModelRun` configuration under `config:` +* A typed run backend configuration under `backend:` +* A postprocessor configuration under `postprocessor:` + +At the CLI level, `rompy pipeline` loads those three sections, builds the matching configuration objects, and executes the built-in local pipeline backend. That pipeline backend then runs the stages in order: generate, run, and postprocess. + +For most users, the CLI is the simplest way to use pipeline integration: + +```bash +# All sections defined inline +rompy pipeline examples/configs/basic_pipeline.yml + +# Reuse the same pipeline but override the backend from a separate file +rompy pipeline examples/configs/basic_pipeline.yml \ + --backend-config examples/configs/local_backend.yml + +# Reuse all sections from separate files via !include +rompy pipeline examples/configs/basic_pipeline_with_includes.yml +``` + +The pipeline config structure is: + +```yaml +config: + run_id: my_pipeline_run + output_dir: ./outputs + period: + start: "2023-01-01T00:00:00" + end: "2023-01-02T00:00:00" + interval: "1H" + config: + model_type: base + +backend: + type: local + timeout: 3600 + +postprocessor: + type: noop + validate_outputs: true +``` + +Use inline sections when a workflow is small and self-contained. Use `!include` when you want to reuse backend or postprocessor settings across multiple pipeline files. + +Programmatic usage follows the same structure, but you pass typed configuration objects directly: + ```python from rompy.pipeline import LocalPipelineBackend -from rompy.backends import LocalConfig, DockerConfig +from rompy.backends import DockerConfig from rompy.postprocess.config import NoopPostprocessorConfig -# Create pipeline with different backends for different stages -backend = LocalPipelineBackend() +# Create the pipeline backend +pipeline = LocalPipelineBackend() -# Configure run backend +# Configure the run stage run_config = DockerConfig( image="swan:latest", cpu=16, memory="32g", - timeout=14400 + timeout=14400, ) -# Configure postprocessor +# Configure the postprocess stage processor_config = NoopPostprocessorConfig( validate_outputs=True, - timeout=3600 + timeout=3600, ) # Execute pipeline -results = backend.execute( +results = pipeline.execute( model_run=model_run, - run_backend=run_config, - processor_config=processor_config, - cleanup_on_failure=False + backend_config=run_config, + processor=processor_config, + cleanup_on_failure=False, + validate_stages=True, ) if results["success"]: print(f"Pipeline completed. Stages: {results['stages_completed']}") else: - print(f"Pipeline failed at stage: {results['stages_completed'][-1]}") + print(f"Pipeline failed during: {results['stage']}") ``` +Notes: + +* `backend_config` must be a backend config object such as `LocalConfig`, `DockerConfig`, or `SlurmConfig` +* `processor` must be a postprocessor config object such as `NoopPostprocessorConfig` +* CLI `--backend-config` and `--processor-config` override any inline `backend:` and `postprocessor:` sections +* `!include` paths are resolved relative to the file that contains the include + ## Notebook Examples For a practical, hands-on example of how to use the different backends, please see the following notebook: