Feature/etl milestone 2

cda-etl/Dockerfile

@@ -7,4 +7,12 @@ RUN pip install --no-cache-dir -r requirements.txt
 
 COPY src/ /app/src/
 
+# cda_etl uses flat imports (its own directory on the path); cda_expander is a
+# proper package imported as cda_expander.*, so /app/src is on the path too.
+# The two never import each other - see src/cda_expander/__init__.py.
+ENV PYTHONPATH=/app/src/cda_etl:/app/src
+
+# Default entry point is the ETL itself. The expander is a second entry point
+# in the same image, run before the ETL to regenerate the literal-id config:
+#     python -m cda_expander --base <config> --templates <templates> --out <generated>
 CMD ["python", "src/cda_etl/main.py"]

cda-etl/README.md

@@ -23,9 +23,11 @@ The data is organized by office, project, and resource type, then written to a f
 
 ## Configuration Overview
 
-The main runtime configuration is stored in a YAML file, defaulting to `regi.yml` in the working directory.
+The main runtime configuration is stored in a YAML file, defaulting to `regi.generated.yml` in the working directory.
 
-The application reads the YAML path from the `REGI_CONFIG_PATH` environment variable. If the variable is not set, it looks for `regi.yml` next to where the process starts.
+The application reads the YAML path from the `REGI_CONFIG_PATH` environment variable. If the variable is not set, it looks for `regi.generated.yml` next to where the process starts.
+
+Every timeseries, rating, and location level in that file carries a **literal id**. cda-etl has no concept of deriving an id at run time. Applications that store their ids indirectly - REGI keeps them in CWMS association properties, and CWMS PublishedTimeSeries/A2W is expected later - resolve them ahead of time with `cda-expander` (see below), which writes the literal-id file that cda-etl reads.
 
 ### Example Structure
 
@@ -51,6 +53,101 @@ offices:
             enabled: true
 ```
 
+## cda-expander
+
+`cda-expander` is a separate preprocessor that lives in the same image (`src/cda_expander/`). It takes a base config plus a file of association templates, resolves the templates against CWMS, and appends the resulting literal ids to the base.
+
+It exists so that cda-etl stays application-agnostic: REGI's `?GLOBAL?` property conventions, and any future scheme, live entirely in this tool. cda-etl never imports it, and it never imports cda-etl - the two meet only through the generated file, and a test enforces that.
+
+### Files
+
+| File | Role |
+| --- | --- |
+| `data/regi/regi.yml` | **Base config.** Hand-edited. An ordinary cda-etl config in the schema above - literal ids only, valid and runnable on its own. |
+| `data/regi/regi.templates.yml` | **Templates.** Hand-edited. Association patterns and nothing else: no offices, no projects, no settings. |
+| `data/regi/regi.generated.yml` | **Output**, and what cda-etl reads. The base with resolved ids appended. Generated, committed, never hand-edited. |
+
+Keeping the base separate means the template file stays small and is only ever about associations, and the base remains a config you can read, validate, and run without the expander.
+
+### Usage
+
+```bash
+python -m cda_expander --base      /data/regi/regi.yml \
+                       --templates /data/regi/regi.templates.yml \
+                       --out       /data/regi/regi.generated.yml
+```
+
+Add `--check` to verify without writing: it exits `1` if the file on disk differs from freshly generated output, which catches a hand-edited file, an edited base, and an underlying association property that changed without a regenerate. `docker-compose` runs the expander as a service that must complete before `cda-etl` starts, and `./gradlew :cda-etl:check` runs the `--check` form when `SOURCE_CDA_URL` is set.
+
+The expander resolves against `SOURCE_CDA_URL` / `SOURCE_CDA_API_KEY` - the same source instance cda-etl stages from.
+
+### Templates file
+
+One entry per association property category, and nothing else:
+
+```yaml
+version: 1
+
+# Also emit "properties: all: true" per category, so the property records
+# themselves are staged and published. Defaults to true.
+stageProperties: true
+
+templates:
+  timeseries:
+    categoryId: LOCATION TIME SERIES ASSOCIATION
+    placeholder: "?GLOBAL?"
+    valuePlaceholder: "?GLOBAL?"
+  ratings:
+    categoryId: LOCATION RATING ASSOCIATION
+    placeholder: "?GLOBAL?"
+    valuePlaceholder: "?GLOBAL?"
+    entry:
+      por: true
+```
+
+That is the whole file. **There is deliberately no way to list individual property ids.** A category is read once and everything in it is used, so nothing can be missed and nothing has to be added here when a property appears in the database later - the next run picks it up.
+
+An earlier design did enumerate ids. An audit against SWT found only 2 of 33 configured entries matched a property that actually existed, which is what a hand-maintained list of ~110 strings drifts to. See `docs/template-audit-swt.md`.
+
+Keys:
+
+- `categoryId`: The CWMS property category to read.
+- `placeholder`: The token a property *name* uses in place of a project id, marking it as applying to every project. REGI uses `?GLOBAL?`, as in `Regi_project_INPUT.Hourly_wind_speed.?GLOBAL?`.
+- `valuePlaceholder`: The token inside a global row's *value* that is replaced with the project id. In SWT's data this is also `?GLOBAL?` — that row's value is `?GLOBAL?.Speed-Wind.Inst.1Hour.0.Ccp-Rev`. It stays a separate setting because nothing guarantees the name-side and value-side tokens agree.
+- `entry`: Extra keys carried onto every appended entry (`por: true`, `download:`).
+
+**Which projects a category applies to: all of them.** The file names no offices and no projects; those come from the *base* config, and each category is resolved once per enabled project of each enabled office found there.
+
+### How one project resolves
+
+Property names follow `{prefix}.{family}.{scope}`, where scope is a project id or the placeholder.
+
+1. **A row named for the project wins outright.** These are not derivable — real SWT rows point a project at a different location's gauge (`KEMP` → `TRUS`, `CHEN` → `MARI`), and two projects can share one (`FOSS` and `FCOB` → `FCSO2`). That is the whole reason the properties exist.
+2. **Globals fill the gaps.** Each `?GLOBAL?` row whose `(prefix, family)` has no project-specific row contributes its value with `valuePlaceholder` replaced by the project id.
+3. Rows with an empty value contribute nothing. 10 of SWT's 84 rows are like that.
+
+Step 1 starting from what exists — rather than iterating globals — matters: SWT has `Regi_project_PRIMARY.Hourly_wind_speed.EUFA` with no matching `PRIMARY` global, and iterating globals would silently drop it.
+
+If a global's value doesn't contain `valuePlaceholder`, that's an error rather than a silent pass. REGI's templates have the same six-part shape as a real `ts_id`, so without the guard a token mismatch would stage a timeseries literally named `?GLOBAL?.Speed-Wind.Inst.1Hour.0.Ccp-Rev`.
+
+### Appending rules
+
+- Resolved ids are appended after whatever the base already declares for that category.
+- **An id already present is not appended again**, so a project's own literal entry always wins, and REGI's heavy aliasing collapses. Seven SWT families all resolve to `Elev.Inst.1Hour.0.Ccp-Rev`; without this, cda-etl would download and publish it seven times per run.
+- Disabled offices and projects are skipped; nothing is ever removed or rewritten.
+
+### Cost of a run
+
+**One request per office per category** — three for SWT, regardless of project count. Everything after the listing is in-memory, and listings are memoized for the run.
+
+For scale: the previous per-id design needed two requests per template per project, which at SWT's ~110 ids and 40 projects was over 1,300 sequential round trips to learn what three requests say.
+
+### Why the generated file is committed
+
+The resolved ids are the mapping from "REGI says this project's storage timeseries" to an actual `ts_id` that gets written to the destination database. Committing the generated file makes that mapping reviewable in a diff before a run happens, rather than something you reconstruct from logs afterwards.
+
+For that to work the output has to be stable, so the header records the expander version and a SHA-256 of each input but deliberately **no timestamp** and **no source URL** - either would make the file change on every regeneration or vary by environment, and the diff would stop meaning anything.
+
 ### YAML Fields
 
 - `version`: Config version. Must be `1`.
@@ -90,31 +187,44 @@ Environment variable values are trimmed. Empty or whitespace-only values are tre
 
 ### Other Runtime Settings
 
-- `REGI_CONFIG_PATH`: Path to the YAML config file. Defaults to `regi.yml`.
+- `REGI_CONFIG_PATH`: Path to the literal-id YAML config cda-etl reads. Defaults to `regi.generated.yml`.
+- `REGI_BASE_CONFIG_PATH`: Path to the hand-edited base config. Only used by the expander.
+- `REGI_TEMPLATES_PATH`: Path to the association templates file. Only used by the expander.
 - `LOG_LEVEL`: Console log level for the application process. Defaults to `INFO`.
 
 ## Docker Usage
 
 ### docker run
 
-Mount the YAML file into the container and point `REGI_CONFIG_PATH` at it.
+Mount the config directory into the container and point `REGI_CONFIG_PATH` at the generated file.
 
 ```powershell
 docker run --rm `
-  -v ${PWD}\data\regi\regi.yml:/app/regi.yml `
-  -e REGI_CONFIG_PATH=/app/regi.yml `
+  -v ${PWD}\data\regi:/data/regi `
+  -e REGI_CONFIG_PATH=/data/regi/regi.generated.yml `
   -e SOURCE_CDA_URL=https://source.example/cwms-data `
   -e SOURCE_CDA_API_KEY=your-source-key `
   -e DEST_CDA_URL=https://dest.example/cwms-data `
   -e DEST_CDA_API_KEY=your-dest-key `
   cwms-data-api/etl
 ```
 
-If you do not want to download from the source API, omit `SOURCE_CDA_URL` and the pipeline will publish only staged files.
+To regenerate the config first, run the same image with the expander entry point:
+
+```powershell
+docker run --rm `
+  -v ${PWD}\data\regi:/data/regi `
+  -e SOURCE_CDA_URL=https://source.example/cwms-data `
+  -e SOURCE_CDA_API_KEY=your-source-key `
+  cwms-data-api/etl `
+  python -m cda_expander --base /data/regi/regi.yml --templates /data/regi/regi.templates.yml --out /data/regi/regi.generated.yml
+```
+
+If you do not want to download from the source API, omit `SOURCE_CDA_URL` and the pipeline will publish only staged files. Note the expander does require it, since that is where association properties are read from.
 
 ### docker-compose
 
-The included `docker-compose.yml` mounts `regi.yml` into the container and sets `REGI_CONFIG_PATH=/app/regi.yml`.
+The included `docker-compose.yml` defines two services: `cda-expander` regenerates `regi.generated.yml` and exits, and `cda-etl` waits for it to complete successfully before starting. Both mount `./cda-etl/data/regi` at `/data/regi`.
 
 You still need to supply the API endpoint environment variables when running Compose.
 

cda-etl/build.gradle

@@ -101,7 +101,7 @@ tasks.register('runEtlUnitTests', Exec) {
     doFirst {
         def args = [
             'docker', 'run', '--rm',
-            '-e', 'PYTHONPATH=/app/src/cda_etl',
+            '-e', 'PYTHONPATH=/app/src/cda_etl:/app/src',
             '-v', "${projectDir}/src:/app/src"
         ]
 
@@ -116,6 +116,46 @@ tasks.register('runEtlUnitTests', Exec) {
     inputs.dir('tests').optional()
 }
 
+tasks.register('checkGeneratedConfig', Exec) {
+    dependsOn 'dockerBuild'
+
+    group = 'verification'
+    description = 'Fails if data/regi/regi.generated.yml is stale or hand-edited.'
+
+    onlyIf {
+        if (!isDockerAvailable(name)) {
+            return false
+        }
+
+        if (!System.getenv('SOURCE_CDA_URL')) {
+            logger.lifecycle("Skipping ${name} because SOURCE_CDA_URL is not set; " +
+                    "the generated config drift check needs a source CDA to resolve against.")
+            return false
+        }
+
+        return true
+    }
+
+    doFirst {
+        commandLine([
+            'docker', 'run', '--rm',
+            '-e', "SOURCE_CDA_URL=${System.getenv('SOURCE_CDA_URL')}",
+            '-e', "SOURCE_CDA_API_KEY=${System.getenv('SOURCE_CDA_API_KEY') ?: ''}",
+            '-v', "${projectDir}/data/regi:/data/regi",
+            imageName.get(),
+            'python', '-m', 'cda_expander',
+            '--base', '/data/regi/regi.yml',
+            '--templates', '/data/regi/regi.templates.yml',
+            '--out', '/data/regi/regi.generated.yml',
+            '--check'
+        ])
+    }
+
+    inputs.dir('src')
+    inputs.dir('data/regi')
+}
+
 tasks.named('check') {
     dependsOn 'runEtlUnitTests'
+    dependsOn 'checkGeneratedConfig'
 }

cda-etl/data/regi/SWT/Clobs/FLOW.EUFA.PROJECT_TOTAL.json

@@ -0,0 +1,4 @@
+{
+  "office-id": "SWT",
+  "id": "FLOW.EUFA.PROJECT_TOTAL"
+}

cda-etl/data/regi/SWT/LocationLevels/EUFA-Dam.Elev.Inst.0.Top of Flood.por.json

@@ -0,0 +1,6 @@
+{
+  "levels": [],
+  "page": "MHx8MHx8MTAw",
+  "page-size": 100,
+  "total": 0
+}

cda-etl/data/regi/SWT/Properties/LOCATION LEVEL ASSOCIATION/Regi_project_INPUT.Evap_pan_to_lake_multiplier.%3FGLOBAL%3F.json

@@ -0,0 +1,5 @@
+{
+  "office-id": "SWT",
+  "category": "LOCATION LEVEL ASSOCIATION",
+  "name": "Regi_project_INPUT.Evap_pan_to_lake_multiplier.?GLOBAL?"
+}

cda-etl/data/regi/SWT/Properties/LOCATION LEVEL ASSOCIATION/Regi_project_INPUT.Evap_pan_to_lake_multiplier.COPA.json

@@ -0,0 +1,6 @@
+{
+  "office-id": "SWT",
+  "category": "LOCATION LEVEL ASSOCIATION",
+  "name": "Regi_project_INPUT.Evap_pan_to_lake_multiplier.COPA",
+  "value": "COPA.Coeff.Inst.0.Pan Evaporation Multiplier"
+}

cda-etl/data/regi/SWT/Properties/LOCATION LEVEL ASSOCIATION/Regi_project_INPUT.Evap_pan_to_lake_multiplier.KEYS.json

@@ -0,0 +1,6 @@
+{
+  "office-id": "SWT",
+  "category": "LOCATION LEVEL ASSOCIATION",
+  "name": "Regi_project_INPUT.Evap_pan_to_lake_multiplier.KEYS",
+  "value": "KEYS.Coeff.Inst.0.Pan Evaporation Multiplier"
+}

cda-etl/data/regi/SWT/Properties/LOCATION LEVEL ASSOCIATION/Regi_project_INPUT.Evap_pan_to_lake_multiplier.GLOBAL.json

@@ -0,0 +1,5 @@
+{
+  "office-id": "SWT",
+  "category": "LOCATION LEVEL ASSOCIATION",
+  "name": "Regi_project_INPUT.Evap_pan_to_lake_multiplier.?GLOBAL?"
+}

cda-etl/data/regi/SWT/Properties/LOCATION LEVEL ASSOCIATION/Regi_project_INPUT.Evap_seasonal_pan_value.COPA.json

@@ -0,0 +1,6 @@
+{
+  "office-id": "SWT",
+  "category": "LOCATION LEVEL ASSOCIATION",
+  "name": "Regi_project_INPUT.Evap_seasonal_pan_value.COPA",
+  "value": "COPA.Coeff.Inst.0.Pan Evaporation Multiplier"
+}

cda-etl/data/regi/SWT/Properties/LOCATION RATING ASSOCIATION/Regi_project_INPUT.Elev_Area.%3FGLOBAL%3F.json

@@ -0,0 +1,7 @@
+{
+  "office-id": "SWT",
+  "category": "LOCATION RATING ASSOCIATION",
+  "name": "Regi_project_INPUT.Elev_Area.?GLOBAL?",
+  "value": "?GLOBAL?.Elev;Area.Linear.Production",
+  "comment": "Description for: Elev Area Rating"
+}

cda-etl/data/regi/SWT/Properties/LOCATION RATING ASSOCIATION/Regi_project_INPUT.Elev_Area.GLOBAL.json

@@ -0,0 +1,7 @@
+{
+  "office-id": "SWT",
+  "category": "LOCATION RATING ASSOCIATION",
+  "name": "Regi_project_INPUT.Elev_Area.?GLOBAL?",
+  "value": "?GLOBAL?.Elev;Area.Linear.Production",
+  "comment": "Description for: Elev Area Rating"
+}

cda-etl/data/regi/SWT/Properties/LOCATION RATING ASSOCIATION/Regi_project_INPUT.Elev_Stor.%3FGLOBAL%3F.json

@@ -0,0 +1,7 @@
+{
+  "office-id": "SWT",
+  "category": "LOCATION RATING ASSOCIATION",
+  "name": "Regi_project_INPUT.Elev_Stor.?GLOBAL?",
+  "value": "?GLOBAL?.Elev;Stor.Linear.Production",
+  "comment": "Description for: Elev Stor Rating"
+}

cda-etl/data/regi/SWT/Properties/LOCATION RATING ASSOCIATION/Regi_project_INPUT.Elev_Stor.GLOBAL.json

@@ -0,0 +1,7 @@
+{
+  "office-id": "SWT",
+  "category": "LOCATION RATING ASSOCIATION",
+  "name": "Regi_project_INPUT.Elev_Stor.?GLOBAL?",
+  "value": "?GLOBAL?.Elev;Stor.Linear.Production",
+  "comment": "Description for: Elev Stor Rating"
+}

cda-etl/data/regi/SWT/Properties/LOCATION TIME SERIES ASSOCIATION/Regi_gate_INPUT.GateSettings_gate_opening.ALTU.json

@@ -0,0 +1,6 @@
+{
+  "office-id": "SWT",
+  "category": "LOCATION TIME SERIES ASSOCIATION",
+  "name": "Regi_gate_INPUT.GateSettings_gate_opening.ALTU",
+  "value": "ALTU.Opening.Inst.0.0.MANUAL"
+}

cda-etl/data/regi/SWT/Properties/LOCATION TIME SERIES ASSOCIATION/Regi_gate_INPUT.GateSettings_gate_opening.ARCA.json

@@ -0,0 +1,6 @@
+{
+  "office-id": "SWT",
+  "category": "LOCATION TIME SERIES ASSOCIATION",
+  "name": "Regi_gate_INPUT.GateSettings_gate_opening.ARCA",
+  "value": "ARCA.Opening-Low_Flow_Gates.Inst.0.0.MANUAL"
+}

cda-etl/data/regi/SWT/Properties/LOCATION TIME SERIES ASSOCIATION/Regi_gate_INPUT.GateSettings_gate_opening.CANT.json

@@ -0,0 +1,6 @@
+{
+  "office-id": "SWT",
+  "category": "LOCATION TIME SERIES ASSOCIATION",
+  "name": "Regi_gate_INPUT.GateSettings_gate_opening.CANT",
+  "value": "CANT.Opening.Inst.0.0.MANUAL"
+}

cda-etl/data/regi/SWT/Properties/LOCATION TIME SERIES ASSOCIATION/Regi_gate_INPUT.GateSettings_gate_opening.CHOU.json

@@ -0,0 +1,6 @@
+{
+  "office-id": "SWT",
+  "category": "LOCATION TIME SERIES ASSOCIATION",
+  "name": "Regi_gate_INPUT.GateSettings_gate_opening.CHOU",
+  "value": "CHOU.Opening.Inst.0.0.MANUAL"
+}

cda-etl/data/regi/SWT/Properties/LOCATION TIME SERIES ASSOCIATION/Regi_gate_INPUT.GateSettings_gate_opening.DENI.json

@@ -0,0 +1,6 @@
+{
+  "office-id": "SWT",
+  "category": "LOCATION TIME SERIES ASSOCIATION",
+  "name": "Regi_gate_INPUT.GateSettings_gate_opening.DENI",
+  "value": "DENI.Opening.Inst.0.0.MANUAL"
+}

cda-etl/data/regi/SWT/Properties/LOCATION TIME SERIES ASSOCIATION/Regi_gate_INPUT.GateSettings_gate_opening.EUFA.json

@@ -0,0 +1,6 @@
+{
+  "office-id": "SWT",
+  "category": "LOCATION TIME SERIES ASSOCIATION",
+  "name": "Regi_gate_INPUT.GateSettings_gate_opening.EUFA",
+  "value": "EUFA.Opening.Inst.0.0.MANUAL"
+}

cda-etl/data/regi/SWT/Properties/LOCATION TIME SERIES ASSOCIATION/Regi_gate_INPUT.GateSettings_gate_opening.HEYB.json

@@ -0,0 +1,6 @@
+{
+  "office-id": "SWT",
+  "category": "LOCATION TIME SERIES ASSOCIATION",
+  "name": "Regi_gate_INPUT.GateSettings_gate_opening.HEYB",
+  "value": "HEYB.Opening.Inst.0.0.MANUAL"
+}

cda-etl/data/regi/SWT/Properties/LOCATION TIME SERIES ASSOCIATION/Regi_gate_INPUT.GateSettings_gate_opening.HUDS.json

@@ -0,0 +1,6 @@
+{
+  "office-id": "SWT",
+  "category": "LOCATION TIME SERIES ASSOCIATION",
+  "name": "Regi_gate_INPUT.GateSettings_gate_opening.HUDS",
+  "value": "HUDS.Opening.Inst.0.0.MANUAL"
+}

cda-etl/data/regi/SWT/Properties/LOCATION TIME SERIES ASSOCIATION/Regi_gate_INPUT.GateSettings_gate_opening.KAWL.json

@@ -0,0 +1,6 @@
+{
+  "office-id": "SWT",
+  "category": "LOCATION TIME SERIES ASSOCIATION",
+  "name": "Regi_gate_INPUT.GateSettings_gate_opening.KAWL",
+  "value": "KAWL.Opening.Inst.0.0.MANUAL"
+}