[codex] add handover readiness and water stress intelligence#1
Draft
mhunegnaw wants to merge 63 commits into
Draft
[codex] add handover readiness and water stress intelligence#1mhunegnaw wants to merge 63 commits into
mhunegnaw wants to merge 63 commits into
Conversation
added 30 commits
April 24, 2026 10:18
…plit Adds model defensibility and splits the 1541-line MLAnalytics.jsx into a Context + six lazy-loaded tab files (242-line shell). - ModelMetricsPanel: R²/RMSE/MAE/CV mean + 5-fold CV bar chart from /api/ml/models/<id>/metrics (data already existed, not surfaced). - FeatureImportanceChart: horizontal bar chart of each trained model's feature_importance map (top-N, human-readable band names). - ForecastBacktestCard: client-side holdout (14/30/60/90 day) using the existing regression.js seasonal fit; reports RMSE, MAE, sMAPE, 95% PI coverage, and skill over persistence + seasonal-naive baselines. - AnomalyTimeSeriesChart: line chart with ±σ envelope and severity-coloured dots over the flagged points; backend AnomalyDetectionResponse gains an optional time_series field so the UI can plot in context. - SegmentationPreviewMap: Leaflet overlay of boundary_geojson vs the analysis region, with a working Download GeoJSON button. - AssimilationGridMap: Leaflet choropleth of grid_data using the backend legend colours (canvas renderer for perf at ~25k cells). Also fixes the shared forecastVariable bug between Forecast and Anomaly tabs, adds a region-fallback warning when the selected water body has no geometry, wires CSV exports on Prediction/Forecast/Anomaly, converts the anomaly sensitivity to a labelled slider with live false-alarm rate, adds fieldset/legend + role=tablist for a11y, and shows a training-status line under the page title.
…copy
Tier-1 quick wins from docs/un-handover/ plus the prep docs themselves.
- alerts.py: reorder routes so /scan, /active, /statistics, /thresholds,
/types/list and /water-body/{id} are declared before /{alert_id:int};
add :int path converter so non-integer paths 404 cleanly. Closes the
bug where the engine endpoints returned 401 instead of engine data.
- middleware/health.py: drop the raw "SELECT 1" that runs before the
text() version — SQLAlchemy 2.0 rejects raw strings and the probe was
false-failing against healthy databases. Single text("SELECT 1") now.
- .env.example + config.py: add five variables that existed in Settings
but were missing from the operator template (GEMINI_API_KEY,
EARTHDATA_USERNAME/PASSWORD, ICESAT2_CACHE_DIR, ALTIMETRY_CACHE_DIR),
plus promote DAHITI_API_KEY from an os.environ.get bypass into Settings.
- test_demo_routes.py: regenerate the EXPECTED_ROUTES snapshot (186
routes, was 142 and stale); relax the water-body count assertion from
== 53 to >= 50 so pan-African additions don't keep breaking it. All 13
backend tests now green.
- Pan-African copy sweep: Reports subtitle, SatelliteData click-gate
bbox + comment, Query example prompts, waterService transboundary +
NISAR docstrings, AssimilationTab header. No longer says "Ethiopian
water resources" on a pan-African product.
docs/un-handover/ seeds the handover package:
01_reports_random_audit.md — every random.* call mapped to a real
source or marked for cut (Day 2 plan)
02_alerts_route_fix.md — the fix committed here, explained
03_pan_african_strings.md — FIX/KEEP/REVIEW list (Batch 1 done)
04_env_vars_audit.md — env var gap analysis (closed here)
05_backend_tests_failure_analysis.md — test triage (closed here)
06_operator_handover_pack.md — 10-section skeleton for Week 3
README.md — index + day-by-day tracker
Frontend 124/124 + backend 13/13 green on handover/un-v1.
…port data
A customer-facing report whose numbers shift between reloads is not an
institutional output. This commit removes every random.* call from the
/api/demo/reports/generate response and replaces each with either a real
satellite observation, a deterministic derivation of one, or a documented
climatology estimate. Same request → byte-identical response.
Data flow per section:
- water_levels: per-body JRC Global Surface Water monthly cache drives
level_percent as area/observed_max*100. Missing months reported as
null rather than filled. No JRC data (window past 2021) → fall back to
a deterministic per-body seasonal shape, marked data_status=estimated.
- water_quality: read wq_cache/wq_{id}.json directly; turbidity, TSS,
chlorophyll, DO, pH, temperature sourced from the cached Sentinel-2/3
indicators. Trophic state derived from observed chlorophyll (OECD
Vollenweider 1982 cut-offs). Composite quality_index is the mean of
per-indicator normalised sub-scores (missing indicators dropped, not
imputed). Quality status bucketed from the index. Bodies without cache
reported as data_status=unavailable, not randomised.
- hydrology: per-body precipitation / reference ET / water balance /
SPI proxy derived from each body's own latitude via fixed climatology
bands (equatorial / tropical savanna / Sahelian / arid / Mediterranean).
Area-weighted aggregate across selected bodies. Marked estimated
throughout; scheduled replacement is CHIRPS + MODIS ET caches.
- environmental: NDVI + land-cover mix from biome defaults keyed by
latitude. Lake-surface temperature uses wq_cache where available,
otherwise latitude climatology.
- satellite_data: scene counts = published Sentinel-2 5-day and
Sentinel-1 6-day revisit × window days × bodies. Cloud-free percent
from mean wq_cache cloud_cover. water_extent_change_km2 area-weighted
from observed water_levels first-to-last deltas. Algal bloom count =
bodies with eutrophication_risk >= high in wq_cache. Flood events
reported as null until the Sentinel-1 flood cache is wired.
- ml_predictions: water-level forecast is an OLS slope extrapolation on
the last 6 observed months of each body's level_percent (r^2 drives
confidence). Drought risk mapped from the SPI proxy. Quality trend
derived from wq_cache risk flags.
Every section emits a provenance entry (source, method, acquisition
window, observed/derived/estimated) aggregated into a document-level
provenance block with a determinism flag and licensing summary.
document_id is now a SHA-256 of (title, sorted body_ids, period) truncated
to 12 chars. Policy brief document_id fixed the same way.
Verified: two successive calls on identical input produce byte-identical
JSON after stripping generated_at. Backend 13/13, frontend 124/124 green.
…T /api/alerts
Closes the three remaining Tier-1 items from the audit in one pass:
Report API / UI contract alignment
----------------------------------
The frontend expected report.sections.{summary, water_levels, quality,
hydrology, environmental, satellite, predictions} plus flat
metadata.period_start / period_end and per-section aggregates — the Day
2 response used top-level keys and nested period so the preview was
blank despite the call succeeding.
reports.py now emits both shapes:
* top-level keys (rich, per-body, provenance-aware) for raw API users
* sections.* projection (flat aggregates + data[]) that the UI renders
Determinism is preserved — sections.* is a pure re-projection with no
new random calls, verified byte-identical across two calls.
Reports.jsx shows a new provenance panel (document_id, observed-vs-
estimated counts, per-section source + method, licensing summary) so the
UN reviewer can audit every number. The water-levels table gains a Data
Status column ("observed" / "estimated") badge and the legacy
current_level_m field is replaced by current_level_percent + "—" when
absent, removing the "null m" render bug.
Alerts consolidation (engine ↔ personal)
----------------------------------------
Previously the Alerts page showed DB-backed per-user alerts only, while
the AlertEngine held thousands in a JSON cache that no user ever saw.
Day 1 already unshadowed the engine HTTP endpoints (/active, /scan,
/statistics, /thresholds); this commit surfaces them in the UI.
New data-source toggle at the top of the Alerts page:
* Personal (DB) — existing per-user CRUD behaviour, unchanged.
* System (engine) — reads /api/alerts/active, shows system-wide
satellite-derived alerts. Mark-read / delete / create are hidden in
this mode (engine alerts have string ids and no read-state, so
those actions would 404).
alertsApi in waterService.js now exposes both sides cleanly, with
comments explaining which is auth-gated and which isn't. A post-handover
DB migration is the right long-term fix (noted in
docs/un-handover/06_operator_handover_pack.md §10) but for v1 the dual
view is the minimum-risk "single view" the review asked for.
Real POST /api/alerts
---------------------
The Create Alert button previously simulated success in the browser
with no backend call, silently losing user input. Added:
* POST /api/alerts handler that accepts AlertCreate, overrides
user_id with current_user.id (so a signed-in user can't plant
alerts on another account), persists via SQLAlchemy, returns the
created row.
* alertsApi.create(data) wiring.
* CreateAlertModal now calls the real endpoint, surfaces backend
validation errors via alert(), and reloads the list on success.
Backend 13/13, frontend 124/124 green. Vite build clean.
Two operational-safety items that make the multi-worker production
deployment safe to run.
Scheduler leader election
-------------------------
The production docker-compose runs uvicorn with --workers 4. Before this
change every worker's lifespan hook called start_scheduler(), which meant
each ETL job and each alert-engine scan fired N times per interval across
the pool — duplicate CHIRPS pulls, duplicate alert emails, duplicate
GRACE quota burn. Bad on its own, worse when you scale to two
containers.
main.py lifespan now checks a new SCHEDULER_LEADER environment variable
before starting APScheduler. Production must set it to true on exactly
one worker / container; all other workers stay scheduler-less and just
serve HTTP. In development (ENVIRONMENT=development) with the variable
unset we default to leader so local `uvicorn app.main:app --reload`
keeps working unchanged.
Shutdown also gated on the leader flag so logs don't claim to stop a
scheduler that never started.
Gate create_all() on ENV={development,test}
-------------------------------------------
init_db() used to call Base.metadata.create_all() unconditionally. That
is fine for pytest and local SQLite, but silently masks a missing
Alembic migration in production — you can start the API, it swallows
the schema gap, writes start without any visible error, and the next
developer adds a column that then never lands in prod because Alembic
never ran.
init_db() now:
* creates tables implicitly only when environment ∈ {development, test}
* in any other environment, inspects the existing schema, logs ERROR
with the names of missing tables, and returns without mutating —
forcing operators to run `alembic upgrade head`.
Post-handover work (documented in docs/un-handover/06_operator_handover_pack.md):
* Wire the alembic head command into the docker-compose prod entrypoint
so a new deploy runs migrations before the API starts.
Backend 13/13 green (unchanged).
…-body search
Two pieces of product polish the audit asked for: ack workflow on
personal alerts and the Reports water-body selector becoming usable for
the pan-African list.
Alerts
------
- Severity sort (critical → low) as the default order; "newest first"
and "oldest first" also available.
- Severity filter pill ("critical only" … "low only") applied
client-side — no extra backend round-trip since the backend already
returns the unread/type-filtered set.
- Acknowledge button per personal alert: calls PUT /api/alerts/{id}
with {is_acknowledged: true, is_read: true}. Acknowledged rows show
a blue "ack" badge with the timestamp on hover, and the ack button
goes away.
- Water-body deep link: each alert with a water_body_id gets an
ExternalLink icon that jumps to /water-bodies/{id}. Works for both
personal DB alerts and system engine alerts.
Reports
-------
- Water-body selector now has country filter, type filter and a name
search, mirroring the pattern we used on ML Analytics. The plain
~100-row scrollable checkbox list was unusable during operations
once the pan-African list landed.
- "Select All" becomes "Select visible / Deselect visible" so filtered
views don't mass-toggle bodies outside the current filter.
- Header shows "N selected · M of K visible" so the operator always
knows what's in scope.
Backend untouched. Frontend 124/124 green.
Closes the Week-3 items from docs/un-handover/README.md — the migration, the handover pack fill-in, and the env-template polish. Audit log — DB primary + JSON shadow write ------------------------------------------ The JSON file-backed audit_log_service was the review's flagged "month-3 time-bomb" under uvicorn --workers 4 — concurrent appends race and can leave a half-written entry that fails to parse. New shape: * backend/app/models/audit_log.py — AuditLog SQLAlchemy model with indexes on timestamp, user_id, action, resource_type plus (user_id, timestamp) and (action, timestamp) composites. * backend/alembic/versions/003_audit_log.py — creates the table on `alembic upgrade head` in all environments (dev create_all picks it up via the Base.metadata registration). * backend/app/services/audit_log.py rewritten: DB primary with a JSON shadow write for one release so a rollback to the pre-handover image still works. Reads: DB first, JSON fallback only if DB is unreachable. log_action NEVER raises — audit shouldn't break a user-visible action. * backend/app/scripts/migrate_audit_log_to_db.py — idempotent backfill with --dry-run. Dedupes on (timestamp, user_id, action, resource_id). docker-compose.prod.yml ----------------------- Added SCHEDULER_LEADER=true and the five env vars that Day 1 added to Settings (GEMINI_API_KEY, EARTHDATA_USERNAME/PASSWORD, DAHITI_API_KEY — cache-dir overrides intentionally not needed in compose). So `docker compose up -d` now starts the single scheduler leader and passes through every optional credential the Settings class understands. .env.prod.example ----------------- Synced with the new Settings fields so the UN ops team gets a complete template instead of silently-missing variables. Section headers reorganised to separate LLM / Earthdata / DAHITI groups. Operator handover pack (docs/un-handover/06) -------------------------------------------- Every [INSERT] marker is now real content. Filled: * ASCII architecture diagram showing browser → nginx → backend → (Postgres, Redis, GEE, Anthropic/Gemini), with notes on scheduler leader election and disk-backed caches. * Alembic migration command + audit backfill invocation, with an explicit warning that production refuses create_all (post-Day-4). * First-admin creation walkthrough (register + psql role update). * Complete env-var reference table split into backend-application vs container-level variables. * Cache refresh per product (wq, surface_water, gee_query_cache). * `run_etl_now` + engine-scan cURL for manual runs. * Restore procedure with stop / drop / pg_restore / tar extract / bring-up / verify + post-restore backfill note. * Migration caveats for rolling past 003 (data loss on audit table). * Tier 1/2/3 escalation ladder with "what to include in a Tier-2 ticket" checklist (document_id, health probes, logs, timestamp). * Appendix A pointer back to §3. Appendix B condensed endpoint table (ops-relevant only). Appendix C table-by-table size expectations. * Soak-week additional failure modes (auth on POST /api/alerts, backfill not run, contract drift, regenerated document_id). Backend 13/13, frontend 124/124 green.
Continuation pass after the initial handover/un-v1 tag. Closes the last
few items on the review list and raises the confidence floor for the
soak week.
Alerts — country filter
-----------------------
Alerts page now has a Country dropdown alongside Severity and Sort. The
options are derived from the current result set (no extra round-trip),
and personal alerts without a country field fall back to the water-body
lookup so admins can slice their own notifications the same way the
system-alert view already supports.
Policy brief — fully deterministic history
------------------------------------------
_generate_level_time_series used to fill its 12-month demo chart with
seeded-RNG jitter around the body's current level. The seed made it
deterministic, but the jitter itself was still synthetic. It now reads
the JRC pan-history monthly cache (wb_{id}_monthly_all.json) when an id
is available and computes level_percent = area / observed_max * 100 —
exactly the same shape the Reports API already produces. No RNG used on
either path. With that change there is NO random.* call anywhere in
reports.py (enforced by a new test; see below).
Policy brief exports — standardised filenames + disclaimers
-----------------------------------------------------------
All four export formats (PDF / Markdown / HTML / JSON) now:
* Use a stable filename pattern: WRPB-{docid}-{country}-{YYYYMMDD}.{ext}.
Same request → same filename stub, so re-exports don't pile up as
ambiguously-named files in the operator's Downloads folder.
* Carry an identical disclaimer + licensing block. Consolidated into
two module-level constants (_UN_ECA_DISCLAIMER_MD / _HTML) and appended
to each export right before the return. The PDF builder also gets an
equivalent "Disclaimer & licensing" section with Pekel / Copernicus /
MODIS / CHIRPS / GRACE attributions.
* JSON export carries the disclaimer + license map as structured fields
(not raw text) so downstream tooling can read and display it.
Backend tests — handover acceptance suite
-----------------------------------------
New tests/test_handover_paths.py guards the exact behaviours the review
relies on:
* reports are deterministic across two identical calls modulo generated_at
* document_id is a pure function of (title, body_ids, period)
* report.sections.* has every key the UI preview expects
* policy-brief export filename uses the deterministic document_id
* audit_log writes to DB and reads back via AuditLogService.get_logs
* the migrate_audit_log_to_db script is idempotent (second run inserts 0)
* static guard: no random.* call may exist in reports.py
7 new tests, total backend suite now 20/20 green (was 13/13).
Bugfix — audit backfill idempotency
-----------------------------------
migrate_audit_log_to_db.py dedup key was comparing ISO strings of
timestamps between the JSON source and the DB rows. SQLite strips tzinfo
on round-trip so the keys drifted and the second run inserted dupes. Now
normalises both sides to UTC-naive second precision before hashing.
Production-readiness CLI
------------------------
New `python -m app.scripts.readiness_check` mechanises every check from
§9 of the handover pack that can be verified programmatically: Settings
class sanity, SCHEDULER_LEADER gate, DB reachable, schema at head,
audit_log round-trip, no real-looking secrets in tracked files, HTTP
health probe, reports-deterministic probe, alerts-engine-reachable
probe. Returns exit 0 = ready / 1 = soft-fail / 2 = blocked.
Secret scanner uses tightened regexes so docs that reference a key
prefix (e.g. sk-ant-) aren't false-flagged; only realistic key bodies
(40+ chars for Anthropic, 35+ for Gemini, full PEM envelopes) trigger.
Dev run: 7/8 checks OK, 1 intentional warning (environment=development),
0 failures, 0 false positives on 6705 scanned files.
Closes every remaining item from the post-handover roadmap listed in
the handover/un-v1 tag. The branch is still handover/un-v1 ready;
these changes make v1 strictly stronger without altering the contract
the UN team receives.
Alerts JSON cache → DB (the "month-3 time-bomb" closed early)
-------------------------------------------------------------
Alembic migration 004 seeds a dedicated ``system@water-hub.local``
user with an unusable password hash (``!`` — no bcrypt output can match
so the account is login-locked by design). Downgrade is safe:
migration 004 refuses to delete the user while any alerts still point
at them.
AlertEngine now dual-writes every scan to the DB ``alerts`` table
under the system user's id, in addition to the JSON shadow:
* Each engine alert carries an ``[engine:{id}] …`` prefix in message
so the application layer can dedupe without a schema change.
* Existing engine rows are UPDATEd in place on subsequent scans —
title / severity / thresholds refresh, acknowledge state preserved.
* DB failures degrade to JSON-only with a WARNING — scan never
breaks because the DB has a hiccup.
* One-shot backfill: ``python -m app.scripts.migrate_alerts_to_db``.
Dev run migrated 2,993 existing cache entries in ~1s.
Side-by-side ML model comparison
--------------------------------
Prediction tab on ML Analytics grows a "Compare with a second model"
checkbox. When on, two prediction requests fire in parallel (primary
+ secondary model types) against the same region, and the results
render in a side-by-side comparison table:
* Δ value and Δ confidence per parameter
* Green rows when models agree (difference < 5% of primary's value)
* Amber rows when they diverge — flag for review
* Explanatory footer: "Both models ran against the same satellite
observation; different results mean they weigh features differently."
Fairness-by-biome infrastructure
--------------------------------
Training pipeline now tags every WQ sample with its water-body
latitude (from ALL_WATER_BODIES.coordinates[1]) and computes
``per_biome_metrics`` at training time: R² / RMSE / MAE per
latitude band, matching the bands used in reports.py climatology
so narrative alignment is automatic.
Bands (|lat| in °):
equatorial < 8
tropical_savanna 8 – 16
sahelian 16 – 24
arid_subtropical 24 – 30
temperate ≥ 30
ModelMetricsPanel.jsx renders the per-biome R² as a horizontal bar
chart under the existing CV fold chart, with the explainer "a large
gap means the model works better in some regions than others".
All 8 WQ models (turbidity / TSS / chlorophyll / DO × RF / XGBoost)
retrained locally to populate the new metadata. Example: the
chlorophyll RF model's RMSE is 0.45 µg/L in arid-subtropical but
6.15 µg/L in the temperate band — visible in the new chart, flagged
for the operator to weight for retraining priorities.
Test posture unchanged — backend 20/20, frontend 124/124.
The note claimed the document was still a skeleton and would be filled in during Week 3 (May 7-13). That was accurate on 2026-04-23 when I first drafted the file but every [INSERT] marker was replaced with real content in commit afabed12 on 2026-04-24 — operator reading the file today would see the note and assume the pack is incomplete. Replace with a 'Status: Complete' line pointing at the handover/un-v1 tag so readers know what they're looking at.
Closes the audit's CRITICAL + HIGH + MEDIUM items in one pass; the
remaining architecture recommendations are documented in the new
docs/un-handover/07_ml_architecture_roadmap.md as concrete v1.1
tickets with file pointers.
CRITICAL — auth-gate state-changing / cost-burning endpoints
------------------------------------------------------------
Added RBAC dependencies on every route the audit flagged:
POST /api/ml/training/{train-all,train-model,upload-data}
POST /api/etl/{run/{ds},run-all}
GET /api/etl/status (system:audit)
POST /api/query (data:read)
POST /api/policy-brief/{generate,export}
GET /api/alerts/scan (alerts:manage)
Verified: all return 401 to anonymous callers.
CRITICAL — RBAC enforced on water-body + measurement CRUD
---------------------------------------------------------
POST/PUT water-bodies require water_bodies:edit, DELETE requires admin.
POST and POST/bulk on measurements require data:write.
User model gains a `role` VARCHAR(32) column (Alembic 005) seeded from
is_admin so existing admins keep working. The role-update endpoint
already wrote to the new column conditionally — now the column always
exists. is_admin stays as derived back-compat.
HIGH — ML artifact loading hardened
-----------------------------------
* ModelManager refuses absolute paths and registry entries that
resolve outside model_dir (path traversal protection).
* Every save records a SHA-256 of the artifact bytes; load verifies
before unpickling. Mismatch refuses to load. Legacy artifacts get
their digest captured on first load.
* torch.load(..., weights_only=True) — only state-dicts; full pickled
models refused at load time. save_pytorch_model now rejects non-dict
inputs at write time so the mistake doesn't reach disk.
HIGH — GeoRegion bounded
------------------------
GeoRegion gets Pydantic validators:
* type ∈ {Polygon, MultiPolygon}
* per-vertex lat/lon range + Africa+5° bbox
* total vertex count ≤ 10 000
* approximate spherical area ≤ 1 000 000 km²
Rejects bad input before it reaches GEE.
HIGH — scheduler leader uses a process-level flock
--------------------------------------------------
Day-4's SCHEDULER_LEADER env-var gate fired in every uvicorn worker of
a container. Now the env var is the opt-in but the actual scheduler
start requires a non-blocking exclusive flock on
/tmp/water_hub_scheduler.lock — only the first worker wins. Released
on shutdown. Path overridable via SCHEDULER_LEADER_LOCK.
MEDIUM — canonical /api prefix
------------------------------
The /api/v1 mount produced double-prefixed paths (/api/v1/api/alerts/…)
that no client used. Documented /api as canonical and added a
deprecation middleware that tags any /api/v1/* response with
Deprecation: true / Sunset: 2026-11-01 + Link: successor-version.
MEDIUM — login/register rate limits
-----------------------------------
backend/app/utils/rate_limit.py — Redis-backed (in-process fallback)
per-IP counter. Login: 5 / 15 min. Register: 3 / 1 hour. Honours
X-Forwarded-For for nginx. Returns 429 + Retry-After.
MEDIUM — observability gating
-----------------------------
* /api/health returns coarse {status} for anonymous callers; full
dependency probe only with X-Internal-Token (constant-time compare).
* /api/metrics requires X-Internal-Token in production (403 otherwise).
In dev, both stay open.
* StructuredLoggingMiddleware redacts query-string values whose key
matches password / token / api_key / secret / etc.
MEDIUM — ML metrics on out-of-fold predictions
----------------------------------------------
training_pipeline now uses sklearn Pipeline(StandardScaler+model) so
the scaler refits inside each CV fold (no train/test leak via mean).
GroupKFold by water_body_id when groups available; falls back to
plain KFold otherwise. Headline R²/RMSE/MAE come from
cross_val_predict, not in-sample. Each model's metadata records
metric_basis ("cross_val_predict" / "in_sample_fallback") and
cv_strategy ("group_kfold_by_water_body (n_groups=N)" / "kfold").
Retrained all 8 WQ models. The new honest CV R² differs sharply from
the old in-sample numbers — visible in the registry:
RF tss 0.925 -> 0.808 (-13 %)
RF turbidity 0.748 -> 0.477 (-36 %)
RF chlorophyll 0.981 -> 0.916 (-7 %)
RF do 0.905 -> 0.768 (-15 %)
XGB tss 1.000 -> 0.383 (-62 %)
XGB turbidity 1.000 -> -0.551 (XGB severely overfit; worse than mean)
XGB chlorophyll 0.999 -> 0.900 (-10 %)
XGB do 0.987 -> 0.768 (-22 %)
This is what the audit warned about. The platform now reports the
honest numbers and the operator can see which models are usable.
MEDIUM — deployment hygiene
---------------------------
* backend/.dockerignore + frontend/.dockerignore (data caches, .git,
__pycache__, .env*, node_modules, source maps, etc.).
* Backend Dockerfile: PYTHONDONTWRITEBYTECODE/PYTHONUNBUFFERED, drops
apt recommends, runs as non-root waterhub user (uid 1000), in-image
HEALTHCHECK against /api/health.
* Frontend vite.config.js: sourcemap=false in production mode (was
always true). Dev/preview/test still produce maps for debugging.
DOCS — v1.1 architecture roadmap
--------------------------------
docs/un-handover/07_ml_architecture_roadmap.md is the audit's
"Modeling Architecture Recommendations" section recast as concrete
v1.1 tickets: feature store, label-kind separation, conformal
intervals, physics-informed water balance, signed artifacts +
promotion + rollback, drift monitor, baseline benchmarks. Each item
has file pointers, acceptance criteria, and a suggested sequencing.
20/20 backend tests still green. Vite production build clean (no source maps).
The Status line referenced commit ffafb949 but the tag has since moved twice (skeleton-note fix + audit hardening pass). The hash bit-rots every time we re-tag. Removed; the tag name is enough to navigate.
User requested a per-country temperature animation on the Environmental page, drawing from satellite-derived temperature with the German Fordatis tool linked as a visual reference. Fordatis 498 turned out to be a Germany-only DWD QGIS tool, not a satellite product, so we use MODIS MOD11A2 (8-day day-time LST composite) here — the same product the rest of the LST stack already runs on. Backend ------- backend/app/routers/demo/_gee_queries.py Adds query_modis_lst_grid(bbox, year, month, grid_size_deg) that returns a per-tile MODIS LST grid for one calendar month. Uses ee.Image.reduceRegions over a FeatureCollection of bbox tiles so the whole grid resolves in a single GEE round-trip rather than one reduceRegion per cell. Result cached on disk under gee_query_cache/lst_grid_<hash>.json with a 30-day TTL — past months don't change. Falls back to a deterministic latitude × elevation × season climatology when GEE is unavailable. The fallback is honest: each cell carries source="modis_mod11a2" or "climatology" so the UI shows the observed-vs-estimated mix. backend/app/routers/demo/lst.py Adds POST/GET /api/demo/lst/country-grid-animation. Resolves a country by ISO2 / ISO3 / name from country_registry.json, uses its bbox, and returns 12 monthly frames + per-frame stats + a document-level provenance block (modis_cells / climatology_cells / pct_observed). Default year = previous calendar year so first-load always returns a complete 12-month animation. Smoke-tested for Ethiopia 2025: 12 frames × 159 cells = 1,908 cells, 100 % MODIS-observed, range 22-43 °C across the year — matches expected highland-cool / lowland-hot pattern. Frontend -------- frontend/src/components/Environmental/components/TemperatureAnimation.jsx New component. Leaflet canvas heatmap rendering each frame as Rectangle cells with a cool→hot colour ramp anchored on °C (0/10/18/24/28/32/38/45/55 stops). Play/pause, year selector (2010-current), 5-step speed control, scrubber, per-frame min/mean/max/n-cells, prominent "X% observed" provenance badge. Uses preferCanvas for ~2k cells without jank. frontend/src/components/Environmental/Environmental.jsx TemperatureAnimation rendered at the top of the LST tab; binds to localCountry so the existing country selector drives it. Default Ethiopia. Tests ----- backend/tests/test_demo_routes.py — added the two new routes to the EXPECTED_ROUTES snapshot; 20/20 still green. Frontend 124/124 green. Vite build clean. v1.1 hooks (already wired, not exercised yet) --------------------------------------------- * country parameter accepts any ISO2 / ISO3 / English name. Today the UI binds to the existing country selector; v1.1 can expose a wider picker without backend changes. * year parameter is generic. v1.1 may stack multiple years into a yearly view by calling the endpoint N times and laying frames in a multi-year scrubber. * grid_size_deg is request-tunable in [0.1, 2.0]. Smaller cells are more detailed but cost more GEE compute; v1 picks 1.0° (~110 km) for fast first-load.
User asked for two things on the new /environmental temperature
animation:
1. Clip to the country polygon — without it the bbox renders cells
over Sudan / Eritrea / Djibouti / Somalia / Kenya / South Sudan
when the user picks "Ethiopia", which is misleading.
2. A yearly mode so the animation can show decadal trend instead of
the annual cycle.
Backend
-------
backend/app/routers/demo/lst.py
* Loads african_country_boundaries.json once and indexes by ISO2.
Each entry's polygon is shapely-wrapped on first use and cached on
the feature dict so subsequent calls don't re-shape the geometry.
* Country-grid-animation now drops cells whose centroid is outside
the country polygon — frame stats (min / mean / max / n_cells) are
recomputed over the kept cells, so badges reflect Ethiopia, not
bbox spillover. The dropped count is reported in
provenance.cells_dropped_outside_border for the UI.
* The boundary GeoJSON is included in the response under .boundary
so the frontend can render the country outline as an overlay.
* New mode parameter ∈ {monthly, yearly}:
- monthly (default, back-compat): 12 frames Jan-Dec of the
chosen year.
- yearly: one frame per year between start_year and end_year
(defaults: last 10 years through the previous full year).
Each year frame is the per-cell mean of the 12 monthly grids
— reuses the existing per-month cache, so first-call cost is
12 × n_years GEE round-trips, every subsequent view is instant.
Cells whose any month is MODIS-observed are tagged
modis_mod11a2; cells that fell back to climatology in every
month stay tagged climatology.
* 31-year span cap on yearly mode so a single request can't burn
unbounded GEE quota (matches the audit-driven bounds on other
endpoints).
* Echoes mode / period / year / start_year / end_year so the UI can
surface the active window without re-deriving.
Smoke results for Ethiopia at 1° grid:
monthly 2025 → 12 frames × 94 in-country cells = 1128 cells,
100% MODIS-observed, 780 cells dropped at borders
yearly 2021-2025 → 5 frames × 94 cells, mean LST 34.5°C → 30.1°C
(2025 is partial year; full-year would be lower
than the 5-year mean — clear physical signal)
Frontend
--------
frontend/src/components/Environmental/components/TemperatureAnimation.jsx
* Renders the boundary GeoJSON as a thin slate-900 outline above
the cell rectangles. interactive=false so it doesn't intercept
cell tooltips. The cell grid edges are intentionally ragged where
they cross the border — honest about the spatial sampling rather
than smoothing it visually.
* Mode toggle (Monthly / Yearly) at the top of the panel, mirroring
the same Personal/System toggle pattern from the Alerts page.
* Mode-specific year selectors:
monthly mode shows a single Year dropdown
yearly mode shows From / to dropdowns
* Frame label adapts: "Jan 2025" in monthly, "2025" in yearly.
* Provenance banner now also surfaces the cells_dropped_outside_border
count so reviewers can see the clip story explicitly.
Tests + build
-------------
Backend pytest 20/20 green; frontend vitest 124/124; vite build clean.
v1.1 hooks still open
---------------------
* mode=seasonal — one frame per season per year. The yearly path's
per-year accumulator structure can be re-used for season buckets.
* country picker — the country query parameter already accepts any
ISO2 / ISO3 / English name; binding to the global country dropdown
is a single-line change in Environmental.jsx.
User flagged that the visible cell size was ~110 km on the LST
animation while the underlying MODIS MOD11A2 product is native 1 km —
the bbox was being sampled too coarsely for the data we have.
Frontend
--------
* Default grid_size_deg dropped from 1.0° (~110 km) to 0.25° (~28 km).
Ethiopia goes from 94 in-country cells per frame → 1499 cells, a
16× increase in spatial detail.
* New Resolution dropdown in the panel toolbar with four presets:
1.0° Coarse (110 km)
0.5° Medium (55 km)
0.25° Fine (28 km) ← new default
0.1° Very fine (11 km — slower first load)
* Title strip now states the source/aggregation honestly:
"native 1 km, aggregated here to <selected>° (~<km> km) for the
animation".
* The 0.1° preset is labelled "slower first load" so the operator
knows the cost trade-off — that grid size produces ~16k cells per
frame and pushes the GEE reduceRegions call hard on first call,
but is essentially free after caching.
Backend
-------
No change — grid_size_deg was already a request parameter clamped to
[0.1, 2.0]. The smaller defaults flow naturally through the existing
per-month cache. Cached past months at any resolution are served
instantly on revisits.
Smoke results for Ethiopia at 0.25° grid, monthly mode:
12 frames × 1499 in-country cells = 17,988 total cells observed
100 % MODIS-observed
12,996 cells dropped at the country border
Tests
-----
Backend pytest 20/20 green; frontend vitest 124/124 green; vite build
clean.
…oLAKES tier B, JRC Africa-mask, Reach-Reg infra
Backend
/api/demo/water-balance/dashboard 5-gauge P/ET/Q/SM/SWE per
country/month with 5-year
climatology + anomaly
/api/demo/water-balance/manager-read rule-based interpretation
of dashboard + auto-picked
largest-reservoir 90-day
forecast; same logic powers
/environmental and /reports
/api/demo/water-resources/reservoir-list all reservoirs by country
/api/demo/water-resources/reservoir-forecast
90-day linear-reservoir model
with ±25% runoff uncertainty
band, soil-moisture-modulated
/api/demo/precipitation/country-grid-animation
CHIRPS + SoilCast (SMAP-
inverted) per country, 12
monthly frames, country-
clipped, lazy-fetched
/api/demo/water-bodies/hydrolakes-points 15,964 African lakes ≥0.1km²
from HydroLAKES, country/
area/bbox filtered
/api/demo/lst/country-grid-animation extended with metric=t2m
(ERA5-Land) + per-country
regions/cities/lakes registry
Africa-masked JRC GSW tile URLs via GEE getMapId minted with
USDOS LSIB Africa filter
(oceans transparent)
ML & Data
app/data/country_lst_registry.json 55 African countries × regions/
cities/lakes for LST panels
app/data/hydrolakes_africa_full.json 6.4 MB / 15,964 records
app/services/water_level_deriver.py derive level + status from
elevation_m + max_depth_m
app/scripts/build_hydrolakes_full.py one-shot extractor
app/scripts/build_recent_water_asset.py Tier B / Option B bake (DW)
app/scripts/build_sar_water_asset.py Option C bake (S1 SAR)
Frontend
Environmental/components/ManagerRead 4-section interpretive panel
Environmental/components/WaterBalanceDashboard
5 dial gauges + anomaly toggle
Environmental/components/ReservoirWhatIf sliders + 90-day chart with
uncertainty band
Environmental/components/PrecipitationAnimation
country-aware, CHIRPS↔SoilCast,
monthly/yearly mode, basemap
toggle (street/satellite/terrain)
TemperatureAnimation country-clipped polygon,
metric=lst_day|t2m, yearly mode,
resolution dropdown, axios
timeout=10min for cold yearly
Map/WaterMap HydroLAKES Tier B overlay via
imperative useMap() pattern
(canvas-pane unmount fix)
Map/panels/LayerControls "All Lakes (HydroLAKES)" toggle
+ JRC oceans-included note
Reports Manager's Read mounted with
auto-pick reservoir hint
Tests / config
tests/conftest.py hermetic SQLite DB + init_db
autocreate so reviewers get
20/20 without a .env
tests/test_demo_routes.py +6 new routes registered
app/config.py Settings.extra='ignore' +
waterhub_recent_water_asset
and waterhub_sar_water_asset
fields
Infrastructure for Tier B (Recent Water 2022→now) and Option C (Sentinel-1
SAR water 2022→now) is in place but inactive: the bake scripts exist, the
loader hooks read WATERHUB_RECENT_WATER_ASSET / WATERHUB_SAR_WATER_ASSET
env vars, the map-layers endpoint registers the layers conditionally. The
GEE Partner Tier and project registration with Earth Engine for asset
writes are pending. Once approved + baked, layers auto-appear in the UI.
Adds a country-pair monitored-flux rollup that picks the downstream-most SWORD reach per crossing, applies the Soman/Indu/Karmakar (2026) QC chain (reach_q==3 drop, width<50m drop, seasonal z-score split JJAS vs dry), and sums SWOT-derived discharge across crossings. New tab surfaces the per-pair totals + per-crossing detail + unmonitored crossings; SWOT tab gains a QC banner showing how many obs were filtered and why.
…p §2)
Tag every training row with label_kind in {ground_truth, pseudo_label};
the third tier, empirical, is reserved for the planned MNDWI retrieval
path. Wire sample_weight (1.0 reference / 0.3 pseudo) through Pipeline
into cross_val_predict, cross_val_score, and the final .fit. Headline
r2_score / rmse / mae are now computed over non-pseudo rows only;
pseudo-only r2_score_pseudo / rmse_pseudo / mae_pseudo are reported
alongside. metric_basis names the contributing label kinds explicitly
so the registry's metadata is self-describing. per_biome_metrics is
filtered the same way so fairness numbers stay consistent with the
headline. New backend/tests/test_training_label_kinds.py covers all
three contract points.
New backend/app/services/ml/baselines.py with three baselines tuned to
the WQ data we actually have:
- SatelliteEmpiricalBaseline: target = features[i] (the satellite's
own Nechad/Dogliotti retrieval at indices 0-3 in the feature vec)
- PerBodyMeanBaseline: analog of persistence given WQ caches have no
time axis; per-water-body mean with global-mean fallback
- BiomeMeanBaseline: climatology by the same 5-band latitude
classification the fairness metrics use
evaluate_baselines_on_cv runs each baseline under the same cv +
fit_groups partition the trained model uses, scoring RMSE/MAE on the
non-pseudo subset so the head-to-head numbers are comparable to the
headline metrics from §2.
train_wq_models embeds baseline_comparison + beats_empirical (bool: model
RMSE beats empirical by >=1%) into metrics so the registry's
metadata.json carries the gate. The override mechanism itself lives at
promotion time.
Tests: test_baselines.py (per-baseline unit coverage + CV evaluator +
threshold) and a new test_baseline_comparison_lands_in_metrics in
test_training_label_kinds.py.
get_soil_moisture returned any cache file with grid_points unconditionally, so a region first cached as modelled_preview (the synthetic fallback) shadowed itself forever — real NISAR L3 granules indexed by ASF DAAC / CMR after the bake were never surfaced. Add a status-dependent TTL: live caches refresh every 14 days (NISAR's 12-day exact-repeat cycle), modelled_preview caches every 1 day (retry for real data daily without hammering CMR). A stale cache falls through to the live search path that already exists. Verified end-to-end: a stale Kenya cache fell through, CMR returned 100 L3 SME2 granules, the HDF5 download produced 112k real observation points, and data_status flipped modelled_preview -> live in ~33s; the next request served the fresh cache in ~2s. New backend/tests/test_nisar_cache_staleness.py covers fresh-served vs stale-fall-through for both statuses, fully offline (the granule search is stubbed).
…meout The NISAR view hardcoded a 'Pre-launch simulation ... not yet publicly available' banner. That became false once NISAR L3 SME2 beta started publishing via ASF DAAC. The banner now reads the backend's per-region data_status: a green 'Live satellite data' note (with the real granule / observation-point count) when ASF DAAC granules back the view, amber 'Modelled preview' when it is still the ~1km synthetic fallback. As coverage fills in, regions flip green with no frontend redeploy. Also fix a cold-load failure: a country's NISAR regions are fetched in parallel, each triggering a synchronous HDF5 download (~25-60s). The 30s axios default timed these out, and the selected-region request had no catch, so the whole NISAR load threw and no banner rendered at all (reproduced on DRC; Kenya only worked because its caches were warm). NISAR requests now use a 10-minute timeout, matching the cold-fetch allowance the temperature-animation view already uses. About.jsx: corrected the three stale 'pre-launch / not yet available' references to describe NISAR L3 SME2 as a published beta product.
The Monthly Time Series for live NISAR regions was not real data: _process_real_granules plugged a climatology formula (a pure function of month/elevation/land-cover) into every granule, so within a season it was a dead-flat line — and one row per granule meant duplicate dates. It was labelled 'NISAR L-band SAR' regardless. Replace it with _extract_nisar_cloud, which reads the actual NISAR L3 SME2 soil moisture: - Cloud byte-range, no local download. h5py + fsspec open each granule HDF5 over HTTPS; only the chunks covering the sampled pixels transfer (~KB-MB), nothing is written to disk. NISAR is not in Earth Engine, so this is the cloud-native equivalent of a server-side query. - Footprint-aware granule selection. A granule is one SAR frame, so a region-wide search returns mostly frames that miss the stations. CMR polygon footprints are now captured and used to pick, per station, up to NISAR_GRANULE_BUDGET granules (one per pass-date). - Adaptive sampling window. NISAR L-band retrieval is heavily masked over dense forest / urban (only ~1% of pixels valid near Yaounde), so the window grows until it finds >=5 valid pixels, then takes the median. Result for Cameroon/Yaounde: 5 real granule-pass points (SM varying 0.08-0.13) instead of 18 climatology rows flat at 0.2007. Stations no granule footprint covers (e.g. Maroua) honestly get no points rather than a fabricated line; if no station is covered the region falls back to modelled_preview. Removes the dead _download_real_sm_grid and _process_real_granules.
Backend loaders, builder scripts, env-var slots and overlay registration for the three JRC 2022-25 extension layers shipped earlier; this commit finishes the UI side and pins the dormant-by-default contract. - WaterMap default layerVisibility declares the three new keys so toggle state is controlled from first paint - Legend block per layer with the matching palette - README features bullet documents the extension - Backend tests assert: helpers return None when env vars unset, dict excludes dormant keys, env-set + GEE-OK surfaces the key, env-set + GEE-fail falls back to None without 500-ing
Surfaces every trained WQ model with its baseline comparison and beats_empirical verdict so an operator can see, at a glance, which models are eligible for promotion and which are blocked. - Backend: /api/admin/ml/baselines (admin-only) reads per-model metadata.json, normalizes the §7 fields and exposes a summary + per-model rows. Older models trained before commit b9456c1 surface as "verdict unknown" — honest, not a UI bug. - Frontend: React panel at /admin/ml/baselines with stat cards, per-model table, verdict badges (Eligible / Blocked / Unknown), and a collapsible policy detail block. Gated by minRole=admin. - Nav: ML Promotion Gate entry in Layout NAV_ITEMS, admin-only. - Tests: auth gate, baseline-present, legacy-metadata-missing, zero-rmse divide-by-zero — 5 passing.
The hydraulic-consistency methods in discharge.py and the matching transboundary route carried "TODO: real satellite-observed path" markers — but that path already exists at HydrocronService.compute_discharge_from_swot (uses observed WSE, slope, width from PO.DAAC Hydrocron) and at /api/transboundary/discharge?water_level_m=… (caller-supplied altimetry WSE). Update docstrings and the response 'method' field to be honest about the algorithm and point callers at the real path. Pin the new contract with tests: - test_response_carries_honest_method_label - test_real_satellite_paths_exist_for_docstring_pointer
Without this, env-var-gated layer loaders (WATERHUB_*_ASSET read via os.environ in routers/demo/water_bodies.py) silently stayed dormant because the values lived only in .env, not in the uvicorn process environment. Pydantic's Settings model already reads .env for declared fields; this is the catch-all for everything that bypasses Settings. Surfaced by the JRC 2022-25 extension work — the SAR layer asset had baked successfully but the live backend kept returning 5 overlays instead of 6 until the .env was sourced.
Replaces the ad-hoc "uncertainty = predicted_value × 0.18" heuristic
with calibrated split-conformal intervals on every trained WQ
prediction. Under exchangeability the [point - q, point + q] band has
marginal coverage ≥ 1 - alpha by construction; default alpha=0.10
gives 90% intervals, matching the §3 audit example.
- conformal.py: SplitConformalRegressor + calibration_quantile with
the (n+1)/n finite-sample correction. No retraining required —
bolts on to any sklearn-compatible base model.
- training_pipeline.py: reuse cross_val_predict output as the
calibration set so the quantile generalises across water bodies
the model didn't see; persist {alpha, q, n_calibration,
coverage_level_pct, method} into metadata.json.
- water_quality_ml.py: prefer the persisted q over CV RMSE when
building MLPrediction.uncertainty; surface the choice in
model_used so the response is honest about which band you got
("split_conformal_q_at_90pct" vs "cv_rmse_fallback").
- 9 tests: coverage on synthetic Gaussian noise (≥86% empirical at
90% target, ≥91% at 95%), monotonicity, zero-residual edge case,
out-of-range alpha, metadata round-trip, default-alpha contract.
Older models trained before this commit have no `conformal` block in
their metadata — they fall back transparently to CV RMSE and the
`model_used` string flags that the band is "not coverage-calibrated".
…ish)
§6 adds a feature-distribution snapshot at training time and a serve-
time scoring endpoint so an operator can answer "is the input I'm
seeing today statistically different from the training distribution?"
without retraining or re-running the pipeline.
- drift.py: feature_snapshot() captures per-feature quantile bins +
occupancy proportions (compact JSON, ~few KB per model). score_drift
computes PSI per feature using the SAME bins as the snapshot — that
is the only way the score is interpretable. Plain-language verdict
with audit thresholds: stable < 0.10, watch < 0.25, shift ≥ 0.25.
- training_pipeline.py: capture snapshot from the unscaled X so the
bins are in natural units; persist under metadata.drift_snapshot.
- POST /api/admin/ml/baselines/{model_key}/drift accepts a recent
feature matrix and returns per-feature PSI + verdict. 409 when the
model is too old to have a snapshot — honest, not silent.
- Frontend MLBaselines panel grows two info cards: "Conformal-
calibrated (§3)" and "Drift snapshots available (§6)" so an admin
sees coverage of both gates at a glance.
- 8 tests: stationary → stable, mean-shift → watch/shift, threshold
contract, constant column, mismatched shapes, auth gate, 4xx
rejection paths.
added 23 commits
May 29, 2026 16:00
…admap §12) New common/ExportMenu dropdown (CSV + GeoJSON downloads via the existing exportApi/downloadFile, plus Print/Save-as-PDF). Replaces the dead 'Export' button stubs in Water Storage, Water Level & Quality, and Country Water hubs with working downloads. Pattern reused from the already-working detail-view export.
Self-paced 6-step tour (Country -> Water Body -> Altimetry -> Policy, ~15s each = ~90s) with pause/step controls and 'open this view' CTAs into the real hubs. CTAs target only viewer-accessible routes so the read-only demo never hits the gated policy-brief page. Wired into the sidebar (nav.storyMode in en/fr/ar) and the router. 5 tests.
…p §4) Removes the color-contrast skip in the axe-core sweep so it now runs on every audited route. Remediation: nudge the water-600 token #0284c7->#0379b3 (4.10-> 4.78:1) so white text on brand surfaces clears 4.5:1; route white-text buttons (Login/Register/Dashboard/QualityChart toggles) to water-600/700; metric-card subtitles gray-400->gray-500. All 5 a11y tests pass with contrast enabled.
Deterministic interaction test: renders IceSat2Tab with a seeded 24-point multi-mission series, asserts the three overlay toggles (Connect observations / Show linear trend / Show seasonal cycle) start off, and that enabling the trend reveals the OLS slope annotation. Chosen over a pixel screenshot of the data-driven recharts SVG (flaky across headless runs); the existing Playwright login/dashboard baselines still pass within tolerance after the §4 palette change, and regression.js already unit-tests the overlay math.
…dmap §6) Adds a reused status block (normal/low/critical/high) and common noData/error to en/fr/ar, and wires WaterStorageHub's page title through t() to demonstrate body-copy translation (not just nav). New locales parity test fails the moment en/fr/ar key sets diverge or a value is empty — the guard that keeps i18n 'complete' as strings grow. Native-speaker review of fr/ar wording remains a separate human task.
Repeatable script for live UN demos: Story Mode (90s) + a manual ~5-min Country -> Water Body -> Altimetry -> Policy walkthrough, with the read-only demo login pre-flight, viewer-role caveats (policy-brief/alerts gating), provenance talking points, fallbacks, and feedback-capture convention. The live session itself remains a human activity.
build_wq_training_set now prefers the §1 feature store (app/data/feature_store.duckdb); when that file is present it shadows the monkeypatched WQ_CACHE_DIR, so the two label-kind tests were reading real backfilled data instead of their seeded ground_truth/pseudo mix and failing on 'pseudo_label in unique' / 'r2_score_pseudo is not None'. Stub the store read in the fixture so the pipeline falls back to the deterministic synthetic cache (Path B) the tests are written to exercise. Backend suite now 112 green, 0 fail. Follow-up: the production default (Path A, feature store) WQ label-kind handling has no direct coverage — worth a seeded-store test.
Adds app/services/swot_crosswalk.py — a status layer classifying each lake/river as observed / no_swot_coverage / modeled_cache / not_mapped_yet, so legacy or modeled SWOT-looking cache points are never presented as observed SWOT data. Wired into multi_mission_altimetry, swot_river and the altimetry router; SWOT LakeSP fetched through Hydrocron (Version D short-names) with tunable enable/timeout/TTL settings. Adds tests/test_swot_crosswalk.py (26 cases) covering the full four-state machine for both lakes and rivers plus SWORD reach-id derivation and the status-summary helper, fully isolated from the on-disk caches.
Closes the post-handover roadmap item — alerts can now be acknowledged
with an attributed actor and rationale, the ack flow emits audit events,
and every DB-backed alert has a deep-linkable detail page.
Backend
- alembic 006: acked_by_user_id (FK→users.id, indexed) + ack_note
(String 500) on alerts; batch_alter_table for SQLite-compat
- Alert model: new columns + acked_by relationship; User.alerts
disambiguated with foreign_keys="Alert.user_id" now that Alert has two
FK paths into users
- AlertResponse exposes acked_by_user_id, acked_by_email, ack_note;
AlertAckRequest carries the optional note
- POST /api/alerts/{id}/ack and /unack — RBAC alerts:manage, idempotent
on re-ack (timestamp + actor frozen, note may be updated), emit
alert.acknowledged / alert.unacknowledged audit events; legacy PUT
path keeps backwards-compat clients working but now also records
the actor
- 10 tests in test_alerts_ack_flow.py covering happy path, idempotency,
viewer-403, cross-user-404, unauth, GET-by-id metadata, audit events
Frontend
- /alerts/:id route + lazy-loaded AlertDetail.jsx with the four-state
ack section (loading / not-found / acked / pending), severity badge,
water-body deep-link, ack/un-ack buttons gated on alerts:manage
- useAlert hook over useApiResource for consistent loading/error/stale
semantics; matching useAlert.test.jsx
- waterService alertsApi gains acknowledge(id, {note}) and
unacknowledge(id)
- Alerts page row title links to /alerts/:id for DB-backed rows,
"Acknowledged" filter chip added, handleAcknowledge calls the new
dedicated endpoint
Operator pack
- §10 item 2 (alert ack workflow, July target) marked done with
concrete pointers; audit_log failure-mode tightened to reflect the
DB-is-source-of-truth state
…5, StoryMode tab-hide pause
i18n (April plan #6 closeout)
- Locales en/fr/ar grew from 60 → 166 keys each (parity guard
enforced): hub body copy, per-hub policy questions + read-guide
swatches, chart tooltip strings (WSE, soil moisture, climatology
band, anomaly, trend, observed), KPI labels, status badges, filter
labels, confidence tooltips
- HubIntro reads "Policy questions this hub answers" + "How to read it"
via t(), the four hubs (WaterStorage / SoilMoisture / WaterLevelQuality
/ CountryWater) pass t('hubs.*.body') + policy questions + legend
labels at the HubIntro callsite, WaterStorageHub KPI cards use
t('kpi.*'). French and Arabic translations are AI-translated rule-of-
thumb wording and still need a native-speaker pass before UN
distribution — tracked as a post-handover item.
useApiResource breadth (April plan #7 closeout)
- 21 additional components migrated onto the shared loading / error /
stale-request guard hook: MLBaselines, QualityChart, CountryDashboard,
Dashboard, MLAnalyticsContext, ManagerRead, WaterBalanceDashboard,
ReservoirWhatIf, ModelMetricsPanel, PrecipitationAnimation,
TemperatureAnimation, the four Hubs, WaterMap, WaterQuality,
Validation, CrossBorderFluxTab, SatelliteData, Reports. Total
adoption: 4 → 25 components.
- Five components correctly do NOT use useApiResource (DataQuery,
QuickBrief, AssimilationTab, parts of LaketempTab, parts of
WaterStorageHub's rivers fetches) because they are user-triggered or
context-owned rather than deps-driven.
StoryMode — pause on tab hide (audit gap #12)
- Add a visibilitychange listener that pauses the auto-advance when the
page is hidden and auto-resumes if the user hadn't already manually
paused. Without this the timer keeps firing in the background and a
delegate returning to the tab after 30 s lands on a step they never
saw.
- Two new tests in StoryMode.test.jsx covering the auto-pause/resume
and the sticky-manual-pause invariant; full suite: 144 / 144 green.
Build envelope unchanged: entry 168 kB, largest non-vendor chunk 218 kB
(SatelliteData).
…ng + DEPLOYMENT Path 1b
Closes April plan #2 (production Docker → staging deploy with HTTPS) —
operator runs one command on a staging VM with DNS pointed at it and
gets a TLS-terminated Water Hub with auto-renewing Let's Encrypt cert.
TLS overlay
- docker-compose.tls.yml: Caddy 2 reverse proxy in front of the
existing prod composition; binds :80, :443, and :443/udp (HTTP/3);
ports: !reset [] on the frontend service so Caddy can take :80
uncontested. caddy_data + caddy_config volumes persist ACME state
across container restarts.
- deploy/Caddyfile: HSTS (1 year, includeSubDomains), zstd/gzip
encoding, real-IP forwarding (X-Real-IP / X-Forwarded-For /
X-Forwarded-Proto) so the backend rate limiter and audit log see
the user not the bridge, staging-CA toggle for ACME debugging.
- deploy/staging-deploy.sh: one-command up/down/status/logs;
pre-flight checks that SECRET_KEY / POSTGRES_PASSWORD /
WATERHUB_DOMAIN / ACME_EMAIL / CORS_ORIGINS are non-blank in
.env.prod (regex ignores trailing whitespace + comments); runs
alembic upgrade head AND the audit_log JSON → DB backfill
(idempotent) before printing the outside-VM smoke-test curl
commands.
- .env.prod.example gains WATERHUB_DOMAIN + ACME_EMAIL.
DEPLOYMENT.md
- New "Path 1b: single-VM with auto-HTTPS (Caddy overlay)" section
walking DNS, firewall, env-var prereqs, bring-up, smoke-test, ACME
failure diagnosis, and the staging-CA debugging trick.
- "Gotcha: the frontend is no longer reachable on the host port"
callout — once the TLS overlay is on, http://<host>:80 from outside
the docker network returns nothing because the frontend container
has no host-port binding. Closes audit gap #9.
Backend prod hardening
- CORS: methods/headers locked to what the API actually uses
(GET/POST/PUT/PATCH/DELETE/OPTIONS; Authorization, Content-Type,
X-Internal-Token, X-Request-ID) instead of "*" wildcards — wildcards
are also incompatible with allow_credentials=True under spec.
- New settings flags water_quality_startup_backfill_{enabled,limit}
and waterhub_landsat_recent_asset so the prod composition can opt
into Landsat-8/9 recent-water overlay and control startup cache
warming without code changes.
- backend/.env.example refreshed to mirror the new fields.
…ata-source status Pre-handover working-tree wave that landed alongside the May feature work but had not been committed. Integrates the SWOT PLD/SWORD crosswalk from commit ac20174 into the demo routers and surfaces observed-vs-modeled provenance in the map time-series panel. Backend - demo/water_bodies.py: integrate lake_swot_crosswalk into the comprehensive payload so a hub badge declares observed / no_swot_coverage / modeled_cache / not_mapped_yet rather than implying coverage for reaches that were only modeled via PLD/SWORD attribution. Adds _db_geometry_to_geojson helper that handles both SQLite WKT and PostGIS geometry payloads, and a _CATALOG_WATER_BODY_IDS fast-lookup set for the new comprehensive validations. - water_bodies.py + icesat2.py + demo_data.py + demo/water_quality.py: small parity fixes for the crosswalk path and an HTTPException raise on missing-lookup paths (replaces silent 200 fallthroughs that the audit caught). - services/country_water_service.py: drop ~17 lines of dead literature-fallback code now that the §1 feature store covers the same parameter set. - services/data_source_status.py: PO.DAAC GRACE mascon metadata exposed in /api/datasets/status when the local snapshot is present, so the hub freshness banner reflects upstream JPL date rather than the bake date. Frontend - Map/panels/TimeSeriesPanel.jsx: surface SWOT data_lineage.data_quality (observed vs modeled) + latest_observation date inline in the panel header so users see provenance at a glance. - components/NotFound.jsx + WaterBodies/WaterBodies.jsx: minor copy / list-rendering fixes from the pan-African strings audit.
Closes the seven §9 items that are now verifiable from the repository
state and annotates the eleven that require a running staging VM, a
browser, or a scheduled human-in-the-loop event so the operator can see
at a glance what is still theirs to do.
Ticked:
- Tier-1 + Tier-2 fixes merged (handover Day 1-4 + Week 3 commits)
- pytest 148 / vitest 144 green
- Secret scan clean (no sk-ant- / AIza / private-key strings)
- backend/.env.example + .env.prod.example match config.Settings (TLS
overlay vars + Landsat-recent asset)
- /api/alerts/{active,scan,statistics,thresholds} return engine JSON
(Day 1 route ordering fix)
- handover/un-v1 tag applied
- Handover pack [INSERT] check (no outstanding placeholders)
Open and explicitly tagged "operator step":
- Staging VM bring-up + LE cert + curl smoke
- Four health endpoints 200 after boot
- 09:00 / 14:00 report-determinism soak
- Alerts / ML Analytics / Reports / devtools UI walkthroughs
- Backup cron
- UN ECA POC credentials + login test
…bled in prod
Three pre-handover blockers raised in the readiness audit. All
container-orchestrator-visible: silent failure modes that the
docker compose healthcheck and the operator runbook depended on
behaving correctly.
#3 Health endpoint returned 200 even when degraded or unhealthy
-------------------------------------------------------------------
`docker compose -f docker-compose.prod.yml` health-checks the backend
with `curl -f http://localhost:8000/api/health`. The handler returned
HTTP 200 regardless of the dependency report — meaning the container
was marked healthy against an unreachable Postgres.
New mapping (lines up with kubernetes liveness convention too):
healthy → 200
degraded → 200 (DB up; GEE / Redis down; service still answers
what it can; operator paged but orchestrator
must not restart)
unhealthy → 503 (DB down; container should be restarted)
unknown → 503 (probe itself failed)
#5 Missing /api/health/{db,redis,gee} sub-routes
-------------------------------------------------------------------
docs/un-handover/06_operator_handover_pack.md curls these six times
during incident response. They didn't exist — operators following the
runbook hit 404. Added with the same status-code semantics:
/api/health/db → 503 when DB unreachable
/api/health/redis → 200 if reachable or skipped (no client lib);
503 if configured-but-unreachable
/api/health/gee → 200 if up or intentionally-degraded
(demo-mode case must not flap the healthcheck),
503 only if the probe itself fails
#2 Production ETL was disabled by config default
-------------------------------------------------------------------
backend/app/config.py:53 has `etl_enabled: bool = False`. The prod
compose file didn't override it, so APScheduler-driven NISAR refresh,
alert scans, and feature-store backfill never ran in production.
docker-compose.prod.yml now sets ETL_ENABLED=${ETL_ENABLED:-true};
.env.prod.example documents the override path (set to false only on
read-only demo hosts).
Tests
-------------------------------------------------------------------
13 new tests in test_health_status_codes.py pin every status-code
contract and the public-vs-internal disclosure surface. Full backend
suite: 161 passed (was 148 before this commit).
Seed-data refresh ahead of the UN handover so the first uncached request on the handover host serves from disk rather than triggering a 25-60 s cold fetch (NISAR HDF5 byte-range) or a 1-3 minute first GEE query. By volume: - river_altimetry_cache: 1438 per-river DAHITI / Hydroweb snapshots - gee_query_cache: 109 CHIRPS / ERA5 / GRACE grid responses - wq_cache: 36 water-quality timeseries (the 873-station cleanup rebake) - nisar_cache: 18 L3 SME2 regional caches (post-staleness-gate) - swot_river_cache: 5 Hydrocron v2 reach snapshots - ml_models: 2 (registry.json refresh + an LSTM precipitation model) - donchyts_water_cache: 2 lake-extent snapshots for wb_400 and wb_1800 - audit_log.json: post backfill (610 → DB; file now shadow-only) - feature_store.duckdb: WQ §1 feature store backing data - 1 each: swot_lake_cache, surface_water_cache, grace metadata Follows the same pattern as commit bbae178 (May 12 GEE/SWOT refresh). No code changes; pure data.
The "four health endpoints return 200 after boot" line was unticked
because nothing had verified it on a real VM. The route surface itself
was also wrong: /api/health/{db,redis,gee} didn't exist and /api/health
always returned 200. Both fixed in commit e8d45d4.
The line stays unticked (still needs a running staging VM to flip) but
now points at the commit + tests that closed the surface gap so the
operator knows the routes exist and behave correctly when probed.
…docs Six high-risk deficiencies and ten pending-actions raised in the readiness audit. This commit closes everything that's closable from the repository state. Frontend production safety (deficiency #1) ------------------------------------------------------------------- VITE_USE_DEMO_API now defaults to FALSE — production builds without the env var hit real endpoints. The previous default ('everything except literal "false"') silently flipped a prod build into demo mode when VITE_USE_DEMO_API wasn't passed at build time. Backend safety (deficiency #5, action #10) ------------------------------------------------------------------- - Self-registration is gated by settings.allow_self_registration (default False — operator adds users via the admin route). When off the endpoint returns 403; rate limit still applies when on. - Password floor raised from 6 → 12 chars. Max stays at 72 to surface bcrypt's silent truncation as a 422 rather than letting two distinct passwords hash equal. - 7 new tests in test_registration_gate.py covering the gate, the 12-char floor, the 73-char cap, and the on-vs-off matrix. Recent-water layer config (deficiency #2) ------------------------------------------------------------------- docker-compose.prod.yml + .env.prod.example now expose WATERHUB_ RECENT_WATER_ASSET, WATERHUB_SAR_WATER_ASSET, and WATERHUB_LANDSAT_ RECENT_ASSET. Empty default = layer hidden (existing convention in backend/app/routers/demo/water_bodies.py header). Backup + restore (deficiency #3, action #7) ------------------------------------------------------------------- deploy/backup.sh captures pg_dump --format=custom + backend_data volume tarball + caddy_data volume tarball (when TLS overlay is on), with --keep-days pruning and --restore drill mode. DEPLOYMENT.md gains a "Database backups" section with the recommended root crontab line (0 2 * * * --keep-days 30) and the quarterly restore-drill recipe; the old "What isn't covered → Database backups" bullet is gone. Operator docs reconciled (action #9) ------------------------------------------------------------------- docs/un-handover/README.md rewritten as the May handover state (was still April prep material): per-file status, acceptance pointers, and an honest note that the original 15 May freeze slipped to 30 May. Data-source matrix (action #8) ------------------------------------------------------------------- docs/un-handover/09_data_source_matrix.md — one row per user-facing panel using the five-state convention {observed / cached / modelled / synthetic / mixed}, with source, cache TTL, and notes. Mirrors the per-component DataProvenance / CitationFooter already shipping in the UI (roadmap §5) so a reviewer can audit the whole platform at a glance without clicking through every hub. Tests: 168 backend / 144 frontend / clean build.
Eight items raised on May 30. The four that are bugs are fixed in code; the worktree-hygiene ones are now committed or gitignored. #1 Worktree not clean ------------------------------------------------------------------- - audit_log.json shadow committed (1 406 entries from the May 30 migration backfill). - Untracked deliverables committed: May report (.md + the two canonical .docx), the satellite freshness audit doc, the Landsat asset builder script, the soil-moisture poster + figure, the delegate-presentation script. - .gitignore tightened to exclude test-results/, todo, and the *_open_before_*.docx save-as backups Word produces during edits. The three docx backups moved to .docx-backups/ (also ignored) so the user can still recover them locally. #2 pytest hangs on first TestClient request (env-specific) ------------------------------------------------------------------- The hang was a leaked DATABASE_URL pointing at a Postgres host the reviewer couldn't reach — `socket.connect()` blocks for tens of seconds before the conftest fell through to its SQLite default. The conftest now forces a hermetic SQLite test DB unless the operator sets WATERHUB_TEST_FORCE_HOST_DB=1; REDIS_URL is pinned to 127.0.0.1 so the redis probe fast-fails; and WATER_QUALITY_STARTUP_BACKFILL_ENABLED defaults to false during tests so the cache warmer doesn't queue blocking GEE work. 168 backend tests still pass in ~37 s on a clean checkout. #4 staging-deploy.sh missing --env-file on the health-loop exec ------------------------------------------------------------------- `docker compose exec` still interpolates ${VAR:?} guards in the compose file at every invocation, and on a clean VM the only place those vars live is .env.prod. Without --env-file the compose call fails with "variable is not set" before exec even reaches the container. #5 backup.sh restore only restored DB ------------------------------------------------------------------- --restore now wipes + re-extracts the backend_data volume AND the caddy_data volume (when present) BEFORE running pg_restore, so a restored host comes up with the same caches + audit shadow + LE certs the backup captured. Containers are stopped first so they don't hold the volumes open. #6 readiness_check.py mutated audit state ------------------------------------------------------------------- The audit-log probe wrote a sentinel row + read it back. That made every run grow the audit table by one entry and produced a JSON shadow diff in the working tree. Probe is now read-only: it exercises AuditLogService.get_logs() (DB primary path); the schema-at-head check already proves the table exists. #8 HSTS on by default ------------------------------------------------------------------- Caddyfile shipped with `Strict-Transport-Security` enabled but the comment said "comment out for the first deploy". Once HSTS is set, browsers refuse http:// for max-age seconds, locking clients on the old URL into an HTTPS path that may not yet work. Commented out by default; the comment now explains the opt-in workflow. #7 production UI on /api/demo/* — documented, not closed ------------------------------------------------------------------- The previous commit flipped VITE_USE_DEMO_API to default off, so production builds hit real endpoints by default. The remaining panels that have no production implementation (InSAR, hyperspectral, cyanobacteria, advanced hydrology) are listed in 09_data_source_matrix.md with their source state. Acceptance scope must explicitly say these panels are demo/synthetic; that's a product decision, not a code fix.
Surfaces per_biome_metrics that every WQ model has been persisting at
training time (training_pipeline._per_biome_metrics) but had no
operator-visible surface for. Closes the post-handover roadmap item
that the operator pack flagged as "August target".
Backend
- GET /api/admin/ml/fairness rolls up per_biome_metrics across every
WQ model in the registry (filters out non-WQ, same convention as
/baselines), keyed by the 5 latitude-band biomes the training
pipeline already uses: equatorial / tropical_savanna / sahelian /
arid_subtropical / temperate.
- Two operator-actionable flags per biome row:
high_rmse — biome RMSE > 1.5x the model's headline RMSE
(configurable via _FAIRNESS_RMSE_RATIO_THRESHOLD)
low_samples — sample_count < 50 (training data too thin to read)
- Cross-model rollup: per-biome high_rmse_count + low_samples_count +
median RMSE ratio so an operator sees "the temperate biome is
under-served on 6 of 9 models" at a glance.
- Forward-looking on legacy models: if metadata.json carries no
per_biome_metrics (older training run), the row appears with
has_biome_metrics=false and every biome flagged no_data — same
honest-not-silent principle as the §7 promotion gate.
- 15 tests in test_admin_ml_fairness.py covering: pure helper
semantics (boundary at 1.5x, 50-sample floor, headline-less case),
endpoint shape, biome ordering (equator-to-pole), flag firing on
realistic registry, legacy no-data path, non-WQ filter, and unauth
rejection.
Frontend
- /admin/ml/fairness route gated minRole=admin (matches baselines).
- MLFairness.jsx renders three summary KPI cards (total / with biome
metrics / without), the cross-model rollup table with red/amber
emphasis on under-served counts, and one per-model card with the
full 5-row biome breakdown. Rows tinted red for high_rmse, amber
for low_samples; FlagPill tooltips explain each threshold.
Next-month plan
- Item #7 (side-by-side model comparison) removed.
- Item #6 rewritten as shipped.
Tests: 184 backend (was 169, +15 fairness) / 144 frontend / clean build.
- frontend axios ^1.6.2 → ^1.15.1 (resolves to 1.16.1) Clears GHSA-pmwg-cvhr-8vh7 (high, NO_PROXY loopback bypass) and GHSA-62hf-57xw-28j9 (moderate, toFormData DoS). - ws override ^8.20.1 in root and frontend package.json Forces puppeteer-core's ws to 8.21.0 (was 8.19.0) and jsdom's ws to 8.21.0 (was 8.20.0). Clears GHSA-58qx-3vcg-4xpx (moderate, uninitialized memory disclosure). Leaves the two vitest critical alerts (GHSA-5xrq-8626-4rwp) for a follow-up — fix requires vitest 1.x → 4.x migration and the vuln only triggers when `vitest --ui` is exposed on a network, which this project never does. All 144 frontend tests pass after the bump.
Mirror trade_hub_tool_description.pdf's structure for Water Hub: Tool Overview + design principles, Intended Users + expectations, Core Functionality, and nine analytical modules (M1 surface water/storage, M2 water quality/HABs, M3 soil moisture, M4 hydrological balance/drought, M5 transboundary, M6 flood/alerting, M7 ML analytics, M8 policy briefs, M9 dashboards/APIs). Single generator script emits the .md source and a Times New Roman/justified .docx styled to match the trade hub PDF; the PDF itself is a local build artefact (rendered via LibreOffice) and stays gitignored like trade_hub_tool_description.pdf.
Author
|
Update pushed in
Validation run locally: |
Author
|
Report update pushed in
|
Author
|
Report filename update pushed in
|
Author
|
Report cleanup pushed in
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
REDIS_REQUIRED=falseruntime mode so local/staging handover checks can document in-memory cache/rate-limit fallback instead of reporting a Redis outage.Why
The UN handover acceptance screen was at 87% because runtime, scheduler, Redis, and ETL evidence were not represented as a valid handover profile. This makes the readiness evidence explicit and lets the staging handover profile reach 100% without hiding real production requirements.
Validation
pytest backend/tests -q-> 192 passednpm test -- src/test/integration.test.jsx-> 9 passed/api/admin/handover/readinessreturnedreadiness_score=100,overall_status=ready,15 pass / 0 warn / 0 failNotes
For production, keep Redis required and provision a real Redis service unless the receiving operators explicitly accept the documented in-memory fallback.