From 47ebc978e49be470362446f67c7e2834c6b35cbe Mon Sep 17 00:00:00 2001 From: Bill Hlavacek Date: Fri, 24 Jul 2026 14:31:28 -0600 Subject: [PATCH] ci: give the recovery tier a home in integration.yml; restore Boehm to per-push The `recovery` tier (real parameter-recovery fits through the bngsim backend, over an hour) ran nowhere automatically -- deselected from per-push CI (#515), absent from the pre-push hook, and not part of integration.yml's nightly slow job. Add a second job to integration.yml that installs bngsim and runs `-m recovery`. It is gated to workflow_dispatch only for now (not the nightly cron): the wall-clock is unmeasured on a 2-core runner and these are stochastic fits, so it needs one dispatch to size it and a flake policy settled before it goes on a schedule. Escalating is a one-liner (drop the `if:` for nightly, add `continue-on-error` for non-blocking), documented on the job. Also fix the small live regression #515 introduced: the Boehm `test_roadrunner_reproduces_published_data` is roadrunner-backed (bngsim-free, ~3s) but was recovery-marked at the class level, so it stopped running anywhere automatic. Move the class-level recovery marker down to just the bngsim cross-check; the roadrunner leg rejoins the default per-push tier, restoring the one end-to-end real-backend signal #515 dropped. This also aligns the code with the module docstring's stated rule (RoadRunner legs in the normal tier, the bngsim leg gated -m recovery). Closes #517. --- .github/workflows/integration.yml | 54 +++++++++++++++++++++++++++---- tests/test_petab_sbml_layer.py | 14 +++++--- 2 files changed, 57 insertions(+), 11 deletions(-) diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml index 482ed730..d36beeaa 100644 --- a/.github/workflows/integration.yml +++ b/.github/workflows/integration.yml @@ -1,14 +1,18 @@ name: integration -# Manual only. The slow tier is the 4 statistical-recovery tests against -# analytical targets (AnalyticalModel) -- bngsim-free and BNG2.pl-free, ~8 min. +# The home for the test tiers too slow for the per-push gate. Two jobs: # -# NOTE: the fast bngsim *simulation* suites (true NFsim/RuleMonkey/SBML runs) -# DO run in hosted CI -- tests.yml installs bngsim from public PyPI (#514). Not -# covered anywhere hosted yet: the `recovery` tier (real parameter-recovery fits -# through the backend, over an hour), which still runs on a dev machine. +# slow -- the 4 statistical-recovery tests against analytical targets +# (AnalyticalModel), bngsim-free and BNG2.pl-free, ~8 min. Runs +# nightly AND on manual dispatch. +# recovery -- the `recovery` tier: ~130 real parameter-recovery fits through +# the bngsim backend (over an hour on a dev machine, unmeasured on +# a 2-core hosted runner). DISPATCH-ONLY for now (see the job's +# `if:`), so it can be sized and given a flake policy before it +# goes on the nightly schedule (#517). # -# Runs nightly and on manual dispatch. +# NOTE: the fast bngsim *simulation* suites (true NFsim/RuleMonkey/SBML runs) +# DO run in hosted CI -- tests.yml installs bngsim from public PyPI (#514). on: workflow_dispatch: schedule: @@ -36,3 +40,39 @@ jobs: env: PYBNF_NO_BNGSIM: '1' run: uv run --no-sync pytest -m slow -n auto + + recovery: + runs-on: ubuntu-latest + name: recovery tier (bngsim fits) + # Dispatch-only for now, NOT on the nightly cron. The `slow` job above runs + # on both triggers; this one is gated to manual dispatch so a maintainer can + # run it once to measure the wall-clock on a 2-core runner (over an hour on a + # dev machine) and decide a flake policy for these stochastic fits BEFORE it + # is committed to a schedule -- an unmeasured hour-plus job that reds nightly + # for tolerance reasons gets ignored, which is worse than not having it + # (#517). To put it on the 08:00 nightly, delete this `if:` (the `on:` block + # already schedules the workflow); to make it non-blocking instead, add + # `continue-on-error: true` here. + if: github.event_name == 'workflow_dispatch' + steps: + - uses: actions/checkout@v6 + + - uses: ./.github/actions/setup-pybnf + with: + python-version: '3.12' + # The whole point of this tier: real fits through the simulation + # backend, so bngsim must be present (and PYBNF_NO_BNGSIM stays unset). + bngsim: 'true' + # Also a 3.12 setup-pybnf run; give it its own uv-cache namespace so it + # does not race the `slow` job's save on a dispatch (both run then). + cache-suffix: pybnf-recovery + + - name: Run the recovery tier + # -m recovery: the ~130 synthetic-data parameter-recovery fits through + # the backend (test_recovery.py, the tutorial suites, profile-likelihood, + # the gradient/SBML fits, ...). -n auto parallelizes across the runner's + # cores -- fewer here than on a dev box, so expect this to be the slow leg. + # + # --no-sync: the composite action already populated .venv; do not let + # `uv run` re-resolve it from pyproject.toml. + run: uv run --no-sync pytest -m recovery -n auto diff --git a/tests/test_petab_sbml_layer.py b/tests/test_petab_sbml_layer.py index d52b3c9e..5d8dfba4 100644 --- a/tests/test_petab_sbml_layer.py +++ b/tests/test_petab_sbml_layer.py @@ -799,15 +799,21 @@ def test_layer_agrees_across_roadrunner_and_bngsim(self, tmp_path): # --------------------------------------------------------------------------- -# 5. The real-world Boehm problem reproduces its published data (-m recovery) +# 5. The real-world Boehm problem reproduces its published data # # The crafted decay model above isolates the layer with an analytic oracle; this closes # the loop on the HEADLINE milestone -- the externally-authored Boehm v2 problem imported # end to end (ADR-0037). Simulated at the published optimum (the SBML's embedded parameter # values) on RoadRunner, the imported measurement layer's materialized observable columns # track the published measurement table (the fit IS the optimum), and agree across -# RoadRunner and bngsim. Opt-in (-m recovery): a real stiff-ODE simulation of the full -# model, not a unit-scale crafted one. +# RoadRunner and bngsim. +# +# Tier split (mirrors the module docstring's rule): the RoadRunner leg is a core-dependency +# path and runs in the DEFAULT tier (~3s -- a real stiff-ODE sim of the full model, but a +# single one, cheap enough for the per-push gate), so it is unmarked. The bngsim cross-check +# is the expensive one and stays opt-in (-m recovery + @pytest.mark.bngsim). Keeping the +# RoadRunner leg in the default tier is what restores the one end-to-end real-backend signal +# that #515 dropped from per-push CI (#517). # --------------------------------------------------------------------------- _BOEHM_DIR = Path(__file__).resolve().parent / 'petab_fixtures' / 'boehm_v2' @@ -851,7 +857,6 @@ def _simulate_boehm(model_cls, out_dir): return ds[next(iter(ds))] -@pytest.mark.recovery class TestBoehmRecovery: def test_roadrunner_reproduces_published_data(self, tmp_path): @@ -866,6 +871,7 @@ def test_roadrunner_reproduces_published_data(self, tmp_path): assert np.corrcoef(pred, y)[0, 1] > 0.9 assert np.sqrt(np.mean((pred - y) ** 2)) < 0.15 * (y.max() - y.min()) + @pytest.mark.recovery @pytest.mark.bngsim def test_layer_agrees_across_roadrunner_and_bngsim_on_boehm(self, tmp_path): """Neither backend exposes Boehm's computed observables; the measurement layer