The recovery tier (130 tests, real backend fits) runs nowhere automatically

[wshlavacek]

Follow-up from #514 / #515.

#515 put the fast bngsim suites on hosted runners. The recovery tier did not come with them, and it is now the only substantial part of the test suite with no automated home at all.

Where it stands

pytest -m recovery collects 130 tests across 16 files — test_recovery.py plus the tutorial suites, test_real_world_examples.py, test_profile_likelihood.py, test_gradient_optimizer.py, test_gradient_sbml.py, test_petab_sbml_layer.py and others. These are real parameter-recovery fits through the simulation backend: simulate at known-true parameters, fit, assert the parameters come back.

Before #515 the tier was selected by CI's -m "not slow" but 129 of the 130 skipped for want of bngsim, so it cost nothing. With bngsim installed those 129 would run for real — well over an hour — so #515 deselected the tier explicitly (-m "not slow and not recovery"). That was the right call for a per-push gate, but it means:

• hosted CI: does not run it (deselected)
• pre-push hook: does not run it (.pre-commit-config.yaml runs only the fast bngsim files, deliberately — a multi-minute hook gets the push reaped by GitHub)
• integration.yml: does not run it — that nightly job is -m slow, which is the 4 analytical-target statistical tests, and it runs bngsim: 'false'

So the fitting loop's only end-to-end verification against a genuine engine happens when a maintainer remembers to type pytest -m recovery by hand. That is the premise #514 was filed about, one tier over.

There is also a small live regression: test_petab_sbml_layer.py::TestBoehmRecovery::test_roadrunner_reproduces_published_data is recovery-marked but bngsim-free (roadrunner-backed), so it was the one test of the 130 actually running in CI before #515. It no longer runs anywhere automatic.

Suggested fix

integration.yml is the natural home — it already exists for exactly this, runs nightly at 08:00 UTC plus manual dispatch, and nobody waits on it. Add a second job that passes bngsim: 'true' to the composite action and runs -m recovery. GitHub's 6-hour job limit is not close to binding.

Worth deciding first

1. Runtime is unmeasured on a hosted runner. Over an hour on a dev machine; a 2-core runner will be slower, and -n auto buys less there. Worth one dispatch to size before it goes on a schedule.
2. Statistical tests on a schedule need a flake policy. These are stochastic fits with recovery tolerances. A nightly that goes red every third run for tolerance reasons gets ignored, which is worse than not having it. Decide up front whether a failure notifies anyone.
3. The Boehm test is separable. If the full tier turns out too slow or too flaky to schedule, that one bngsim-free test could simply lose its recovery marker and come back to the per-push job, restoring what ci: install bngsim from PyPI so the simulation suites run in CI (#514) #515 cost.