Skip to content

ci: install bngsim from PyPI so the simulation suites run in CI (#514) - #515

Merged
wshlavacek merged 1 commit into
mainfrom
ci/run-bngsim-suite-from-pypi
Jul 24, 2026
Merged

ci: install bngsim from PyPI so the simulation suites run in CI (#514)#515
wshlavacek merged 1 commit into
mainfrom
ci/run-bngsim-suite-from-pypi

Conversation

@wshlavacek

Copy link
Copy Markdown
Collaborator

Closes #514.

tests.yml disabled the entire bngsim surface on the premise that bngsim is a private, macOS-only wheel. It is neither — bngsim 0.11.35, exactly the version pyproject.toml pins, is on public PyPI with manylinux_2_28_x86_64 wheels for cp310–cp313.

The published wheel is fully capable

Installed from PyPI into a clean venv, every capability PyBNF gates on comes back available:

feature published wheel
nfsim, rulemonkey
libsbml, sbml_import, sbml_ssa, sbml_psa
codegen, output_sensitivities
SteadyStateResult.output_sensitivities
antimony_import needs the companion antimony PyPI package (manylinux wheel exists)

So the action installs bngsim[antimony] — the bngsim_antimony-marked tests unskip along with everything else.

Changes

  • setup-pybnf grows a bngsim input (default true). A leg that must stay bngsim-less passes 'false' and keeps PYBNF_NO_BNGSIM=1: py3.14 (no cp314 wheel — it would fall back to an sdist build), the docs job (autodoc never simulates), the native-BNGL petab leg, and integration.yml's analytical slow tier.
  • The pytest matrix moves to include so 3.14 can carry no-bngsim: '1'. That leg is no longer an accident of packaging: it is now the deliberate guard that the bngsim-free code paths still work, which is worth a leg regardless. On the other legs PYBNF_NO_BNGSIM is empty, which _bngsim_caps.py reads as unset.
  • The marker expression becomes -m "not slow and not recovery". Naming -m on the command line replaces the addopts default, so the old command was also selecting the recovery tier — harmless only because 129 of its 130 tests skip without bngsim. With bngsim present they are real parameter-recovery fits and the job would run for over an hour. The cost of deselecting is the 130th (a roadrunner-backed Boehm oracle, bngsim-free); noted in the comment.
  • One real bug the new coverage exposed. test_fd_acceptance_gate_estimated_scale is @pytest.mark.bngsim and fits a loguniform_var parameter, whose log-space chain rule autodiffs through jax — an extra CI does not install. Its twelve siblings in the file declare pytest.importorskip('jax'); this one did not, so it failed rather than skipped the moment bngsim arrived. Declared.
  • Stale "private / mac-only" rationale corrected in .pre-commit-config.yaml, integration.yml, docs.yml and tests/README_integration.md. ADRs left alone as historical records.

Verification

The CI install was mirrored exactly — bngsim from PyPI, not the local dev wheel — and the job's own command run against it:

3221 passed, 78 skipped, 0 failed in 4m28s

No bngsim-family test skips remain. The 78 residual skips are all optional extras CI has never installed (jax/blackjax, arviz, matplotlib) — test_hmc.py alone loses ~40 tests to that, which is worth a separate issue.

Landing as a hard gate rather than behind continue-on-error: the local run is 0-failure against the published wheel, and a non-blocking gate is the failure mode #514 is about.

Follow-ups (not in this PR)

  • The composite action installs a hand-mirrored dep list that omits the tests extra's jax / arviz / matplotlib, so those oracles skip in CI.
  • The recovery tier still runs nowhere automatically.

🤖 Generated with Claude Code

tests.yml ran the whole suite under PYBNF_NO_BNGSIM=1 on the premise that
bngsim was a private, macOS-only wheel. It is neither: bngsim 0.11.35 --
exactly the version pyproject.toml pins -- is on public PyPI with
manylinux_2_28_x86_64 wheels for cp310-cp313. The premise cost 323
bngsim/nfsim/rulemonkey/sbml-marked tests, including 14 dedicated
tests/test_bngsim_*.py files, none of which had ever run on a hosted runner;
their only gate was one maintainer's pre-push hook.

The published wheel is fully capable -- NFsim, RuleMonkey, libsbml, codegen,
forward and steady-state output sensitivities all compile in. Only
antimony_import needs a companion package (antimony, also a manylinux wheel),
so the composite action installs bngsim[antimony] and the bngsim_antimony
tests unskip too.

- setup-pybnf grows a `bngsim` input (default true). A leg that must stay
  bngsim-less passes 'false' and keeps PYBNF_NO_BNGSIM=1: py3.14 (no cp314
  wheel, and it now doubles as the guard on the bngsim-free code paths), docs
  (autodoc never simulates), the native-BNGL petab leg, and integration.yml's
  analytical slow tier.

- The pytest matrix moves to `include` so 3.14 can carry no-bngsim: '1'. On the
  other legs PYBNF_NO_BNGSIM is empty, which _bngsim_caps reads as unset.

- The marker expression becomes `-m "not slow and not recovery"`. Naming -m
  REPLACES the addopts default, so the old command was also selecting the
  recovery tier -- free only because 129 of its 130 tests skip without bngsim.
  With bngsim present they are real parameter-recovery fits and the job would
  run over an hour.

- test_fd_acceptance_gate_estimated_scale is bngsim-marked and fits a
  loguniform parameter, whose log-space chain rule autodiffs through jax --
  an extra CI does not install. Its twelve siblings declare
  pytest.importorskip('jax'); this one did not, so it failed rather than
  skipped the moment bngsim arrived. Declared.

Verified by mirroring the CI install exactly (PyPI bngsim, not the dev wheel)
and running the job's command: 3221 passed, 78 skipped, 0 failed. No
bngsim-family test skips remain; the residual skips are the jax/arviz/
matplotlib extras CI has never installed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@wshlavacek
wshlavacek merged commit acb3e0a into main Jul 24, 2026
8 checks passed
@wshlavacek
wshlavacek deleted the ci/run-bngsim-suite-from-pypi branch July 24, 2026 18:47
wshlavacek added a commit that referenced this pull request Jul 24, 2026
…o per-push (#518)

The `recovery` tier (real parameter-recovery fits through the bngsim backend,
over an hour) ran nowhere automatically -- deselected from per-push CI (#515),
absent from the pre-push hook, and not part of integration.yml's nightly slow
job. Add a second job to integration.yml that installs bngsim and runs
`-m recovery`. It is gated to workflow_dispatch only for now (not the nightly
cron): the wall-clock is unmeasured on a 2-core runner and these are stochastic
fits, so it needs one dispatch to size it and a flake policy settled before it
goes on a schedule. Escalating is a one-liner (drop the `if:` for nightly, add
`continue-on-error` for non-blocking), documented on the job.

Also fix the small live regression #515 introduced: the Boehm
`test_roadrunner_reproduces_published_data` is roadrunner-backed (bngsim-free,
~3s) but was recovery-marked at the class level, so it stopped running anywhere
automatic. Move the class-level recovery marker down to just the bngsim
cross-check; the roadrunner leg rejoins the default per-push tier, restoring the
one end-to-end real-backend signal #515 dropped. This also aligns the code with
the module docstring's stated rule (RoadRunner legs in the normal tier, the
bngsim leg gated -m recovery).

Closes #517.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI runs bngsim-less on a stale premise: bngsim is on public PyPI, not a private mac-only wheel

1 participant