Repository navigation
Add opt-in nightly Debian package probe - #3
Closed
nej1gotnochill wants to merge 3 commits into
Closed
nej1gotnochill wants to merge 3 commits into
nej1gotnochill wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved critical command-injection risk and additional provenance, validation, and documentation issues remain.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (1)
What changed in this PR
Adds an opt-in GitHub Actions job to validate published nightly OpenMS Debian packages independently of PR-SHA builds.
Changes:
- Resolves and downloads dated or latest nightly packages.
- Records provenance, validates dependencies, and probes runtime behavior.
- Reuses the Smoke benchmark and uploads diagnostics.
| File | Summary |
|---|---|
.github/workflows/benchmark.yml |
Adds inputs and an isolated nightly Debian package probe job. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
nej1gotnochill
force-pushed
the
feat/nightly-deb-probe
branch
from
September 25, 2026 15:40
c2dc868 to
436c8d3
Compare
Opt-in workflow_dispatch input nightly_deb_probe (+ optional nightly_deb_date pin) driving an isolated run-nightly-deb-probe job: resolve a nightly archive date dir, download the x86_64 .deb, record size/SHA256 + best-effort develop-SHA provenance, satisfy declared Depends, dpkg-deb -x into RUNNER_TEMP, runtime-probe loader (FileMerger --help) and shared-data (Digestor) resolution with no env workarounds, run the unchanged smoke benchmark against the package, enforce the smoke.json verdict, and upload diagnostics (if: always()). Validated end-to-end against the real 2026-09-24 nightly on WSL2 Ubuntu 24.04 (glibc 2.39): loader/data resolution need no env vars; unchanged smoke runner -> verdict: pass. Loader probe uses --help, not --version (TOPP tools reject --version, exit 6, on healthy packages). Isolated by design: no needs:, excluded from render-report, cannot gate PR validation. Nightlies track develop and contain no ONNX; the probe is a nightly packaging/Smoke regression signal, NOT PR-SHA validation.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
The .deb filename is scraped from remote archive HTML and stored in a
step output, from where it was textually expanded into two generated
run: scripts (provenance and summary steps). Restrict the captured
basename to a shell-safe charset with an anchored re-validation, pass
it to consumers via step env (PACKAGE_FILENAME) instead of ${{ }}
expansion, and only record an API-provided develop SHA if it is a
40-hex commit SHA.
nej1gotnochill
force-pushed
the
feat/nightly-deb-probe
branch
from
September 25, 2026 15:44
436c8d3 to
50b6a8e
Compare
nej1gotnochill
added a commit
that referenced
this pull request
Sep 29, 2026
Stage 1 of the runtime-provider architecture on top of the reviewed nightly .deb probe (PR #3). A guarded package-runtime provider resolves the published nightly Debian package, verifies its SHA256, extracts it into $RUNNER_TEMP (never installed system-wide), and derives exact provenance from the package's openms_package_version.h. Smoke and OpenSwath run mode-conditionally against either the source build (pre-Stage-1 behavior, preserved byte-identically - reverse-diff round-trip proven) or the package runtime via the unchanged OPENMS_BIN/ENGINES_DIR/OPENMS_SHA runner contract. ONNX/ProSE jobs gate on the ONNX source build and do not run in package mode; no capability is inferred from binary availability. Normalized results carry explicit runtime provenance (filename, SHA256, archive date, embedded OpenMS SHA), package runs are excluded from SHA-ancestor baseline selection, and render-report uses a mode-conditional artifact pattern and report label so package mode never references the skipped build's empty SHA. Includes review fixes: the pinned default openms_sha is treated as unset by the package strict-SHA guard, the requested SHA is charset-validated before the GitHub API, and the renderer no longer compares a --current run against itself (or drops the _file reference of promoted v1 runs).
timosachsenberg
pushed a commit
that referenced
this pull request
Sep 29, 2026
* Add OpenSwath DIA benchmark prototype
- Copy OpenSwathWorkflow_1 test fixtures from OpenMS repo (7 peptides, 5 SWATH windows)
- Add run_openswath_benchmark.sh following smoke benchmark conventions
- Include correctness metrics: feature count, overall quality, intensity sum, QC JSON
- Update PROVENANCE.md with OpenSwath fixture provenance
This benchmark exercises DIA proteomics (complementing DDA smoke benchmark),
covering SWATH data reading, chromatogram extraction, MRM scoring, and feature picking.
Fixtures: 320 KB total, deterministic, no external dependencies.
Expected runtime: 5-15s in CI.
* Add OpenSwath benchmark to CI workflow
- Add run-openswath-benchmark job alongside smoke benchmark
- Uses same OpenMS artifact from build-openms job
- Installs runtime dependencies and sets up library paths
- Uploads results as separate artifact
This enables CI validation of the OpenSwath DIA benchmark prototype.
* Add v2 schema support with backward compatibility
- Update normalize_smoke and normalize_proteobench to produce v2 output
- Add v1->v2 auto-promotion in load_results
- Support both v1 (openms/tools) and v2 (benchmark-specific) directory layouts
- Add comprehensive test suite for v2 schema changes
* Refactor renderer to be benchmark-agnostic using v2 schema fields
- Remove duplicate v1 fields from normalize_smoke and normalize_proteobench output
- Renderer reads exclusively from v2 fields (identity, run, performance, metrics)
- Dynamic stage discovery: no hard-coded stage names (decoy_database, comet, percolator)
- Dynamic metric comparison: discovers shared metrics between runs
- Optional sections: handles missing stages/build/correctness/tool gracefully
- No benchmark-specific if/elif branches in renderer
- Add comprehensive test suite (11 tests) covering all benchmark types
- Add v2 data access helpers (_identity, _run, _performance, _metrics, etc.)
- Fix tool result labels to use _run_id() instead of raw get("run_id")
- Remove _v1_compat from promote output (clean v2 only)
The renderer now works with Smoke, ProteoBench, OpenSwath, and any future
benchmark type without code changes.
* Fix comparison table regression: include stage/build metrics
The comparison table lost all stage and build metrics when the renderer
was refactored to use _metrics() only. Smoke comparison showed only
"verdict" instead of stage wall times, build time, and stage statuses.
Add _flat_compare() that generically flattens a result's metrics,
performance.build, performance.stages, and correctness into a single
comparable dict — without duplicating data in the stored schema.
Also fix stage regression detection which was silently broken because
stage.*.status keys never appeared in the comparison dict.
Tests: 16/16 pass (5 new tests for _flat_compare, Smoke comparison,
stage regression detection, and ProteoBench no-spurious-keys).
* Add minimal benchmark definition files
Add JSON definition files for Smoke, OpenSwath, and ProteoBench
benchmarks. Each definition contains id, name, description, runner
path, fixtures, and normalization function. Also add a lightweight
validation script that checks referenced files exist and functions
are defined.
These definition files provide discovery and documentation benefits
without hiding execution logic. The runner scripts and normalization
functions remain unchanged. The renderer remains benchmark-agnostic.
* Improve benchmark result normalization and generic reporting
- Add normalize_openswath: extract OpenSwath normalization from runner script
into report_generate.py as a proper CLI subcommand
- Remove tool_versions from canonical v2 result format (normalize_smoke,
normalize_openswath, _promote_v1); identity.software.version remains
the source for primary software version
- Add CI normalization steps: Smoke and OpenSwath results are now normalized
into the canonical v2 format after benchmark execution
- Add 4 tests for normalize_openswath: basic, load_results integration,
missing files graceful handling, required stage failure
- Preserve raw benchmark output for debugging alongside canonical results
- Generic renderer remains benchmark-agnostic; no schema migration introduced
* Harden OpenSwath normalization and CI artifact sizing
Fix missing/empty stages producing incorrect pass verdicts via all([]),
make TSV numeric parsing robust to empty/malformed values, and default
CI artifact size to 0 when du fails. Add 4 regression tests.
* Add ProSE + PeptDeep entrapment benchmark runner (fixture-driven)
Two arms over the same mzML/FASTA/config: baseline (ProSE -> Percolator)
vs ProSE with PeptDeep MS2+RT prediction features (PR #9975). Extraction
of q-values, target/entrapment classification and unique-peptide counting
lives in benchmark/prose_metrics.py (shared with unit tests); the runner
follows the OpenSwath runner conventions (/usr/bin/time -v per stage).
The peptdeep arm fails the verdict unless all five PeptDeep features land
in the idXML, so an ONNX-less build cannot silently pass as baseline.
Includes the v2 normalization subcommand, benchmark definition, design
report and synthetic-idXML unit tests; no dataset is vendored yet.
* Add entrapment dataset generation and fix ProSE runner CLI/verdict handling
- Add make_entrapment_fasta.py (deterministic P02769 target + ENTRAPMENT_
E. coli K-12 targets, rev_ decoys excluded) and prepare_prose_inputs.sh
(FileMerger BSA1+2+3 merge into git-ignored benchmark/generated/prose/)
- Fix peptdeep:enable CLI handling: OpenMS maps the string param to a
no-value flag, so the baseline arm must omit it and the peptdeep arm
must pass it bare
- Record a clean verdict=fail prose.json instead of crashing when a search
produces no PeptideIdentification elements
- Document dataset provenance in the definition note; include pending
design-doc, normalizer, and prose_metrics test updates
Validated against OpenMS PR #9975 (e4b9609c95): both arms pass, Percolator
runs in both, all five PeptDeep features present, target-vs-entrapment
separation clean (67 BSA vs 1 entrapment vs 2 decoy at q<=0.05).
* Add ONNX CI job for ProSE PeptDeep benchmark
* Fix ONNX Runtime CMake discovery in CI
* Fix XercesC dependency in CI
* Fix OpenMS runtime libraries in ONNX benchmark artifact
* Fix OpenMS shared data path in ONNX benchmark
* Add PXD028735 Phase-2 input preparation (HYE target + SoCe entrapment)
Reproducible preparation path for the scientific benchmark dataset
(PXD028735 ProteoBench QE HF-X HYE DDA, CC0): pinned archive.openms.de
downloads with byte-size + sha256 fail-closed verification, deterministic
target + ENTRAPMENT_-prefixed S. cellulosum FASTA assembly, and a
PROVENANCE.txt record. The FASTA generator refuses decoy-marked sources
(including the SoCe file's _rev suffix decoys, which a rev_-prefix check
misses) - ProSE generates decoys via -Search:decoys auto. Validated
against the unmodified runner: verdict pass, both arms, Percolator
q-values, five-feature coverage 79,113/79,113, +5.18% target PSMs at
1% FDR with PeptDeep on prepared inputs.
* Add PXD028735 ProSE benchmark CI job
* Add ProteoBench external comparison tool
* Update README to document current benchmarking workflows
Rewrite the README as a project-level entry point following the main
OpenMS README's information architecture: CI badge, project overview,
table of contents, Features, per-benchmark sections (smoke, OpenSwath
DIA, ProteoBench reference results), a layered architecture diagram,
local run commands, results/reporting with schema v2 and v1 promotion,
the three CI jobs with ccache and artifacts, a Development section on
benchmark definitions and adding a benchmark, Testing, an updated
repository structure, present-tense scope, and Resources.
* Remove private project-documentation link from Resources
The referenced notebook repository is personal and not publicly
visible, so it does not belong in the public README.
* Add OpenSwath results to benchmark reporting
- add `normalize openswath` subcommand: maps raw CI openswath.json into the
common v2 schema (stages, correctness, verdict, identity), failing loudly
on non-finite stage metrics
- store the first OpenSwath baseline from real CI run 32995654115
(build time and artifact size measured from the same CI run)
- point the OpenSwath definition's normalization at the real function so
validate_definitions.py verifies the wiring
- history table gains a generic Benchmark column (two benchmarks now share
the results tree)
- 3 new behavioral tests (19 total): normalization contract, non-finite
rejection, two-run baseline comparison end-to-end
OpenSwath now participates in the same lifecycle as smoke:
runner -> raw JSON -> normalize -> stored v2 baseline -> generic renderer.
* Add real OpenSwath comparison baseline
Store the second real OpenSwath run (CI run 35456974761, OpenMS
95fb768fa79298ca70ebadbf7fa08a8f54724ae4) alongside the existing
baseline (run 32995654115, f1768367fa66f7901b4fa78a9ebece64b2ce9024),
so the generic renderer now performs a real version-to-version
comparison: correctness identical, no regression detected. Fix the
comparison-card template line that emitted a trailing-whitespace line
when no cache note is present; regenerated report.html is now
git-diff-check clean.
* Add post-vcpkg build compatibility to benchmark workflow
OpenMS moved its CI dependency model to vcpkg ("Only use dependencies
from vcpkg"): newer SHAs no longer install libraries via their
deps-ubuntu.sh, so the classic build fails at CMake configure. Detect
the dependency model from the checked-out SHA itself (deps-script
content + vcpkg mechanism presence, no hardcoded boundary), and for
vcpkg-era SHAs initialize the vcpkg submodule, bootstrap it with the
same upstream mechanism (lukka/run-vcpkg), provide a manifest-hash-keyed
binary cache, and configure with OPENMS_USE_VCPKG=ON and the upstream
linux triplet. Pre-vcpkg SHAs keep the existing path unchanged; the
build/ output layout and all benchmark build flags are preserved, so
the runners need no changes. A SHA with neither working model fails
loudly.
* Fix vcpkg runtime libraries in benchmark artifacts
Post-vcpkg OpenMS builds link TOPP tools against dynamic vcpkg
libraries under build/vcpkg_installed/x64-linux-dynamic/lib, which
did not cross the artifact boundary: Smoke and OpenSwath failed at
their first TOPP invocation with exit 127 (loader cannot resolve
libboost/libxerces shared objects). Upload the vcpkg runtime lib
directory with the binary artifact and add it to LD_LIBRARY_PATH in
both benchmark jobs, preserving the existing build/lib path.
* Add GitHub Pages site for ProteoBench comparison
Static stdlib-only generator reusing the ProteoBench comparison layer:
a pinned reference view over the vendored snapshot (default) and an
optional latest view from a CI shallow clone of upstream HEAD, with
local Exp-1/Exp-2 results rendered alongside external submissions.
* Fix artifact-size step on legacy builds
The Phase-2 port made the vcpkg runtime-lib operand unconditional in
the du -sb artifact-size steps, so on legacy (pre-vcpkg) builds du
exited 2 on the missing directory and pipefail failed the benchmark
jobs after they had already passed. Make the operand conditional on
the directory existing.
* Add required vcpkg manifest features to vcpkg builds
OpenMS SHAs in the vcpkg model install the wnetalign headers
(pylmcf/wnet/wnetalign) and the onnxruntime overlay port only through
VCPKG_MANIFEST_FEATURES, matching upstream's linux-x64-ci preset.
Without them the plain build fails at the CMake generate step
(PYLMCF/WNET/WNETALIGN_INC NOTFOUND, seen in CI run 35700644257) and
the ONNX build cannot satisfy find_package(onnxruntime CONFIG
REQUIRED), which the vcpkg branch of OpenMS' CMakeLists requires
instead of the module/tarball path used in legacy mode.
* Use prebuilt ONNX Runtime for vcpkg benchmark
The onnxruntime vcpkg overlay port builds ORT from source (dbg+rel),
which exhausted the runner disk (0 MB during the debug build) and the
90-minute job budget. Drop the onnxruntime manifest feature and resolve
find_package(onnxruntime CONFIG REQUIRED) against the pinned, SHA256-
verified prebuilt release archive instead, staged at
$RUNNER_TEMP/ort-prefix so the generated config package's lib64 /
include-onnxruntime layout resolves real paths.
* Automate normalize metadata in CI and SHA-ancestor baseline selection
Baseline selection (issue #2, task 1):
- pick_baseline now prefers the latest stored run of the same benchmark
whose OpenMS version is an ancestor of the current run's version,
resolved via the GitHub compare API (compare(base...head) status
"ahead"), with per-render caching and graceful fallback to the
previous timestamp rule when ancestry is unknown, offline, or
rate-limited (403/429 disables further API use).
- render gains --baseline (explicit run id | OpenMS SHA | result path)
and --no-sha-baseline.
Normalize + render in CI (issue #2, task 2):
- all five normalizer invocations record real --run-at timestamps;
smoke/openswath derive cache state from the actual vcpkg cache
restore instead of hardcoding "none" (new build-openms output
vcpkg_cache_restored).
- new render-report job merges all benchmark-* result artifacts,
renders report.html, uploads it, and writes the verdict into the
job summary.
Cold-cache acceleration:
- both vcpkg build jobs seed .vcpkg-cache from upstream's published
binary cache at archive.openms.de on Actions-cache miss (same
manifest hash: hashFiles over vcpkg.json + vcpkg-configuration.json
reproduces the Linux-X64-g++-vcpkg-0fa92544... tree), cutting
from-source provisioning from ~90 min to minutes. Misses and
unreachable archives fall back to source builds.
Tests: 6 new renderer tests (28/28); proteobench + prose suites and
4/4 definitions still pass.
* Add opt-in nightly Debian .deb probe (isolated, Smoke-based)
Opt-in workflow_dispatch input nightly_deb_probe (+ optional
nightly_deb_date pin) driving an isolated run-nightly-deb-probe job:
resolve a nightly archive date dir, download the x86_64 .deb, record
size/SHA256 + best-effort develop-SHA provenance, satisfy declared
Depends, dpkg-deb -x into RUNNER_TEMP, runtime-probe loader (FileMerger
--help) and shared-data (Digestor) resolution with no env workarounds,
run the unchanged smoke benchmark against the package, enforce the
smoke.json verdict, and upload diagnostics (if: always()).
Validated end-to-end against the real 2026-09-24 nightly on WSL2 Ubuntu
24.04 (glibc 2.39): loader/data resolution need no env vars; unchanged
smoke runner -> verdict: pass. Loader probe uses --help, not --version
(TOPP tools reject --version, exit 6, on healthy packages).
Isolated by design: no needs:, excluded from render-report, cannot gate
PR validation. Nightlies track develop and contain no ONNX; the probe
is a nightly packaging/Smoke regression signal, NOT PR-SHA validation.
* Update regex to match OpenMS Debian package names
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Harden nightly deb probe against untrusted archive filename
The .deb filename is scraped from remote archive HTML and stored in a
step output, from where it was textually expanded into two generated
run: scripts (provenance and summary steps). Restrict the captured
basename to a shell-safe charset with an anchored re-validation, pass
it to consumers via step env (PACKAGE_FILENAME) instead of ${{ }}
expansion, and only record an API-provided develop SHA if it is a
40-hex commit SHA.
* Add package runtime mode (runtime_source=source|package)
Stage 1 of the runtime-provider architecture on top of the reviewed
nightly .deb probe (PR #3). A guarded package-runtime provider resolves
the published nightly Debian package, verifies its SHA256, extracts it
into $RUNNER_TEMP (never installed system-wide), and derives exact
provenance from the package's openms_package_version.h. Smoke and
OpenSwath run mode-conditionally against either the source build
(pre-Stage-1 behavior, preserved byte-identically - reverse-diff
round-trip proven) or the package runtime via the unchanged
OPENMS_BIN/ENGINES_DIR/OPENMS_SHA runner contract. ONNX/ProSE jobs gate
on the ONNX source build and do not run in package mode; no capability
is inferred from binary availability. Normalized results carry explicit
runtime provenance (filename, SHA256, archive date, embedded OpenMS
SHA), package runs are excluded from SHA-ancestor baseline selection,
and render-report uses a mode-conditional artifact pattern and report
label so package mode never references the skipped build's empty SHA.
Includes review fixes: the pinned default openms_sha is treated as
unset by the package strict-SHA guard, the requested SHA is
charset-validated before the GitHub API, and the renderer no longer
compares a --current run against itself (or drops the _file reference
of promoted v1 runs).
* Fix workflow parse failure from an empty expression in a comment
The nightly-probe comment used a literal \${{ }} to refer to GitHub
expression expansion. GitHub substitutes expressions in run blocks
before the shell sees them, so the empty expression made the whole
workflow unparseable ("An expression was expected") and broke every
dispatch mode, including source builds. Reworded; comment-only change,
no behavioral difference. Introduced by the probe hardening commit
50b6a8e, which therefore never ran in Actions.
* Emit genuinely empty LD_LIBRARY_PATH/OPENMS_DATA_PATH in package mode
The `cond && '' || fallback` idiom never yields an empty value: GitHub
Actions treats '' as falsy, so package mode leaked the source-mode
library/data paths into the benchmark env. The run still passed (the
leaked paths do not exist on a package runner, and the package binaries
resolve everything via RUNPATH), but the env contract did not match the
documented one. Invert the condition: package mode now evaluates to the
empty string, source mode is unchanged (truthy arm, byte-identical
value). Found by inspecting the package-mode CI run 36312329290 logs.
* Fix pages.yml startup_failure and publish benchmark report on the site
The upstream-clone step echoed `success()` from inside a run: block; the
workflow expression parser only accepts that function in if: conditionals,
so every pages.yml run failed at parse time with zero jobs. Drop the dead
line (the next step already checks `[ -d upstream ]` directly) and also
render the benchmark report from the committed results tree into
site/report.html, linked from the index page.
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
timosachsenberg
added this pull request to stack #8
October 1, 2026 09:43
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. OpenMS is on CodeRabbit Free, which includes PR summaries. Ask your admin to upgrade for code reviews.
ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Free Run ID: 📒 Files selected for processing (1)
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe workflow adds an opt-in job that resolves and downloads an OpenMS nightly Debian package, extracts and probes it, and runs the Smoke benchmark. The job records package provenance, requires a passing Smoke verdict, and uploads diagnostics after failures. ChangesNightly Debian probe
Sequence Diagram(s)sequenceDiagram
participant Dispatch as Workflow dispatch
participant Archive as Nightly archive
participant Package as Debian package
participant Probes as FileMerger and Digestor
participant Smoke as Smoke benchmark
participant Artifact as Diagnostics artifact
Dispatch->>Archive: Resolve dated directory and select package
Archive->>Package: Provide package listing
Package->>Probes: Provide extracted binaries and shared data
Probes->>Smoke: Provide probe result and optional data path
Smoke->>Artifact: Provide Smoke results
Probes->>Artifact: Provide probe logs and provenance
Security Architecture ReviewSecurity architecture risk: 🔵 Low · up to The probe is default-off and isolated from existing benchmark jobs. It nevertheless executes externally published packages with normal runner access. Package authenticity beyond HTTPS and the effective repository-token privileges remain unresolved. Retained concerns
Security review detailsSecurity Blast Radius
Security Findings and Attack Paths
Trust Boundaries and Controls
Resilience and Maintainability Implications
Hardening Proposals
A rabbit checks the nightly trail, Comment |
|
feel free to merge your PRs. I think we can best judge once we get some stable benchmarks being collected over time and visualized |
Collaborator
Author
Thank you for letting me know! I’m working in that direction and will definitely keep you updated as things progress. Thank you once again for trusting me with these responsibilities and for giving me the opportunity to work on them. I really appreciate it, and I’ll do my best to handle them well! Have a nice day (; |
Collaborator
Author
|
Closing this PR as the Debian nightly probe functionality has already been incorporated into main through PR #4. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The upstream nightly Debian packages offer an independent packaging/runtime regression signal that no source build provides: they exercise the shipped packaging itself (bundled libraries,
RUNPATH,share/OpenMSlayout, THIRDPARTY engines) on a stock system. They cannot replace the PR-SHA source build, because nightlies trackdevelopand are built without the ONNX Runtime configuration required by the ProSE/PeptDeep validation path.This PR adds an opt-in probe job to the Benchmark workflow:
workflow_dispatchinputnightly_deb_probe(defaultfalse); the optionalnightly_deb_dateinput pins the archive upload-day directory (YYYY.MM.DD, format- and index-validated), empty selects the newest at dispatch. User input reaches the scripts viaenv:, never inline interpolation..deband records filename, byte size, self-computed SHA256, and build-night vs upload-day dates inPROVENANCE.txt(written early so it survives later failures); the develop HEAD is recorded best-effort and never invented.Dependsviaapt-get satisfy --no-install-recommends(verbatim field, so alternative groups are parsed natively) and extracts withdpkg-deb -xinto$RUNNER_TEMP- the package is never installed system-wide.env -i:FileMerger --helpproves the bundled libraries load via the package'sRUNPATH=$ORIGIN/../lib/with noLD_LIBRARY_PATH(--help, not--version: TOPP tools reject--versionwith exit 6 even on a healthy package);Digestoron a synthetic FASTA provesshare/OpenMSdata discovery with noOPENMS_DATA_PATH. If a data-path failure were ever detected,OPENMS_DATA_PATHwould be exported only for the smoke run and the finding recorded in provenance.benchmark/run_smoke_benchmark.shunmodified against the packaged binaries and THIRDPARTY engines; the verdict is re-read fromsmoke.jsonwith explicit pass/fail enforcement, and diagnostics are uploaded as thenightly-deb-probe-<pkg_date>artifact (if: always()).needs:edges, is not inrender-report's needs graph, and is default-off - it cannot gate or alter any existing job or the PR validation path.Evidence (run 36113957688): validated manually on Ubuntu 24.04 (WSL2) first, then in CI; the probe job completed green in 53 s on the identical package bytes (
OpenMS-3.6.0-pre-nightly-2026-09-24-Debian-Linux-x86_64.deb, SHA2565ad43d8dda039206640b06d3f14765996e140e2412d36281bae8112bfabced1b); loader and data probes passed with no environment help; the existing Smoke benchmark returnedVerdict: pass; the artifactnightly-deb-probe-2026-09-24contains provenance, logs, archive metadata and the Smoke results.Scope limitations: this is a develop-snapshot packaging-health check only. It is not PR-SHA validation, it is Smoke-only (no OpenSwath/ProSE/PeptDeep claims), and ONNX/ProSE/PeptDeep are unavailable in the nightly package.
Stacked on
feat/openswath-normalization-and-cleanup(PR #1) because the probe builds on it; GitHub retargets this PR automatically when #1 merges.Checklist
bash -non all run blocks, and CI run 36113957688)How can I get additional information on failed tests during CI
Click to expand
If your PR is failing you can check outIf you click in the column that lists the failed tests you will get detailed error messages.
Advanced commands (admins / reviewer only)
Click to expand
/reformat(experimental) applies the clang-format style changes as additional commit. Note: your branch must have a different name (e.g., yourrepo:feature/XYZ) than the receiving branch (e.g., OpenMS:develop). Otherwise, reformat fails to push.rebuild jenkinswill retrigger Jenkins-based CI buildsSecurity note: review finding resolved in
90a7e6e+c2dc868- the archive-scraped filename is restricted to a shell-safe charset, anchor-revalidated, and consumed via stepenv(never textual${{ }}expansion); the API-sourced develop SHA is recorded only if it is a 40-hex commit SHA.Summary by CodeRabbit
develop, are not tied to the pull request commit, and provide Smoke-only evidence; ONNX is not included.