feat: add schema-2 campaigns and FODB benchmark evidence - #40
Conversation
Introduce immutable dataset packages, deterministic experiment plans, support audits, isolated run bundles, resumable GCP execution, and canonical publication artifacts. Add FODB and controlled resolution-quality workloads, RAM preflight, frozen environments, and provenance validation. Remove historical runners, result matrices, plotting tools, and generated benchmark figures.
Close the file descriptor returned by tempfile.mkstemp before invoking gcloud. Long resume campaigns otherwise exhaust the VM open-file limit while restoring committed bundles.
Raise the bootstrap descriptor limit and prefer the GCloud CLI inside the mounted Snap. This avoids transient DBus scope failures during high-churn GCS checkpoint restore.
Record execution zone as provenance while comparing matching machine and CPU identities across capacity retries. Legacy zone-bound bundles remain valid and publication output retains their exact zones and platform IDs.
Allow committed GCS artifact layouts to be rebuilt locally and validated without rerunning a benchmark.
Filter the exact 3,000-run benchmark matrix, enforce claim-specific protocols, and generate auditable workload, decision, worker-transfer, coverage, version, and figure assets.\n\nAdd focused tests for common-support accounting, provenance, and deterministic ranking helpers.
Shortlist decoders within 10% of local loader leaders before applying the targeted 277-file robustness audit. Generate the practical recommendation table, coverage receipt, and full platform-workload decision surface.
Report malformed empty-DHT recovery separately from four-component normalized-RGB handling while retaining the combined recommendation gate.
Extend only cells whose initial worker sweep peaked at eight. Test workers=12 first and reserve workers=16 for a second boundary-selected stage.
Merge the frozen workers=12 cells with the original FODB matrix, treat location as provenance, and emit the boundary-selected candidates for a possible workers=16 stage.
Generate audited decoder and worker recommendations per platform-workload cell, report staged worker peaks, and derive workload bundle coverage from the validated evidence.
Keep speed evidence, worker curves, and the narrow compatibility audit while dropping the obsolete recommendation and Pillow-migration outputs.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
ternaus
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 47ca8102f5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| def run_worker(run_spec_path: str | Path, artifact_root: str | Path) -> Path: | ||
| spec = load_run_spec(run_spec_path) | ||
| configuration = spec.identity.configuration | ||
| environment, platform_descriptor = _load_provenance(spec) |
There was a problem hiding this comment.
When --worker-python points to a different environment than --environment-descriptor—including when the CLI default is accidentally retained—this only validates the supplied descriptor against the run identity; decoding still uses packages from the current interpreter, and the bundle persists the supplied descriptor unchanged. The resulting valid, publishable bundle can therefore claim package versions and native backends that were not used. Before measuring, compare the running interpreter and installed distributions with the descriptor, or obtain the interpreter directly from the provisioned environment.
Useful? React with 👍 / 👎.
|
|
||
| def _load_selected_items(spec: RunSpec) -> tuple[ResidentItem, ...]: | ||
| configuration = spec.identity.configuration | ||
| package = open_dataset_package(spec.package_descriptor) |
There was a problem hiding this comment.
For a package produced by dataset materialize, every isolated benchmark run opens it without trust_ready=True, so open_dataset_package rehashes and audits the entire archive and read_workload_items rehashes every selected blob. With the documented FODB limit of up to 2 GiB and hundreds of run specs, this adds repeated terabytes of unnecessary I/O and counts against each worker timeout, despite materialization having already written .READY.json and per-workload verification markers. Use the verified-marker path here as the support-audit worker already does.
Useful? React with 👍 / 👎.
1 participant
Scope
This draft contains the full 18-commit
schema2-greenfieldseries againstmain, plus the final publication cleanup.Validation
.venv/bin/pytest -q— 160 passedimread_benchmarkThe arXiv source archive was prepared and compiled separately from a clean extracted tree; it is intentionally local and ignored by the repository.