corpus: refuse a run that produced no numbers instead of printing zeros - #3040
Merged
Conversation
Closes #3025 A tool that could not be spawned, a capture file that could not be written and a child killed by a signal are not rejections. They met `== Some(0)` at every call site identically to a compile error, so the ENOSPC run that opened the issue published a corpus-wide collapse in acceptance and exited green. `Unresolved` names the four non-verdicts. The three machine-wide ones refuse the run: no percentages, no --per-spec table, exit 2, and JSON that carries no acceptance key at all -- `verilog_build: 0` reads as a measurement of zero rather than as the absence of one. A timeout does not refuse; it is a lower bound a re-run on an idle machine improves. Second door, found by reproducing rather than by reading the report: a corpus over ZERO specs printed `{"specs":0,"verilog_build":0,...}` and exited 0. A mistyped --specs-dir reached it identically, because the walk opens the tree with `read_dir(..).else { continue }`. Both refuse with `refused: no_specs` now. Mutation, both halves: deleting the empty-population guard kills the new end-to-end test; keeping the guard but re-adding one acceptance key to the refusal kills it too. Neither moves any other test in the file, which is also the evidence that the six tests written for the machine-wide half did not cover the empty one. Not fixed here, and not made worse: #3034, the six fixed artefact names in a shared TMPDIR. The tests give every child its own TMPDIR keyed by pid and a counter, so they neither create nor inherit that race.
Contributor
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
This was referenced Sep 3, 2026
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
PR DashboardGenerated at: 2026-09-03 18:48:13 UTC
Summary
Seal Status
|
Owner
Author
The red
|
| binary | alone | 8-way concurrent |
|---|---|---|
verilog_real_arithmetic |
0/8 | 32/64 |
verilog_range_bound |
0/8 | 24/64 |
backend_behaviour |
0/8 | 63/64 |
The CI trace matches: three threads at one instant, two panicking at line 86 (assert!(out.status.success()) inside emit — the child failed) and one at 136 on empty emitted text.
tri harness scratch, which exists to find this, answers none: its conjunction requires a remove_dir_all, because it was written from a case where the shared thing was a directory. These share a file. Filed as #3046 with the census and the proposed widening.
Correction to this PR's own text
The body says the new tests "neither create nor inherit that race", about #3034's fixed artefact names. That is still literally true — the tests that write corpus artefacts set their own TMPDIR, keyed by pid and a counter. But four of the six child spawns in the new file do not set TMPDIR, and the binary spawns t27c children during a 73-target --no-fail-fast run, so it plausibly widened the window on a pre-existing race elsewhere without being its cause. The green rerun is the evidence for "widened, not caused"; I would rather state the mechanism than let "not made worse" stand unqualified.
The probe above runs multiple processes of one binary; CI's failure was multiple threads in one process. Same shared path, so the same defect — but that is the difference between what was observed and what was reproduced.
Contributor
PR DashboardGenerated at: 2026-09-03 19:07:21 UTC
Summary
Seal Status
|
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
PR DashboardGenerated at: 2026-09-03 19:26:45 UTC
Summary
Seal Status
|
Contributor
PR DashboardGenerated at: 2026-09-03 19:44:46 UTC
Summary
Seal Status
|
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
PR DashboardGenerated at: 2026-09-03 20:15:36 UTC
Summary
Seal Status
|
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
PR DashboardGenerated at: 2026-09-03 22:01:55 UTC
Summary
Seal Status
|
Contributor
PR DashboardGenerated at: 2026-09-03 22:11:00 UTC
Summary
Seal Status
|
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3025
The defect
A tool that could not be spawned, a capture file that could not be written, and a child killed by a signal are not rejections. All three met
== Some(0)at every call site identically to a compile error, so the ENOSPC run that opened #3025 published a corpus-wide collapse in acceptance and exited green.Unresolvednames the four non-verdicts (Timeout,NotSpawned,HostIo,Signalled). The three machine-wide ones refuse the run: no percentages, no--per-spectable, exit 2, and JSON that carries no acceptance key at all —verilog_build: 0reads as a measurement of zero rather than as the absence of one. A timeout deliberately does not refuse: it is a lower bound that a re-run on an idle machine improves.The second door, which the report did not have
Reproduced rather than read. A corpus over zero specs printed
— the constant 0 in the format of a measurement, which is the exact sentence the machine-wide refusal exists to prevent. A mistyped
--specs-dirreached it identically, because the walk opens the tree withread_dir(..).else { continue }, so a path that does not exist is indistinguishable from a tree with no specs in it.Both now refuse with
{"refused":"no_specs",...}and exit 2.Evidence
--test corpus_unresolved--bins(whole crate)Mutation, both halves of the new guard:
specs.is_empty()guarda_corpus_over_zero_specs_refuses_instead_of_printing_zerosFAILED, 6 passed / 1 failedNeither mutant moved any other test — which is also the evidence that the six tests written for the machine-wide half did not cover the empty-population half.
Two corrections to the report this branch came from
run_depthnow counts four non-verdicts separately" — false. There is one counter incremented identically at three sites, printed as oneUNRESOLVED (no verdict)line. The line is honest as it stands (printed unconditionally, so0is visible on a clean run) but it is one number, not four.run_depthalso does not refuse; filed as follow-up rather than widened into this PR.verilog_build" — no such ratchet exists in this repo.grep -rl verilog_buildoutsidetarget/returns four paths: this new test,service.rsitself, and two docs. The consumer that actually copies these numbers is prose (docs/theory/IGLA-FORMAL-RESULTS.md), which is a reason for the refusal, not against it — but the automated reader was assumed, not found.Not fixed here, and not made worse
#3034 — six fixed artefact names in a shared
$TMPDIR. The new tests give every child its ownTMPDIR, keyed by pid and an atomic counter, so they neither create nor inherit that race. Fixing it properly changes the four artefact basenames thateach_backend_artefact_write_is_guarded_at_its_own_call_sitepre-creates by hand, which is its own PR with its own control.