tests: a scratch key needs a per-call component AND a per-process one - #3050
Merged
Conversation
Closes #3046 Four test binaries shared a scratch path under $TMPDIR with no per-process component. Measured with a control -- alone, then 16 copies at once: verilog_real_arithmetic 0/8 alone 41/64 concurrent backend_behaviour 0/12 30/32 verilog_range_bound 0/12 24/64 generic_type_application 0/12 10/192 The four-arm experiment on verilog_real_arithmetic says which half does what: key 1 process, 4 threads 16 processes neither 6 / 150 41 / 64 process::id only 7 / 150 0 / 64 counter only 0 / 150 29 / 64 both 0 / 150 0 / 64 So the fix is per file, not uniform. real_arithmetic and range_bound have several tests calling one helper with the same literal, so they need both halves. backend_behaviour and generic_type_application already pass a distinct tag or stem per test, so they need only the pid -- and backend_behaviour's comment asks for a predictable directory so a failing run leaves findable artefacts, which a counter would take away and the pid does not. `tri harness scratch` advised "an AtomicUsize counter, not the pid". The first half is right and the second is the 29/64 row. The advice now asks for both and prints the table, and it moved into a function so it can be tested: one test fails if the old sentence returns, one fails if an arm of the table is dropped. Both were mutation-checked. After the fix all four read 0/150 intra-process and 0/64 inter-process. That pair is the acceptance criterion. A green ordinary run is not: every one of these passed alone, every time, before and after. Not changed here: the detector's population. It requires a remove_dir_all and so cannot see any of these four, and widening it makes the gate report 17 files of which 4 are measured -- a gate that lands red gets disabled rather than obeyed, which this repository already recorded. Left in #3046 with the full table.
Contributor
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
PR DashboardGenerated at: 2026-09-03 20:08:54 UTC
Summary
Seal Status
|
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
This was referenced Sep 3, 2026
gHashTag
enabled auto-merge (squash)
September 3, 2026 20:30
Contributor
PR DashboardGenerated at: 2026-09-03 20:41:14 UTC
Summary
Seal Status
|
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
gHashTag
added a commit
that referenced
this pull request
Sep 3, 2026
Refs #3059 #3050 landed `advice_tests` at the tail of this file while this branch was appending `population_tests` there. Both kept. The first resolution dropped the closing brace of the incoming module -- the compiler said so immediately ('unclosed delimiter', naming line 332), which is the cheapest reader there is for this kind of merge. And a stale reading in the same minute: `tri harness scratch --gate` reported rc=0 while the build was failing, because the binary on disk was the previous one. An exit code from a tool that did not just build is not a reading of the code you are holding.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3046
Measured, with a control
Release test binaries, run alone and then 16 copies at once. The alone column is what makes the other one mean something — none of these ever failed alone.
verilog_real_arithmeticbackend_behaviourverilog_range_boundgeneric_type_applicationWhich half does what
Four arms on one file, one variable each:
process::idonlyThe counter separates the threads of one run. The pid separates concurrent runs — two agents, two worktrees, or a
cargo testbeside a manual run all share$TMPDIR. Neither alone is enough, and the 7/150 in the second row is noise at that N, not a regression: the pid is identical across the threads of one process, so that arm cannot differ from the first.So the fix is per file, not uniform
verilog_real_arithmetic— four tests callemit("gen-verilog")and two callemit("gen-verilog-for-simulation"), plus a fixed_simpath. Both halves.verilog_range_bound— six tests callemit("gen-verilog"), same fixed_simpath. Both halves.backend_behaviour— the tag is already distinct per test, so nothing collides inside one process. Pid only — and deliberately no counter: the helper's own comment asks for a predictable directory "so a failing run leaves its artefacts behind to look at instead of a random name nobody can find again". A counter would take that away; the pid keeps it and still separates runs.generic_type_application— two distinct stems, so intra-process is fine. Confirmed by running it with--test-threads=1, which still failed 6 / 128 with 16 copies concurrent. Pid only.After: all four read 0/150 intra-process and 0/64 inter-process. That pair is the acceptance criterion. A green ordinary run is not — every one of these passed alone, every time, before and after.
The advice this repository prints was half wrong
tri harness scratchsaid "key the directory by an AtomicUsize counter, not by the pid (shared by every test in the binary)". The first half is right. The second half is the 29/64 row.It now asks for both and prints the table. It also moved into
advice()so it can be tested, because a printed measurement is a claim:the_advice_asks_for_both_components...FAILEDcounter only 29/64armthe_advice_carries_the_measurement...FAILEDcargo test -p tri --bins→ 526 passed, 0 failed (524 before).tri harness scratch --self-checkand--gateboth pass; the four fixed files were not flagged before and are not flagged now.Where the two instruments disagreed, which is the interesting part
A read-only fan-out (48 agents, 43 candidates, 33 refuted) and my concurrency probe answer different questions, and the disagreements are informative rather than embarrassing:
verilog_r_si_1.rs:75— CONFIRMED by reading: one string literal, two callers,File::createtruncating, no per-call component. Measured: 0 of 64 concurrent runs, and 0 occurrences of its owntest inconclusiveline (its callersreturnearly on failure, and an early return from a#[test]is a pass — so I checked for the silent form too). Reading proves the structure of a race; running shows whether the window opens. Not fixed here, and named rather than quietly dropped.backend_behaviour.rs:37— REFUTED by reading, correctly, on the only axis reading was asked about (two tests, one process). It is 30/32 on the other axis.Not changed here: the detector's population
tri harness scratchrequires aremove_dir_allin the file, so it cannot see any of these four. Dropping that clause and adding the per-process axis makes it flag 17 files, of which 4 are measured red, 5 cannot be measured at all (they fail unconditionally on this machine, presumably among master's 383 known-failing tests), and 8 measure zero.This command is a gate (
.github/workflows/harness-scratch.ymlruns--gate). Seventeen findings would land it red, and this repository already recorded what happens then — "a gate that lands red gets disabled rather than obeyed". The full table is in #3046 for whoever stages that properly.