The recorded figure is a fail-fast artefact
.github/workflows/corpus-ratchet.yml records, as the reason the test step was reverted:
on master the same command is 1602 passed / 13 failed / 2 ignored
Measured today on origin/master e53b9d048, CARGO_TARGET_DIR warm:
| command |
targets run |
passed |
failed |
cargo test -p t27c --tests |
1 |
1621 |
13 |
cargo test -p t27c --tests --no-fail-fast |
73 |
2031 |
383 |
cargo test stops after the first failing target. The unit-test target fails (those are the
13), so the other 72 targets compile and never run. Every integration test in
bootstrap/tests/ is in that group.
This is why the guards added this week are dead, and the missing CI step was only half
the reason. Adding cargo test -p t27c to CI without --no-fail-fast would have run
exactly the 13 and nothing else — including, for example,
on_clock_plain_assign.rs::on_clock_emits_assignment_whose_rhs_does_not_reference_the_target,
the regression guard for #2363. It passes when actually run; it had simply never been
reached.
It also explains the --bins observation on #2363: that measurement could not have covered
the guard either, by a different mechanism.
The 383 is not 383 problems
bitnet_dma.rs dma_local_addr_autoincrement_both_paths
bitnet_layer.rs spec_first_layer2_packs_two_neuron_trits
bitnet_mlp.rs spec_first_mlp2_two_layer_inference_matches_reference
bitnet_mlp3.rs spec_first_mlp3_three_layer_inference_matches_reference
bitnet_neuron_nchunk.rs spec_first_neuron_n_accumulates_and_quantizes
verilog_array_literal_expr.rs r_ca_2_synthetic_no_comment_only_call_argument
verilog_array_param_index.rs array_param_index_is_element_part_select
verilog_initial_decl.rs r_vd_1_synthetic_no_integer_decl_inside_initial
verilog_initial_decl.rs r_vd_1_real_uart_spec_no_integer_decl_inside_initial
verilog_r_si_1.rs r_si_1_emitter_injects_mul_noop_helper
verilog_translate_off.rs r_tr_1_synthetic_no_inline_translate_marker
verilog_translate_off.rs r_tr_1_real_uart_spec_no_inline_translate_marker
dma_local_addr_autoincrement_both_paths deserves a look first: it lives in the file
#2345 changed (bootstrap/src/bitnet_dma.rs, the write-address pairing fix). I have not
established whether it predates that PR — do not read this as a regression claim, it is
a lead. The remaining eleven are emitter assertions whose age is likewise unknown.
Cost
--no-fail-fast, warm target dir: 1 min 49 s wall. Cold would be dominated by the
t27c build, which several fpga-* jobs already pay.
What this does not establish
Proposed
A ratchet over failing test names — not a count, and not a plain gate. 13 passes when
one is fixed and another appears; a name set does not. Land it as its own job with its own
check name: putting it inside corpus-ratchet.yml places it behind a job that is red on
master, where a new failure is indistinguishable from the standing one.
Whether icarus_lowerable.rs belongs in the baseline or should be excluded pending #2325
is a judgement worth making explicitly rather than by default — baselining 358 failures
makes the file's return to health invisible.
Related: #2292, #2376, #2325, #2363, #2348.
The recorded figure is a fail-fast artefact
.github/workflows/corpus-ratchet.ymlrecords, as the reason the test step was reverted:Measured today on
origin/mastere53b9d048,CARGO_TARGET_DIRwarm:cargo test -p t27c --testscargo test -p t27c --tests --no-fail-fastcargo teststops after the first failing target. The unit-test target fails (those are the13), so the other 72 targets compile and never run. Every integration test in
bootstrap/tests/is in that group.This is why the guards added this week are dead, and the missing CI step was only half
the reason. Adding
cargo test -p t27cto CI without--no-fail-fastwould have runexactly the 13 and nothing else — including, for example,
on_clock_plain_assign.rs::on_clock_emits_assignment_whose_rhs_does_not_reference_the_target,the regression guard for #2363. It passes when actually run; it had simply never been
reached.
It also explains the
--binsobservation on #2363: that measurement could not have coveredthe guard either, by a different mechanism.
The 383 is not 383 problems
tests/icarus_lowerable.rs— 0 passed of 358.iverilogis installed here (/opt/homebrew/bin/iverilog), so this is not a missingtool: it is the Verilog emitter defect gen-verilog: 26 of 32 modules reference struct fields under a prefix declared nowhere — no single prefix can be correct #2325 (the same one giving 26-of-32
non-elaborating modules on the FPGA path), counted once per spec.
dma_local_addr_autoincrement_both_pathsdeserves a look first: it lives in the file#2345 changed (
bootstrap/src/bitnet_dma.rs, the write-address pairing fix). I have notestablished whether it predates that PR — do not read this as a regression claim, it is
a lead. The remaining eleven are emitter assertions whose age is likewise unknown.
Cost
--no-fail-fast, warm target dir: 1 min 49 s wall. Cold would be dominated by thet27cbuild, which severalfpga-*jobs already pay.What this does not establish
just-regressed need different responses, and I did not walk the history.
iverilogpresent),not by reading a failure message from each.
Proposed
A ratchet over failing test names — not a count, and not a plain gate. 13 passes when
one is fixed and another appears; a name set does not. Land it as its own job with its own
check name: putting it inside
corpus-ratchet.ymlplaces it behind a job that is red onmaster, where a new failure is indistinguishable from the standing one.
Whether
icarus_lowerable.rsbelongs in the baseline or should be excluded pending #2325is a judgement worth making explicitly rather than by default — baselining 358 failures
makes the file's return to health invisible.
Related: #2292, #2376, #2325, #2363, #2348.