Skip to content

cargo test -p t27c runs 1 of 73 targets: the 13 known failures hide 60 clean targets and 12 unexamined ones #2382

Description

@gHashTag

The recorded figure is a fail-fast artefact

.github/workflows/corpus-ratchet.yml records, as the reason the test step was reverted:

on master the same command is 1602 passed / 13 failed / 2 ignored

Measured today on origin/master e53b9d048, CARGO_TARGET_DIR warm:

command targets run passed failed
cargo test -p t27c --tests 1 1621 13
cargo test -p t27c --tests --no-fail-fast 73 2031 383

cargo test stops after the first failing target. The unit-test target fails (those are the
13), so the other 72 targets compile and never run. Every integration test in
bootstrap/tests/ is in that group.

This is why the guards added this week are dead, and the missing CI step was only half
the reason. Adding cargo test -p t27c to CI without --no-fail-fast would have run
exactly the 13 and nothing else — including, for example,
on_clock_plain_assign.rs::on_clock_emits_assignment_whose_rhs_does_not_reference_the_target,
the regression guard for #2363. It passes when actually run; it had simply never been
reached.

It also explains the --bins observation on #2363: that measurement could not have covered
the guard either, by a different mechanism.

The 383 is not 383 problems

bitnet_dma.rs                 dma_local_addr_autoincrement_both_paths
bitnet_layer.rs               spec_first_layer2_packs_two_neuron_trits
bitnet_mlp.rs                 spec_first_mlp2_two_layer_inference_matches_reference
bitnet_mlp3.rs                spec_first_mlp3_three_layer_inference_matches_reference
bitnet_neuron_nchunk.rs       spec_first_neuron_n_accumulates_and_quantizes
verilog_array_literal_expr.rs r_ca_2_synthetic_no_comment_only_call_argument
verilog_array_param_index.rs  array_param_index_is_element_part_select
verilog_initial_decl.rs       r_vd_1_synthetic_no_integer_decl_inside_initial
verilog_initial_decl.rs       r_vd_1_real_uart_spec_no_integer_decl_inside_initial
verilog_r_si_1.rs             r_si_1_emitter_injects_mul_noop_helper
verilog_translate_off.rs      r_tr_1_synthetic_no_inline_translate_marker
verilog_translate_off.rs      r_tr_1_real_uart_spec_no_inline_translate_marker

dma_local_addr_autoincrement_both_paths deserves a look first: it lives in the file
#2345 changed (bootstrap/src/bitnet_dma.rs, the write-address pairing fix). I have not
established whether it predates that PR — do not read this as a regression claim, it is
a lead. The remaining eleven are emitter assertions whose age is likewise unknown.

Cost

--no-fail-fast, warm target dir: 1 min 49 s wall. Cold would be dominated by the
t27c build, which several fpga-* jobs already pay.

What this does not establish

Proposed

A ratchet over failing test names — not a count, and not a plain gate. 13 passes when
one is fixed and another appears; a name set does not. Land it as its own job with its own
check name: putting it inside corpus-ratchet.yml places it behind a job that is red on
master, where a new failure is indistinguishable from the standing one.

Whether icarus_lowerable.rs belongs in the baseline or should be excluded pending #2325
is a judgement worth making explicitly rather than by default — baselining 358 failures
makes the file's return to health invisible.

Related: #2292, #2376, #2325, #2363, #2348.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions