Skip to content

Four tests that CI passes are baselined as failing, so a real Linux regression in them would be invisible #2387

Description

@gHashTag

What

Four tests are in scripts/ci/test-baseline.txt as failing, and the Linux runner passes
them. From the first test-ratchet run on master (e109851b):

73 targets ran; 379 failing test(s) (baseline 383)
4 baselined test(s) now PASS — prune them from the baseline:
  - spec_first_layer2_packs_two_neuron_trits            (tests/bitnet_layer.rs)
  - spec_first_mlp2_two_layer_inference_matches_reference (tests/bitnet_mlp.rs)
  - spec_first_mlp3_three_layer_inference_matches_reference (tests/bitnet_mlp3.rs)
  - spec_first_neuron_n_accumulates_and_quantizes       (tests/bitnet_neuron_nchunk.rs)

They fail on macOS/arm64, where the baseline was generated, and pass on ubuntu-latest.

Why it matters

A CI-passing test left in the baseline is invisible if it breaks. The ratchet fails only
on names that are new relative to the baseline, so these four could start failing on Linux
tomorrow and the gate would stay green — the exact blind spot the ratchet exists to remove,
reintroduced in its own input.

The general rule this fixes: the baseline must describe the platform that gates. A
baseline taken somewhere else describes a different suite.

What this does not establish

  • Why they differ across platforms was not investigated. Endianness, float formatting,
    path separators and filesystem ordering are all plausible and none is confirmed. The four
    are spec_first_* inference-vs-reference comparisons, which suggests a numeric or ordering
    difference rather than a build problem, but that is a guess and is labelled as one.
  • Pruning them means a macOS developer running the suite locally will now see four
    failures the ratchet does not know about. That is the correct trade — CI is what gates —
    but it is a trade, not a free win, and it is worth a note in the file, which this adds.
  • It does not fix the four. If they should pass on macOS too, that is separate work.

Related: #2382, #2383, #2386.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions