What
Four tests are in scripts/ci/test-baseline.txt as failing, and the Linux runner passes
them. From the first test-ratchet run on master (e109851b):
73 targets ran; 379 failing test(s) (baseline 383)
4 baselined test(s) now PASS — prune them from the baseline:
- spec_first_layer2_packs_two_neuron_trits (tests/bitnet_layer.rs)
- spec_first_mlp2_two_layer_inference_matches_reference (tests/bitnet_mlp.rs)
- spec_first_mlp3_three_layer_inference_matches_reference (tests/bitnet_mlp3.rs)
- spec_first_neuron_n_accumulates_and_quantizes (tests/bitnet_neuron_nchunk.rs)
They fail on macOS/arm64, where the baseline was generated, and pass on ubuntu-latest.
Why it matters
A CI-passing test left in the baseline is invisible if it breaks. The ratchet fails only
on names that are new relative to the baseline, so these four could start failing on Linux
tomorrow and the gate would stay green — the exact blind spot the ratchet exists to remove,
reintroduced in its own input.
The general rule this fixes: the baseline must describe the platform that gates. A
baseline taken somewhere else describes a different suite.
What this does not establish
- Why they differ across platforms was not investigated. Endianness, float formatting,
path separators and filesystem ordering are all plausible and none is confirmed. The four
are spec_first_* inference-vs-reference comparisons, which suggests a numeric or ordering
difference rather than a build problem, but that is a guess and is labelled as one.
- Pruning them means a macOS developer running the suite locally will now see four
failures the ratchet does not know about. That is the correct trade — CI is what gates —
but it is a trade, not a free win, and it is worth a note in the file, which this adds.
- It does not fix the four. If they should pass on macOS too, that is separate work.
Related: #2382, #2383, #2386.
What
Four tests are in
scripts/ci/test-baseline.txtas failing, and the Linux runner passesthem. From the first
test-ratchetrun on master (e109851b):They fail on macOS/arm64, where the baseline was generated, and pass on
ubuntu-latest.Why it matters
A CI-passing test left in the baseline is invisible if it breaks. The ratchet fails only
on names that are new relative to the baseline, so these four could start failing on Linux
tomorrow and the gate would stay green — the exact blind spot the ratchet exists to remove,
reintroduced in its own input.
The general rule this fixes: the baseline must describe the platform that gates. A
baseline taken somewhere else describes a different suite.
What this does not establish
path separators and filesystem ordering are all plausible and none is confirmed. The four
are
spec_first_*inference-vs-reference comparisons, which suggests a numeric or orderingdifference rather than a build problem, but that is a guess and is labelled as one.
failures the ratchet does not know about. That is the correct trade — CI is what gates —
but it is a trade, not a free win, and it is worth a note in the file, which this adds.
Related: #2382, #2383, #2386.