Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 109 additions & 0 deletions .claude/skills/ci-gates/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -10380,3 +10380,112 @@ The check as written above still runs — the fetch is needed to learn the live
— but its second half, the body diff, is only reached when the ids say something
actually moved. On its first use after being written, this section turned a
re-read into two commands.


## 411. Date a survey from outside its own text

A competitive table is a claim with a date on it, and the date is usually
nowhere in the file. Here it was: an arXiv identifier has had the form
`YYMM.NNNNN` since 2007, and `YYMM` is the month of FIRST submission --
unchanged by later versions. So the id dates the PAPER, not the reading, and the
survey behind `specs/igla/coder/benchmark.t27` can be read for age with no
network at all.

newest paper CITED BY A RECORD 2026-06 2 month(s) ago
2026-05 9 #########
2026-06 49 ########################################
2026-07 0
2026-08 0

Forty-nine records in one month, then nothing. The property that makes this
work is that the population is defined OUTSIDE the repository -- by arXiv's
numbering scheme -- so no matcher of mine can move it, which is exactly what
§383 says a countable population needs.

Two design decisions the reading needed. The gap is pinned by `--as-of YYYY-MM`
rather than the system clock, because a number that changes while nobody edits
anything cannot be asserted in a test. And there is **no `--gate`**: a gate
that reddens because a month passed, with nobody having changed anything, is a
gate that gets muted.

## 412. A gap is not a defect until a counterexample closes it

The empty months above are equally consistent with "the survey stopped looking"
and "the field published nothing", and from inside the repository those two are
indistinguishable. Printing the gap and calling it staleness would have been a
cause invented to fit a correlation.

What separates them is one **counterexample**: a paper in the gap that belongs
in the table. Two exist -- `arXiv:2607.13079` (ChipVerilog, a Pass@k benchmark
for exactly the systems this table scores) and `arXiv:2607.18519`. That
converts "no recent entries" from an observation into a measurement.

The command says this and refuses to conclude, and the counterexamples live in
the issue rather than in the code: a hardcoded known-missing paper goes vacuous
the day somebody adds it, and then the check quietly stops testing anything.

## 413. A claim needs the survey that covers ITS population

`docs/BITNET_STACK.md` says a differentiator "no competitor has", and "None
generates a network from a ternary-native spec-first compiler -- that is this
stack's unique position". The evidence directly above it is a **four-row** table.

Two other documents in the same repository state the standard:
`POSITIONING_CONFORMANCE_LAYER.md:122` -- "We do not claim 'first' or 'only'
anything (banned-hype rule)" -- and `COMPETITIVE_ANALYSIS_SCIENTIFIC_FOUNDATIONS.md:337`
-- "**Avoid:** 'No competitor uses similar mathematics' -- not established
without exhaustive survey."

The interesting part is not that the rule is broken. It is that the repository
holds a 168-record survey which could not license that sentence **however fresh
it were**, because it catalogues LLMs that write Verilog and the claim is about
ternary spec-first compilers. A third document surveys HDL toolchains and is
current to July 2026. Three surveys, three populations, and the claim rests on
none of them.

Before reading a survey as backing for a superlative, ask what the survey
enumerates and whether the claim quantifies over the same set. Freshness is the
second question; population is the first.

## 414. An honesty gate that prints a count and a pointer

`rings-rust.yml` opens by declaring itself "an *honesty gate* that surfaces
real per-crate compile state without yet enforcing it". Its `summary` job
printed the crate COUNT and a link to a hand-maintained markdown file, and never
read the matrix result it had just produced 17 times.

2026-08-20 conclusion=success 17 of 19 jobs failed
2026-08-06 conclusion=success 17 of 19 jobs failed
... seven such master runs, 2026-05-23 to 2026-08-20

Every ring crate failed to compile for three months. All seven runs were green,
and `rings/COMPILE_STATUS.md` -- "the honest, living per-crate compilation
status", last updated 2026-05-22 -- said they compile. Two instruments, both
silent, and the crates were repaired without either having said they broke.

`continue-on-error: true` was **not** the defect: the workflow states why in
its own header, and that reason is sound. The defect is a summary that reports
a population size instead of a result. When a job is deliberately non-blocking,
its summary is the ONLY place the state is stated, and `steps.<id>.outcome` --
the step result taken before `continue-on-error` is applied -- is the honest
value to print.

## 415. Measure the detector before you ship it, and run it on its own example

I was one commit from shipping `tri claims superlative`. Measuring first killed
it: across 2149 first-party markdown files the word `only` occurs **3437**
times and `first` **3152**, so no loose matcher is usable. Narrowed to
documents with a competitor heading it scored **2 real claims in 6 hits** -- and
two of the four false positives were the sentences that STATE the rule
("**Avoid:** 'No competitor uses...'"). Three false positives in six is how a
check dies, so the finding went into an issue as prose and no command shipped.

The narrowing had its own bug, and it is the one worth remembering: the first
matcher **missed the very document that motivated it**. The heading is
`## Competitors` and the pattern was `competitor`, whose word boundary
fails before the `s`. It reported six documents and the seventh -- the one I
was hunting -- was not among them.

Always run a new detector against the example that made you write it, and
confirm that example is in the output. Without that check, "found nothing here"
and "cannot see this shape at all" print identically.
61 changes: 55 additions & 6 deletions .github/workflows/rings-rust.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,14 @@
# non-blocking (continue-on-error: true) on purpose: this is an *honesty gate*
# that surfaces real per-crate compile state without yet enforcing it.
#
# It did not surface it. The summary printed a crate COUNT and a pointer to
# rings/COMPILE_STATUS.md, and never read what the matrix had just measured.
# Between 2026-05-23 and 2026-08-20 seven master runs had all 17 crate jobs
# failing; every one concluded `success`, and COMPILE_STATUS.md -- last updated
# 2026-05-22 and calling itself "honest, living" -- said they compile. The
# crates were repaired by 2026-08-28, and nothing in either instrument had said
# they were broken. Each matrix job now writes its own verdict row.
#
# Anchor: phi^2 + 1/phi^2 = 3
name: rings-rust

Expand Down Expand Up @@ -74,13 +82,43 @@ jobs:
cargo --version

- name: cargo check
id: check
working-directory: ${{ matrix.path }}
run: cargo check --all-targets --locked || cargo check --all-targets

- name: cargo test
id: test
working-directory: ${{ matrix.path }}
run: cargo test --locked || cargo test

# The job is `continue-on-error`, so a crate that does not compile leaves
# no mark on the run's conclusion. Between 2026-05-23 and 2026-08-20,
# SEVEN master runs had all 17 crate jobs failing and every one of them
# concluded `success` -- while rings/COMPILE_STATUS.md, the file this
# gate's summary points readers at, still said they compile. Two
# instruments, three months, both silent.
#
# `outcome` is the step's result BEFORE continue-on-error is applied, so
# this row is the honest one. Each matrix job appends its own line and
# GitHub concatenates them into one table on the run summary page.
- name: Record this crate's verdict
if: always()
env:
CRATE: ${{ matrix.crate }}
CHECK: ${{ steps.check.outcome }}
TEST: ${{ steps.test.outcome }}
run: |
# A list line, not a table row: the matrix jobs finish in no fixed
# order and GitHub concatenates their summaries in completion order,
# so a header written anywhere would not stay above the rows.
if [ "$CHECK" = "success" ] && [ "$TEST" = "success" ]; then
mark="ok"
else
mark="DID NOT BUILD"
fi
printf -- '- `%s` -- %s (check: %s, test: %s)\n' \
"$CRATE" "$mark" "$CHECK" "$TEST" >> "$GITHUB_STEP_SUMMARY"

summary:
name: Compile gate summary
needs: [discover, build]
Expand All @@ -89,9 +127,20 @@ jobs:
steps:
- name: Report
run: |
echo "## rings-rust compile gate" >> "$GITHUB_STEP_SUMMARY"
echo "" >> "$GITHUB_STEP_SUMMARY"
echo "Discovered crates: ${{ needs.discover.outputs.count }}" >> "$GITHUB_STEP_SUMMARY"
echo "" >> "$GITHUB_STEP_SUMMARY"
echo "This is a non-blocking honesty gate (Wave 13)." >> "$GITHUB_STEP_SUMMARY"
echo "See rings/COMPILE_STATUS.md for the living per-crate table." >> "$GITHUB_STEP_SUMMARY"
{
echo "## rings-rust compile gate"
echo ""
echo "Discovered crates: ${{ needs.discover.outputs.count }}"
echo ""
echo "This is a non-blocking honesty gate (Wave 13): the run stays"
echo "green whatever the crates do. The per-crate rows above are"
echo "therefore the only place the compile state is stated -- read"
echo "them, not this run's conclusion."
echo ""
echo "\`check\` and \`test\` are step outcomes taken BEFORE"
echo "continue-on-error, so a \`failure\` row means that crate"
echo "really did not build."
echo ""
echo "rings/COMPILE_STATUS.md is hand-maintained and can lag this"
echo "table; where they disagree, this table was measured today."
} >> "$GITHUB_STEP_SUMMARY"
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# NOW -- The honesty gate printed a count and a pointer, and never read what it measured (2026-08-31)

## The honesty gate printed a count and a pointer, and never read what it measured (Closes #2981)

- rings-rust's summary job printed the crate count and a pointer to rings/COMPILE_STATUS.md, and never read the matrix result. Seven master runs from 2026-05-23 to 2026-08-20 had all 17 crate jobs failing to compile, and every one concluded success.
- COMPILE_STATUS.md, which the summary points at and which calls itself the honest living per-crate status, was last updated 2026-05-22 and said throughout that they compile. Two instruments, three months, both silent; the crates were repaired by 2026-08-28 with neither having said they were broken.
- continue-on-error is NOT the defect and is not touched: the workflow states why in its own header, and a gate that lands red on the default branch is one nobody can merge past. The defect is that the state was computed 17 times a run and discarded.
- Each matrix job now appends its own verdict line using steps.<id>.outcome -- the result taken BEFORE continue-on-error is applied. A list line, not a table row: matrix jobs finish in no fixed order and GitHub concatenates summaries in completion order, so a header would not stay above the rows. Exercised on all four outcome combinations.
- Found by a six-lens adversarial sweep, and the skeptic corrected it twice: continue-on-error has a written reason (so not a defect), and the compile breakage is already repaired (so the repair is preventive, not urgent). 14 of 14 agents returned, 0 errors -- checked before reading the result as a sweep.
11 changes: 10 additions & 1 deletion rings/COMPILE_STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,16 @@
> CI workflow: [`.github/workflows/rings-rust.yml`](../.github/workflows/rings-rust.yml)
> Toolchain: pinned via [`Dockerfile.rust`](../Dockerfile.rust) -- `rust:1.83-bookworm`

This file is the **honest, living** per-crate compilation status for every
> **This file is hand-maintained and does not read CI.** It was last updated
> **2026-05-22**. Between **2026-05-23 and 2026-08-20** seven master runs of
> `rings-rust` had all 17 crate jobs failing to compile while this file said
> they compile -- and every one of those runs concluded `success`, because the
> build job is `continue-on-error` on purpose. The crates were repaired by
> 2026-08-28. Each matrix job now writes its own verdict line into the run
> summary, so where this file and a run disagree, **the run was measured and
> this file was remembered**.

This file is the per-crate compilation status for every
`rings/ring-*-rust/` crate. Wave 13 introduces the **Toolchain & Compilation
Gate**: a non-blocking GitHub Actions matrix that runs `cargo check` and
`cargo test` against the pinned 1.83 toolchain. Results here are updated
Expand Down
Loading