diff --git a/.claude/skills/ci-gates/SKILL.md b/.claude/skills/ci-gates/SKILL.md index 5f8d5feac5..000f996908 100644 --- a/.claude/skills/ci-gates/SKILL.md +++ b/.claude/skills/ci-gates/SKILL.md @@ -10380,3 +10380,112 @@ The check as written above still runs — the fetch is needed to learn the live — but its second half, the body diff, is only reached when the ids say something actually moved. On its first use after being written, this section turned a re-read into two commands. + + +## 411. Date a survey from outside its own text + +A competitive table is a claim with a date on it, and the date is usually +nowhere in the file. Here it was: an arXiv identifier has had the form +`YYMM.NNNNN` since 2007, and `YYMM` is the month of FIRST submission -- +unchanged by later versions. So the id dates the PAPER, not the reading, and the +survey behind `specs/igla/coder/benchmark.t27` can be read for age with no +network at all. + + newest paper CITED BY A RECORD 2026-06 2 month(s) ago + 2026-05 9 ######### + 2026-06 49 ######################################## + 2026-07 0 + 2026-08 0 + +Forty-nine records in one month, then nothing. The property that makes this +work is that the population is defined OUTSIDE the repository -- by arXiv's +numbering scheme -- so no matcher of mine can move it, which is exactly what +§383 says a countable population needs. + +Two design decisions the reading needed. The gap is pinned by `--as-of YYYY-MM` +rather than the system clock, because a number that changes while nobody edits +anything cannot be asserted in a test. And there is **no `--gate`**: a gate +that reddens because a month passed, with nobody having changed anything, is a +gate that gets muted. + +## 412. A gap is not a defect until a counterexample closes it + +The empty months above are equally consistent with "the survey stopped looking" +and "the field published nothing", and from inside the repository those two are +indistinguishable. Printing the gap and calling it staleness would have been a +cause invented to fit a correlation. + +What separates them is one **counterexample**: a paper in the gap that belongs +in the table. Two exist -- `arXiv:2607.13079` (ChipVerilog, a Pass@k benchmark +for exactly the systems this table scores) and `arXiv:2607.18519`. That +converts "no recent entries" from an observation into a measurement. + +The command says this and refuses to conclude, and the counterexamples live in +the issue rather than in the code: a hardcoded known-missing paper goes vacuous +the day somebody adds it, and then the check quietly stops testing anything. + +## 413. A claim needs the survey that covers ITS population + +`docs/BITNET_STACK.md` says a differentiator "no competitor has", and "None +generates a network from a ternary-native spec-first compiler -- that is this +stack's unique position". The evidence directly above it is a **four-row** table. + +Two other documents in the same repository state the standard: +`POSITIONING_CONFORMANCE_LAYER.md:122` -- "We do not claim 'first' or 'only' +anything (banned-hype rule)" -- and `COMPETITIVE_ANALYSIS_SCIENTIFIC_FOUNDATIONS.md:337` +-- "**Avoid:** 'No competitor uses similar mathematics' -- not established +without exhaustive survey." + +The interesting part is not that the rule is broken. It is that the repository +holds a 168-record survey which could not license that sentence **however fresh +it were**, because it catalogues LLMs that write Verilog and the claim is about +ternary spec-first compilers. A third document surveys HDL toolchains and is +current to July 2026. Three surveys, three populations, and the claim rests on +none of them. + +Before reading a survey as backing for a superlative, ask what the survey +enumerates and whether the claim quantifies over the same set. Freshness is the +second question; population is the first. + +## 414. An honesty gate that prints a count and a pointer + +`rings-rust.yml` opens by declaring itself "an *honesty gate* that surfaces +real per-crate compile state without yet enforcing it". Its `summary` job +printed the crate COUNT and a link to a hand-maintained markdown file, and never +read the matrix result it had just produced 17 times. + + 2026-08-20 conclusion=success 17 of 19 jobs failed + 2026-08-06 conclusion=success 17 of 19 jobs failed + ... seven such master runs, 2026-05-23 to 2026-08-20 + +Every ring crate failed to compile for three months. All seven runs were green, +and `rings/COMPILE_STATUS.md` -- "the honest, living per-crate compilation +status", last updated 2026-05-22 -- said they compile. Two instruments, both +silent, and the crates were repaired without either having said they broke. + +`continue-on-error: true` was **not** the defect: the workflow states why in +its own header, and that reason is sound. The defect is a summary that reports +a population size instead of a result. When a job is deliberately non-blocking, +its summary is the ONLY place the state is stated, and `steps..outcome` -- +the step result taken before `continue-on-error` is applied -- is the honest +value to print. + +## 415. Measure the detector before you ship it, and run it on its own example + +I was one commit from shipping `tri claims superlative`. Measuring first killed +it: across 2149 first-party markdown files the word `only` occurs **3437** +times and `first` **3152**, so no loose matcher is usable. Narrowed to +documents with a competitor heading it scored **2 real claims in 6 hits** -- and +two of the four false positives were the sentences that STATE the rule +("**Avoid:** 'No competitor uses...'"). Three false positives in six is how a +check dies, so the finding went into an issue as prose and no command shipped. + +The narrowing had its own bug, and it is the one worth remembering: the first +matcher **missed the very document that motivated it**. The heading is +`## Competitors` and the pattern was `competitor`, whose word boundary +fails before the `s`. It reported six documents and the seventh -- the one I +was hunting -- was not among them. + +Always run a new detector against the example that made you write it, and +confirm that example is in the output. Without that check, "found nothing here" +and "cannot see this shape at all" print identically. diff --git a/.github/workflows/rings-rust.yml b/.github/workflows/rings-rust.yml index a5b96aa7a1..cc852a59a3 100644 --- a/.github/workflows/rings-rust.yml +++ b/.github/workflows/rings-rust.yml @@ -4,6 +4,14 @@ # non-blocking (continue-on-error: true) on purpose: this is an *honesty gate* # that surfaces real per-crate compile state without yet enforcing it. # +# It did not surface it. The summary printed a crate COUNT and a pointer to +# rings/COMPILE_STATUS.md, and never read what the matrix had just measured. +# Between 2026-05-23 and 2026-08-20 seven master runs had all 17 crate jobs +# failing; every one concluded `success`, and COMPILE_STATUS.md -- last updated +# 2026-05-22 and calling itself "honest, living" -- said they compile. The +# crates were repaired by 2026-08-28, and nothing in either instrument had said +# they were broken. Each matrix job now writes its own verdict row. +# # Anchor: phi^2 + 1/phi^2 = 3 name: rings-rust @@ -74,13 +82,43 @@ jobs: cargo --version - name: cargo check + id: check working-directory: ${{ matrix.path }} run: cargo check --all-targets --locked || cargo check --all-targets - name: cargo test + id: test working-directory: ${{ matrix.path }} run: cargo test --locked || cargo test + # The job is `continue-on-error`, so a crate that does not compile leaves + # no mark on the run's conclusion. Between 2026-05-23 and 2026-08-20, + # SEVEN master runs had all 17 crate jobs failing and every one of them + # concluded `success` -- while rings/COMPILE_STATUS.md, the file this + # gate's summary points readers at, still said they compile. Two + # instruments, three months, both silent. + # + # `outcome` is the step's result BEFORE continue-on-error is applied, so + # this row is the honest one. Each matrix job appends its own line and + # GitHub concatenates them into one table on the run summary page. + - name: Record this crate's verdict + if: always() + env: + CRATE: ${{ matrix.crate }} + CHECK: ${{ steps.check.outcome }} + TEST: ${{ steps.test.outcome }} + run: | + # A list line, not a table row: the matrix jobs finish in no fixed + # order and GitHub concatenates their summaries in completion order, + # so a header written anywhere would not stay above the rows. + if [ "$CHECK" = "success" ] && [ "$TEST" = "success" ]; then + mark="ok" + else + mark="DID NOT BUILD" + fi + printf -- '- `%s` -- %s (check: %s, test: %s)\n' \ + "$CRATE" "$mark" "$CHECK" "$TEST" >> "$GITHUB_STEP_SUMMARY" + summary: name: Compile gate summary needs: [discover, build] @@ -89,9 +127,20 @@ jobs: steps: - name: Report run: | - echo "## rings-rust compile gate" >> "$GITHUB_STEP_SUMMARY" - echo "" >> "$GITHUB_STEP_SUMMARY" - echo "Discovered crates: ${{ needs.discover.outputs.count }}" >> "$GITHUB_STEP_SUMMARY" - echo "" >> "$GITHUB_STEP_SUMMARY" - echo "This is a non-blocking honesty gate (Wave 13)." >> "$GITHUB_STEP_SUMMARY" - echo "See rings/COMPILE_STATUS.md for the living per-crate table." >> "$GITHUB_STEP_SUMMARY" + { + echo "## rings-rust compile gate" + echo "" + echo "Discovered crates: ${{ needs.discover.outputs.count }}" + echo "" + echo "This is a non-blocking honesty gate (Wave 13): the run stays" + echo "green whatever the crates do. The per-crate rows above are" + echo "therefore the only place the compile state is stated -- read" + echo "them, not this run's conclusion." + echo "" + echo "\`check\` and \`test\` are step outcomes taken BEFORE" + echo "continue-on-error, so a \`failure\` row means that crate" + echo "really did not build." + echo "" + echo "rings/COMPILE_STATUS.md is hand-maintained and can lag this" + echo "table; where they disagree, this table was measured today." + } >> "$GITHUB_STEP_SUMMARY" diff --git a/docs/now/2026-08-31-the-honesty-gate-printed-a-count-and-a-pointer-and-never-rea.md b/docs/now/2026-08-31-the-honesty-gate-printed-a-count-and-a-pointer-and-never-rea.md new file mode 100644 index 0000000000..0340975424 --- /dev/null +++ b/docs/now/2026-08-31-the-honesty-gate-printed-a-count-and-a-pointer-and-never-rea.md @@ -0,0 +1,9 @@ +# NOW -- The honesty gate printed a count and a pointer, and never read what it measured (2026-08-31) + +## The honesty gate printed a count and a pointer, and never read what it measured (Closes #2981) + +- rings-rust's summary job printed the crate count and a pointer to rings/COMPILE_STATUS.md, and never read the matrix result. Seven master runs from 2026-05-23 to 2026-08-20 had all 17 crate jobs failing to compile, and every one concluded success. +- COMPILE_STATUS.md, which the summary points at and which calls itself the honest living per-crate status, was last updated 2026-05-22 and said throughout that they compile. Two instruments, three months, both silent; the crates were repaired by 2026-08-28 with neither having said they were broken. +- continue-on-error is NOT the defect and is not touched: the workflow states why in its own header, and a gate that lands red on the default branch is one nobody can merge past. The defect is that the state was computed 17 times a run and discarded. +- Each matrix job now appends its own verdict line using steps..outcome -- the result taken BEFORE continue-on-error is applied. A list line, not a table row: matrix jobs finish in no fixed order and GitHub concatenates summaries in completion order, so a header would not stay above the rows. Exercised on all four outcome combinations. +- Found by a six-lens adversarial sweep, and the skeptic corrected it twice: continue-on-error has a written reason (so not a defect), and the compile breakage is already repaired (so the repair is preventive, not urgent). 14 of 14 agents returned, 0 errors -- checked before reading the result as a sweep. diff --git a/rings/COMPILE_STATUS.md b/rings/COMPILE_STATUS.md index ad77e76e92..c519e1b9e4 100644 --- a/rings/COMPILE_STATUS.md +++ b/rings/COMPILE_STATUS.md @@ -5,7 +5,16 @@ > CI workflow: [`.github/workflows/rings-rust.yml`](../.github/workflows/rings-rust.yml) > Toolchain: pinned via [`Dockerfile.rust`](../Dockerfile.rust) -- `rust:1.83-bookworm` -This file is the **honest, living** per-crate compilation status for every +> **This file is hand-maintained and does not read CI.** It was last updated +> **2026-05-22**. Between **2026-05-23 and 2026-08-20** seven master runs of +> `rings-rust` had all 17 crate jobs failing to compile while this file said +> they compile -- and every one of those runs concluded `success`, because the +> build job is `continue-on-error` on purpose. The crates were repaired by +> 2026-08-28. Each matrix job now writes its own verdict line into the run +> summary, so where this file and a run disagree, **the run was measured and +> this file was remembered**. + +This file is the per-crate compilation status for every `rings/ring-*-rust/` crate. Wave 13 introduces the **Toolchain & Compilation Gate**: a non-blocking GitHub Actions matrix that runs `cargo check` and `cargo test` against the pinned 1.83 toolchain. Results here are updated