tools: rank what is actually wrong with the generated C

docs/now/2026-09-08-what-is-biggest-now.md

@@ -0,0 +1,9 @@
+# NOW -- What is biggest now (2026-09-08)
+
+## What is biggest now (Refs #3464)
+
+- Three passes followed one error class from 1729 down to 522 while a family ten times its size sat uncounted, because the first question of each pass was «what is left of what I was doing» rather than «what is biggest now». `tools/c_error_classes.py` asks the second one: it ranks the classes of the generated C corpus, with the cap off and the compiler's identity printed beside the numbers.
+- It found in one run what three passes had not. The `Trit` family -- `unexpected type name 'Trit'` **642**, `use of undeclared identifier 'POS'` **412**, `'NEG'` **319** -- is **1373 errors**, more than twice the `__auto_type` class I have been shrinking. It has never been looked at.
+- The 2057 undeclared-function errors split by cause, which the totals could not do: **A 487 (23%)** declared as `fn` in some spec and not reaching this file (`len` 142, `mean` 40); **B 149 (7%)** the name of a MODULE or spec FILE called as a function (`adder_tree` 58, `ternary_gemm` 50); **C 1421 (69%)** a name nothing in the tree declares (`cast_i8` 210, `compose` 76). Three causes, three different repairs, and the counts had them in one bucket.
+- **I misread my own output on the way there.** A list I printed «excluding len and expect» I then read as if it were bucket A, and concluded `cast_i8` was declared somewhere -- while a grep two commands earlier had said zero files. The contradiction was visible in the same terminal. `cast_i8` is in bucket C, as the grep said.
+- The tool is a READER, not a gate, and deliberately not wired into CI: it rebuilds the whole corpus, and nothing it prints is a pass/fail claim. Its self-check is the usual pair -- a planted error must be counted, a clean file must count zero -- because a report of «0» and a report that could not run look identical.

tools/c_error_classes.py

@@ -0,0 +1,175 @@
+#!/usr/bin/env python3
+"""What is actually wrong with the generated C, ranked.
+
+Three passes were spent shrinking one error class from 1729 to 522 while a
+family TEN TIMES its size sat uncounted, because the first question of each
+pass was "what is left of what I was doing" rather than "what is biggest now".
+This answers the second one.
+
+TWO THINGS IT INSISTS ON, both learned the hard way:
+
+  * `-ferror-limit=0`. Clang stops at twenty errors per file by default and
+    141 corpus files reach it, so every total measured without this is a
+    FLOOR -- 3849 reported where the real count was 15188 (#3448).
+  * the compiler's identity, printed beside the numbers. The same command
+    gives 20 on Apple clang and 50 on gcc for the same input; a diagnostic
+    count measures the code, the instrument AND the machine (#3450).
+
+It is a READER, not a gate, and deliberately not wired into CI: it rebuilds the
+whole corpus, which is minutes, and nothing here is a pass/fail claim. Run it
+at the start of a pass to decide what the pass is about.
+
+Usage:
+  tools/c_error_classes.py                ranked classes
+  tools/c_error_classes.py --undeclared   split the undeclared symbols by cause
+  tools/c_error_classes.py --self-check   negative control
+
+Exit codes:
+  0  the report printed
+  2  COULD NOT RUN (no t27c, no cc, no specs)
+"""
+
+import collections
+import os
+import re
+import subprocess
+import sys
+import tempfile
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def t27c() -> str:
+    for rel in ("target/release/t27c", "bootstrap/target/release/t27c"):
+        p = os.path.join(ROOT, rel)
+        if os.path.isfile(p) and os.access(p, os.X_OK):
+            return p
+    env = os.environ.get("TRI_T27C", "")
+    if env and os.access(env, os.X_OK):
+        return env
+    print("c_error_classes: t27c not built. Exit 2 = COULD NOT RUN.", file=sys.stderr)
+    print("  cargo build --release -p t27c, or set TRI_T27C.", file=sys.stderr)
+    sys.exit(2)
+
+
+def cc_uncap_flag() -> str:
+    """The flag that ACTUALLY removes the cap, per vendor -- gcc rejects
+    `-ferror-limit` and clang ACCEPTS `-fmax-errors=0` while ignoring it."""
+    v = subprocess.run(["cc", "--version"], capture_output=True, text=True).stdout
+    return "-ferror-limit=0" if "clang" in v.lower() else "-fmax-errors=0"
+
+
+def diagnose(binary: str, out_dir: str) -> str:
+    specs = sorted(
+        os.path.join(r, f)
+        for r, _, fs in os.walk(os.path.join(ROOT, "specs"))
+        for f in fs
+        if f.endswith(".t27")
+    )
+    if not specs:
+        print("c_error_classes: no specs. Exit 2.", file=sys.stderr)
+        sys.exit(2)
+    flag = cc_uncap_flag()
+    chunks = []
+    for s in specs:
+        h = os.path.join(out_dir, os.path.relpath(s, ROOT).replace("/", "_") + ".h")
+        with open(h, "w") as fh:
+            subprocess.run([binary, "gen-c", s], stdout=fh, stderr=subprocess.DEVNULL)
+        if os.path.getsize(h) == 0:
+            continue
+        r = subprocess.run(
+            ["cc", "-std=c11", "-Wall", "-Wextra", "-Wno-unused-parameter",
+             flag, "-fsyntax-only", "-x", "c", h],
+            capture_output=True, text=True,
+        )
+        chunks.append(f"### {os.path.basename(h)}\n" + r.stdout + r.stderr)
+    return "\n".join(chunks)
+
+
+def undeclared_split(text: str) -> None:
+    """Why is each undeclared function undeclared? The causes need different
+    repairs, and the totals do not separate them."""
+    names = collections.Counter(re.findall(r"call to undeclared function '(\w+)'", text))
+    fns, mods, files = set(), set(), set()
+    for r, _, fs in os.walk(os.path.join(ROOT, "specs")):
+        for f in fs:
+            if not f.endswith(".t27"):
+                continue
+            files.add(f[:-4])
+            for line in open(os.path.join(r, f), encoding="utf-8", errors="replace"):
+                m = re.match(r"\s*(?:pub\s+)?fn\s+(\w+)\s*\(", line)
+                if m:
+                    fns.add(m.group(1))
+                m = re.match(r"\s*module\s+([\w\-]+)", line)
+                if m:
+                    mods.add(m.group(1).replace("-", "_"))
+    cats, ex = collections.Counter(), collections.defaultdict(list)
+    for n, c in names.items():
+        if n in fns:
+            k = "A  declared as `fn` in some spec, not reaching this file"
+        elif n in files or n in mods:
+            k = "B  the name of a MODULE or spec FILE, called as a function"
+        else:
+            k = "C  a name nothing in the tree declares"
+        cats[k] += c
+        ex[k].append((n, c))
+    total = sum(names.values()) or 1
+    print(f"\nundeclared functions: {total} errors across {len(names)} names")
+    for k in sorted(cats):
+        print(f"  {cats[k]:5} ({100 * cats[k] // total:2}%)  {k}")
+        print("        " + ", ".join(f"{n}({v})" for n, v in sorted(ex[k], key=lambda t: -t[1])[:6]))
+
+
+def self_check(binary: str) -> int:
+    """A planted error must be counted, and a clean file must count zero.
+    Without both, a report of "0" and a report that could not run look alike."""
+    ok = True
+    with tempfile.TemporaryDirectory() as d:
+        bad = os.path.join(d, "bad.h")
+        open(bad, "w").write("int f(void){ return undefined_zzz; }\n")
+        r = subprocess.run(
+            ["cc", "-std=c11", cc_uncap_flag(), "-fsyntax-only", "-x", "c", bad],
+            capture_output=True, text=True,
+        )
+        n = len(re.findall(r"error:", r.stdout + r.stderr))
+        print(f"  planted 1 error -> counted {n} {'PASS' if n >= 1 else 'FAIL'}")
+        ok &= n >= 1
+        good = os.path.join(d, "good.h")
+        open(good, "w").write("int f(void){ return 0; }\n")
+        r = subprocess.run(
+            ["cc", "-std=c11", cc_uncap_flag(), "-fsyntax-only", "-x", "c", good],
+            capture_output=True, text=True,
+        )
+        n = len(re.findall(r"error:", r.stdout + r.stderr))
+        print(f"  clean file      -> counted {n} {'PASS' if n == 0 else 'FAIL'}")
+        ok &= n == 0
+    return 0 if ok else 2
+
+
+def main() -> int:
+    binary = t27c()
+    if subprocess.run(["cc", "--version"], capture_output=True).returncode != 0:
+        print("c_error_classes: no cc. Exit 2 = COULD NOT RUN.", file=sys.stderr)
+        return 2
+    if "--self-check" in sys.argv:
+        return self_check(binary)
+    with tempfile.TemporaryDirectory() as d:
+        text = diagnose(binary, d)
+    v = subprocess.run(["cc", "--version"], capture_output=True, text=True).stdout
+    print(f"instrument: {v.strip().splitlines()[0][:64]}  {cc_uncap_flag()}")
+    files = text.count("### ")
+    errors = len(re.findall(r"error:", text))
+    clean = sum(1 for c in text.split("### ")[1:] if "error:" not in c)
+    print(f"translation units {files}   errors {errors}   compiling {clean}\n")
+    classes = collections.Counter(
+        re.sub(r"\d+", "N", m) for m in re.findall(r"error: ([^[\n]*)", text)
+    )
+    for msg, n in classes.most_common(12):
+        print(f"  {n:5}  {msg.strip()[:70]}")
+    if "--undeclared" in sys.argv:
+        undeclared_split(text)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())