diff --git a/.claude/skills/ci-gates/SKILL.md b/.claude/skills/ci-gates/SKILL.md index d7471ddef6..57251c1ec2 100644 --- a/.claude/skills/ci-gates/SKILL.md +++ b/.claude/skills/ci-gates/SKILL.md @@ -11330,3 +11330,60 @@ seal files** -- specs here are twinned, several `.trinity/seals/*.json` per spec the same commit" is already written down three times on this page; what was missing this time was not the rule but the arithmetic that connects a delta in one unit to a gate that counts in another. + +## 444. Three readings of one question, and only the third measured the gate + +The question: **which gates report success over a tree with nothing in it?** +Three attempts, three answers, and the first two were artefacts of the +instrument. + +**34 of 34 refuse.** Every gate copied into an empty directory and run under +`timeout 25`. Clean, plausible, and entirely false: **`timeout` does not exist +on macOS**, so all 34 exited 127 with `command not found`, and the classifier +put every one of them in the `refused` bucket. The tell was the table itself -- +*a classifier that puts 34 of 34 in one bucket is describing its input*, which +is a rule already on this page and is what made me look. The output had been +discarded; re-running to print it took ten seconds and showed 34 identical +shell errors. + +**12 of 38 pass.** Real runs this time, by script name. Also wrong, and in the +more interesting way: CI does not invoke a script, it invokes a **command +line**. Seven of those twelve are written `--require` in the workflow, which is +precisely the flag that turns their `SKIP: t27c is not built` branch into a +failure. Measuring `tools/check_verilog_widths.py` when the workflow writes +`tools/check_verilog_widths.py --require` measures a program that is not run +anywhere. + +**5 of 36.** The invocations as the workflows write them. Four of the five are +self-contained self-tests, green anywhere by construction; the fifth prints +`tracked files read 0` before it says the tree is clean, which is a gate +telling a reader it read nothing. **A clean audit is a result**, and after +three passes that each found a defect it is worth saying plainly rather than +hunting until something breaks. + +**A gate is what it is called with.** When the population is "the gates", the +unit is the invocation, not the file -- the same distinction as an issue's +headline versus the issue, one layer down. + +## 445. An empty tree that carries the data is not an empty tree + +The command shipped for the above copies the scripts into a scratch directory +and runs them there. Its first version copied **every** file out of `tools/` and +`scripts/`, which carried `tools/withdrawn.txt` and every baseline along with +them. The result was not an error -- it was a plausible table with two rows +inverted: `check_withdrawn_live.py`, whose whole job is to refuse when its +register is missing, found the register and passed; `check_conflict_markers.py` +read as a refusal. + +It was caught by **running the two by hand and disagreeing with my own +command** -- the free control this page keeps naming, used deliberately for +once rather than by accident. Two readings of one question, and the difference +is in the observer. + +The repair is a filter, and the test is the filter's own mutation: plant +`gate.py`, `helper.sh`, `withdrawn.txt` and `baseline.json`, copy, and demand +exactly the two scripts arrive. Removing the extension check turns it red. + +**The general shape:** when a probe builds a world to run something in, the +world is a component with its own defects, and the cheapest check on it is to +ask what it contains rather than what you meant to put there. diff --git a/cli/tri/src/gates.rs b/cli/tri/src/gates.rs index 4aeb3609eb..661f2e4bed 100644 --- a/cli/tri/src/gates.rs +++ b/cli/tri/src/gates.rs @@ -174,6 +174,12 @@ pub enum GatesCmd { #[arg(long, default_value = "origin/master")] base: String, }, + /// Run every gate CI runs, in an EMPTY tree, and report what still passes. + Empty { + /// Print what each passing invocation printed. + #[arg(long)] + verbose: bool, + }, Required { /// owner/repo. Defaults to the repository of the working directory. #[arg(long)] @@ -2297,6 +2303,7 @@ pub fn run(cmd: &GatesCmd) -> Result<()> { }; unmeasured(&list, *stale_days) } + GatesCmd::Empty { verbose } => empty(*verbose), GatesCmd::Preview { base } => preview(base), GatesCmd::Required { repo } => required(repo.as_deref()), GatesCmd::Dead { repos, min_runs } => { @@ -4654,3 +4661,316 @@ mod preview_tests { assert!(!Reading::Unavailable.is_pass()); } } + +// --------------------------------------------------------------------------- +// `tri gates empty` -- what still passes when there is nothing to check. +// --------------------------------------------------------------------------- + +/// One invocation this repository's CI writes, or a reason it was left out. +#[derive(Debug, PartialEq, Eq)] +pub enum Invocation { + /// A complete command line, runnable as written. + Run(String), + /// Found and deliberately not run: `(the text, why)`. + Skipped(String, String), +} + +/// Every `python3 …` / `bash …` gate invocation a workflow writes, verbatim. +/// +/// **The population is the COMMAND LINE, not the script.** Measuring by script +/// name gave 12 of 38 passing in an empty tree; measuring the lines CI actually +/// writes gave **5 of 36**, because seven of those twelve are invoked with +/// `--require`, which is precisely the flag that turns their SKIP branch into a +/// failure. A gate is what it is called with. +/// +/// Exclusions are RETURNED rather than dropped: a line continued with `\`, a +/// line inside a `$( … )` substitution, and a `--self-check` or `--self-test` +/// run (whose subject is the gate itself, not the tree) cannot be reproduced +/// here as written, and a population that silently shrinks is the defect this +/// command exists to look for. +pub fn ci_invocations(yaml: &str) -> Vec { + let mut out = Vec::new(); + for raw in yaml.lines() { + let line = raw.trim(); + let Some(start) = find_invocation(line) else { + continue; + }; + let cmd = line[start..].trim().to_string(); + if cmd.ends_with('\\') { + out.push(Invocation::Skipped( + cmd, + "continued onto the next line".into(), + )); + } else if cmd.contains(')') { + out.push(Invocation::Skipped( + cmd, + "inside a command substitution".into(), + )); + } else if cmd.contains("--self-check") || cmd.contains("--self-test") { + out.push(Invocation::Skipped( + cmd, + "its subject is the gate itself, not the tree".into(), + )); + } else { + out.push(Invocation::Run(cmd)); + } + } + out.sort_by(|a, b| text_of(a).cmp(text_of(b))); + out.dedup_by(|a, b| text_of(a) == text_of(b)); + out +} + +fn text_of(i: &Invocation) -> &str { + match i { + Invocation::Run(s) => s, + Invocation::Skipped(s, _) => s, + } +} + +/// Where a `python3 tools/…` / `bash scripts/…` command starts on a line. +fn find_invocation(line: &str) -> Option { + for lead in ["python3 ", "python ", "bash "] { + let mut from = 0usize; + while let Some(rel) = line[from..].find(lead) { + let i = from + rel; + let rest = &line[i + lead.len()..]; + if rest.starts_with("tools/") || rest.starts_with("scripts/") { + return Some(i); + } + from = i + 1; + } + } + None +} + +fn empty(verbose: bool) -> Result<()> { + let root = repo_root()?; + let wf = root.join(".github/workflows"); + let mut found: Vec = Vec::new(); + for e in std::fs::read_dir(&wf) + .with_context(|| format!("{}: cannot read the workflow directory", wf.display()))? + .flatten() + { + let p = e.path(); + if p.extension().and_then(|x| x.to_str()) != Some("yml") { + continue; + } + if let Ok(text) = std::fs::read_to_string(&p) { + found.extend(ci_invocations(&text)); + } + } + found.sort_by(|a, b| text_of(a).cmp(text_of(b))); + found.dedup_by(|a, b| text_of(a) == text_of(b)); + if found.is_empty() { + anyhow::bail!( + "no gate invocation found in {}. A population of zero here would print \ + as \"every gate refuses an empty tree\", which is the shape this \ + command was written to catch.", + wf.display() + ); + } + + // An empty tree with the SCRIPTS present and no data: every precondition a + // gate has -- a baseline, a built compiler, specs, seals -- is absent at + // once. Copied rather than aimed with a flag, so nothing here adds a way to + // point a live gate somewhere harmless. + let tmp = std::env::temp_dir().join(format!("tri_gates_empty_{}", std::process::id())); + let _ = std::fs::remove_dir_all(&tmp); + std::fs::create_dir_all(tmp.join("tools"))?; + std::fs::create_dir_all(tmp.join("scripts/ci"))?; + copy_dir(&root.join("tools"), &tmp.join("tools"))?; + copy_dir(&root.join("scripts"), &tmp.join("scripts"))?; + copy_dir(&root.join("scripts/ci"), &tmp.join("scripts/ci"))?; + let _ = std::process::Command::new("git") + .args(["init", "-q"]) + .current_dir(&tmp) + .status(); + + let mut passed: Vec<(String, String)> = Vec::new(); + let mut refused = 0usize; + for inv in &found { + let Invocation::Run(cmd) = inv else { continue }; + let parts: Vec<&str> = cmd.split_whitespace().collect(); + let out = std::process::Command::new(parts[0]) + .args(&parts[1..]) + .current_dir(&tmp) + .stdin(std::process::Stdio::null()) + .output(); + match out { + Ok(o) if o.status.success() => { + let text = String::from_utf8_lossy(&o.stdout).to_string(); + passed.push((cmd.clone(), text)); + } + Ok(_) => refused += 1, + Err(_) => refused += 1, + } + } + let _ = std::fs::remove_dir_all(&tmp); + + let runnable = found + .iter() + .filter(|i| matches!(i, Invocation::Run(_))) + .count(); + println!("EVERY GATE CI RUNS, AGAINST A TREE WITH NOTHING IN IT\n"); + println!(" invocations found in .github/workflows {}", found.len()); + println!(" run here {runnable}"); + println!(" refused the empty tree {refused}"); + println!( + " PASSED over nothing {}", + passed.len() + ); + for (cmd, text) in &passed { + println!(" {cmd}"); + if verbose { + for l in text.lines().take(3) { + println!(" {l}"); + } + } + } + let skipped: Vec<&Invocation> = found + .iter() + .filter(|i| matches!(i, Invocation::Skipped(_, _))) + .collect(); + if !skipped.is_empty() { + println!( + "\n found and NOT run ({}), with the reason:", + skipped.len() + ); + for i in skipped { + if let Invocation::Skipped(c, why) = i { + println!(" {c}\n {why}"); + } + } + } + println!( + "\n A pass here is not automatically a defect: a self-contained self-test\n \ + is green anywhere, and a gate that prints the size of what it walked --\n \ + `tracked files read 0` -- has told the reader it read nothing. What this\n \ + looks for is the third kind: green, silent, and having checked nothing.\n\n \ + The population is the COMMAND LINE, not the script. By script name this\n \ + reads 12 of 38; by the lines CI actually writes it reads {} of {runnable},\n \ + because `--require` is what turns a SKIP branch into a failure and seven\n \ + gates carry it. A gate is what it is called with.", + passed.len() + ); + Ok(()) +} + +/// Copy the SCRIPTS out of a directory and nothing else. +/// +/// The first version copied every file, which put `tools/withdrawn.txt` and +/// every baseline into the "empty" tree -- so a gate that refuses a missing +/// register found it, and `check_withdrawn_live.py` read as a pass while +/// `check_conflict_markers.py` read as a refusal. Both were backwards, and it +/// was caught by running the two by hand and disagreeing with my own command. +/// **An empty tree that carries the data is not an empty tree**, and the +/// symptom was a plausible table, not an error. +fn copy_dir(from: &std::path::Path, to: &std::path::Path) -> Result<()> { + let Ok(rd) = std::fs::read_dir(from) else { + return Ok(()); + }; + for e in rd.flatten() { + let p = e.path(); + let is_script = matches!( + p.extension().and_then(|x| x.to_str()), + Some("py") | Some("sh") + ); + if p.is_file() && is_script { + let _ = std::fs::copy(&p, to.join(p.file_name().unwrap())); + } + } + Ok(()) +} + +#[cfg(test)] +mod empty_tests { + use super::*; + + fn run_texts(y: &str) -> Vec { + ci_invocations(y) + .into_iter() + .filter_map(|i| match i { + Invocation::Run(s) => Some(s), + _ => None, + }) + .collect() + } + fn skipped(y: &str) -> Vec<(String, String)> { + ci_invocations(y) + .into_iter() + .filter_map(|i| match i { + Invocation::Skipped(s, w) => Some((s, w)), + _ => None, + }) + .collect() + } + + /// The whole finding in one assertion: measuring by SCRIPT gave 12 of 38 + /// passing an empty tree, measuring by the line CI writes gave 5 of 36, + /// and the difference is `--require`. + #[test] + fn the_arguments_are_part_of_the_invocation() { + let y = " - run: python3 tools/check_verilog_widths.py --require\n"; + assert_eq!( + run_texts(y), + vec!["python3 tools/check_verilog_widths.py --require"] + ); + } + + #[test] + fn what_cannot_be_reproduced_is_named_rather_than_dropped() { + let y = " - run: python3 scripts/ci/test_ratchet.py \\\n"; + let s = skipped(y); + assert_eq!(s.len(), 1, "a continued line must be reported, not dropped"); + assert!(s[0].1.contains("continued")); + assert!(run_texts(y).is_empty()); + + let y = " M=$(python3 scripts/ci/rings_matrix.py)\n"; + let s = skipped(y); + assert_eq!(s.len(), 1); + assert!(s[0].1.contains("substitution")); + + let y = " - run: python3 tools/check_json_parses.py --self-check\n"; + let s = skipped(y); + assert_eq!(s.len(), 1); + assert!(s[0].1.contains("gate itself")); + } + + #[test] + fn only_this_repositorys_gate_paths_are_a_population() { + // A python call that is not a gate in this tree is not an invocation. + assert!(run_texts(" - run: python3 setup.py build\n").is_empty()); + assert!(run_texts(" - run: python3 -m pip install x\n").is_empty()); + // And one that is, however it is indented or prefixed. + assert_eq!( + run_texts(" run: bash scripts/ci/loop-tools-tracked.sh\n").len(), + 1 + ); + } + + /// An empty tree that carries the data is not an empty tree. The first + /// version copied every file, and two gates read backwards because of it. + #[test] + fn only_scripts_are_carried_into_the_empty_tree() { + let from = std::env::temp_dir().join(format!("tri_copy_from_{}", std::process::id())); + let to = std::env::temp_dir().join(format!("tri_copy_to_{}", std::process::id())); + let _ = std::fs::remove_dir_all(&from); + let _ = std::fs::remove_dir_all(&to); + std::fs::create_dir_all(&from).unwrap(); + std::fs::create_dir_all(&to).unwrap(); + std::fs::write(from.join("gate.py"), "x").unwrap(); + std::fs::write(from.join("helper.sh"), "x").unwrap(); + std::fs::write(from.join("withdrawn.txt"), "x").unwrap(); + std::fs::write(from.join("baseline.json"), "x").unwrap(); + copy_dir(&from, &to).unwrap(); + let mut got: Vec = std::fs::read_dir(&to) + .unwrap() + .flatten() + .map(|e| e.file_name().to_string_lossy().to_string()) + .collect(); + got.sort(); + let _ = std::fs::remove_dir_all(&from); + let _ = std::fs::remove_dir_all(&to); + assert_eq!(got, vec!["gate.py".to_string(), "helper.sh".to_string()]); + } +} diff --git a/docs/now/2026-09-03-every-gate-ci-runs-against-a-tree-with-nothing-in-it.md b/docs/now/2026-09-03-every-gate-ci-runs-against-a-tree-with-nothing-in-it.md new file mode 100644 index 0000000000..769237423c --- /dev/null +++ b/docs/now/2026-09-03-every-gate-ci-runs-against-a-tree-with-nothing-in-it.md @@ -0,0 +1,7 @@ +# NOW -- Every gate CI runs, against a tree with nothing in it (2026-09-03) + +## Every gate CI runs, against a tree with nothing in it (Refs #2994) + +- tri gates empty extracts the exact command lines the workflows write, runs each in a tree holding the scripts and no data, and reports what still exits 0. +- Of 36 runnable invocations, 31 refuse the empty tree and 5 pass -- four are self-contained self-tests and the fifth prints 'tracked files read 0'. This axis is healthy, and a clean audit is a result. +- The population is the COMMAND LINE, not the script: by script name it reads 12 of 38, because seven gates are invoked with --require, which is what turns their SKIP branch into a failure.