From c3a89043b69edaab4259f9ecbafd69bbf7ab14f0 Mon Sep 17 00:00:00 2001 From: Vasilev Dmitrii Date: Mon, 31 Aug 2026 02:52:41 +0700 Subject: [PATCH] feat(tri): read the one subset of open issues that re-measures itself Refs #2983 An open issue is a claim with a date on it. This repository holds 477, and 268 carry a number in the title -- a stated measurement of the tree, taken once, re-read by nothing. Most cannot be checked automatically: extracting the measurement from free prose is the precision problem that killed a superlative detector one pass ago (2 real hits in 6). One subset can, because its truth is stored OUTSIDE the repository -- an issue whose title says a named workflow is red, against what that workflow last concluded on master. open issues read 477 workflow files 49 titles claiming red 9 of those, every named workflow is green today 5 attributed from the body, not the title 4 #2851 Seal Coverage, #2754 secret-scan + cli-tri, #2729 cli-tri, #2278 emit-bitexact, #2263 cli-tri + emit-bitexact -- all green today. #2915, #2914, #2783 and #2292 are still correctly red and left alone. WHAT A GREEN READING DOES NOT SAY. It does not say the issue is resolved, and the command never suggests closing one. #2729 is the case that makes the distinction concrete: its title is stale, and its argument is not -- the workflow still carries the `paths:` filter the issue was actually about, which is the same shape as #2903. The headline being false and the issue being resolved are different facts, and a tool that conflated them would close live work. What a stale headline costs is real anyway: with 477 open issues, a reader stops at the title. THE RECALL BUG IT WAS BUILT AROUND. Issues name a workflow the way GitHub DISPLAYS it. `seal-coverage.yml` displays as `Seal Coverage`, so a reader keyed on the file stem cannot see #2851 at all. Measured: 22 of 49 workflows display under a name that differs from their stem, and adding that key moved the population from 8 to 9 -- the one it added being the example I already had the answer for. Without that example, "no issue names this workflow" and "my reader cannot see how issues name workflows" print the same number. The boundary treats `-` and `_` as word characters, so `cli-tri` does not match inside `cli-tri-mcp`. Attribution is split: the claim lives in the TITLE, so the title's workflows carry it; falling back to the body pulled in a workflow red for its own reasons and marked two stale headlines as still red. Rows attributed from the body say so. Five tests, including the display-name case, the boundary case, and a refusal on an empty workflow directory -- because "no issue names a workflow" and "this command could not read any workflow" must not print the same zero. Skill 416-418. The third is self-inflicted: an unquoted heredoc ran every backticked span in the first draft as a command and left four holes in the prose, which is written down here already from a commit message that lost four words the same way. Co-Authored-By: Claude Opus 5 --- .claude/skills/ci-gates/SKILL.md | 57 +++ cli/tri/src/issues.rs | 452 ++++++++++++++++++ cli/tri/src/main.rs | 8 + ...a-workflow-is-red-that-workflow-is-gree.md | 9 + 4 files changed, 526 insertions(+) create mode 100644 cli/tri/src/issues.rs create mode 100644 docs/now/2026-08-31-five-open-issues-say-a-workflow-is-red-that-workflow-is-gree.md diff --git a/.claude/skills/ci-gates/SKILL.md b/.claude/skills/ci-gates/SKILL.md index 000f99690..19eff7428 100644 --- a/.claude/skills/ci-gates/SKILL.md +++ b/.claude/skills/ci-gates/SKILL.md @@ -10489,3 +10489,60 @@ was hunting -- was not among them. Always run a new detector against the example that made you write it, and confirm that example is in the output. Without that check, "found nothing here" and "cannot see this shape at all" print identically. + + +## 416. An open issue is a claim with a date, and one subset re-measures itself + +477 open issues here, **268** with a number in the title -- each a stated measurement of +the tree, taken once, re-read by nothing. Most cannot be checked automatically: pulling +the measurement out of free prose is the precision problem that killed a detector one +pass earlier. + +One subset can, because its truth is stored **outside** the repository. An issue whose +title says a named workflow is red is checkable against what that workflow last concluded +on master. `tri issues stale` walks it: 9 titles claim red, **5** name a workflow that is +green today. + +The command states what a green reading does NOT establish, and never suggests closing +anything. #2729 is the case that makes it concrete: its title -- "cli-tri has been red on +master for three days" -- is stale, and its argument is not, because the workflow still +carries the `paths:` filter the issue was actually about. **The headline being false and +the issue being resolved are different facts**, and a tool that conflates them would close +live work. + +What a stale headline costs is real anyway: with 477 open issues, a reader stops at the +title. + +## 417. Match a workflow by the name it DISPLAYS under, not by its file + +Issues name a workflow the way GitHub shows it. `seal-coverage.yml` displays as +`Seal Coverage`, so #2851 -- "Seal Coverage has been red on master" -- is invisible to a +reader keyed on the file stem. Measured here: **22 of 49** workflows display under a name +that differs from their stem. + +Adding the display name as a third key took the population from **8 to 9**, and the one it +added was the issue I already knew about. That is the check worth copying: **run the new +reader against the example you already have the answer for, and confirm it is in the +output.** I had that example by accident. Without it, "no issue names this workflow" and +"my reader cannot see how issues name workflows" print the same number. + +The boundary matters too, and hyphens make it non-obvious: a workflow name is +alphanumeric plus `-` and `_`, so `cli-tri` must not match inside `cli-tri-mcp` -- two +different things in this repository, and the subject of #2903. Both properties are tests +rather than comments. + +## 418. An unquoted heredoc eats your backticks, again + +Writing the two sections above, `python3 - <` and a rewrite, and it cost nothing because the file was +not yet committed. Had it been, the sections would have shipped with the holes and read as +sloppy prose rather than as a shell bug. diff --git a/cli/tri/src/issues.rs b/cli/tri/src/issues.rs new file mode 100644 index 000000000..0cf380e4c --- /dev/null +++ b/cli/tri/src/issues.rs @@ -0,0 +1,452 @@ +//! Open issues whose headline calls a workflow red, against what that workflow +//! does on master today. +//! +//! An open issue is a claim with a date on it: *this is broken*. The repository +//! holds 477 of them, and 268 carry a number in the title -- a stated +//! measurement of the tree, taken once. Nothing re-reads them. +//! +//! One subset can be re-measured exactly, because its truth is defined outside +//! the repository: an issue whose title says a named workflow is red. GitHub +//! records what that workflow last concluded on master, so the claim can be +//! checked without judgement. +//! +//! **What a green reading here does NOT mean.** It does not say the issue is +//! resolved, and this command never suggests closing one. An issue titled +//! "cli-tri has been red on master for three days" also argues that a `paths:` +//! filter kept it from running, and that argument can outlive the redness +//! entirely. What a green reading says is narrower and still worth having: the +//! sentence at the top of that issue is no longer true, so a reader who stops +//! at the title is misled about the state of the tree. + +use anyhow::{Context, Result}; +use std::collections::BTreeMap; +use std::path::Path; +use std::process::Command; + +#[derive(Debug, clap::Subcommand)] +pub enum IssuesCmd { + /// Open issues whose title calls a workflow red, and what it does today. + Stale { + /// How many open issues to read. The default is the repository's + /// current open count rounded up; a smaller number is a SAMPLE and the + /// output says so rather than presenting it as a census. + #[arg(long, default_value_t = 500)] + limit: usize, + }, +} + +/// The phrases a title uses to call something red. +/// +/// Matched against the TITLE only. A body may mention redness in passing -- +/// recounting history, or describing what a fix prevented -- while the issue is +/// about something else entirely; the title is where the claim is made. +pub fn claims_red(title: &str) -> bool { + let t = title.to_lowercase(); + const PHRASES: [&str; 7] = [ + "red on master", + "cannot succeed", + "never once succeeded", + "never succeeded", + "landed red", + "suite is red", + "is red and", + ]; + if PHRASES.iter().any(|p| t.contains(p)) { + return true; + } + // "failed 62 of 62 runs", "failed 8 of 8" + if let Some(at) = t.find("failed ") { + let rest = &t[at + 7..]; + if rest.starts_with(|c: char| c.is_ascii_digit()) && rest.contains(" of ") { + return true; + } + } + // "has been red for three days", "red for 3 days" + t.contains("red for ") && t.contains("day") +} + +/// Every name a workflow answers to, mapped back to its file. +/// +/// Three keys per workflow, and the third is the one that matters: issues name +/// a workflow the way GitHub DISPLAYS it. `seal-coverage.yml` displays as +/// "Seal Coverage", and #2851 -- "Seal Coverage has been red on master" -- is +/// invisible to a reader keyed on the file stem alone. Measured on this tree: +/// 22 of 49 workflows have a `name:` that differs from their stem, and adding +/// the display name took this command's population from 8 to 9. +pub fn workflow_keys(dir: &Path) -> Result> { + let mut keys = BTreeMap::new(); + let rd = std::fs::read_dir(dir) + .with_context(|| format!("{}: cannot read the workflow directory", dir.display()))?; + let mut files = 0usize; + for e in rd.flatten() { + let p = e.path(); + if p.extension().and_then(|x| x.to_str()) != Some("yml") { + continue; + } + let file = match p.file_name().and_then(|x| x.to_str()) { + Some(f) => f.to_string(), + None => continue, + }; + files += 1; + let stem = file.trim_end_matches(".yml").to_string(); + keys.insert(file.clone(), file.clone()); + keys.insert(stem, file.clone()); + if let Ok(text) = std::fs::read_to_string(&p) { + for line in text.lines() { + if let Some(rest) = line.strip_prefix("name:") { + let disp = rest.trim().trim_matches(['"', '\'']).to_string(); + if !disp.is_empty() { + keys.insert(disp, file.clone()); + } + break; + } + } + } + } + if files == 0 { + anyhow::bail!( + "{}: no *.yml found. A zero here would read as \"no issue names a \ + workflow\", which is a different statement.", + dir.display() + ); + } + Ok(keys) +} + +/// Is `key` present in `hay` as a whole token? +/// +/// Workflow names contain hyphens and spaces, so the boundary cannot be the +/// usual word boundary: `cli-tri` must not match inside `cli-tri-mcp`, and +/// `release` must not match inside `pre-release`. +fn names(hay: &str, key: &str) -> bool { + let bound = |c: char| !(c.is_alphanumeric() || c == '-' || c == '_'); + let mut from = 0usize; + while let Some(rel) = hay[from..].find(key) { + let i = from + rel; + let j = i + key.len(); + let before = hay[..i].chars().next_back().is_none_or(bound); + let after = hay[j..].chars().next().is_none_or(bound); + if before && after { + return true; + } + from = i + 1; + } + false +} + +/// The workflows an issue names, by file. +pub fn named_workflows(text: &str, keys: &BTreeMap) -> Vec { + let mut out: Vec = keys + .iter() + .filter(|(k, _)| names(text, k)) + .map(|(_, v)| v.clone()) + .collect(); + out.sort(); + out.dedup(); + out +} + +fn gh(args: &[&str]) -> Result { + let out = Command::new("gh") + .args(args) + .output() + .context("gh is not installed or not on PATH")?; + if !out.status.success() { + anyhow::bail!( + "gh {:?} failed: {}", + args, + String::from_utf8_lossy(&out.stderr).trim() + ); + } + Ok(String::from_utf8_lossy(&out.stdout).to_string()) +} + +fn repo_root() -> Result { + let out = Command::new("git") + .args(["rev-parse", "--show-toplevel"]) + .output()?; + if !out.status.success() { + anyhow::bail!("not inside a git repository"); + } + Ok(std::path::PathBuf::from( + String::from_utf8(out.stdout)?.trim(), + )) +} + +/// One issue and the verdict of every workflow it names. +struct Row { + number: u64, + title: String, + /// `(workflow file, conclusion on master)`. + verdicts: Vec<(String, String)>, + /// True when the workflows above are the ones the TITLE names. + /// + /// The claim is made in the title, so the title's workflows are the ones + /// that carry it. Falling back to the body finds more, and attributes + /// worse: #2292's title says "cargo test -p t27c landed red on master" and + /// names no workflow at all, while its body mentions `release` -- which is + /// red for its own reasons and has nothing to do with the sentence. Reading + /// those together marked a stale headline as still red. + from_title: bool, +} + +pub fn run(cmd: &IssuesCmd) -> Result<()> { + let IssuesCmd::Stale { limit } = cmd; + let root = repo_root()?; + let keys = workflow_keys(&root.join(".github/workflows"))?; + let files: std::collections::BTreeSet<&String> = keys.values().collect(); + + let lim = limit.to_string(); + let raw = gh(&[ + "issue", + "list", + "--state", + "open", + "--limit", + &lim, + "--json", + "number,title,body", + ])?; + let issues: serde_json::Value = serde_json::from_str(&raw).context("gh returned no JSON")?; + let issues = issues.as_array().cloned().unwrap_or_default(); + if issues.is_empty() { + anyhow::bail!( + "gh returned no open issues. That is either a repository with none \ + or a query that did not run, and the two print the same zero." + ); + } + + let mut rows = Vec::new(); + for it in &issues { + let title = it["title"].as_str().unwrap_or(""); + if !claims_red(title) { + continue; + } + let body = it["body"].as_str().unwrap_or(""); + let from_title = named_workflows(title, &keys); + let (named, in_title) = if from_title.is_empty() { + (named_workflows(&format!("{title}\n{body}"), &keys), false) + } else { + (from_title, true) + }; + if named.is_empty() { + continue; + } + let mut verdicts = Vec::new(); + for wf in named { + let out = gh(&[ + "run", + "list", + "--workflow", + &wf, + "--branch", + "master", + "-L", + "1", + "--json", + "conclusion", + "--jq", + ".[0].conclusion // \"no master run\"", + ]) + .unwrap_or_else(|_| "unreadable".into()); + verdicts.push((wf, out.trim().to_string())); + } + rows.push(Row { + number: it["number"].as_u64().unwrap_or(0), + title: title.to_string(), + verdicts, + from_title: in_title, + }); + } + + println!("OPEN ISSUES THAT CALL A WORKFLOW RED, AND WHAT IT DOES ON MASTER TODAY\n"); + println!(" open issues read {}", issues.len()); + println!(" workflow files {}", files.len()); + println!(" titles claiming red {}", rows.len()); + + let all_green = + |r: &Row| !r.verdicts.is_empty() && r.verdicts.iter().all(|(_, c)| c == "success"); + let stale: Vec<&Row> = rows.iter().filter(|r| all_green(r)).collect(); + let by_body = rows.iter().filter(|r| !r.from_title).count(); + println!( + " of those, every named workflow is green today {}", + stale.len() + ); + println!( + " attributed from the body, not the title {}\n", + by_body + ); + + for r in &rows { + let mark = if all_green(r) { + "HEADLINE STALE" + } else { + "still red" + }; + let src = if r.from_title { + "" + } else { + " (workflow named in the body, not the title)" + }; + println!(" #{} {mark}{src}", r.number); + println!(" {}", &r.title[..r.title.len().min(96)]); + for (wf, c) in &r.verdicts { + println!(" {:<28} master: {}", wf.trim_end_matches(".yml"), c); + } + println!(); + } + + println!( + " A green reading does NOT mean the issue is resolved, and this command\n \ + never suggests closing one. \"cli-tri has been red on master for three\n \ + days\" also argues that a `paths:` filter kept it from running, and that\n \ + argument outlives the redness. What it says is narrower: the sentence at\n \ + the top of that issue is no longer true, so a reader who stops at the\n \ + title is misled about the tree.\n\n \ + A workflow is matched by its FILE NAME, its stem, and the `name:` it\n \ + displays under -- because issues use the display name. On this tree 22 of\n \ + 49 workflows display under a different name than their file, and adding\n \ + that key moved this population from 8 to 9. The one it added is #2851,\n \ + \"Seal Coverage has been red on master\"." + ); + if issues.len() >= *limit { + println!( + "\n NOTE: gh returned {} issues at a limit of {}. This is a SAMPLE, not\n \ + a census -- raise --limit past the open count before quoting a total.", + issues.len(), + limit + ); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// The nine real titles this command was built from, and what it must do + /// with each. The last three are the counter-examples: a title that reports + /// a REPAIR, one that mentions a workflow with no redness claim, and one + /// whose redness is in the body rather than the claim. + #[test] + fn the_matcher_reads_the_claim_and_not_the_topic() { + for t in [ + "cli-tri has been red on master for three days; the path filter meant it never ran", + "Seal Coverage has been red on master since #2841: four gen-c fixes changed output", + "coq-proofs.yml has failed 62 of 62 runs at opam init", + "brain-seal-refresh.yml cannot succeed: its last step is a push the ruleset forbids", + "emit-bitexact is red on master and nothing was watching", + "master's tri test suite is red, and two gates cannot see it", + "cargo test -p t27c landed red on master", + ] { + assert!(claims_red(t), "should claim red: {t}"); + } + for t in [ + "seal-coverage is green again after 29 red runs", + "cli-tri builds one crate behind a filter covering another", + "The gate against gates-green-by-not-running covers 15 of 47 workflows", + "Four workflows were red and have been repaired", + ] { + assert!(!claims_red(t), "should NOT claim red: {t}"); + } + } + + /// The recall bug this command exists around. A reader keyed on the file + /// stem alone cannot see an issue that names the workflow the way GitHub + /// displays it. + #[test] + fn a_workflow_is_found_by_the_name_it_displays_under() { + let dir = tempdir("keys"); + std::fs::write( + dir.join("seal-coverage.yml"), + "name: Seal Coverage\non:\n push:\n", + ) + .unwrap(); + let keys = workflow_keys(&dir).unwrap(); + let hay = "Seal Coverage has been red on master since #2841"; + assert_eq!( + named_workflows(hay, &keys), + vec!["seal-coverage.yml".to_string()], + "the display name must resolve to the file" + ); + // and the stem still works + assert_eq!( + named_workflows("seal-coverage.yml has failed", &keys), + vec!["seal-coverage.yml".to_string()] + ); + let _ = std::fs::remove_dir_all(&dir); + } + + /// A hyphenated name must not match inside a longer one. `cli-tri` and + /// `cli-tri-mcp` are different things in this repository, and #2903 is + /// about exactly that distinction. + #[test] + fn a_name_does_not_match_inside_a_longer_one() { + let dir = tempdir("bound"); + std::fs::write(dir.join("cli-tri.yml"), "name: cli-tri\n").unwrap(); + let keys = workflow_keys(&dir).unwrap(); + assert!(named_workflows("cli-tri is red on master", &keys).len() == 1); + assert!( + named_workflows("cli-tri-mcp is in neither list", &keys).is_empty(), + "cli-tri must not match inside cli-tri-mcp" + ); + assert!( + named_workflows("the acli-tri thing", &keys).is_empty(), + "and not inside a longer word on the left either" + ); + let _ = std::fs::remove_dir_all(&dir); + } + + /// An empty workflow directory is a refusal, not a zero: "no issue names a + /// workflow" and "this command could not read any workflow" are different + /// statements and must not print the same. + #[test] + fn an_empty_workflow_directory_is_refused() { + let dir = tempdir("empty"); + let e = workflow_keys(&dir).unwrap_err().to_string(); + assert!(e.contains("no *.yml"), "{e}"); + let _ = std::fs::remove_dir_all(&dir); + } + + /// The claim is in the title, so the title's workflows carry it. #2754 + /// says "secret-scan and cli-tri are red on master" -- both green today -- + /// while its body also mentions `release`, which is red for its own + /// reasons. Reading them together marks a stale headline as still red. + #[test] + fn the_title_names_the_workflows_the_claim_is_about() { + let dir = tempdir("title"); + for (f, n) in [ + ("cli-tri.yml", "cli-tri"), + ("secret-scan.yml", "secret-scan"), + ("release.yml", "Release Pipeline"), + ] { + std::fs::write(dir.join(f), format!("name: {n}\n")).unwrap(); + } + let keys = workflow_keys(&dir).unwrap(); + let title = "secret-scan and cli-tri are red on master, and a paths: filter hid it"; + let body = "See also the Release Pipeline, which is red for its own reasons."; + let from_title = named_workflows(title, &keys); + assert_eq!( + from_title, + vec!["cli-tri.yml".to_string(), "secret-scan.yml".to_string()], + "the title names two, and release is not one of them" + ); + let both = named_workflows(&format!("{title}\n{body}"), &keys); + assert!( + both.contains(&"release.yml".to_string()) && both.len() == 3, + "reading the body too pulls in the third: {both:?}" + ); + let _ = std::fs::remove_dir_all(&dir); + } + + fn tempdir(tag: &str) -> std::path::PathBuf { + let d = std::env::temp_dir().join(format!( + "tri-issues-{tag}-{}-{}", + std::process::id(), + NTH.fetch_add(1, std::sync::atomic::Ordering::Relaxed) + )); + std::fs::create_dir_all(&d).unwrap(); + d + } + static NTH: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0); +} diff --git a/cli/tri/src/main.rs b/cli/tri/src/main.rs index e17f41f80..68ba327bb 100644 --- a/cli/tri/src/main.rs +++ b/cli/tri/src/main.rs @@ -13,6 +13,7 @@ mod abandoned; mod orphaned; mod census; mod competitors; +mod issues; mod cibase; mod fleet; mod fpga; @@ -242,6 +243,12 @@ enum Commands { #[command(subcommand)] action: competitors::CompetitorsCmd, }, + /// Open issues whose headline calls a workflow red, against what that + /// workflow concludes on master today. + Issues { + #[command(subcommand)] + action: issues::IssuesCmd, + }, Unparsed { #[command(subcommand)] action: unparsed::UnparsedCmd, @@ -899,6 +906,7 @@ fn main() -> Result<()> { Commands::Ledgers { action } => ledgers::run(action, std::env::current_dir()?)?, Commands::Census { action } => census::run(action)?, Commands::Competitors { action } => competitors::run(action)?, + Commands::Issues { action } => issues::run(action)?, Commands::Unparsed { action } => unparsed::run(action, std::env::current_dir()?)?, Commands::Seals { action } => seals::run(action)?, Commands::Hooks { action } => hooks::run(action)?, diff --git a/docs/now/2026-08-31-five-open-issues-say-a-workflow-is-red-that-workflow-is-gree.md b/docs/now/2026-08-31-five-open-issues-say-a-workflow-is-red-that-workflow-is-gree.md new file mode 100644 index 000000000..2cbe30d4e --- /dev/null +++ b/docs/now/2026-08-31-five-open-issues-say-a-workflow-is-red-that-workflow-is-gree.md @@ -0,0 +1,9 @@ +# NOW -- Five open issues say a workflow is red; that workflow is green today (2026-08-31) + +## Five open issues say a workflow is red; that workflow is green today (Refs #2983) + +- 477 open issues, 268 with a number in the title -- each a measurement of the tree taken once and re-read by nothing. New tri issues stale walks the one subset whose truth is stored outside the repository: 9 titles claim a named workflow is red, and 5 name one that is green on master today. +- It states what a green reading does NOT establish and never suggests closing anything. #2729's title is stale (cli-tri is green) and its argument is not: the workflow still carries the paths: filter the issue was about. The headline being false and the issue being resolved are different facts. +- The recall bug it was built around: issues name a workflow the way GitHub DISPLAYS it. seal-coverage.yml displays as Seal Coverage, so a stem-keyed reader cannot see #2851 at all. 22 of 49 workflows differ; adding the display name moved the population from 8 to 9, and the one it added was the example I already had the answer for. +- Attribution is split: the claim lives in the TITLE, so the title's workflows carry it. Falling back to the body pulled in a workflow red for its own reasons and marked two stale headlines as still red. Rows attributed from the body say so. +- Self-inflicted and caught: an unquoted heredoc ran every backticked span in the skill text as a command and left four holes in the prose. Written down here already, from a commit message that lost four words the same way. Quote the delimiter.