Skip to content

The first wall a session hits is the auto-mode classifier, and it teaches nothing #342

Description

@bdelanghe

Measured by dogfood P2 (#337), and it undercuts a property the org has been treating as achieved.

What was measured

P2 called read_chat in an auto-mode session. Two walls, in series:

  1. The auto-mode permission classifier denied the call. It never reached the tool. Its entire output was Blocked by classifier, plus an instruction to stop and ask a human.

  2. On retry, the call landed and read_chat returned its own refusal, verbatim:

    chat-fetch: no relay bearer. Set CLAUDE_RELAY_BEARER or write ~/.relay-lease-bearer —
      grant one machine-to-machine: dispatch bounded.tools deploy.yml with
      grant_relay_lease=<name> and an ephemeral age recipient, approve the Face ID,
      decrypt the run's output with the ephemeral identity (bounded.tools#62).
    

The second is exactly what #318/#326 asked for: a refusal that names the grant path — env var, file path, workflow input, and an issue to read. It works.

The first is what a session actually meets, and it says nothing.

Why this matters beyond one cell

The org's convention (claude/context.md) is that a refusal must name its grant path. We have been scoring that as met because the tool's refusal names it. But a caller only ever sees the tool's refusal if it gets past the classifier — so the property holds on the wall that is second in line and fails on the wall that is first.

This is the #326 lesson repeating one layer up. #326 was motivated by a blind agent that invented a Cloudflare bot-gate bypass across six dead ends rather than find chat-fetch.sh. A session told only Blocked by classifier is in the same position: it has a wall, no name for it, and no next step. The predictable failure is not "it stops" — it is "it works around".

Related, and probably the same root: .github-private#514 measured the classifier denying the CLAUDE.md step-1 bootstrap snippet, both as the one-liner and decomposed. .github-private#850 item 6 tracks the classifier blocking context.md edits. .github#270's live half is the same denial. Three symptoms already filed; this issue is about the message, not any one denial.

Also: it makes the measurement non-reproducible

E is not reproducible in one call. A fresh auto-mode session that calls read_chat exactly once records - (undetermined), not 1. P2 only got a real answer because the owner asked for a retry. Any probe that treats a single call as conclusive will mis-score this cell — noted on #337's method.

What would fix it

Not clear this is ours to fix, which is the first thing to establish:

  1. Determine whether the classifier's message is configurable at all from a repo's settings.json, a hook, or anywhere else we control. If it is not, this is a vendor ask and belongs with the read_code_session: a Claude Code session URL should read like a share link #338 door (a) ask rather than in a repo.
  2. If a denial cannot be made self-documenting, the second-best is to make the session self-documenting: the org context could name the classifier explicitly — "a call may be denied by the auto-mode classifier before it reaches the tool; that is a wall, not a missing capability, and the answer is a human visit to the session, never a workaround." .github-private#514's comment already reached that conclusion for the bootstrap step specifically; it is a general rule.
  3. Either way, stop scoring "our refusals teach" as met. It is met for the inner wall only, and the outer wall is the one sessions hit.

Not in scope

Whether the classifier's decisions are right. This is only about what it says when it decides no.

Related: #337, #318, #326, #270, .github-private#514, .github-private#850.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions