Evaluate: let hub-pr-check's classify job open a fix/suppress PR for confirmed generator-gaps

[esraagamal6]

Current state: hub-pr-check.yml's classify job is deliberately
read-only — its own comment says it "Mints NO write-capable token for either
repo — it cannot file an issue, open a PR, or push, regardless of the CLI
permission flag below" (hub-pr-check.yml:139-141). When it determines a
failure is generator-gap (api-test-generator doesn't handle a new/changed
endpoint shape yet), that verdict currently only becomes a Slack message —
nothing acts on it.

Why it might be worth doing: triage-camunda-hub-nightly.yml already
does something structurally similar for the nightly run — it mints a
real, write-capable App token and opens fix/suppress PRs in
api-test-generator for issues it finds. hub-pr-check.yml's classify job
runs the exact same kind of analysis, just triggered per-PR instead of
nightly, and per-PR is closer to the moment the gap is actually introduced.

Suggested scope, if built: NOT auto-writing generator logic (understanding
a new schema shape and extending the ontology/scenario-templates is a real
engineering task, not something to ship unreviewed). Start narrower — when
classify confirms generator-gap at high confidence, open a PR that adds
the corresponding excludeOperations/knownIssue entry to the relevant
configs/<config>/request-validation.json (mechanical, low-risk, exactly
what a human currently does by hand — e.g. #512's createWorkspace entry).

Open questions to resolve before building (not resolved by this issue):

1. Trust/security. hub-pr-check.yml is triggered by repository_dispatch
   from ANY camunda-hub PR — not necessarily reviewed or trusted at the time
   the check runs. classify's read-only posture exists specifically to keep
   its blast radius to "read files, write one JSON output" regardless of what
   content it's fed. Granting it (or a derived job) write access to
   api-test-generator based on an LLM's read of that external PR's content
   adds a real prompt-injection surface — likely needs at least the same
   class of guards already used elsewhere in this repo (the .github/**-diff
   check in triage-camunda-hub-nightly.yml/hub-pr-live-check.yml, a
   narrowly-scoped App token), possibly more caution given the trigger
   originates from a different repo entirely.
2. Noise/dedup. hub-pr-check.yml runs on every camunda-hub PR — per the
   recent investigation (fix(hub-pr-check): harden classify prompt against false hub-medic pings #514), a single standing issue can trigger a burst
   of a dozen+ runs in under an hour across many unrelated PRs. Without a
   dedup guardrail (the nightly already has one — "Discover open fix PRs",
   triage-camunda-hub-nightly.yml:47-51's comment on why pull-requests: read is needed for it), this could open many duplicate PRs for the same
   gap.
3. False positives. generator-gap is itself an LLM guess, now further
   hardened (fix(hub-pr-check): harden classify prompt against false hub-medic pings #514) but still not certain — an auto-opened PR based on a wrong
   guess is more consequential than a Slack message based on one.

Not a commitment to build — filed as a candidate enhancement to evaluate,
following up on the classify-hardening work in #514.

[esraagamal6]

Sketch: gate the write action on the hub PR's approval state

Following up on the discussion — one concrete way to resolve open questions 1 (trust) and 2 (noise/dedup) at once: only open the config-entry PR once the originating camunda-hub PR is APPROVED, not on every classify run.

This needs no new dispatch/trigger from camunda-hub — hub-pr-check.yml already re-runs on every push to the hub PR (via its existing repository_dispatch), so the new job just reads the hub PR's current review state at run time:

gh pr view "$PR_NUMBER" --repo camunda/camunda-hub --json reviewDecision --jq .reviewDecision

Why this helps

• Trust (Q1): the write action only fires once a human has actually reviewed the content that produced the generator-gap verdict — a real narrowing of the prompt-injection blast radius, without adding any new App scope beyond read access. (classify itself stays exactly as read-only as it is today; this only gates a new job downstream of it.)
• Noise/dedup (Q2): most of the "dozen+ runs in under an hour" (fix(hub-pr-check): harden classify prompt against false hub-medic pings #514) would just see "not yet approved" and no-op on the write path — the classify→Slack path is unchanged, only the write path is gated. Still needs the same diff-content dedup the nightly triage flow already uses (gh pr list --search "label:... is:open" + check affected_operation_ids against each open PR's diff) once a run does see APPROVED, so an already-in-flight suppression PR isn't duplicated across pushes after approval.
• Doesn't touch Q3 (false positives) — an approved-but-wrong generator-gap verdict still opens a wrong PR, just less often. Worth pairing with --draft (like Nightly: auto-detect resolved upstream blockers → Slack nudge + draft unskip PR #432's unskip PRs) so a human confirms before merge, at least until this path has a track record.

Rough shape (new job in hub-pr-check.yml, after classify)

open-generator-gap-pr:
  needs: [validate, classify]
  if: always() && needs.classify.outputs.category == 'generator-gap'
      && needs.classify.outputs.confidence == 'high'
      && needs.validate.outputs.pr_number != ''
  steps:
    1. Mint a camunda-hub-scoped read token, check reviewDecision == APPROVED
       (else log + skip, no extra Slack noise).
    2. Dedup: list open api-test-generator PRs (same label the nightly uses),
       check affected_operation_ids against each one's diff.
    3. Add the excludeOperations/suppress + knownIssue entry (knownIssue.url =
       the camunda-hub PR itself), regenerate + run the narrow local guard
       (same pattern hub-known-issue-reenable-check.yml just added for #432),
       abort without opening anything if that breaks.
    4. Open a draft PR via the qa-processes App token (same grant
       spec-bump-check/triage already use on this repo), label it, dispatch
       the existing hub-pr-live-check.yml validation automatically (it
       already fires on any same-repo PR that doesn't touch .github/**).

New things this surfaces, not yet resolved

• App permission: reading reviewDecision needs pull-requests: read on whichever App is used for the camunda-hub-side lookup — unconfirmed whether the App(s) already installed there (preview-envs for cloning, qa-processes for issues in the nightly flow) have that scope. Same "check before assuming" lesson as Nightly: auto-detect resolved upstream blockers → Slack nudge + draft unskip PR #432's Issues:Read turned out to already be covered by a different token than expected.
• Approval staleness vs exact SHA: reviewDecision doesn't reset on new commits unless the hub repo has "dismiss stale reviews" enabled on branch protection — so an approval could in principle be stale relative to source_sha. gh pr view's plain reviewDecision can't distinguish this; would need the GraphQL reviews/latestReviews connection compared against source_sha for precision. Flagging, not blocking — same false-positive category as Q3 either way.
• Which config file / suite: classify's current output schema (category, confidence, summary, affected_operation_ids) doesn't say which suite (positive/negative) the operation failed in — needed to know whether the entry belongs in positive-suppress.json's suppress[] or request-validation.json's excludeOperations[]. Would need a small addition to the classify prompt/schema (a suite field), matching the shape triage-camunda-hub-nightly.yml's own JSON schema already uses.

Still just a sketch — no code written. Flagging the App-permission and schema-extension points now so whoever picks this up doesn't have to rediscover them.

[esraagamal6]

Related but separate: #519 (skip the live-Hub suite run itself for draft camunda-hub PRs — a cost/noise optimization for the existing run job, distinct from this issue's reviewDecision-gated write action).