Add Agent Framework pull request risk classification

.github/workflows/classify-pull-request-risk.md

@@ -0,0 +1,108 @@
+---
+description: Classifies Agent Framework pull request risk when deterministic rules are inconclusive
+tracker-id: classify-pull-request-risk
+on:
+   workflow_call:
+      inputs:
+         pr_number:
+            description: "Pull request number to classify"
+            required: true
+            type: number
+   workflow_dispatch:
+      inputs:
+         pr_number:
+            description: "Pull request number to classify"
+            required: true
+            type: number
+concurrency:
+   group: "gh-aw-${{ github.workflow }}-${{ github.repository }}-${{ inputs.pr_number }}"
+   cancel-in-progress: true
+permissions:
+   contents: read
+   issues: read
+   pull-requests: read
+   copilot-requests: write
+inlined-imports: true
+network: defaults
+tools:
+   cli-proxy: true
+   github:
+      mode: gh-proxy
+      toolsets: [default]
+      min-integrity: unapproved
+safe-outputs:
+   noop:
+      report-as-issue: false
+   add-labels:
+      allowed:
+         - risk:low
+         - risk:medium
+         - risk:high
+         - failed-auto-risk
+      # A successful classification adds one risk; a failure adds only its marker.
+      max: 1
+      target: "${{ inputs.pr_number }}"
+   remove-labels:
+      allowed:
+         - risk:low
+         - risk:medium
+         - risk:high
+         - failed-auto-risk
+      max: 4
+      target: "${{ inputs.pr_number }}"
+timeout-minutes: 10
+---
+
+# Agent Framework Pull Request Risk Classifier
+
+Classify pull request `${{ inputs.pr_number }}` in `${{ github.repository }}` only when the evidence supports one risk level with high confidence.
+
+The deterministic classifier already handled unambiguous low-risk changes. This agent reviews every production or otherwise ambiguous change. A wrong risk label is worse than abstaining. Before this agent starts, the deterministic stage adds `failed-auto-risk` and clears every existing risk label. A confident classification must therefore actively add one risk label and remove the marker.
+
+## Risk Levels
+
+- `risk:low`: Limited blast radius and straightforward rollback. Examples include documentation, comments, examples, tests that do not alter production behavior, isolated internal fixes, and patch dependency updates with no meaningful runtime impact.
+- `risk:medium`: Contained production impact with a reasonable rollback. Examples include backwards-compatible API additions, bounded feature or bug-fix behavior, minor dependency upgrades, retries, timeouts, streaming, tool invocation, error handling, or refactoring across several packages.
+- `risk:high`: Large blast radius, difficult rollback, security implications, or compatibility risk. Examples include authentication, authorization, secrets, tool permissions or code execution, breaking APIs, core orchestration, checkpointing, persistence, serialization compatibility, significant concurrency changes, major dependency upgrades, or inadequate tests for the possible impact.
+
+## Signals
+
+Use all available evidence rather than treating any single label as decisive:
+
+- Assess the regression risk introduced by this proposed change, not merely how sensitive the touched component is. A small, well-tested safeguard or bounded bug fix in a sensitive area is usually `risk:medium`, not automatically `risk:high`.
+- `size:*` indicates review surface, not semantic risk by itself.
+- `kind:*` distinguishes code, tests, documentation, examples, dependencies, and CI.
+- `area:*` identifies affected Agent Framework subsystems and providers.
+- Changed paths reveal workflow state, checkpoint, shell/tool-approval, concurrency, authentication, and serialization boundaries.
+- For dependency changes, inspect whether the update is patch, minor, or major and whether it changes runtime behavior.
+- `public-api-change` is positive evidence that exported API changed, but it does not prove the change is breaking. Its absence is not proof that no API changed because the parity workflow may still be running.
+- Review the actual diff and test changes to distinguish an isolated fix from a broad behavioral change.
+
+## Confidence Gates
+
+Apply a risk label only when the actual diff provides concrete evidence for that level:
+
+- Use `risk:low` only when there is no meaningful production behavior change, or the change is obviously isolated, directly tested, and straightforward to roll back.
+- Use `risk:medium` only when production impact is clearly bounded, compatibility is preserved, tests cover the changed behavior, and no high-risk criterion is present.
+- Use `risk:high` only when the diff itself shows a concrete high-risk property such as a breaking API, weakened trust boundary, persistence or serialization incompatibility, broad orchestration or concurrency impact, major dependency upgrade, difficult rollback, or inadequate tests for the potential impact. A sensitive path alone is not sufficient.
+
+Abstain by adding `failed-auto-risk` when any of these are true:
+
+- Required files, patches, labels, or dependency-version details cannot be read.
+- Classification depends on assumptions not established by the diff.
+- More than one risk level remains reasonably plausible.
+- The test evidence or rollback scope is unclear.
+- You cannot cite concrete evidence for the selected level.
+
+Do not guess, choose a default, or round uncertainty up to a higher risk level.
+
+## Process
+
+1. Use GitHub tools to read the PR title, body, existing labels, changed files, and relevant diff patches. Do not execute pull request code.
+2. Apply the confidence gates above. Select exactly one risk level only when one level is clearly supported; otherwise abstain.
+3. On success, add the selected risk label if missing, remove the other two risk labels, and remove `failed-auto-risk` if present. Do not remove labels outside this allowlist.
+4. If the selected risk label is already the only risk label and no failure marker is present, use `noop`.
+5. Do not add comments or reviews.
+6. If the PR cannot be read or classified confidently, leave `failed-auto-risk` in place and do not add a risk label. Use `noop` if no label changes are needed.
+
+The workflow validates the final state after this agent finishes. A valid automatic result is either exactly one risk label with no failure marker, or `failed-auto-risk` with no risk labels. Missing, conflicting, or mixed states are converted to the unable marker and fail the workflow check.

.github/workflows/classify-pull-request.yml

@@ -24,7 +24,7 @@ concurrency:
 
 jobs:
   classify-size-area:
-    uses: microsoft/go-infra/.github/workflows/classify-pull-request.yml@35d34d21089773bebd2dd5efc6197a38f16490bf
+    uses: microsoft/go-infra/.github/workflows/classify-pull-request.yml@cf095ee7dc1ed628f4b3d7c225011e858ad7733e
     permissions:
       contents: read
       issues: write
@@ -55,7 +55,7 @@ jobs:
   classify-kind:
     if: github.event_name != 'workflow_dispatch' || github.event.inputs.pr-number != '0'
     needs: classify-size-area
-    uses: microsoft/go-infra/.github/workflows/classify-pull-request-kind.lock.yml@35d34d21089773bebd2dd5efc6197a38f16490bf
+    uses: microsoft/go-infra/.github/workflows/classify-pull-request-kind.lock.yml@cf095ee7dc1ed628f4b3d7c225011e858ad7733e
     secrets: inherit
     permissions:
       actions: read
@@ -65,3 +65,127 @@ jobs:
       copilot-requests: write
     with:
       pr_number: ${{ github.event.pull_request.number || fromJSON(github.event.inputs.pr-number || '0') }}
+
+  classify-risk-rules:
+    if: github.event_name != 'workflow_dispatch' || github.event.inputs.pr-number != '0'
+    needs: classify-kind
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      issues: write
+      pull-requests: write
+    outputs:
+      needs-agent: ${{ steps.classify.outputs.needs-agent }}
+    steps:
+      # pull_request_target resolves github.workflow_sha from the trusted base
+      # branch, so this never builds or executes pull request code.
+      - name: Checkout classifier
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          ref: ${{ github.workflow_sha }}
+          persist-credentials: false
+
+      - name: Set up Go
+        uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
+        with:
+          go-version-file: go.mod
+          cache: false
+
+      - name: Build risk classifier
+        run: go build -o "$RUNNER_TEMP/prriskclassifier" ./cmd/prriskclassifier
+
+      - name: Apply deterministic risk rules
+        id: classify
+        env:
+          GH_TOKEN: ${{ github.token }}
+          PR_NUMBER: ${{ github.event.pull_request.number || github.event.inputs.pr-number }}
+        run: |
+          "$RUNNER_TEMP/prriskclassifier" \
+            -repo "${{ github.repository }}" \
+            -pr-number "$PR_NUMBER"
+
+  classify-risk-agent:
+    if: needs.classify-risk-rules.outputs.needs-agent == 'true'
+    needs: classify-risk-rules
+    uses: ./.github/workflows/classify-pull-request-risk.lock.yml
+    secrets: inherit
+    permissions:
+      actions: read
+      contents: read
+      issues: write
+      pull-requests: write
+      copilot-requests: write
+    with:
+      pr_number: ${{ github.event.pull_request.number || fromJSON(github.event.inputs.pr-number || '0') }}
+
+  validate-risk-result:
+    if: >-
+      always() &&
+      needs.classify-risk-rules.result == 'success' &&
+      (needs.classify-risk-agent.result == 'success' || needs.classify-risk-agent.result == 'skipped')
+    needs: [classify-risk-rules, classify-risk-agent]
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      issues: write
+      pull-requests: write
+    steps:
+      - name: Checkout classifier
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          ref: ${{ github.workflow_sha }}
+          persist-credentials: false
+
+      - name: Set up Go
+        uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
+        with:
+          go-version-file: go.mod
+          cache: false
+
+      - name: Build risk classifier
+        run: go build -o "$RUNNER_TEMP/prriskclassifier" ./cmd/prriskclassifier
+
+      - name: Enforce automatic risk result
+        env:
+          GH_TOKEN: ${{ github.token }}
+          PR_NUMBER: ${{ github.event.pull_request.number || github.event.inputs.pr-number }}
+        run: |
+          "$RUNNER_TEMP/prriskclassifier" \
+            -repo "${{ github.repository }}" \
+            -pr-number "$PR_NUMBER" \
+            -validate-result
+
+  mark-risk-failure:
+    if: >-
+      always() &&
+      (needs.classify-risk-rules.result == 'failure' ||
+       needs.classify-risk-agent.result == 'failure' ||
+       needs.validate-risk-result.result == 'failure')
+    needs: [classify-risk-rules, classify-risk-agent, validate-risk-result]
+    runs-on: ubuntu-latest
+    permissions:
+      issues: write
+      pull-requests: write
+    steps:
+      - name: Mark automatic risk classification failure
+        env:
+          GH_TOKEN: ${{ github.token }}
+          PR_NUMBER: ${{ github.event.pull_request.number || github.event.inputs.pr-number }}
+        run: |
+          gh label create failed-auto-risk \
+            --repo "${{ github.repository }}" \
+            --color 6E7781 \
+            --description "Automatic risk classification was inconclusive or failed" \
+            --force
+          gh api --method POST \
+            "repos/${{ github.repository }}/issues/$PR_NUMBER/labels" \
+            -f 'labels[]=failed-auto-risk'
+          risk_labels="$(gh api \
+            "repos/${{ github.repository }}/issues/$PR_NUMBER" \
+            --jq '.labels[].name | select(. == "risk:low" or . == "risk:medium" or . == "risk:high")')"
+          while IFS= read -r label; do
+            if [[ -n "$label" ]]; then
+              gh api --method DELETE \
+                "repos/${{ github.repository }}/issues/$PR_NUMBER/labels/${label/:/%3A}"
+            fi
+          done <<< "$risk_labels"

cmd/prriskclassifier/classify.go

@@ -0,0 +1,71 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+package main
+
+import (
+	"path/filepath"
+	"slices"
+	"strings"
+)
+
+const (
+	riskLow    = "risk:low"
+	riskMedium = "risk:medium"
+	riskHigh   = "risk:high"
+)
+
+type deterministicDecision struct {
+	Label  string
+	Reason string
+}
+
+func classifyDeterministically(files, labels []string) deterministicDecision {
+	kinds := labelsWithPrefix(labels, "kind:")
+
+	if isClearlyLowRisk(kinds, labels) && len(files) > 0 && !slices.ContainsFunc(files, isPotentialProductionFile) {
+		return deterministicDecision{
+			Label:  riskLow,
+			Reason: "changes are limited to documentation, tests, or examples",
+		}
+	}
+
+	return deterministicDecision{Reason: "production or ambiguous changes require confidence-gated semantic review"}
+}
+
+func isClearlyLowRisk(kinds, labels []string) bool {
+	if len(kinds) == 0 ||
+		!slices.ContainsFunc(labels, func(label string) bool { return strings.HasPrefix(label, "size:") }) ||
+		slices.Contains(labels, "public-api-change") ||
+		slices.Contains(labels, "failed-auto-classify") ||
+		slices.Contains(labels, "size:xlarge") {
+		return false
+	}
+	for _, kind := range kinds {
+		if kind != "kind:docs" && kind != "kind:tests" && kind != "kind:examples" {
+			return false
+		}
+	}
+	return true
+}
+
+func isPotentialProductionFile(path string) bool {
+	path = filepath.ToSlash(path)
+	base := filepath.Base(path)
+	return !strings.HasPrefix(path, "docs/") &&
+		!strings.HasPrefix(path, "examples/") &&
+		!strings.Contains(path, "/testdata/") &&
+		!strings.HasSuffix(path, "_test.go") &&
+		!strings.HasSuffix(path, ".md") &&
+		base != "LICENSE"
+}
+
+func labelsWithPrefix(labels []string, prefix string) []string {
+	return slices.Collect(func(yield func(string) bool) {
+		for _, label := range labels {
+			if strings.HasPrefix(label, prefix) && !yield(label) {
+				return
+			}
+		}
+	})
+}