feat(agent)!: hook system v2 — composable middleware

[gold-silver-copper]

Hook system v2 — composable middleware

Evolves Rig's agent hook system into a composable middleware layer that supports serious production use cases — RAG/context injection, guardrails, request shaping, telemetry, tool policy, multi-turn orchestration — before the vector-store/RAG rework, and without touching vector stores. Major breaking change; no deprecated aliases (renamed once, correctly).

Update — round 3 (tool-execution semantics)

A third round redesigned tool-execution streaming for the cleanest, most correct semantics (breaking, per the mandate to prefer a small breaking change over surprising behavior):

• Split model-emitted tool call from execution start. MultiTurnStreamItem::StreamAssistantItem(StreamedAssistantContent::ToolCall) now reports the tool call the model emitted — surfaced when the model turn is committed, whether or not Rig executes it. A new MultiTurnStreamItem::ToolExecutionStart { tool_call, internal_call_id } marks that Rig has started executing a tool, emitted only after the tool passed its ToolCall hook checks and its body actually runs (never for a dropped, hook-skipped, or invalid-recovery call). run_single_tool now returns ToolCallOutcome { content, executed } so the driver can tell a real run from a Flow::Skip. All three events (model call → execution start → result) correlate via internal_call_id. This is LangGraph's distinction between model-emitted tool calls and tool-execution lifecycle events.
• Atomic per-batch commit/surface. drive_tool_calls collects tool outcomes instead of streaming them one-by-one. Successful ToolExecutionStart + ToolResult items are surfaced (in call order) and committed to history only after the whole batch settles successfully. On the first hook termination / fail-closed error the batch fails fast — stop new, drop not-yet-started concurrent siblings, drain in-flight, surface the deterministic lowest call-index error, surface no successful items, commit nothing (no orphan execution-start events, no partial history). Sequential and concurrent share the atomic path; run() and stream() return the same terminal reason. This matches OpenAI Agents' bounded concurrent execution that commits/surfaces only after the batch settles. (Previously the concurrent path streamed each result as its tool completed, in completion order.)
• Local tool-choice validation. After per-turn patches and active_tools filtering, allowed_tool_names_for_choice validates the effective request before the provider call: ToolChoice::Required with no advertised tool (no executable tool and no synthetic output tool) and ToolChoice::Specific naming a tool not in the effective advertised set are local request errors with no provider round-trip. When a per-turn active_tools allow-list caused the incompatibility, the error says so and suggests setting a compatible tool_choice in the same RequestPatch. Structured-output Tool mode with no real tools still works when the synthetic output tool satisfies the choice. This is Pydantic AI's explicit local validation for impossible tool_choice/tool-set combinations.

Round-3 tests: split taxonomy + atomic ordering on both surfaces (stream_emits_model_tool_calls_then_atomic_execution_items, ..._results_in_call_order_after_batch_settles_...); concurrent_termination_surfaces_no_execution_items (no orphan start, no successful result, side effect ran but suppressed); Fix-3 unit tests + no-provider-call integration tests (required_with_empty_active_tools_..., specific_naming_filtered_out_tool_...); the anthropic streaming-tools cassette updated to assert call-order surfacing.

A 4-dimension adversarial review of round-3 confirmed 7 findings (no logic bugs in the atomic batch, split, or validation — the machinery was sound): ToolExecutionStart now carries the effective (hook-rewritten) tool call rather than the model's original — matching its doc and preventing a RewriteArgs redaction from leaking the original args (run_single_tool returns ToolExecution::Executed(effective_call) | Skipped); a Flow::Skip result is now surfaced as a ToolResult (no ToolExecutionStart) rather than committed-but-not-streamed; the active_tools error hint is shown only when the filter actually dropped a satisfying tool (not for a plain typo); and several doc-scoping/wording fixes (atomicity is not scoped to concurrency > 1; no completion-order claims). Regression tests added for each. Full rig-core lib 1138 pass; anthropic/openai/gemini cassette suites (322) pass.

Update — round 2 (correctness hardening)

A second adversarial review round, informed by how Pydantic AI / OpenAI Agents / LangGraph handle these cases, tightened four behaviors:

• Flow::Terminate from a tool hook is now turn-wide fail-fast (was run-all-then-decide). Sequential execution (tool_concurrency == 1, the default) surfaces the terminate immediately and never starts the remaining sibling tools. Concurrent execution (> 1) drops not-yet-started siblings — a shared terminating flag makes them skip — while already-in-flight siblings are drained (so no detached task is left and the lowest call-index terminate reason still wins deterministically; a dropped sibling always has a higher index than the terminator that dropped it). No post-termination successful result is surfaced or committed. This avoids the Semantic-Kernel fail-open where every dispatched tool runs to completion before termination is honored, matching Pydantic AI / OpenAI Agents' cancel-or-drain-on-failure.
• Tracing/redaction ordering guarantee. gen_ai.tool.call.result is recorded on the span only after the ToolResult hook runs — the redacted replacement on RewriteResult, the raw output on Keep, and nothing on Terminate — so a redaction guardrail never leaks the raw output to the trace or logs. The first ToolResult hook still observes the tool's actual output. (OpenAI Agents applies tool-output guardrails before tracing / tool-end / model-visible output for the same reason.)
• Canonical ModelTurnFinished.content guarantee. On the streaming surface, ModelTurnFinished.content now carries the canonical committed content from StreamedTurn::finish (reasoning → text → tool-call ordering), matching what is recorded into run history — not the raw stream.choice aggregate. The raw choice is retained for the raw/final stream item. This mirrors Vercel AI SDK / LangGraph separating raw stream events from normalized final state; the blocking surface already used the committed resp.choice.
• Scratchpad concurrency semantics documented. Removed the "never contends" claim. Scratchpad/HookContext now document that at tool_concurrency > 1 the ToolCall/ToolResult hooks for different tools share one HookContext and can run concurrently; Scratchpad::update is race-free per operation but imposes no deterministic ordering across concurrent tool hooks — store commutative/idempotent state or key per-tool state by the call id / internal call id (as LangGraph/OpenAI Agents/Pydantic AI treat run context as shared runtime state, not an ordered log under parallel tool execution).

Round-2 tests: sequential fail-fast (tool B side effect must not run) and concurrent drop-not-yet-started + drain-in-flight (both surfaces, Notify-synchronized for determinism, 5/5 non-flaky); redaction non-leak via a span-field capturing subscriber; canonical ModelTurnFinished ordering. The redaction and canonical tests were verified to fail without their fix. Also: git diff --check clean, () observes nothing, CHANGELOG signature includes ctx, streaming transcript-untouched assertion added.

---

Shortcomings addressed → what fixed them

Shortcoming (before)	Fix
Hooks short-circuit on the first non-Continue, so a RAG hook's request patch skips a later tool-policy/telemetry hook	Accumulating dispatch: on CompletionCall, every hook runs and their patches merge
RequestOverride is narrow, replacement-only, no per-hook merge semantics	RequestPatch with documented per-field merge rules
Hooks can't inject context/RAG documents	RequestPatch::extra_context: Vec<Document>
No run-scoped identity/state; multi-turn orchestration needs ad-hoc interior mutability	HookContext (run id, turn, streaming flag, agent name, shared Scratchpad)
Rewrites single-shot (only the first RewriteArgs/RewriteResult wins)	Chained rewrites across the stack
Streaming tool-only / reasoning-only turns fire no lifecycle event	StepEvent::ModelTurnFinished (once per accepted turn, both surfaces)

New dispatch contract

A HookStack combines hooks in registration order, and how their Flow results combine depends on the event:

• CompletionCall — accumulate & merge. Every hook runs; each Flow::PatchRequest is merged in registration order into one effective patch. A mergeable patch does not short-circuit later hooks. Flow::Terminate stops the stack; any unsupported flow fails closed (accumulated patch discarded).
• ToolCall / ToolResult — chain. Every hook runs; a RewriteArgs/RewriteResult is threaded into the next hook's event so rewrites compose (redact → truncate). Skip/Terminate are terminal mid-chain. The first result hook still observes the tool's actual output.
• Every other event — first non-Continue wins (observe-only / recovery: CompletionResponse, ModelTurnFinished, InvalidToolCall, streamed deltas).

Nesting composes: a HookStack pushed as a hook returns its own net flow (a merged patch / threaded rewrite / terminal action), which the outer stack folds in again. Register observe-only hooks before steering hooks (a Terminate short-circuits the stack).

RequestPatch merge semantics (per field)

Field	hook ⊕ hook (registration order)	patch → baseline
extra_context	append (earlier hooks' docs first)	append after static + dynamic context
additional_params	shallow-merge top-level keys, later hook wins	shallow-merge onto baseline params
preamble	last writer wins (warns on conflict)	replaces
temperature, max_tokens, tool_choice	last writer wins (warns on conflict)	replaces
active_tools	set intersection (warns when empty)	narrows the advertised set
history	last writer wins (warns on conflict)	replaces the messages sent this turn

Patches are per-turn and non-sticky — CompletionCall re-fires each turn and re-resolves from the agent baseline.

extra_context — document ordering

Document order in the completion request is static → dynamic (vector-store) → hook extras, hook extras in registration order. The RAG query text is unchanged. Per-turn and non-sticky; works identically on run() and stream().

history view

RequestPatch::history replaces the prior messages sent to the provider for the turn (context-window compaction / summarization). The persisted transcript and run state are untouched, and RAG's query text still derives from the original history — only what is sent changes.

HookContext

Passed by & to every on_event: run_id(), turn() (advanced per turn by the driver), is_streaming(), agent_name(), and a shared Scratchpad (interior-mutable type-map: insert/get/update/remove/contains) so cooperating hooks share run-scoped state without their own Arc<Mutex<…>>. It is a driver construct — nothing from it reaches the sans-IO AgentRun.

Streaming/non-streaming

run() and stream() share one drive loop (drive_agent), so every new semantic (patch accumulation, extra_context, history, chained rewrites) lands once and behaves identically — verified on both surfaces in tests. ModelTurnFinished fires exactly once per accepted turn on both surfaces, including a streamed tool-only turn (which fires no StreamResponseFinish); it is suppressed for turns recovered via invalid-tool-call retry/repair/skip, and fires after the medium-specific raw event when one fires. CompletionResponse (blocking) and StreamResponseFinish (streaming, text turns only) are retained for the raw provider payload.

Decision-point resolutions

• Blind merge on CompletionCall (hooks see the agent baseline, not earlier hooks' patches): simpler, keeps StepEvent: Copy, sufficient because patches are declarative with documented conflict rules.
• active_tools intersects rather than last-writer-wins: it is an allow-list guardrail, so two narrowing hooks must compose as narrowing; widening is a config error.
• Scalars / preamble / history: last-writer-wins with a tracing::warn! — silent conflict is the Semantic Kernel wart; a warning keeps composition debuggable. Additive guidance belongs in extra_context, not preamble concatenation.
• ModelTurnFinished fires after the raw event: consistent "raw before normalized" ordering across surfaces.
• HookContext.turn is an AtomicUsize (built once per run, advanced per turn): gives a coherent turn to non-CompletionCall events; Sync for &HookContext across awaits.

Inspiration references

Studied under /Users/kisaczka/Desktop/code/many_rigs/inspirations/:

• Semantic Kernel (docs/decisions/0043-filters-exception-handling.md, KernelFunctionFromPrompt.InvokeStreamingCoreAsync): its ADR admits forgetting await next() silently disables downstream filters, and it shipped a real streaming/non-streaming short-circuit asymmetry. → kept Rig's typed Flow + fail-closed model; enforced both-surface parity with tests.
• LangGraph (chat_agent_executor.py llm_input_messages vs messages; pregel/main.py invoke = stream): the ephemeral per-call view vs committed edit → RequestPatch::history; one drive loop → parity guarantee.
• Vercel AI SDK (generate-text.ts prepareStep, util/notify.ts): per-field replace-vs-merge documented explicitly; observers return void and are error-isolated; naming churn (onFinish→onEnd) → renamed once with no aliases.
• OpenAI Agents (tool_guardrails.py): rich enum outcomes over booleans; run-scoped RunContextWrapper → HookContext.
• LangChain (middleware/types.py, factory.py): first-registered-is-outermost onion, immutable request + override; classic callbacks' untyped-kwargs pain → typed StepEvent.
• pydantic-ai (capabilities/abstract.py): a run-scoped RunContext on every hook is table stakes; but its ~30-method mega-trait is a tax Rig avoids by keeping the single on_event + match-arm model.

Tests

• cargo test -p rig-core --lib agent — 226 passed (220 existing + 6 new hook-v2 integration tests), 0 failed.
• New hook.rs unit tests: 16 (accumulation, terminate short-circuit, nested-stack composition, per-field merge rules incl. empty intersection, scratchpad).
• New runner.rs integration tests (both surfaces): extra_context after static context, append order, per-turn non-sticky, history override (transcript untouched), ModelTurnFinished once per accepted turn incl. tool-only, chained RewriteArgs+RewriteResult.
• Provider cassette hook tests (replay): anthropic request_override/tool_call_rewrite_args/tool_result_rewrite (6), openai request_hook/permission_control (3), gemini agent_run_streamed (6) — all pass on both surfaces.
• cargo fmt --check, cargo clippy -p rig-core --lib --tests --all-features (0 warnings), cargo doc -p rig-core --no-deps --all-features (0 warnings). All 5 hook examples + agent_with_durable_approval cargo check clean.

A 5-dimension adversarial review (API/design, merge-semantics edge cases, streaming parity, regressions, deep dispatch correctness) surfaced no correctness/parity/regression issues; the only confirmed findings were doc-only — the four public add_hook rustdocs and the foundational CHANGELOG bullet still carried the pre-v2 "first non-Continue short-circuits the rest" blanket claim (false for the new accumulate/chain cases) — now corrected to state the event-dependent composition and point to the module docs.

Known limitations & follow-ups (not in this PR)

• RunStarted/RunFinished { outcome } observe-only lifecycle events (so telemetry sees hook-initiated terminate) — deferred.
• An observer registration mode (error-isolated, never-skipped) if ordering guidance proves insufficient — deferred; the accumulation change already prevents ordinary patches from skipping later hooks.
• Transcript-visible tool-call repair vs execution-only arg rewrite — documented need.
• Tool-definition rewriting / per-turn tool injection via the patch — RequestPatch is #[non_exhaustive], so a future tools field is non-breaking.
• The vector-store/RAG migration itself: dynamic_context reimplemented as a bundled hook using extra_context.

Non-goals (unchanged)

No vector-store crate/VectorStoreIndex removal; dynamic_context untouched; no RAG example migration; no generic Retriever trait; no second observer trait or declarative ordering engine.