Skip to content

fix(knowledge): close the learning feedback loop#1722

Merged
1 commit merged into
mainfrom
fix/knowledge-close-feedback-loop
Jul 6, 2026
Merged

fix(knowledge): close the learning feedback loop#1722
1 commit merged into
mainfrom
fix/knowledge-close-feedback-loop

Conversation

@zaxbysauce

@zaxbysauce zaxbysauce commented Jul 5, 2026

Copy link
Copy Markdown
Collaborator

Closes #1715

Summary

The knowledge feedback loop was an open circuit — outcome signals were recorded but nothing acted on them. Three gaps fixed (G1/G2/G3 from the gap catalog):

  • G1 — phase-failure outcomes now record. phase-complete previously called updateRetrievalOutcome(dir, name, true) with a hardcoded true at a point in the flow before success was determined. The call now runs after the real outcome is finalized and passes that value, so a failed phase (policy=enforce + missing required agents) emits 'failure' outcome events and failed_after_shown_count can finally increment.
  • G2 — confidence-floor action. Both feedback bridges (applySkillUsageFeedback, applyKnowledgeVerdictFeedback) bumped entry.confidence but nothing acted on low confidence — the bumps dead-ended at the floor. Now bumpKnowledgeConfidenceBatch runs a post-bump sweep that demotes (confidence_floor_demoted flag → strips retrieval statusBoost) or quarantines per config entries that are floor-clamped AND have a net-negative outcome signal over ≥3 outcome events. Reversible by default.
  • G3 — contradiction signals unified. contradicted_count (incremented only via knowledge_receipt) and the curator's flag_contradiction (tag-only) were disconnected. The curator now emits contradicted events post-transaction (with agent: 'curator' for audit), and a new maybeQuarantineOnContradiction auto-quarantines entries whose in-window contradicted count crosses the threshold (default 3 in 30d). Counts come from the RAW event log, not the mtime-keyed rollup cache, to avoid stale-count risk.

The genuinely-closed sub-loop was previously only the violation-count → escalator → enforce-block path. This change closes the confidence/contradiction/phase-failure halves.

Invariant audit

  • 1 (plugin init): not touched — no plugin-registration changes.
  • 2 (runtime portability): not touched — pure TS, no runtime-specific code.
  • 3 (subprocesses): not touched.
  • 4 (.swarm containment): not touched — new artifacts (contradicted events, confidence_floor_demoted flag) write only to existing .swarm/ knowledge + events files; no new locations.
  • 5 (plan durability): not touched.
  • 6 (test_runner safety): not touched.
  • 7 (test writing): touched — 4 new/updated test files co-located with source per convention; each runs in its own process to avoid the cross-file contamination tracked in coverage job's full-suite single-process run shows ~189 files (16%) with test failures — likely systemic mock/state contamination, not independent bugs #1712.
  • 8 (session state): not touched.
  • 9 (guardrails/retry): not touched — fail-open semantics preserved (bumpKnowledgeConfidenceBatch and the new floor-action are best-effort, never throw).
  • 10 (chat/system msg): not touched — the G2 statusBoost gate is read-only on retrieval scoring.
  • 11 (tool registration): not touched.
  • 12 (release/cache): touched — release fragment added at docs/releases/pending/1715-close-knowledge-feedback-loop.md; no version files edited.

Test plan

Validation commands run + results:

$ bun run typecheck                    # tsc --noEmit
(empty — clean)

$ bun run lint                         # biome lint .
Checked 2313 files. No fixes applied.

# Each affected test file in its own process (per repo convention; the
# combined single-process run has pre-existing cross-file contamination
# tracked in #1712 — see proof below).
$ bun --smol test tests/unit/config/knowledge-config.test.ts            # 43 pass / 0 fail
$ bun --smol test tests/unit/hooks/knowledge-reader.test.ts             # 20 pass / 0 fail
$ bun --smol test tests/unit/hooks/knowledge-events.test.ts             # 24 pass / 0 fail
$ bun --smol test tests/unit/hooks/knowledge-store.test.ts              # 39 pass / 0 fail
$ bun --smol test tests/unit/hooks/knowledge-verdict-feedback.test.ts   # 10 pass / 0 fail
$ bun --smol test tests/unit/hooks/curator.test.ts                      # 118 pass / 0 fail
$ bun --smol test tests/unit/hooks/knowledge-escalator.test.ts          # 5 pass / 0 fail
$ bun --smol test tests/unit/hooks/knowledge-floor-action.test.ts       # 6 pass / 0 fail   (NEW)
$ bun --smol test tests/unit/hooks/knowledge-contradiction.test.ts      # 8 pass / 0 fail   (NEW)
$ bun --smol test tests/unit/hooks/search-knowledge.test.ts             # 14 pass / 0 fail
$ bun --smol test tests/unit/tools/phase-complete-outcome-regression.test.ts  # 3 pass / 0 fail  (NEW)
                                                                        # = 290 pass / 0 fail

Pre-existing-failure proof (combined run is NOT a regression): the combined single-process run produces 24 failures — but I verified this is identical on clean origin/main (worktree at HEAD c5eb59c, before any of these edits): same 24 failures, same pass count. This is the documented cross-file mock/state contamination tracked in #1712, not caused by this change.

Independent review: GLM-5.2 reviewer (fresh context) returned NEEDS_REVISION with 5 findings (2 HIGH broken config fixtures + missing tests; 1 MED hive quarantine no-op; 2 LOW). All addressed. GLM-5.2 critic (separate fresh context) returned APPROVE with 2 optional LOW improvements, both applied (floor-action console.warn, behavioral retrieval-ranking test).

New test coverage:

  • G2: floor-demote on net-negative signal, min-outcomes gate, net-positive skip, recovery clears stale flag, none=legacy, idempotency, retrieval down-ranking.
  • G3: curator emits contradicted event, tag preserved (back-compat), threshold quarantine, below-threshold skip, idempotency on already-quarantined, window respected, curator→quarantine end-to-end, tag_only=legacy.
  • G1: source-level regression guard (no literal true, passes success var, old false-premise comment gone). Companion behavioral test (false'failure' event) already exists at knowledge-reader.test.ts Test 12.

How to use

New config knobs (all optional, conservative defaults):

  • knowledge.confidence_floor_action: 'none' | 'demote' | 'quarantine' (default 'demote').
  • knowledge.confidence_floor_min_outcomes: default 3.
  • knowledge.confidence_floor_signal_threshold: default 0 (net-negative).
  • knowledge.contradiction_threshold_action: 'tag_only' | 'quarantine' (default 'quarantine').
  • knowledge.contradiction_quarantine_threshold: default 3.
  • knowledge.contradiction_quarantine_window_days: default 30.

For the legacy dead-end behavior: confidence_floor_action: 'none' + contradiction_threshold_action: 'tag_only'.

Migration

No migration required. New config fields default to conservative behavior. Existing entries without confidence_floor_demoted read as undefined → not-demoted (legacy). Hive floor-entries get the flag (honored by retrieval) but not quarantine (the quarantine mechanism reads only the swarm file — documented in the release fragment).

Review in cubic

Three gaps that left the knowledge feedback loop open are now closed:

- G1: phase-complete now records the REAL phase outcome (success/failure)
  via updateRetrievalOutcome, not a hardcoded `true`. Previously the call
  ran before `success` was determined, so failed phases could never emit
  'failure' outcome events and the negative half of the outcome signal
  was dead.
- G2: confidence feedback no longer dead-ends. A just-bumped entry clamped
  to the floor (0.1) with a net-negative outcome signal is now demoted
  (confidence_floor_demoted flag, strips retrieval statusBoost) or
  quarantined per config. Conservative defaults: requires >=3 outcome
  events and a net-negative signal.
- G3: contradiction signals unified. Curator flag_contradiction now emits
  a `contradicted` event post-transaction (in addition to the existing
  tag), and maybeQuarantineOnContradiction auto-quarantines entries whose
  in-window contradicted count crosses the threshold (default 3 in 30d).

The genuinely-closed sub-loop was previously only the violation-count ->
escalator -> enforce-block path. Confidence/contradiction/phase-failure
halves were wired to record signal but nothing consumed it.

Reviewed: independent GLM-5.2 reviewer (NEEDS_REVISION -> all 5 findings
addressed) + independent GLM-5.2 critic (APPROVE).

Validation: typecheck clean, lint clean, 276 tests pass / 0 fail across
11 affected files (each run in its own process per repo convention; the
combined-run failures are pre-existing on main, tracked in #1712).
@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Drift check report

✅ No drift detected across skills, tools, commands, and agents.

@zaxbysauce

Copy link
Copy Markdown
Collaborator Author

🤖 Multi-Stage PR Review

Pipeline: Friday-35B (Qwen3.6-35B-A3B) (context pack) → Mellum2-12B (explore → candidates) → GLM-5-turbo (challenge + blind-spot)
Commit reviewed: 8c8d90808f31


🔍 PR Intent

Reconstructed obligation list from PR text, issue #1715, commit messages, changed tests, changed docs, and changed interfaces.

  • O-001 phase-complete must call updateRetrievalOutcome with the real success variable, not a hardcoded true (G1).
  • O-002 When a confidence-bumped entry is clamped to the floor (0.1) with a net-negative outcome signal and ≥3 outcome events, it must be demoted (default) or quarantined per config (G2).
  • O-003 Curator flag_contradiction actions must emit contradicted events post-transaction to unify the two previously-disconnected contradiction signals (G3).
  • O-004 When an entry's in-window contradicted count crosses the configured threshold, it must be auto-quarantined (G3).
  • O-005 New config knobs must have correct Zod schema definitions with proper defaults.
  • O-006 The confidence_floor_demoted flag must suppress statusBoost in retrieval ranking.
  • O-007 Recovery above the floor must clear a stale confidence_floor_demoted flag.
  • O-008 Backward compatibility: entries without confidence_floor_demoted treated as not-demoted; none/tag_only config options preserve legacy dead-end behavior.

📦 Implementation Summary

Three independent feedback-loop gaps are closed. G1: The updateRetrievalOutcome call in phase-complete.ts is moved after success is finalized and passes that variable instead of true; a source-level regression test guards against re-introduction. G2: bumpKnowledgeConfidenceBatch now returns the set of touched entries from applyConfidenceDeltas and runs a post-bump applyConfidenceFloorAction sweep that reads event rollups once, checks signal/evidence thresholds, and persists confidence_floor_demoted flags (and optionally quarantines) via separate transactKnowledge calls for swarm and hive tiers. Retrieval scoring in search-knowledge.ts suppresses statusBoost when the flag is set. G3: applyCuratorKnowledgeUpdates collects flag_contradiction actions during its transaction, then emits contradicted receipt events post-transaction (avoiding lock deadlock) and calls maybeQuarantineOnContradiction which counts raw events and auto-quarantines at the threshold. Six new config fields are added to the Zod schema with conservative defaults. New type ConfidenceFloorOptions threads the config through both feedback bridges.


✅ / ⚠️ / ❌ Intended vs Actual

Obligation Status Evidence (file:line)
O-001 SUPPORTED src/tools/phase-complete.ts:1383 — passes success variable; old literal-true call removed
O-002 SUPPORTED src/hooks/knowledge-store.ts:906–1089applyConfidenceFloorAction with signal/evidence gates, demote/quarantine/none paths
O-003 SUPPORTED src/hooks/curator.ts:1718–1745 — emits contradicted events with agent:'curator' post-transaction
O-004 SUPPORTED src/hooks/knowledge-escalator.ts:280–352maybeQuarantineOnContradiction with raw-event counting, idempotency guard
O-005 SUPPORTED src/config/schema.ts:1102–1131 — six new fields with correct Zod types and defaults matching docs
O-006 SUPPORTED src/hooks/search-knowledge.ts:487–491!entry.confidence_floor_demoted gate on statusBoost
O-007 SUPPORTED src/hooks/knowledge-store.ts:943–944,1014recovered array clears stale flag via entry.confidence_floor_demoted = false
O-008 SUPPORTED undefined is falsy → not-demoted (legacy); floorAction:'none' early-returns; tag_only skips quarantine call

🚨 Confirmed Findings

None found.


🔬 Unverified but Plausible Risks

None found.


🧪 Test / Coverage Gaps

None found. All three gaps (G1 source guard + behavioral companion, G2 demote/quarantine/recovery/idempotency/none, G3 event emission/tag-backcompat/threshold/below-threshold/idempotency/window/end-to-end/tag_only) are covered. The retrieval-ranking test confirms the demoted entry actually sinks.


📋 Shipped-vs-Claimed Gaps

None found. 290 tests pass (all in isolated processes per repo convention). New config defaults match the schema, the release fragment, and the PR description exactly.


🔁 Validation provenance

  • Confirmed findings kept: 0 (the reviewer-confirmed list was empty).
  • Confirmed findings dropped: N/A.
  • Blind-spot findings added: 0. I traced every + line across all seven changed source files, verified cross-file type contracts (ConfidenceFloorOptions wiring), checked lock ordering (sequential, no deadlock), validated the touched-array propagation from applyConfidenceDeltasapplyConfidenceFloorAction, confirmed hive-tier quarantine is intentionally omitted (documented caveat), and verified the G1 guard condition is a semantic tightening consistent with the function's purpose. No defects found.

📝 Merge Recommendation

APPROVE

Clean, well-structured closure of three documented feedback-loop gaps with conservative defaults, proper fail-open semantics, correct cross-file wiring, and thorough test coverage.

Check Result
No CRITICAL findings
No unresolved STEALTH_CHANGE
No UNSUPPORTED obligations
Test coverage adequate
No hardcoded secrets
All async errors handled
Input validation present
No broken agent role boundaries
Prompt format contracts intact
Lockfile consistent

🔒 Reviewed by a multi-stage local-first funnel (architect context pack → explorer candidates → critic challenge/author) for high recall with low false-positive noise. Findings are advisory — verify before acting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(knowledge): close the learning feedback loop — phase-failure outcomes, confidence-floor action, unified contradiction signal

1 participant