fix(parser): recover mid-turn operator messages and stop self-inflicted parse noise - #47
Draft
m-szymanska wants to merge 4 commits into
Draft
fix(parser): recover mid-turn operator messages and stop self-inflicted parse noise#47m-szymanska wants to merge 4 commits into
m-szymanska wants to merge 4 commits into
Conversation
added 4 commits
July 31, 2026 07:39
…user turns A message the operator submits while a turn is running exists in a Claude session JSONL only as a `queue-operation` enqueue record. The harness writes a `type:"user"` row only for what it delivers between turns, so a mid-turn submission that is interrupted or superseded never becomes one. The adapter consumed the record as `metadata_record` and discarded its body, making that text unrecoverable for every downstream consumer. The record stays consumed as `metadata_record` — the frozen taxonomy and the oracle fixture do not move — and its `enqueue` text is now additionally projected as an ordinary `TurnKind::UserMsg` turn, timestamped from the record and referencing the record's own evidence. No schema change: markdown renders the standard `**[HH:MM:SS] user:**` header. Two dedupe layers, both observed on live sessions, keep the projection from doubling a message: re-submitting the same text keeps the first enqueue, and a queued message the harness later delivers as a real `user` row loses to that row. Dropping a projected turn renumbers turns, tool events, skill invocations and segment bounds so the model stays internally consistent. Empty bodies and `<task-notification` heads are harness chatter, not operator input; `remove`/`dequeue`/`popAll` repeat text `enqueue` already carried. Authored-By: claude <agents@vetcoders.io> session_id: 77601750-3f2c-4edb-8e99-07c3a4011a56 timestamp: 2026_0731_0024_CEST runtime: claude-code
…ion contract Contract §2.1 states the rule the adapter fix implements: a unit's consumption kind decides its accounting, never whether its body may be discarded. Where a recognized metadata record is the only carrier of literal operator input, the adapter must project that text — spelled out for Claude `queue-operation`, with the loss class (mid-turn submissions), the dedupe layers, and the reason the loss is invisible to coverage. Adds the end-to-end guard for the same path: a fixture session whose only carrier of a mid-turn message is an enqueue record must surface that message through `to_conversation`, at the moment it was submitted. Authored-By: claude <agents@vetcoders.io> session_id: 77601750-3f2c-4edb-8e99-07c3a4011a56 timestamp: 2026_0731_0026_CEST runtime: claude-code
…adata_record The harness moved Claude file-history tracking from whole snapshots to per-message `file-history-delta` records, which the frozen taxonomy never declared. Each one terminated as `skipped(unknown_payload_type)` with a typed warning and raised `unsupported_visible_event`, so a perfectly healthy modern session was flagged as carrying unsupported visible events purely from its own bookkeeping. The record was verified against live sources to be rewind/backup bookkeeping only — a backup descriptor (`backupFileName`, `version`, `backupTime`, `realParentDir`), the message ids it belongs to, and a tracking path. No conversation content, so it carries no visible event and belongs with its older sibling `file-history-snapshot` as `metadata_record`. This is a deliberate edit of the frozen taxonomy: the declared list, its description, the contract's Claude physical-unit row, and a fixture unit all move together, and the contract gate stays green (42 kinds). Measured on a live 3995-unit session: 48 records moved from skipped(unknown_payload_type) to consumed(metadata_record), skipped went 48 -> 0, and unknown-payload warnings dropped 448 -> 400. The residual 400 come from an unrelated pre-existing path (assistant `thinking` blocks the harness emits with an empty body) and are untouched here. Authored-By: claude <agents@vetcoders.io> session_id: 77601750-3f2c-4edb-8e99-07c3a4011a56 timestamp: 2026_0731_0104_CEST runtime: claude-code
…ng blocks silently Since 2026-07 the Claude harness emits thinking blocks signature-only: the block carries `thinking: ""` plus a signature, and the reasoning text never reaches the JSONL at all. Live evidence on a 3995-unit session: 400 of 400 thinking blocks were empty. The adapter read an empty body through `string_field`, which cannot tell an empty string from a missing key, so every such block fell to the unrecognized shape path — a typed `unknown_payload_type` warning plus `unsupported_visible_event`. The result was that any session which reasoned at all reported itself as carrying unsupported visible events, draining the flag of its meaning exactly where it was supposed to be a signal. A known block with no body is now consumed silently, the same semantics an empty text block already had in `emit_text_turn`. This is a silent-consume of a known-empty block, not a new kind and not a second redacted_thinking path: accounting is untouched (still `consumed(thinking_block)`), only the turn projection is skipped. A block with a body is unchanged, and a block missing the field entirely stays an unsupported shape — the fix distinguishes the two cases `string_field` had conflated. Measured on the same live session: unknown-payload warnings 400 -> 0 and `unsupported_visible_event` true -> false, with consumed/skipped/turn counts unchanged (3995/0/1180). Authored-By: claude <agents@vetcoders.io> session_id: 77601750-3f2c-4edb-8e99-07c3a4011a56 timestamp: 2026_0731_0148_CEST runtime: claude-code
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Three same-class classification fixes in the Claude JSONL adapter, found by a full JSONL-vs-extract audit (208 utterances, live session):
2ae831b). Messages the operator sends while the agent is working exist in the JSONL only asqueue-operationrecords (text in.content, older builds.prompt) and are delivered insidetool_resultentries — they never become standaloneuserrows, soextract --conversationsilently dropped them. The audit proved this is the extractor's only systematic loss class (assistant losses: 0, standalone-user losses: 0), and it skews toward the most valuable words: decisions and course corrections typed during dense tool activity. Enqueues are now projected as regularUserMsgturns (consumption staysmetadata_record; frozen taxonomy untouched), with a two-layer dedupe: re-enqueued text keeps the first occurrence, and a message later delivered as a realuserrow wins over its queue synthetic.file-history-deltais recognized as metadata (5d888ab). The harness replacedfile-history-snapshotwith delta records (rewind/backup bookkeeping — zero conversational content). Unrecognized, each one counted asskipped(unknown_payload_type)+ warning.thinkingblocks are consumed silently (a6dc10f). The harness stores thinking blocks with an empty body and only a signature (400/400 blocks in the probe session). Each one raisedUnknownPayloadTypeand latchedunsupported_visible_event, so every modern reasoning session degraded its own parse status — the flag was dead as a signal. Empty body is now consumed like emptytext; a missing/non-string body still raises the flag (that narrowing is deliberate and tested).9750b3brecords the projection inPARSER_NORMATIVE_CONTRACT.md(§2.1 "Consumption kind is not a licence to drop the body").Measured on a live 5 MB session JSONL
skipped(unknown_payload_type)recordsUnknownPayloadTypewarningsunsupported_visible_eventtrue(permanently)falseTest plan
cargo test -p aicx-parser— 0 failed (22 tests inclaude_adapter, incl. 7 new for the queue projection + dedupe, 3 for empty thinking, 1 for file-history-delta)raw_unit_taxonomy.tomlmatch rules unchanged forqueue-operation;file-history-deltaadded with a fixture unit so the new kind is actually exercised by the gatesrc/extraction/tests.rs: fixture JSONL →to_conversationcarries the mid-turn messagemake test,make fmt-check,make clippy(-D warnings), pre-push gate (clippy + test + semgrep) — all greenNotes for review
main; the same commits also exist stacked onfeat/live-window-hot-pathasfeat/midturn-queue-user-turns(local) — provenance, not a competing line.validate_evidence_referencechecks evidence ids against coverage membership only, so a turn may reference ametadata_record-consumed unit as long as theevidence_event_idmatches the consumption exactly.🤖 Generated with Claude Code