parser: dedupe the twin string/Buffer large-JSONL scanners#698
Conversation
iamtoruk
left a comment
There was a problem hiding this comment.
Verified hands-on since CI does not run tests: tsc clean, full suite green apart from the pre-existing #681 failures, and a cold-cache month-scale report over real data (22k calls including many 32KB-plus lines and the advisor fixtures) is byte-identical between this branch and main. Mutation checks: corrupting the shared number reader is killed by the parity suite's absolute pins, and the advisor large-line tests pass on both scanner entry paths. One nit for a follow-up or a quick push: the generic findJsonValueBounds wrapper is dead code, defined with zero callers; the shared higher-level family calls findObjectFieldValue directly. Nice net reduction on the riskiest duplicated code in the repo.
|
Dead wrapper removed — |
Problem
src/parser.ts carried two structurally identical hand-rolled JSON scanners — a string family and a Buffer twin of every function, ~340 lines duplicated. Any scanner fix had to land in both or parsing diverged between small-string and large-buffer inputs, surfacing as silent size-dependent miscounts. (#643)
Fix
One scanner implementation generic over an indexable
JsonSource. The hot loops stay monomorphic: low-level primitives (string-end/container-end/value-bounds scans) branch once per call on source type into type-specific loops; all higher-level logic (field lookup, capped string reads, usage/tool-block extraction) exists once. Net −67 lines with the parity test included; the eleven*Buffertwins of the higher-level family are gone.Careful spots, called out for review:
Performance
Measured against the old implementation (same methodology as the differential check; ~300KB assistant lines, 50-iteration averages):
Tests
tests/parser-large-json-scanner-parity.test.ts(13 cases): identical results asserted through both entry paths on escaped quotes/backslashes, unicode escapes, nested braces in strings, raw multi-byte UTF-8, emoji at cap boundaries, truncated lines, and whitespace variants.npx tsc --noEmitclean; full suite green except the three pre-existingapp/electron/cli.test.tsenvironment failures noted in the Extract shared token estimation helper, migrate all CHARS_PER_TOKEN sites #677 review.Closes #643