Skip to content

parser: dedupe the twin string/Buffer large-JSONL scanners#698

Merged
iamtoruk merged 2 commits into
getagentseal:mainfrom
avs-io:refactor/643-dedupe-jsonl-scanner
Jul 16, 2026
Merged

parser: dedupe the twin string/Buffer large-JSONL scanners#698
iamtoruk merged 2 commits into
getagentseal:mainfrom
avs-io:refactor/643-dedupe-jsonl-scanner

Conversation

@avs-io

@avs-io avs-io commented Jul 16, 2026

Copy link
Copy Markdown
Member

Problem

src/parser.ts carried two structurally identical hand-rolled JSON scanners — a string family and a Buffer twin of every function, ~340 lines duplicated. Any scanner fix had to land in both or parsing diverged between small-string and large-buffer inputs, surfacing as silent size-dependent miscounts. (#643)

Fix

One scanner implementation generic over an indexable JsonSource. The hot loops stay monomorphic: low-level primitives (string-end/container-end/value-bounds scans) branch once per call on source type into type-specific loops; all higher-level logic (field lookup, capped string reads, usage/tool-block extraction) exists once. Net −67 lines with the parity test included; the eleven *Buffer twins of the higher-level family are gone.

Careful spots, called out for review:

  • Byte vs UTF-16 semantics: capped reads count UTF-16 output units while Buffer offsets are bytes; the escape-scanner only yields ASCII-safe segment boundaries, so caps can't split a multi-byte character. The parity test covers raw multi-byte UTF-8 through the Buffer path, surrogate pairs at cap boundaries, escapes-before-cap + multibyte-after, and truncation mid-string / mid-escape / mid-UTF-8-sequence.
  • One deliberate unification: large lines with leading whitespace used to parse on the Buffer path but return null on the string path. Both now parse (the permissive behavior). This is the only behavioral divergence between old and new, verified by differential execution of the old implementation against the new one across all the nasty-input cases.
  • Rebased over the advisor-iterations and web-fetch usage additions; both are ported into the unified extraction and covered by their existing tests.

Performance

Measured against the old implementation (same methodology as the differential check; ~300KB assistant lines, 50-iteration averages):

Case Old New
string large line 6.06 ms 5.56 ms
buffer large line 4.47 ms 3.19 ms
string pathological cap 4.70 ms 4.66 ms
buffer pathological cap 4.25 ms 2.77 ms

Tests

  • New tests/parser-large-json-scanner-parity.test.ts (13 cases): identical results asserted through both entry paths on escaped quotes/backslashes, unicode escapes, nested braces in strings, raw multi-byte UTF-8, emoji at cap boundaries, truncated lines, and whitespace variants.
  • npx tsc --noEmit clean; full suite green except the three pre-existing app/electron/cli.test.ts environment failures noted in the Extract shared token estimation helper, migrate all CHARS_PER_TOKEN sites #677 review.

Closes #643

@avs-io
avs-io marked this pull request as ready for review July 16, 2026 19:02

@iamtoruk iamtoruk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified hands-on since CI does not run tests: tsc clean, full suite green apart from the pre-existing #681 failures, and a cold-cache month-scale report over real data (22k calls including many 32KB-plus lines and the advisor fixtures) is byte-identical between this branch and main. Mutation checks: corrupting the shared number reader is killed by the parity suite's absolute pins, and the advisor large-line tests pass on both scanner entry paths. One nit for a follow-up or a quick push: the generic findJsonValueBounds wrapper is dead code, defined with zero callers; the shared higher-level family calls findObjectFieldValue directly. Nice net reduction on the riskiest duplicated code in the repo.

@avs-io

avs-io commented Jul 16, 2026

Copy link
Copy Markdown
Member Author

Dead wrapper removed — findJsonValueBounds deleted, the String/Buffer variants stay as the direct callees. tsc clean, parser + parity suites green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

parser: dedupe the twin string/Buffer large-JSONL scanners

2 participants