Skip to content

feat(parsing): frame CESR 2.0 (v2) streams (stacked on #402) - #403

Open
dhh1128 wants to merge 6 commits into
WebOfTrust:mainfrom
dhh1128:feat/cesr-stream-parser-v2
Open

dhh1128 wants to merge 6 commits into
WebOfTrust:mainfrom
dhh1128:feat/cesr-stream-parser-v2

Conversation

@dhh1128

@dhh1128 dhh1128 commented Jul 22, 2026

Copy link
Copy Markdown

Summary

Adds CESR 2.0 / KERI 2.x support to the stream parser introduced in #402.
The parser now frames both CESR-1 and CESR-2 streams to the same resilience
posture, ground-truthed against keripy 2.0.0-dev6.

Stacked on #402. This branch is based on feat/cesr-stream-parser, so
until #402 merges the diff here shows both commits; the change owned by this
PR is the second commit (feat(parsing): frame CESR 2.0 (v2) streams via genus dispatch). Please review/merge #402 first — this will then narrow to just the
v2 commit and can retarget cleanly onto main.

What it does

  • parseVersion now parses the v2 version string
    (proto + protocol-version + embedded CESR-genus-version + kind +
    base64 size + . terminator) and returns the genus.
  • frameGroup dispatches by CESR genus. Native v2 counter framing (from
    keripy's CtrDex_2_0) is added because signify-ts's Counter carries only the
    v1 table and silently misframes v2 codes — they collide with v1 code
    strings but denote different groups (e.g. v1 -C is NonTransReceiptCouples;
    v2 -C is the AttachmentGroup wrapper). Every v2 group counts its body in
    quadlets, so it self-frames as count*4 bytes: decomposition is uniform,
    even unrecognized codes are unknown-but-framed, and the v1 -V
    resilience-boundary rule generalizes to all groups.
  • Primitives still delegate to Matter/Indexer even under v2 — primitive
    codes are genus-stable, so a v2 AID/SAID/key/signature parses unchanged.
  • AttachmentGroup gains a genus field, since a code's meaning depends on it.

Tests

Adds v2 parseVersion, framing, and resilience/corruption tests, plus a real
keripy-generated v2 witness-OOBI vector (PII-free synthetic). All four CI gates
(build, lint, pretty:check, tests) pass locally.

Provenance

Developed and proven in a downstream viewer against a corpus of keripy
2.0.0-dev6 streams, then contributed upstream.

🤖 Generated with Claude Code

signify-ts has the primitive classes (Matter/Counter/Indexer) and EMITS framed
streams (eventing.messagize) but has no parser that CONSUMES a stream. Add
src/keri/core/parsing.ts: a deterministic v1 stream walker that frames messages
by version-string size + attachment counters (never by sniffing a leading '{',
so binary CBOR/MGPK bodies work too), delegating all primitive/counter/indexer
sizing to the existing classes. Every node carries byte-span provenance, and the
walk is resilient — on a code it cannot frame it stops and reports, keeping
everything parsed so far. Body decoders are pluggable (JSON built in). Exported
via src/exports.ts; 30 inline keripy-vector tests in test/core/parsing.test.ts.

Scope: CESR v1 only; the v2 count/genus/version-string machinery is a separate
follow-up.

Signed-off-by: Daniel Hardman <daniel.hardman@gmail.com>
@daidoji

daidoji commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

I'm not sure this one is working either. The SAD Path example from the CESR spec fails to parse (as do many other streams that I think should work).

    it('SAD Path example from CESR spec', () => {
        const stream = bytesOf('{"v":"ACDCCAACAAJSONAAIe.","t":"acm","d":"EO3117lnAbjDt66qe2PtgHooXKAYQT_C6SIbESMcJ5lN","i":"EEDGM_DvZ9qFEAPf_FX08J3HX49ycrVvYVXe9isaP5SW","s":"EGU_SHY-8ywNBJOqPKHr4sXV9tOtOwpYzYOM63_zUCDW","a":{"d":"ED1wMKzV72L7YI1yJ3NXlClPUgvEerw4jRocOYxaZGtH","i":"ECsGDKWAYtHBCkiDrzajkxs3Iw2g-dls3bLUsRP4yVdT","dt":"2025-06-09T17:35:54.169967+00:00","personal":{"name":"John Doe","home":"Atlanta"},"p":[{"ref0":{"name":"Amy","i":"ECmiMVHTfZIjhA_rovnfx73T3G_FJzIQtzDn1meBVLAz"}},{"ref1":{"name":"Bob","i":"ECWJZFBtllh99fESUOrBvT3EtBujWtDKCmyzDAXWhYmf"}}]}}');
        const { messages, errors } = parse(stream);
        assert.isEmpty(messages);
        assert.equal(errors, []);
    });

https://trustoverip.github.io/kswg-cesr-specification/#sad-path-examples

…erable

Nothing compared a DECLARED length against the bytes actually available, so a
truncated stream parsed as a complete one. Reported by @daidoji on WebOfTrust#402:
`{"v":"KERI10JSON100000_"}` — 25 bytes claiming 0x100000 — returned one
message, no errors, and a `consumed` and `span.end` a megabyte past the end of
the buffer, because subarray clamps silently and the clamped slice happened to
be valid JSON.

The same omission ran through the attachment framing (a truncated -V framed
`state: 'known'` past the buffer; a truncated -A framed unframable-group, which
condemns bytes that were merely absent), and produced a hang: a version string
with size 0 and a serialization with no registered decoder never advanced the
cursor, pushing a message per pass until the heap died — about 40s to 4 GB in
node, from 25 bytes of input.

- ParseError.permanent is a boolean; `incomplete` carries false, the one
  failure more bytes can cure. New `invalid-version-size` rejects a size too
  small to hold its own version string, which makes forward progress an
  invariant of the parse loop rather than a special case for 0.
- Primitive and counter shortage is read from the Matter/Indexer/Counter size
  tables BEFORE construction — a throw could never tell short from malformed.
- An incomplete message is not emitted and `consumed` stays at its first byte,
  so a caller appends bytes and re-parses with no double-emit.
- A truncation sweep parses nine vectors cut at every byte offset, asserting
  termination and that no span or `consumed` ever exceeds the input: the class
  of bug, not the one example, is what went undetected.

One corner is left: a version string cut in half still reports
no-version-string, since deciding that needs a "could these bytes still become
a legal message start" test rather than a length check. The sweep pins its
exact bound rather than allowing any permanent error.

Signed-off-by: Daniel Hardman <daniel.hardman@gmail.com>
@dhh1128
dhh1128 force-pushed the feat/cesr-stream-parser-v2 branch from 3ecbb53 to ee410e3 Compare September 3, 2026 16:32
The Matter/Indexer size tables store a variable-size code's `fs` as null while
typing it `fs?: number`, so the preceding commit's `fs === undefined || fs < 0`
let null through (null >= 0 is true) and handed back a null length. It only
looked harmless because the empty slice made the constructor throw and the
catch called it 'bad'. Tested by value with typeof now.

Found by cesrview's 100% coverage gate on the same code, which flagged that the
bounds checks split several one-line failure paths in two and left six new
branches untested. Tests added for the distinctions the split created, each a
real one: a primitive that is the right SIZE but the wrong bytes (sizing says
yes, decoding says no); a counter whose bytes are all present but are not ASCII;
a compound group whose NESTED group is malformed vs cut off; a primitive hard
code cut off vs one in no size table vs one that is variable-size; a big
counter's 3-character hard code cut off; and a -V whose inner group is wholly
present but overflows the wrapper, which is not shortage — it belongs to no one
— so the wrapper still stands as a resilience boundary.

Signed-off-by: Daniel Hardman <daniel.hardman@gmail.com>
Stacks on the v1 stream parser (WebOfTrust#402): the parser now also handles CESR 2.0 /
KERI 2.x streams, ground-truthed to keripy 2.0.0-dev6.

- parseVersion parses the v2 version string (proto + protocol-version + embedded
  CESR-genus-version + kind + base64 size + '.') and returns the genus.
- frameGroup dispatches by genus. Native v2 counter framing (from keripy's
  CtrDex_2_0) is added because signify-ts's Counter carries only the v1 table and
  silently misframes v2 codes (they collide with v1 strings but denote different
  groups). Every v2 group counts its body in quadlets, so it self-frames as
  count*4 bytes — decomposition is uniform and even unrecognized codes are
  unknown-but-framed, with the -V resilience-boundary rule generalized to all
  groups. Primitives still delegate to Matter/Indexer (genus-stable).
- AttachmentGroup gains `genus`, since a code's meaning depends on it.

Adds v2 parseVersion, framing, and resilience/corruption tests, plus a real
keripy-generated v2 witness-OOBI vector. All four CI gates green locally.

The v2 framing carries the same bounds-checking the preceding commit added for
v1: parseV2Counter distinguishes a header cut off by the end of the stream from
one whose soft count is not base64, and a v2 group whose count*4 quadlets run
past the end is short rather than framed. Self-framing is exactly the property a
truncated stream makes a lie of, so the truncation sweep runs over the v2 OOBI
vector too.

Signed-off-by: Daniel Hardman <daniel.hardman@gmail.com>
@dhh1128
dhh1128 force-pushed the feat/cesr-stream-parser-v2 branch from ee410e3 to 6caf3be Compare September 3, 2026 16:45
@dhh1128

dhh1128 commented Sep 3, 2026

Copy link
Copy Markdown
Author

Thanks for the catches, @daidoji . Fixed now.

@dhh1128

dhh1128 commented Sep 5, 2026

Copy link
Copy Markdown
Author

@daidoji — following up on the SAD-path example, now that the fix is pushed (head 6caf3be).

That stream parses cleanly. Against the current branch head:

parseVersion → { proto: 'ACDC', version: '2.0', kind: 'JSON', size: 542, genus: 2, length: 19 }
parse        → messages: 1, errors: [], consumed: 542, span: { start: 0, end: 542 }

with the full sad decoded, ilk: 'acm', said: 'EO3117lnAbjDt66qe2PtgHooXKAYQT_C6SIbESMcJ5lN', and no attachments. What was broken before was the size field: AAIe is 542, and the parser wasn't reading the v2 version string, so it never framed the body.

One note on the snippet as written, in case you rerun it: assert.isEmpty(messages) asserts the absence of a message, so it fails on a stream that parses correctly, and assert.equal(errors, []) compares a fresh array by identity and can never pass. assert.lengthOf(messages, 1) and assert.isEmpty(errors) are what I think you meant.

On "many other streams that I think should work" — I'd like to chase those. If you have concrete ones, paste them (or point me at the corpus) and I'll add each as a test case.

`CesrMessage.sn` was populated from the body's `s` field for every
protocol. Under KERI that field is the sequence number, a hex-encoded
integer; under ACDC the same field name holds the schema SAID, an
unrelated value. So parsing an ACDC handed the caller a 44-character
digest in a field documented as a sequence number, and any consumer
that compared or ordered on `sn` would act on it.

Gate the assignment on the protocol and say so in the field's doc
comment. Nothing is lost for ACDC callers: the schema SAID is still on
`sad.s`, where it belongs.

Signed-off-by: Daniel Hardman <daniel.hardman@gmail.com>
The spec's SAD Path Examples section carries a bare genus-2 ACDC with
no attachments — a shape the suite did not cover, since every other v2
vector is a KERI event and most carry attachment groups. @daidoji
reported it failing to parse on PR WebOfTrust#403, which it did until the v2
version string was read; this pins the behaviour so it cannot regress.

Covers parseVersion on an ACDC v2 string (base64 size AAIe = 542) and
full framing: one message, no errors, consumed == length, the nested
attribute block intact, and `sn` null because `s` here is the schema.

Signed-off-by: Daniel Hardman <daniel.hardman@gmail.com>
@dhh1128 dhh1128 mentioned this pull request Sep 5, 2026
@daidoji

daidoji commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Sure, I was testing by hand and hadn't pulled v2 streams out of keripy into a testable corpus (it was always the intention to pull them out into https://github.com/WebOfTrust/cesr-test-vectors/ but we never got around to it, just the primitives and cesr appropriate messages but not necessarily the keri appropriate ones). Like these should parse with cesr but most aren't keri protocol field map payloads. https://github.com/WebOfTrust/cesr-test-vectors/tree/main/example_payloads/version2/json

The ones I was trying were the version 2 field maps from the keri, cesr, acdc specs though as those are the easiest to reach for. I may have made errors in minifying them and running them by hand though so didn't include them explicitly. The counterexample I found was just the easiest. If it becomes difficult I can come back and pull some arbitrary keri protocol messages of any length for you from our implementation. I find it a bit more difficult to do this from keripy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants