Conversation
signify-ts has the primitive classes (Matter/Counter/Indexer) and EMITS framed
streams (eventing.messagize) but has no parser that CONSUMES a stream. Add
src/keri/core/parsing.ts: a deterministic v1 stream walker that frames messages
by version-string size + attachment counters (never by sniffing a leading '{',
so binary CBOR/MGPK bodies work too), delegating all primitive/counter/indexer
sizing to the existing classes. Every node carries byte-span provenance, and the
walk is resilient — on a code it cannot frame it stops and reports, keeping
everything parsed so far. Body decoders are pluggable (JSON built in). Exported
via src/exports.ts; 30 inline keripy-vector tests in test/core/parsing.test.ts.
Scope: CESR v1 only; the v2 count/genus/version-string machinery is a separate
follow-up.
Signed-off-by: Daniel Hardman <daniel.hardman@gmail.com>
|
I'm not sure this one is working either. The SAD Path example from the CESR spec fails to parse (as do many other streams that I think should work). it('SAD Path example from CESR spec', () => {
const stream = bytesOf('{"v":"ACDCCAACAAJSONAAIe.","t":"acm","d":"EO3117lnAbjDt66qe2PtgHooXKAYQT_C6SIbESMcJ5lN","i":"EEDGM_DvZ9qFEAPf_FX08J3HX49ycrVvYVXe9isaP5SW","s":"EGU_SHY-8ywNBJOqPKHr4sXV9tOtOwpYzYOM63_zUCDW","a":{"d":"ED1wMKzV72L7YI1yJ3NXlClPUgvEerw4jRocOYxaZGtH","i":"ECsGDKWAYtHBCkiDrzajkxs3Iw2g-dls3bLUsRP4yVdT","dt":"2025-06-09T17:35:54.169967+00:00","personal":{"name":"John Doe","home":"Atlanta"},"p":[{"ref0":{"name":"Amy","i":"ECmiMVHTfZIjhA_rovnfx73T3G_FJzIQtzDn1meBVLAz"}},{"ref1":{"name":"Bob","i":"ECWJZFBtllh99fESUOrBvT3EtBujWtDKCmyzDAXWhYmf"}}]}}');
const { messages, errors } = parse(stream);
assert.isEmpty(messages);
assert.equal(errors, []);
});https://trustoverip.github.io/kswg-cesr-specification/#sad-path-examples |
…erable Nothing compared a DECLARED length against the bytes actually available, so a truncated stream parsed as a complete one. Reported by @daidoji on WebOfTrust#402: `{"v":"KERI10JSON100000_"}` — 25 bytes claiming 0x100000 — returned one message, no errors, and a `consumed` and `span.end` a megabyte past the end of the buffer, because subarray clamps silently and the clamped slice happened to be valid JSON. The same omission ran through the attachment framing (a truncated -V framed `state: 'known'` past the buffer; a truncated -A framed unframable-group, which condemns bytes that were merely absent), and produced a hang: a version string with size 0 and a serialization with no registered decoder never advanced the cursor, pushing a message per pass until the heap died — about 40s to 4 GB in node, from 25 bytes of input. - ParseError.permanent is a boolean; `incomplete` carries false, the one failure more bytes can cure. New `invalid-version-size` rejects a size too small to hold its own version string, which makes forward progress an invariant of the parse loop rather than a special case for 0. - Primitive and counter shortage is read from the Matter/Indexer/Counter size tables BEFORE construction — a throw could never tell short from malformed. - An incomplete message is not emitted and `consumed` stays at its first byte, so a caller appends bytes and re-parses with no double-emit. - A truncation sweep parses nine vectors cut at every byte offset, asserting termination and that no span or `consumed` ever exceeds the input: the class of bug, not the one example, is what went undetected. One corner is left: a version string cut in half still reports no-version-string, since deciding that needs a "could these bytes still become a legal message start" test rather than a length check. The sweep pins its exact bound rather than allowing any permanent error. Signed-off-by: Daniel Hardman <daniel.hardman@gmail.com>
3ecbb53 to
ee410e3
Compare
The Matter/Indexer size tables store a variable-size code's `fs` as null while typing it `fs?: number`, so the preceding commit's `fs === undefined || fs < 0` let null through (null >= 0 is true) and handed back a null length. It only looked harmless because the empty slice made the constructor throw and the catch called it 'bad'. Tested by value with typeof now. Found by cesrview's 100% coverage gate on the same code, which flagged that the bounds checks split several one-line failure paths in two and left six new branches untested. Tests added for the distinctions the split created, each a real one: a primitive that is the right SIZE but the wrong bytes (sizing says yes, decoding says no); a counter whose bytes are all present but are not ASCII; a compound group whose NESTED group is malformed vs cut off; a primitive hard code cut off vs one in no size table vs one that is variable-size; a big counter's 3-character hard code cut off; and a -V whose inner group is wholly present but overflows the wrapper, which is not shortage — it belongs to no one — so the wrapper still stands as a resilience boundary. Signed-off-by: Daniel Hardman <daniel.hardman@gmail.com>
Stacks on the v1 stream parser (WebOfTrust#402): the parser now also handles CESR 2.0 / KERI 2.x streams, ground-truthed to keripy 2.0.0-dev6. - parseVersion parses the v2 version string (proto + protocol-version + embedded CESR-genus-version + kind + base64 size + '.') and returns the genus. - frameGroup dispatches by genus. Native v2 counter framing (from keripy's CtrDex_2_0) is added because signify-ts's Counter carries only the v1 table and silently misframes v2 codes (they collide with v1 strings but denote different groups). Every v2 group counts its body in quadlets, so it self-frames as count*4 bytes — decomposition is uniform and even unrecognized codes are unknown-but-framed, with the -V resilience-boundary rule generalized to all groups. Primitives still delegate to Matter/Indexer (genus-stable). - AttachmentGroup gains `genus`, since a code's meaning depends on it. Adds v2 parseVersion, framing, and resilience/corruption tests, plus a real keripy-generated v2 witness-OOBI vector. All four CI gates green locally. The v2 framing carries the same bounds-checking the preceding commit added for v1: parseV2Counter distinguishes a header cut off by the end of the stream from one whose soft count is not base64, and a v2 group whose count*4 quadlets run past the end is short rather than framed. Self-framing is exactly the property a truncated stream makes a lie of, so the truncation sweep runs over the v2 OOBI vector too. Signed-off-by: Daniel Hardman <daniel.hardman@gmail.com>
ee410e3 to
6caf3be
Compare
|
Thanks for the catches, @daidoji . Fixed now. |
|
@daidoji — following up on the SAD-path example, now that the fix is pushed (head That stream parses cleanly. Against the current branch head: with the full One note on the snippet as written, in case you rerun it: On "many other streams that I think should work" — I'd like to chase those. If you have concrete ones, paste them (or point me at the corpus) and I'll add each as a test case. |
`CesrMessage.sn` was populated from the body's `s` field for every protocol. Under KERI that field is the sequence number, a hex-encoded integer; under ACDC the same field name holds the schema SAID, an unrelated value. So parsing an ACDC handed the caller a 44-character digest in a field documented as a sequence number, and any consumer that compared or ordered on `sn` would act on it. Gate the assignment on the protocol and say so in the field's doc comment. Nothing is lost for ACDC callers: the schema SAID is still on `sad.s`, where it belongs. Signed-off-by: Daniel Hardman <daniel.hardman@gmail.com>
The spec's SAD Path Examples section carries a bare genus-2 ACDC with no attachments — a shape the suite did not cover, since every other v2 vector is a KERI event and most carry attachment groups. @daidoji reported it failing to parse on PR WebOfTrust#403, which it did until the v2 version string was read; this pins the behaviour so it cannot regress. Covers parseVersion on an ACDC v2 string (base64 size AAIe = 542) and full framing: one message, no errors, consumed == length, the nested attribute block intact, and `sn` null because `s` here is the schema. Signed-off-by: Daniel Hardman <daniel.hardman@gmail.com>
|
Sure, I was testing by hand and hadn't pulled v2 streams out of keripy into a testable corpus (it was always the intention to pull them out into https://github.com/WebOfTrust/cesr-test-vectors/ but we never got around to it, just the primitives and cesr appropriate messages but not necessarily the keri appropriate ones). Like these should parse with cesr but most aren't keri protocol field map payloads. https://github.com/WebOfTrust/cesr-test-vectors/tree/main/example_payloads/version2/json The ones I was trying were the version 2 field maps from the keri, cesr, acdc specs though as those are the easiest to reach for. I may have made errors in minifying them and running them by hand though so didn't include them explicitly. The counterexample I found was just the easiest. If it becomes difficult I can come back and pull some arbitrary keri protocol messages of any length for you from our implementation. I find it a bit more difficult to do this from keripy. |
Summary
Adds CESR 2.0 / KERI 2.x support to the stream parser introduced in #402.
The parser now frames both CESR-1 and CESR-2 streams to the same resilience
posture, ground-truthed against keripy
2.0.0-dev6.What it does
parseVersionnow parses the v2 version string(
proto+ protocol-version + embedded CESR-genus-version +kind+base64
size+.terminator) and returns the genus.frameGroupdispatches by CESR genus. Native v2 counter framing (fromkeripy's
CtrDex_2_0) is added because signify-ts'sCountercarries only thev1 table and silently misframes v2 codes — they collide with v1 code
strings but denote different groups (e.g. v1
-CisNonTransReceiptCouples;v2
-Cis theAttachmentGroupwrapper). Every v2 group counts its body inquadlets, so it self-frames as
count*4bytes: decomposition is uniform,even unrecognized codes are unknown-but-framed, and the v1
-Vresilience-boundary rule generalizes to all groups.
Matter/Indexereven under v2 — primitivecodes are genus-stable, so a v2 AID/SAID/key/signature parses unchanged.
AttachmentGroupgains agenusfield, since a code's meaning depends on it.Tests
Adds v2
parseVersion, framing, and resilience/corruption tests, plus a realkeripy-generated v2 witness-OOBI vector (PII-free synthetic). All four CI gates
(build, lint,
pretty:check, tests) pass locally.Provenance
Developed and proven in a downstream viewer against a corpus of keripy
2.0.0-dev6streams, then contributed upstream.🤖 Generated with Claude Code