api,web: stop a quiet capture source reading as a stalled path - #802
Open
juan-malbeclabs wants to merge 19 commits into
Open
api,web: stop a quiet capture source reading as a stalled path#802juan-malbeclabs wants to merge 19 commits into
juan-malbeclabs wants to merge 19 commits into
Conversation
A stalled series says nothing about whose silence it is, and on a per-event feed it is usually nobody's: a sports capture source is one market, and a market that closes mid-window goes quiet on every path at once. Both paths of edge-kalshi-sports-tob read 'stalled 1/29' on the same instance, and that verdict outranks 'behind', so it hid the one finding on the row that was about the path. demoteEdgeMulticastQuietCaptureSources compares the paths recording a capture source at one node — the path-parity key, with the channel folded out for the same reason — and flags a stall every path there shares as the source's. A vantage with one path is left alone, and a path is only excused at one source while it is delivering at another, so a feed that stopped everywhere still reads stalled rather than advancing. Also: the Sequence count meant two things in one slot. '6/31' is faults over series and a bare '3' was the series count, which reads as three faults. A count with no faults now takes the multiplier the Heard column already uses.
|
🔗 Preview: https://pr-802.data.malbeclabs.com |
…es a cycle Msg/s, Peer and Sequence are folded from refresher payloads on a ten-minute cycle, and nothing on the page bounded how old that could get. readPageCache does not look at updated_at, the folds do not use readPageCacheWithAge, and STALE_AFTER_SECS was only ever applied to the counter buckets. The age lived inside the Sequence tooltip and nowhere at all for Msg/s and Peer, so a refresher that died left the page asserting a verdict indefinitely with no visible tell. Observed on staging: a sequence payload aging 9m, 10m, 11m with byte-identical numbers, and a gapped verdict rendering as current throughout. Past STALE_AFTER_SECS the column header now carries the age in amber and the values below it dim — the two-part treatment the Ingress column already gives a rate bucket that missed its window. An absent stamp is deliberately not staleness: that is a payload written before the API carried the clock, and dimming over it would invent the reading rather than report it. Msg/s and Peer get their own stamp, observations_as_of, rather than reading sequence_as_of. That one is the older of the two SEQUENCE legs, so sharing it would dim two columns over the market-by-price leg's staleness — a payload they do not come from and cannot be made stale by.
Measured on mainnet: of the three recorders of edge-kalshi-perps, one held ~91% of what the other two held, on both paths, sustained in every minute of the window. Nothing on the page could say so. Capture-node parity is the only recorder-side check there is, its floor is half the median — 0.91 sails past it — and the group verdict it feeds is not rendered anyway. edgeMulticastNodeCoverage computes the same question on the observations plane, where the counts are exact. It is the transpose of path parity: that one fixes the vantage and compares the paths, this one fixes the path and compares the vantages, and the pair is what makes either result attributable — a deficit in both is a path short at one recorder, a deficit only here is the recorder. It renders on the group row beside the reconcile link, because a node short on every path is a statement about the vantage and no publisher line owns it. Two bounds. A node is listed only when it is behind on every path of the group it records: a deficit confined to one path IS that path's finding, Peer already carries it on that line, and repeating it here would name the recorder for what a publisher's branch did. And the floor is 0.95 rather than the path check's 0.98, because the window has no exclusion of its trailing edge and every node is filtered by one clock, so a recorder whose ingest lags reads as a deficit of exactly that lag — at 0.98 eighteen seconds of it would report as loss.
Four findings from reading the live page against the feeds tables. A gap measured at ONE recorder cannot name whose loss it is. Comparing the two paths there clears that recorder's host and nothing more: the branch into it is upstream of the comparison and downstream of everything else, so a loss on the branch reads exactly like a loss on the path. Measured: 13 books gapped at the only node recording market-by-price, while the plane with three vantages found that same path intact at a second one — the loss was two burst minutes on the branch. Market-by-price runs one recorder on every group, so every gap this page reports is single-vantage. GapNodes carries the count and the tooltip says so; the verdict stays gapped, because data was lost either way. 'behind' had the same one-instance sensitivity the stalled verdict just lost. A sports node compares 29-33 capture sources and one failing pair marked the whole line. edgeMulticastPathParityBehindShare gates the verdict on Behind/Compared at a quarter: a path with one comparison still fires at 1 of 1, and a real branch deficit is indiscriminate and clears the floor nearly everywhere. What tipped it off is sports reading 0.988 and 0.967 on the two paths at once, which one systematic deficit cannot produce — the better path of each pair is 1.0 by construction, so each is losing at a different market. The ratio is now truncated toward zero rather than rounded, so the digit shown is never better than the measurement: a 0.9797 printed as "0.980" beside a badge saying it was under the 0.98 floor. And the instance tooltip carries gap_messages/messages as a loss rate where there were gaps. Not the banned display of gap_messages as a fault count — a rate is not a count — and the only severity available, since gap_books saturates at the channel's instrument count: thirteen of thirteen and one of thirteen print the same badge. It stays in the tooltip; putting it on the badge would change what the column asserts.
ben-dz
reviewed
Aug 24, 2026
ben-dz
left a comment
ben-dz
left a comment
Contributor
There was a problem hiding this comment.
One blocking issue: the quiet-capture-source demotion excuses a recorder that dies mid-window as the venue going quiet, so a dead recorder's stalls leave the tally and the lines read advancing — keying the aliveness guard on (path, node) closes it. The new recorder_coverage badge is also the one folded signal with no staleness treatment, which is the tell the observations_as_of commit exists to add.
Findings not anchored to the current diff:
web/src/components/edge-multicast-page.tsx:853— Minor —recorder_coverageis folded from the same cache entry theobservations_as_ofcommit exists to age, but its badge carries no age and never dims, so a dead refresher leaves an amber1/3 recordersverdict on the row indefinitely.observationsAsOfAgeis already a prop onGroupRow: pass it in, appendcomputedLine(...)to the tooltip and dim onpayloadStale.
A recording node that stops ingesting mid-window is the same shape as a venue going quiet, one level down: every series it holds goes stale together, so every pair at that vantage is quiet on both paths. Keyed on the path alone the paths still looked alive — they are delivering at the OTHER recorders — so a dead recorder was excused as the venue: stalled=0, capture_source_quiet=4, and the lines read advancing over it. The aliveness guard is now keyed on (path, recording node): a path is only excused where it is itself delivering. And the recorder badge is folded from the same cache entry Msg/s and Peer are, so it now ages with them. An amber verdict that cannot go stale is one that outlives the refresher that produced it.
Contributor
Author
|
Both findings addressed in The blocking one — the aliveness guard. You were right about the shape and the fix. One consequence worth flagging since it is a deliberate trade: on a group with a single capture source there is nothing at that vantage to prove the recorder is alive, so a genuinely closed market keeps its stall. That is the conservative side — no evidence, no excuse — and it matches how the rest of these checks refuse to decide without a comparison. The recorder badge. Fixed, and the irony is noted: it was the one folded signal with no staleness treatment, in the PR that adds staleness treatment. It now takes CI is green on the head commit, including |
ben-dz
approved these changes
Aug 24, 2026
ben-dz
left a comment
ben-dz
left a comment
Contributor
There was a problem hiding this comment.
LGTM, except that the recorder-side check did not get the share gate its sibling just gained.
The sibling check gained a share gate one commit ago and this one did not. Breadth and share answer different questions: behind on every path is what separates a bad vantage from a bad path, and breadth alone is not enough, because a node short at ONE capture source is short on both paths there and clears the every-path test on two comparisons out of the ~58 a sports group makes. The group row went amber over a single market while every publisher line stayed green.
The Sequence column could say a path lost data but not when, and its gap count saturates: perps carries 13 instruments and one lost datagram on the delta port un-anchors most of them, so it pins at 13/13 after a single episode and reads as total failure where the truth was ~10 losses of a few seconds each. Measured on mainnet the two Kalshi perps paths never gapped in the same second — 10 seconds of loss on one, 64 on the other, none on both — so the redundancy covered every episode and the feed itself lost nothing. No counter can express that; one axis shared by a group's publisher lines shows it. The episodes come from one more aggregate on the scan the L2 coverage refresher already runs (1.8s for every source), collapsed into contiguous runs. The top-of-book plane has no gap marker, so it carries no timeline rather than an empty one.
Gap-free share, gaps per hour, worst recovery and time since the last gap, all derived from the episodes already in the payload — no query, no new field. No loss percentage, deliberately. An episode is time a book spent un-anchored, not a count of datagrams that failed to arrive, so the gap marker cannot supply the denominator. The wire sequence can: frame_sequence is a dense per-(publisher, channel, port) counter and perps reads 0.367% against 0.369% on its two paths. But on every sports channel the two independent paths are missing the SAME frame numbers — 193 against 192, 180 against 180 — which is the numbering having holes, not the network dropping packets. Shipping it would paint 29 sports feeds at 12% loss permanently.
Adds the loss half of the gap metrics: parts per million lost, missing
updates per minute, sequence gaps per hour, worst break and p99 break, on
top of the gap-free share and recovery time already derived from the
timeline.
The counter matters more than the metrics. Three candidates were measured
on mainnet and only one survives:
- status_after = 'gap' counts TIME a book spent un-anchored and has no
denominator at all.
- frame_sequence looks like the obvious one and is not. Its holes are
identical on both redundant paths of every sports feed — 193 against
192, 180 against 180 — and two independent paths do not lose the same
datagrams. Reading them as loss reports ~12% on 29 healthy feeds.
- per_instrument_seq is dense per (path, book, reset generation). Golf
reads zero on both paths where frame_sequence claimed 11.6%, and where
there is real loss the paths differ: perps at 36 against 9.
One more query on the same refresher, 1.5s fleet-wide. A lane with no
sequence reports no denominator rather than a rate of zero, so "measured
clean" stays distinguishable from "not measured".
…at once The gap timeline could say a path lost data but not whose loss it was. Each recorder now gets its own line on the same axis, and a last line marks the seconds where TWO OR MORE lost at the same moment. That reading is the point: a mark on one recorder with clear tracks beside it is that recorder's branch, and a mark on several at once is not. Measured on mainnet, was-rec1 lost on BOTH paths of tob_edge_kalshi_perps in the same second — 97 and 87 messages — which is a recorder-side fault no per-path view can name. The reference is the union of what the nodes recorded, and that bounds the claim: a message no node received is in nobody's set and can never be reported missing. So "all of them" is not a state this can reach, and two or more is as close as the plane gets to a loss upstream of the recorders. It also sidesteps what blocks absolute gap counting here — a row exists only where the top of the book changed, so a message that moved nothing leaves a legitimate hole. That hole is absent at EVERY recorder, so it never enters the reference. Top-of-book only, and not by choice: market-by-price runs a single recorder on every group, and one node has no peer to be measured against.
The query emitted a row only for a node that was missing something, which drops the line the comparison is made of. "was lost 267" says nothing without "cmh lost 0" beside it, and a window in which one node alone lost anything rendered no comparison at all — the strip needs two lines to be one. Now it array-joins the path's whole node universe and counts the misses per node, so a recorder that recorded everything its peers did gets a row with an empty track. Paths with a single recorder are filtered out instead: no peer, no reference, and every hole in their numbering is the top-of-book plane's own legitimate hole. That filter makes it cheaper too — 1.2s against 2.2s fleet-wide, since it prunes every single-recorder sports path.
…line apart Two display faults on the recorder strips. The metro was resolved with an any() over a group that spans every node of a sequence, so one arbitrary metro landed on all three rows: they rendered cmh/cmh/cmh, and a comparison whose rows cannot be told apart is not a comparison. It is a property of the node, so it is resolved per node and joined on. And the 2+ line was amber, which on this page means degraded — so the line that says a loss is NOT one recorder's branch read as a milder fault than the red rows above it, the exact inversion of what it means. Violet instead: different family for a different kind of statement, with a rule separating it from the observations it is a conclusion about.
The first strip carried no label, stacked among labelled ones, and read as their total or as a fourth recorder. It is neither: it is time the publisher's books spent un-anchored, from the market-by-price plane at one recording node, where the rows below are messages a recorder did not receive on the top-of-book plane at three. It is labelled "book" — by what it measures, not by a vantage it does not have — and a rule now separates the three statements: book impact, per-recorder loss, and the crossing. Rows also sit a pixel apart instead of two. The marks are read by vertical alignment, so the distance between rows is the distance between the two things being compared.
The query died on every ten-minute refresh in production — `(total) memory limit exceeded: would use 7.20 GiB` — so the strips this branch adds never rendered there. CI could not see it: the tests run against a ClickHouse with six rows, not 1.7M aggregate groups. It is a genuinely large aggregate: one group per (path, sequence) over the window, 1.7M of them across the 29 top-of-book capture sources, wanting more than 1.2 GiB. Unbounded it competed for the server's whole budget and lost. SETTINGS max_memory_usage + max_bytes_before_external_group_by lets the GROUP BY spill to disk instead: measured 3.7s inside 700 MiB against the live table, same six rows. Same shape and numbers as networkHealthQuerySettings, which exists for this reason. Narrowing the scan was tried first and does not help, which is worth recording so nobody tries it again: 28 of the 29 sources have a single recorder and can never produce a comparison, but filtering them out — by path tuple, or by a source IN subquery — left memory unchanged at ~480 MiB, because the cost is the group count and ClickHouse builds it before either filter prunes. A bitmask in place of groupUniqArray saved 2%. And the failure was invisible, which was my design error: the measurement is additive so a failure is a WARN, and the page rendered a failed comparison exactly like an inapplicable one — nothing. It now carries recorder_loss_unavailable and the strip reads "not measured", the same distinction the Sequence column already makes between a gap-checked ok and an advancing one.
Of the 29 top-of-book capture sources, one has three recorders and 28 have one. Those 28 can never produce a row — every path of theirs fails the `length(all_nodes) > 1` test at the end — but they contributed 954k of the 1.7M (path, sequence) groups the aggregate built to get there. A cheap pre-query names the sources worth comparing, and the main query filters on those literals. It has to be a separate query. As a path-tuple `IN`, or as a `source IN (SELECT ...)` subquery, ClickHouse builds the full aggregate before either can prune and the memory does not move; explicit literals prune. The pre-query groups by source rather than by sequence, so its state is one entry per source holding three node ids. This does NOT fix the OOM seen on the PR preview, and the previous commit's settings do not either. That environment proxies through a local 8 GiB / 2 vCPU ClickHouse pod which is already pinned at its CPU limit and failing other queries (`bulk device metrics` dies against the same ceiling); the query needs more headroom than is free there, narrowed or not. Production has no local ClickHouse — lake-prod runs API, indexer, neo4j and temporal only, and talks to Cloud directly, where this measures 3.7s inside 700 MiB. Both changes stand on their own regardless: a third of the rows read, and the aggregate no longer built over sources that cannot answer the question.
A feed travels two redundant paths, so a gap on one is the redundancy working and reports as that path's own. A gap on BOTH in the same second is data the feed did not deliver, and no publisher line can say it — a line only ever sees its own loss. That is exactly what the group row is for on this page: what no line can report. It intersects the per-publisher gap seconds already in the payload, so it costs no query. Keyed on (capture source, recording node) before intersecting, for the two reasons edgeMulticastPathParity is keyed the same way: without the node a recorder that stopped ingesting has every series go stale together and reads as every path failing at once, and without the capture source two unrelated losses at two markets in the same second read as one shared outage. The channel stays out — the paths publish under different ids and would never meet. Measured on mainnet over six hours of edge-kalshi-perps-mbp: 22 seconds where both paths lost together against 83 and 84 where one did. Rare enough to be a finding, frequent enough to be worth a badge. Also gives the single-recorder case words. Three states — compared, failed, and nothing to compare — had two renderings between them, so a sports group with one vantage rendered exactly like a broken feature, and that is how the first person to look at one read it. It now says so, and that absence is an operational fact in its own right: with a single vantage a gap cannot be attributed to the path rather than to that recorder's own branch, which is the same limit GapNodes already bounds.
kalshi_bbo_observations carries the top-of-book series AND the BBO a market-by-price publisher derives, and perps runs three recorders on each, so both can be compared. Only tob was, which left the mbp rows — the ones that actually gap — with no recorder comparison at all. That extension was held back on an unexplained reading: measured 2026-08-24, cmh and was were each missing ~9.7% of the mbp plane while dub missed 3 of 42,878, and Dublin receiving better than Columbus is backwards enough not to ship. It does not reproduce now that was is off its saturated t3.medium: all three sit under 0.4% (cmh 0.315%, dub 0.039%, was 0.025%). The earlier reading was a symptom of the box, not of the plane. Unlike the Sequence column, this needs no gap marker and so has no reason to prefer one plane: the measurement is which sequence numbers a node recorded that its peers did. It also required rekeying the fold. A publisher line is now (destination group, publisher address) rather than the address alone, because the tob and mbp halves of a feed are two addresses on ONE tunnel — the same publisher appears in both groups, and keying on the address would have summed its losses across them and printed the total on each of its rows. Pinned by a test, since the failure reads as plausible numbers rather than as an error.
The measurement kept dying in the preview: ~2M (path, sequence) groups wanting about 5 GiB against a 7.2 GiB proxy shared with other queries that were already failing. Bounding it and narrowing the scan both helped and neither was enough, because the shape was wrong. The strip is 96 pixels wide for a fifteen-minute window. One pixel is already about nine seconds, so a per-second comparison keyed on individual sequence numbers computed a resolution nobody can see. At a ten-second bucket the same query returns ~1,100 rows and completes in 5.8s inside the preview pod that was OOMing — verified there, not against the feeds service, which is the mistake that sent the first two attempts after the wrong bottleneck. It also drops a real source of noise. A message arriving 12:00:00.999 at one recorder and 12:00:01.001 at another is not loss, but per-second bucketing counts it as a deficit in one bucket and a surplus in the next; at ten seconds the boundary is crossed a tenth as often against the same signal. Two things change meaning, so the cache key goes to v3. The comparison is now a message count rather than a set difference over sequence numbers, and the reference is the BEST-PLACED recorder rather than the union of what all of them saw — the same reference edgeMulticastNodeCoverage already uses, and for the same reason: against a mean, a pair sinks with its faulty half and reports both as roughly fine. `reference_seqs` is `reference_messages` accordingly, because it no longer counts sequences. What it gives up is naming WHICH message a node missed, which nothing displayed. What it keeps is how much each recorder is short, and when.
This reverts commit b4a232f. The bucket comparison was cheap and wrong, and it put a wall of red on the page that meant nothing. Comparing message COUNTS per bucket has a noise floor above the signal it exists to find. Three recorders never receive exactly the same count in a ten-second window — normal jitter leaves one a few messages below the max — so "short if below the best" marked two of three nodes in nearly every bucket and the 2+ line fired constantly. The real loss it was meant to surface is 0.02%; the method's own noise is several times that. Bucketing the EXACT comparison does not save it either, and it is worth recording why. Bucketed on arrival time, the same message lands in different buckets at different recorders — 12:00:09.999 here, 12:00:10.001 there — and each counts as missing from the other: measured, 66-86 of 90 buckets short against 30 messages of real loss. Bucketed on the sequence instead, which every recorder agrees on, that noise goes away (0-14 buckets) but the partial buckets at the window's edges produce their own, and the totals still do not match the whole-window comparison. The exactness was the point. It is what let this find a recorder losing 8.2% and then confirm it back down to 0.02%, and it is why the strips could be trusted at all. The per-sequence set difference is expensive because it is exact; where it cannot run, "not measured" is the honest answer and the page now says so.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Corrections to what the Edge Multicast page claims, each found by reading the live page against its own rules and then checking the reading against the feeds tables. The common thread is attribution: several verdicts named a path or a feed for something a recorder, a branch, or a venue did.
A quiet capture source is not a stalled path
edge-kalshi-sports-tobreadstalled 1/29on the same instance on mainnet, andstalledoutranksbehind, so it hid the one finding on that row which was about the path.demoteEdgeMulticastQuietCaptureSourcesflags a stall every path at one vantage shares as the source's. The instance keepsstatus: stalled— the reading is unchanged, only its attribution moves — and thecapture_source_quiettally keeps it out ofstalled, which is what the verdict reads.(capture source, recording node), the path-parity key: the node is in it so a recorder quiet on everything is not mistaken for the venue, and the channel is out of it because the two paths publish under different channel ids.stalledinstead of excusing itself intoadvancing.A gap measured at one recorder cannot name whose loss it is
GapNodescarries the count, and the tooltip says so. The verdict staysgapped— data was lost either way — and what narrows is the sentence the page is allowed to say about whose loss it was. The real fix is a second market-by-price recorder, which is not work this repo can do.behindhad the same one-instance sensitivitystalledjust lostedgeMulticastPathParityBehindSharegates the verdict onBehind / Comparedat a quarter: a path with one comparison — every perps group — still fires at 1 of 1, and a real branch deficit is indiscriminate and clears the floor nearly everywhere rather than at a single market.0.988and0.967on the two paths at once, which one systematic deficit cannot produce — the better path of each pair is 1.0 by construction, so each path is losing at a different capture source, the signature of per-source noise near the volume floor.A recording node short on every path is now reported
edge-kalshi-perps, one held ~91% of what the other two held, on both paths, sustained in every minute of the window. Nothing on the page could say so — capture-node parity is the only recorder-side check there is, its floor is half the median, and the group verdict it feeds is not rendered anyway.edgeMulticastNodeCoverageasks the same question on the observations plane, where the counts are exact. It is the transpose of path parity — that one fixes the vantage and compares the paths, this one fixes the path and compares the vantages — and the pair is what makes either result attributable.Peercarries it there. The floor is 0.95, not the path check's 0.98, because the window does not exclude its trailing edge — at 0.98, eighteen seconds of ingest lag would report as loss.The folded columns age, and dim when the refresher misses a cycle
readPageCachedoes not look atupdated_at, andSTALE_AFTER_SECSwas only ever applied to the counter buckets. Observed on staging: a payload aging 9m → 10m → 11m with byte-identical numbers, and agappedverdict rendering as current throughout.STALE_AFTER_SECSthe column header carries the age in amber and the values dim. An absent stamp is deliberately not staleness — that is a payload written before the API carried the clock.observations_as_of;sequence_as_ofis the older of the two sequence legs, so sharing it would dim two columns over a payload they do not come from.Two display corrections
0.9797printed as0.980beside a badge saying it was under the0.98floor.6/31is faults over series, but a bare3was the series count and read as three faults. A count with no faults takes the×Nmultiplier the Heard column already uses.gap_messages / messagesas a loss rate where there were gaps. Not the banned display ofgap_messagesas a fault count, and the only severity available, sincegap_bookssaturates at the channel's instrument count: thirteen of thirteen and one of thirteen print the same badge. It stays in the tooltip — putting it on the badge changes what the column asserts.Diff Breakdown
Five measurement rules with their guards, the tests that pin them, and the plumbing that carries two clocks and a receiver-side finding to the UI.
Key files (click to expand)
api/handlers/edge_multicast_observations.go— the recorder-side check, the parity share gate, and the payload's own clockweb/src/components/edge-multicast-page.tsx— staleness for the folded columns, the recorder badge, the ratio truncation, the×Ncount, and the tooltips that carry every narrowed claimapi/handlers/edge_multicast_sequence.go— the quiet-capture-source demotion,GapNodes, and the loss-rate numeratorapi/handlers/edge_multicast.go—observations_as_ofandrecorder_coverageon the payloadTesting Verification
edge_multicast_quiet_source_test.goover the real grading path — a market closed on both paths, one path stalled while its peer records, a vantage with no peer, a feed stopped everywhere, and a gapped series a quiet peer must not excuse — plus two onGapNodes, including that a plane with no gap marker corroborates nothing. The two-path case uses the +100 channel offset, so it fails if the channel is ever keyed back into the comparison.edge_multicast_node_coverage_test.go, the first seeded with the measured mainnet counts (125,467 / 125,050 / 125,393 / 125,385 / 114,605 / 112,914): it asserts the lagging recorder is named at 0.90 and that the recorder carrying a 0.33% deficit on one path is not. The rest pin the refusals — one path short is not the node, a lone recorder is no comparison, a thin instance is not judged, channels are summed before vantages are compared.edge_multicast_publisher_health_test.go.TestGetEdgeMulticast_ObservationsCarryTheirOwnAsOfseeds the two cache entries thirty minutes apart and asserts each column reports its own clock.TestEdgeMulticastPathParity_SlowRecorderDoesNotFaultBothPaths, which is what establishes that a lagging recorder cancels out of the path ratio rather than contaminating it.