fix: IATA-filtered packet lists stop paginating early - #100
Merged
446564 merged 1 commit intoAug 19, 2026
Merged
Conversation
A packet repeats once per observer in the site scan, so grouping can collapse a scan_depth window to fewer packets than the page asked for. That short page set hasMore=false and stranded all older history: YOW dead-ended after 45 packets, ~6 minutes back. Report scan saturation and the floor the scans covered, and treat a saturated short page as more data. Clamp the cursor to the floor so paging on cannot skip the band the scan never read.
446564
approved these changes
Aug 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
The IATA-filtered packet list from #84 walks each site's observation log and dedups. A packet appears once per observer, so a scan of
(limit+1)*8observations can collapse to fewer distinct packets than the page asked for — and a short page was being read as end of data:That's a hard stop, not a display quirk. On my instance YOW (27 observers, ~8–9 hear a given packet) dead-ended after 3 pages / 292 packets with 13 hours of history still in the table and unreachable by scrolling. It flips with page size rather than failing consistently —
limit=10andlimit=20returned full pages and still saidhasMore: false, whilelimit=100was fine.The query now reports whether any site's scan hit
scan_depth(scan_saturated) and the timestamp above which every requested site is covered (scan_floor). A saturated short page keepshasMoretrue, and the cursor is clamped to the floor so paging on can't skip the band a saturated scan never read — worst case a packet repeats and the client dedups by hash.Scan depth itself is unchanged; this is only about not claiming the data has run out.
Same walk back through YOW, before and after:
limit=200: dead end at page 3 (292 packets) → reaches a 13.5h-old packet at page 15 (2988 packets)limit=50: dead end at page 3 → page 59Page latency unchanged at ~41ms.