Repository navigation
fix: stabilize flaky snapshot bootstrapping tests - #1953
Open
LautaroPetaccio wants to merge 2 commits into
Open
LautaroPetaccio wants to merge 2 commits into
LautaroPetaccio wants to merge 2 commits into
Conversation
Each test asserted on server2's post-bootstrap state synchronously right after the bootstrap-finished signal, but bootstrap keeps settling: under CI load a transient inter-server fetch failure makes the first attempt fail and schedules a retry. Poll those assertions via the existing `awaitUntil` helper instead of reading once. Also replace the exact `markSnapshotAsProcessed` call-count assertions with distinct-set-of-marked-hashes checks: with `has` mocked false, a retry re-marks the same snapshots, so the count varies across runs while the set does not.
LautaroPetaccio
enabled auto-merge (squash)
July 23, 2026 19:21
decentraland-bot
requested changes
Jul 23, 2026
decentraland-bot
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for stabilizing this suite. I found one reliability issue that should be addressed before merge.
Findings
- P1: The new set-based spy assertions can pass as soon as every expected hash has been observed once, but they do not prove the bootstrap retry work has settled before the spy is reset and reused for the second sync phase. Since the PR explains that transient bootstrap fetch failures can schedule retry work after the bootstrap-finished signal, late calls from the first phase can either contaminate the second phase after
mockReset()or let the test move forward before the condition it is trying to stabilize is actually drained. Please wait on a real settled/idle condition for the bootstrap retry work before resetting the spy, or assert durable DB/deployment state for the phase instead of reusing this spy across unsettled async work.
Security review: no security issues found; this is test-only.
CI: validations/title/quay passed; test-content is still pending.
Reviewed by Jarvis 🤖 · Requested by Lautaro Petaccio (<@U025WCHLMN3>) via Slack
…e spy The set-based spy assertion passed as soon as each hash was observed once and did not drain the bootstrap retry work, so a late phase-1 re-mark could fire after mockReset() and contaminate the second sync phase. Drop the markSnapshotAsProcessed spy entirely and poll server2's processed_snapshots DB state per phase instead — idempotent under retries and scoped to each phase.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
stabilizes
test/integration/syncronization/bootstrapping.spec.ts, a pre-existing flaky sync suite. surfaced while validating catalyst#1952 (a bootstrapping test failed in its ci) but unrelated to that bump — the suite stubs the validator.root cause
each test asserts on server2's synced state immediately after
startProgramAndWaitUntilBootstrapFinishes, but bootstrap keeps settling after that signal: under ci load a transient inter-server fetch (ENOENT/fetch failed) makes the first bootstrap attempt fail and schedules a retry. so the assertions read incomplete state, and which test trips varies per run (which is why patching one assertion just moved the failure to a sibling).fix
awaitUntilhelper (20s window, well within the 120s integration timeout) instead of reading once, so the bootstrap retry has time to complete.markSnapshotAsProcessedcall-count assertions (toBeCalledTimes(3)/(2)) with distinct-set-of-marked-hashes checks. the test mockssnapshotStorage.has → false, so a bootstrap retry re-marks the same snapshots and the call count varies across runs; the distinct set of marked hashes does not. asserting the set preserves the "exactly these snapshots processed, nothing extra" intent while tolerating retries.verification
ran the suite repeatedly locally — green. before the fix it failed ~1 in 2 (either an early-read on a state assertion, or the count assertion polling out at 20s because a retry pushed the count past the expected value).
no production code touched; test-only.