Skip to content

fix: stabilize flaky snapshot bootstrapping tests - #1953

Open
LautaroPetaccio wants to merge 2 commits into
mainfrom
fix/stabilize-bootstrapping-sync-tests
Open

LautaroPetaccio wants to merge 2 commits into
mainfrom
fix/stabilize-bootstrapping-sync-tests

Conversation

@LautaroPetaccio

Copy link
Copy Markdown
Contributor

stabilizes test/integration/syncronization/bootstrapping.spec.ts, a pre-existing flaky sync suite. surfaced while validating catalyst#1952 (a bootstrapping test failed in its ci) but unrelated to that bump — the suite stubs the validator.

root cause

each test asserts on server2's synced state immediately after startProgramAndWaitUntilBootstrapFinishes, but bootstrap keeps settling after that signal: under ci load a transient inter-server fetch (ENOENT / fetch failed) makes the first bootstrap attempt fail and schedules a retry. so the assertions read incomplete state, and which test trips varies per run (which is why patching one assertion just moved the failure to a sibling).

fix

  • poll every post-bootstrap "settled-state" assertion — processed snapshots, reported deployments, persisted failed deployments — via the existing awaitUntil helper (20s window, well within the 120s integration timeout) instead of reading once, so the bootstrap retry has time to complete.
  • replace the exact markSnapshotAsProcessed call-count assertions (toBeCalledTimes(3) / (2)) with distinct-set-of-marked-hashes checks. the test mocks snapshotStorage.has → false, so a bootstrap retry re-marks the same snapshots and the call count varies across runs; the distinct set of marked hashes does not. asserting the set preserves the "exactly these snapshots processed, nothing extra" intent while tolerating retries.

verification

ran the suite repeatedly locally — green. before the fix it failed ~1 in 2 (either an early-read on a state assertion, or the count assertion polling out at 20s because a retry pushed the count past the expected value).

no production code touched; test-only.

Each test asserted on server2's post-bootstrap state synchronously right
after the bootstrap-finished signal, but bootstrap keeps settling: under
CI load a transient inter-server fetch failure makes the first attempt
fail and schedules a retry. Poll those assertions via the existing
`awaitUntil` helper instead of reading once.

Also replace the exact `markSnapshotAsProcessed` call-count assertions
with distinct-set-of-marked-hashes checks: with `has` mocked false, a
retry re-marks the same snapshots, so the count varies across runs while
the set does not.
@LautaroPetaccio
LautaroPetaccio enabled auto-merge (squash) July 23, 2026 19:21

@decentraland-bot decentraland-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for stabilizing this suite. I found one reliability issue that should be addressed before merge.

Findings

  • P1: The new set-based spy assertions can pass as soon as every expected hash has been observed once, but they do not prove the bootstrap retry work has settled before the spy is reset and reused for the second sync phase. Since the PR explains that transient bootstrap fetch failures can schedule retry work after the bootstrap-finished signal, late calls from the first phase can either contaminate the second phase after mockReset() or let the test move forward before the condition it is trying to stabilize is actually drained. Please wait on a real settled/idle condition for the bootstrap retry work before resetting the spy, or assert durable DB/deployment state for the phase instead of reusing this spy across unsettled async work.

Security review: no security issues found; this is test-only.

CI: validations/title/quay passed; test-content is still pending.


Reviewed by Jarvis 🤖 · Requested by Lautaro Petaccio (<@U025WCHLMN3>) via Slack

…e spy

The set-based spy assertion passed as soon as each hash was observed once
and did not drain the bootstrap retry work, so a late phase-1 re-mark
could fire after mockReset() and contaminate the second sync phase. Drop
the markSnapshotAsProcessed spy entirely and poll server2's
processed_snapshots DB state per phase instead — idempotent under retries
and scoped to each phase.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants