Repository navigation
[Prism] Fix flaky TestPersistBundle_ResidualResumeDelay bound - #40993
jonathansudhakar1 wants to merge 1 commit into
Conversation
The "parks with hold and schedules" subtest checked that the residual's release time is within (0, delay] of a nanosecond-precision time.Now() taken before PersistBundle. PersistBundle computes the release time from its own processing time, which is read later and truncated to milliseconds. When a millisecond boundary falls between the two reads, the release time lands a few microseconds past before+delay and the test fails (13 of 20,000 local runs; also seen on master CI). Bound the release time by the delay after before and after PersistBundle, both truncated to milliseconds the same way, which is the exact range of valid results. Fixes apache#40992 Generated-by: Claude Code
There was a problem hiding this comment.
📋 Review Summary
This PR resolves a flaky test failure in TestPersistBundle_ResidualResumeDelay/parks_with_hold_and_schedules (#40992) caused by comparing a millisecond-truncated mtime.Time sampled inside PersistBundle against a single nanosecond-precision time.Now() taken before the call. Bounding fireAt between [mtime.FromTime(before).Add(delay), mtime.FromTime(after).Add(delay)] accurately reflects the runner's millisecond-granularity processing time calculation while keeping the assertion strict.
🔍 General Feedback
- The root cause analysis and bounds check adjustment are sound, clean, and well-tested.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
|
Assigning reviewers: R: @jrmccluskey for label go. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #40993 +/- ##
===========================================
- Coverage 79.83% 56.60% -23.23%
===========================================
Files 494 877 +383
Lines 83575 154949 +71374
===========================================
+ Hits 66722 87710 +20988
- Misses 16853 65381 +48528
- Partials 0 1858 +1858
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
jrmccluskey
left a comment
There was a problem hiding this comment.
Broadly LGTM, although I do think the (0, delay] form for the error message in the failure case is a little friendlier for human readability.
| // PersistBundle schedules the release at the delay after its own processing | ||
| // time, which it reads between before and after, truncated to milliseconds. |
There was a problem hiding this comment.
Nit: we don't really need comments explaining the behavior here
Fixes #40992.
TestPersistBundle_ResidualResumeDelay/parks_with_hold_and_schedulesis flaky. It failed on master in this "Go tests" run withresidual release scheduled 5.000001877s after persisting, want within (0, 5s].Cause. The test takes
before := time.Now(), at nanosecond precision, and checks that the scheduled release timefireAtis within(0, delay]of it.PersistBundlecomputesfireAtasemNow.Add(delay). WithEnableRTC,emNowismtime.Now(), which is read a little later and truncated to milliseconds. When a millisecond boundary falls between the two reads,fireAtlands a few microseconds pastbefore + delay. The runner behaves correctly; only the test's bound is wrong.Change (test only). The test now also takes
after := time.Now()oncePersistBundlereturns. It checksfireAtagainst[mtime.FromTime(before).Add(delay), mtime.FromTime(after).Add(delay)]. That is the exact rangePersistBundlecan produce, so the check is still strict: a release time that is immediate, doubled, or off by a millisecond still fails.Testing
Run from
sdks/on macOS/arm64 with Go 1.26:go test ./go/pkg/beam/runners/prism/internal/engine/ -run 'TestPersistBundle_ResidualResumeDelay/parks_with_hold_and_schedules' -count=20000failed 13 times, with the same message as CI (for example,5.000001s after persisting). A separate-count=3000run failed once.-count=20000, run twice: 0 failures,ok.partitionResiduals, each run once against the new check, all failed as expected:fireAt := emNow.Add(r.Delay + time.Millisecond)failed.fireAt := emNow.Add(r.Delay - time.Millisecond)failed.fireAt := emNow.Add(2 * r.Delay)failed.go test ./go/pkg/beam/runners/prism/internal/engine/:ok(53.5s).go vetandgofmt -lon the package are clean.🤖 Generated with Claude Code