Fix durable task flaky integration test#1784
Conversation
Signed-off-by: Siri Varma Vegiraju <siri.varma@outlook.com>
|
@javier-aliaga durable task has a flaky test. Fixed it as part of this PR |
|
nvm. It failed again |
Signed-off-by: Siri Varma Vegiraju <siri.varma@outlook.com>
… clock The assertion on the internal sub-timer count (counter >= 4) was flaky under CI load. The absolute timer deadline was computed on the test's wall clock at method start, but the number of sub-timers created for a long timer depends on the span from the orchestration's *creation time* to that deadline. Worker startup + scheduling latency shrinks that span, and when it exceeded ~1s one sub-timer was dropped, producing counter == 3 and failing 'expected <true> but was <false>' at line 387. Anchor the deadline to the orchestration's own replay-safe clock via ctx.getCurrentInstant() (equivalent to what createTimer(Duration) does internally), so the timer always spans exactly the intended delay regardless of startup latency, and assert the splitting invariant (counter >= 3) instead of a latency-sensitive exact count. Signed-off-by: Siri Varma Vegiraju <siri.varma@outlook.com>
|
@javier-aliaga need one review on this pr please |
Wrap the flaky integration/validation steps in nick-fields/retry@v3 (max_attempts: 2) to absorb transient CI failures: - build-durabletask: retry the mvnw integration-tests step; switch the failure gate to the action's outcome output and bump the job timeout to 45m to fit two attempts. - validate: retry each mm.py example validation step (dir folded into the command since uses: steps can't set working-directory). Signed-off-by: Siri Varma Vegiraju <siri.varma@outlook.com>
Set per-attempt timeout_minutes to 30 on all nick-fields/retry blocks (durable task + auto-validate) and revert the build-durabletask job timeout back to its original 30m. The 30m cap is applied per step; no job-level cap is added to the validate job since it runs 19 example validations sequentially. Signed-off-by: Siri Varma Vegiraju <siri.varma@outlook.com>
Collapse the 19 per-example nick-fields/retry blocks into a single retry step that runs all example validations. On failure the whole validation batch reruns once, matching the job-level retry granularity (no per-example retries). Signed-off-by: Siri Varma Vegiraju <siri.varma@outlook.com>
Wrap the sb3.5 and sb4 integration-test commands in nick-fields/retry (max_attempts: 2, timeout_minutes: 30). continue_on_error is left off so an exhausted retry still fails the step, keeping the existing failure() artifact uploads and job-failure behavior intact. Signed-off-by: Siri Varma Vegiraju <siri.varma@outlook.com>
There was a problem hiding this comment.
Pull request overview
This PR reduces flakiness in Durable Task integration tests by anchoring timer behavior to the orchestration’s replay-safe clock and making a timeout-based test deterministic, and it adds a CI retry wrapper around selected flaky integration/validation commands.
Changes:
- Update
DurableTaskClientITto compute timestamp timers fromctx.getCurrentInstant()and relax assertions to invariant-based checks. - Make
waitForInstanceStartThrowsExceptiondeterministic by scheduling an instance with a future start time and assertingTimeoutException. - Wrap Durable Task integration tests, Spring Boot integration tests, and example validations in
nick-fields/retry@v3to reduce transient CI failures.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| durabletask-client/src/test/java/io/dapr/durabletask/DurableTaskClientIT.java | Fixes two flaky Durable Task integration tests by removing timing races and using orchestration time. |
| .github/workflows/validate.yml | Consolidates example validations into a single retried step to mitigate transient failures. |
| .github/workflows/build.yml | Adds retries around integration test steps and adjusts failure gating logic for Durable Task tests. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| - name: Fail the job if tests failed | ||
| if: env.TEST_FAILED == 'true' | ||
| if: steps.durabletask_tests.outputs.outcome == 'failure' | ||
| run: exit 1 |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1784 +/- ##
=========================================
Coverage 77.02% 77.02%
Complexity 2324 2324
=========================================
Files 245 245
Lines 7198 7198
Branches 760 760
=========================================
Hits 5544 5544
Misses 1288 1288
Partials 366 366 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Description
Fixes flaky Durable Task integration tests and adds a CI retry safety net so transient infrastructure hiccups don't fail otherwise-green runs.
Test fixes (
DurableTaskClientIT)longTimeStampTimer— The timer's fire time was computed from the test's wall clock at method start, and the test then asserted an exact internal sub-timer count (counter >= 4). The number of sub-timers a long timer is split into depends on the span from the orchestration's creation time to the deadline; worker startup + scheduling latency shrank that span under CI load, occasionally dropping a sub-timer and failingexpected <true> but was <false>. Fix: anchor the timer to the orchestration's own replay-safe clock viactx.getCurrentInstant()(whatcreateTimer(Duration)does internally) so the span is always exactlydelay, and assert the splitting invariant (counter >= 3) instead of a latency-sensitive exact count.waitForInstanceStartThrowsException— Scheduling the instance on a background thread racedwaitForInstanceStart: if the wait reached the sidecar first it failed with "no such instance exists"; if the schedule won, the worker started the instance within 2s and nothing was thrown. Fix: schedule with a future start time so the instance deterministically exists but staysPending, guaranteeingwaitForInstanceStartblocks to its 2s deadline and throwsTimeoutException.CI retries (
build.yml,validate.yml)Wrap the flaky integration/validation commands in
nick-fields/retry@v3(max_attempts: 2,timeout_minutes: 30) at job granularity:mvnw ... verifystep;continue_on_errorkeeps the sidecar-log upload and the outcome-gated fail step intact.continue_on_erroris left off so an exhausted retry still fails the step, preserving the existingfailure()artifact uploads and job-failure behavior.Retries are a safety net for transient failures (sidecar/gRPC connect races, example runs), not a substitute for the root-cause test fixes above.
Issue reference
[add issue number if this closes one]
Checklist
Please make sure you've completed the relevant tasks for this PR, out of the following list: