Skip to content

Fix durable task flaky integration test#1784

Open
siri-varma wants to merge 10 commits into
dapr:masterfrom
siri-varma:users/svegiraju/fix-durable-task-test
Open

Fix durable task flaky integration test#1784
siri-varma wants to merge 10 commits into
dapr:masterfrom
siri-varma:users/svegiraju/fix-durable-task-test

Conversation

@siri-varma

@siri-varma siri-varma commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Description

Fixes flaky Durable Task integration tests and adds a CI retry safety net so transient infrastructure hiccups don't fail otherwise-green runs.

Test fixes (DurableTaskClientIT)

longTimeStampTimer — The timer's fire time was computed from the test's wall clock at method start, and the test then asserted an exact internal sub-timer count (counter >= 4). The number of sub-timers a long timer is split into depends on the span from the orchestration's creation time to the deadline; worker startup + scheduling latency shrank that span under CI load, occasionally dropping a sub-timer and failing expected <true> but was <false>. Fix: anchor the timer to the orchestration's own replay-safe clock via ctx.getCurrentInstant() (what createTimer(Duration) does internally) so the span is always exactly delay, and assert the splitting invariant (counter >= 3) instead of a latency-sensitive exact count.

waitForInstanceStartThrowsException — Scheduling the instance on a background thread raced waitForInstanceStart: if the wait reached the sidecar first it failed with "no such instance exists"; if the schedule won, the worker started the instance within 2s and nothing was thrown. Fix: schedule with a future start time so the instance deterministically exists but stays Pending, guaranteeing waitForInstanceStart blocks to its 2s deadline and throws TimeoutException.

CI retries (build.yml, validate.yml)

Wrap the flaky integration/validation commands in nick-fields/retry@v3 (max_attempts: 2, timeout_minutes: 30) at job granularity:

  • Durable Task — retry the mvnw ... verify step; continue_on_error keeps the sidecar-log upload and the outcome-gated fail step intact.
  • Spring Boot 3.5 / 4.0 build legs — retry the integration-test command; continue_on_error is left off so an exhausted retry still fails the step, preserving the existing failure() artifact uploads and job-failure behavior.
  • Auto Validate — a single retry around all example validations (reruns the batch once on failure) rather than per-example.

Retries are a safety net for transient failures (sidecar/gRPC connect races, example runs), not a substitute for the root-cause test fixes above.

Issue reference

[add issue number if this closes one]

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

  • Code compiles correctly
  • Created/updated tests
  • Extended the documentation

Signed-off-by: Siri Varma Vegiraju <siri.varma@outlook.com>
@siri-varma siri-varma requested review from a team as code owners July 9, 2026 22:10
@siri-varma

Copy link
Copy Markdown
Contributor Author

@javier-aliaga durable task has a flaky test. Fixed it as part of this PR

@siri-varma

Copy link
Copy Markdown
Contributor Author

nvm. It failed again

Signed-off-by: Siri Varma Vegiraju <siri.varma@outlook.com>
… clock

The assertion on the internal sub-timer count (counter >= 4) was flaky under
CI load. The absolute timer deadline was computed on the test's wall clock at
method start, but the number of sub-timers created for a long timer depends on
the span from the orchestration's *creation time* to that deadline. Worker
startup + scheduling latency shrinks that span, and when it exceeded ~1s one
sub-timer was dropped, producing counter == 3 and failing 'expected <true> but
was <false>' at line 387.

Anchor the deadline to the orchestration's own replay-safe clock via
ctx.getCurrentInstant() (equivalent to what createTimer(Duration) does
internally), so the timer always spans exactly the intended delay regardless of
startup latency, and assert the splitting invariant (counter >= 3) instead of a
latency-sensitive exact count.

Signed-off-by: Siri Varma Vegiraju <siri.varma@outlook.com>
@siri-varma

Copy link
Copy Markdown
Contributor Author

@javier-aliaga need one review on this pr please

Wrap the flaky integration/validation steps in nick-fields/retry@v3
(max_attempts: 2) to absorb transient CI failures:

- build-durabletask: retry the mvnw integration-tests step; switch the
  failure gate to the action's outcome output and bump the job timeout
  to 45m to fit two attempts.
- validate: retry each mm.py example validation step (dir folded into
  the command since uses: steps can't set working-directory).

Signed-off-by: Siri Varma Vegiraju <siri.varma@outlook.com>
@siri-varma siri-varma changed the title Fix durable task flaky integration teest Fix durable task flaky integration test Jul 10, 2026
Set per-attempt timeout_minutes to 30 on all nick-fields/retry blocks
(durable task + auto-validate) and revert the build-durabletask job
timeout back to its original 30m. The 30m cap is applied per step; no
job-level cap is added to the validate job since it runs 19 example
validations sequentially.

Signed-off-by: Siri Varma Vegiraju <siri.varma@outlook.com>
Collapse the 19 per-example nick-fields/retry blocks into a single
retry step that runs all example validations. On failure the whole
validation batch reruns once, matching the job-level retry granularity
(no per-example retries).

Signed-off-by: Siri Varma Vegiraju <siri.varma@outlook.com>
Wrap the sb3.5 and sb4 integration-test commands in nick-fields/retry
(max_attempts: 2, timeout_minutes: 30). continue_on_error is left off
so an exhausted retry still fails the step, keeping the existing
failure() artifact uploads and job-failure behavior intact.

Signed-off-by: Siri Varma Vegiraju <siri.varma@outlook.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces flakiness in Durable Task integration tests by anchoring timer behavior to the orchestration’s replay-safe clock and making a timeout-based test deterministic, and it adds a CI retry wrapper around selected flaky integration/validation commands.

Changes:

  • Update DurableTaskClientIT to compute timestamp timers from ctx.getCurrentInstant() and relax assertions to invariant-based checks.
  • Make waitForInstanceStartThrowsException deterministic by scheduling an instance with a future start time and asserting TimeoutException.
  • Wrap Durable Task integration tests, Spring Boot integration tests, and example validations in nick-fields/retry@v3 to reduce transient CI failures.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
durabletask-client/src/test/java/io/dapr/durabletask/DurableTaskClientIT.java Fixes two flaky Durable Task integration tests by removing timing races and using orchestration time.
.github/workflows/validate.yml Consolidates example validations into a single retried step to mitigate transient failures.
.github/workflows/build.yml Adds retries around integration test steps and adjusts failure gating logic for Durable Task tests.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 99 to 101
- name: Fail the job if tests failed
if: env.TEST_FAILED == 'true'
if: steps.durabletask_tests.outputs.outcome == 'failure'
run: exit 1
@codecov

codecov Bot commented Jul 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.02%. Comparing base (26b2327) to head (006f4db).

Additional details and impacted files
@@            Coverage Diff            @@
##             master    #1784   +/-   ##
=========================================
  Coverage     77.02%   77.02%           
  Complexity     2324     2324           
=========================================
  Files           245      245           
  Lines          7198     7198           
  Branches        760      760           
=========================================
  Hits           5544     5544           
  Misses         1288     1288           
  Partials        366      366           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants