Skip to content

[Dataflow Streaming] Invalidate local state cache on KeyCommitTooLargeException. - #40956

Merged
scwhittle merged 1 commit into
apache:masterfrom
arunpandianp:invalidateCaches
Nov 3, 2026
Merged

scwhittle merged 1 commit into
apache:masterfrom
arunpandianp:invalidateCaches

Conversation

@arunpandianp

Copy link
Copy Markdown
Contributor

When a work item commit exceeds operational commit size limits in StreamingModeExecutionContext, a KeyCommitTooLargeException is recorded and the commit request is truncated, discarding state updates, timers, and output messages.

However, the worker's WindmillStateCache and ReaderCache previously retained the dirty state updates and cached reader for the key. Because Windmill commits the truncation request with CommitStatus.OK, neither onCompleteCommit nor processWork's exception handler would invalidate the caches, leaving uncommitted/stale state in the worker cache.

This change invokes invalidateCache() when KeyCommitTooLargeException occurs, ensuring all cached state and readers for the key are invalidated.

When a work item commit exceeds operational commit size limits in StreamingModeExecutionContext, a KeyCommitTooLargeException is recorded and the commit request is truncated, discarding state updates, timers, and output messages.

However, the worker's WindmillStateCache and ReaderCache previously retained the dirty state updates and cached reader for the key. Because Windmill commits the truncation request with CommitStatus.OK, neither onCompleteCommit nor processWork's exception handler would invalidate the caches, leaving uncommitted/stale state in the worker cache.

This change invokes invalidateCache() when KeyCommitTooLargeException occurs, ensuring all cached state and readers for the key are invalidated.
@arunpandianp

Copy link
Copy Markdown
Contributor Author

R: @scwhittle

@github-actions github-actions Bot added runners dataflow size/s S: 10-49 lines changed labels Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

@codecov

codecov Bot commented Nov 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.98%. Comparing base (12bbb17) to head (ef1a3f7).
⚠️ Report is 39 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master   #40956      +/-   ##
============================================
- Coverage     79.83%   75.98%   -3.86%     
- Complexity        0     2277    +2277     
============================================
  Files           494      259     -235     
  Lines         83575    14385   -69190     
  Branches          0     1482    +1482     
============================================
- Hits          66722    10930   -55792     
+ Misses        16853     2764   -14089     
- Partials          0      691     +691     
Flag Coverage Δ
java 75.98% <100.00%> (?)
python ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@scwhittle

Copy link
Copy Markdown
Contributor

I don't think that this would actually trigger in the common truncation case because we would not used the cached objects due to non-advancing work token, see ReaderCache.acquireReader and WindmillStateCache.ForKey.updateTokens.

But it seems better to clean it up where we detect the error than wait for or rely on that.

@scwhittle
scwhittle merged commit 6ef970c into apache:master Nov 3, 2026
18 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dataflow runners size/s S: 10-49 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants