Repository navigation
[Dataflow Streaming] Invalidate local state cache on KeyCommitTooLargeException. - #40956
Conversation
When a work item commit exceeds operational commit size limits in StreamingModeExecutionContext, a KeyCommitTooLargeException is recorded and the commit request is truncated, discarding state updates, timers, and output messages. However, the worker's WindmillStateCache and ReaderCache previously retained the dirty state updates and cached reader for the key. Because Windmill commits the truncation request with CommitStatus.OK, neither onCompleteCommit nor processWork's exception handler would invalidate the caches, leaving uncommitted/stale state in the worker cache. This change invokes invalidateCache() when KeyCommitTooLargeException occurs, ensuring all cached state and readers for the key are invalidated.
|
R: @scwhittle |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #40956 +/- ##
============================================
- Coverage 79.83% 75.98% -3.86%
- Complexity 0 2277 +2277
============================================
Files 494 259 -235
Lines 83575 14385 -69190
Branches 0 1482 +1482
============================================
- Hits 66722 10930 -55792
+ Misses 16853 2764 -14089
- Partials 0 691 +691
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
I don't think that this would actually trigger in the common truncation case because we would not used the cached objects due to non-advancing work token, see ReaderCache.acquireReader and WindmillStateCache.ForKey.updateTokens. But it seems better to clean it up where we detect the error than wait for or rely on that. |
When a work item commit exceeds operational commit size limits in StreamingModeExecutionContext, a KeyCommitTooLargeException is recorded and the commit request is truncated, discarding state updates, timers, and output messages.
However, the worker's WindmillStateCache and ReaderCache previously retained the dirty state updates and cached reader for the key. Because Windmill commits the truncation request with CommitStatus.OK, neither onCompleteCommit nor processWork's exception handler would invalidate the caches, leaving uncommitted/stale state in the worker cache.
This change invokes invalidateCache() when KeyCommitTooLargeException occurs, ensuring all cached state and readers for the key are invalidated.