UseStorageLock does not throttle failed-message retries across multiple instances — poisoned messages are retried at FailedRetryInterval/replicas cadence

[tamazbagdavadzespacege]

Summary

With UseStorageLock = true and N pods of the same service, a message whose subscriber consistently throws is retried roughly every FailedRetryInterval / N seconds instead of once per FailedRetryInterval. With 12 replicas and FailedRetryInterval = 60 we observe retries spaced ~2–15 s apart on the same MessageId, with Retries incrementing 1‑per‑attempt, originating from different pods.

Steps to Reproduce

1. Configure CAP with PostgreSQL + Kafka, defaults, UseStorageLock = true.
2. Add a subscriber that always throws
3. Start ≥ 2 replicas of the same service
4. Publish one message to its topic.
5. After Retries >= 3 (the in-process inner retry loop is exhausted) and Added > FallbackWindowLookbackSeconds ago, observe in your logs that the subscriber is invoked roughly every 60 / N seconds, not every 60 seconds.

Expected Behavior

With UseStorageLock = true, a single failed message should be retried at most once per FailedRetryInterval cluster-wide, regardless of replica count. (Or, equivalently, there should be a documented mechanism to achieve this.)

Actual Behavior

No response

Log Output

CAP Configuration

Transport Used

Kafka

Storage Provider

PostgreSQL

Environment

CAP 8.4.1

Additional Context

No response

[yang-xiaodong]

Thanks for the clear report .

What you’re seeing is consistent with CAP’s current retry scheduling model when UseStorageLock = true:

it coordinates who runs the retry scan at a given moment (single scanner at a time),

but it does not currently guarantee a strict per-message cluster-wide cooldown of exactly one retry per FailedRetryInterval.

So this is not “concurrent duplicate execution” of the same retry attempt, but rather different replicas acquiring the lock at different time slices, which can make retries appear more frequent as replica count increases.

For now, we consider this an expected limitation of the current design, not a correctness bug in locking itself.

That said, your expectation is valid, and we agree this should be clearer in docs. We will:

clarify the UseStorageLock semantics in documentation, and

evaluate an optional “strict retry throttle” mode.

Implementing strict cluster-wide pacing would likely require stronger centralized coordination semantics (for example, lease/visibility-style control) and/or additional persisted retry scheduling state (e.g., extra database fields and update rules), which introduces notable design and maintenance complexity.
Because of that, we do not plan to change the default behavior at this time.

Thanks again for raising this — it’s valuable feedback.