Add opt-in stale-on-error recovery from retained Redis values

[lan17]

Priority

P2 — Medium — opt-in availability behavior for selected use cases. Default behavior remains unchanged.

Summary

Allow DialCache to return a retained Redis value when the source of truth (SoT) rejects.

For an opted-in use case:

• F is the resolved remote ttlSec and therefore the logical freshness age.
• M is the resolved staleOnErrorMaxAgeSec and therefore the absolute recovery limit measured from the Redis frame's creation time.

age < F        fresh
F <= age < M   stale but recoverable after an SoT rejection
age >= M       unavailable

Redis keeps one existing value key for up to M. The normal read path uses the frame's Redis-server created_at timestamp to enforce F. DialCache tries the SoT after a logical miss and only rereads the same key with recovery age M after that SoT attempt rejects.

This is stale-on-error, not stale-while-revalidate. It does not return stale while the SoT is healthy and does not introduce background refresh.

Baseline: origin/main at v0.14.1.

Current behavior

The Redis frame already contains the data needed for logical age:

[version: u8][created_at_ms: u64 big-endian][encoding: u8][serialized payload]

Writes obtain created_at_ms from Redis TIME. Today the Redis TTL is also the freshness boundary: untracked reads ignore created_at, and tracked reads unpack it only for watermark comparison. Once Redis expires the value, a later SoT rejection propagates even if retaining the prior payload would have been acceptable for that use case.

Relevant source:

• Redis frame and Lua scripts
• Semantic Redis client boundary
• Redis cache read/write path
• Source fallback and publication path
• Fallback error/timer telemetry
• Shared one-year duration ceiling

Goals

• Keep stale-on-error explicit, per-use-case, runtime-overridable, and default-off.
• Reuse one Redis value key and the existing envelope timestamp.
• Return fresh data normally and stale data only after an actual SoT rejection.
• Preserve invalidation watermarks as the authority for tracked values.
• Preserve existing cache-plumbing fail-open and caller-wait bounds.
• Never turn a recovered stale value into a fresh publication.
• Make recovery, resource cost, and failure outcomes observable.

Non-goals

• Stale-while-revalidate or background refresh.
• A circuit breaker, SoT health state, retry cooldown, or second SoT attempt.
• Separate fresh and stale Redis data keys.
• Extending Redis retention on reads.
• Returning stale after the initial Redis read itself failed or timed out.
• Weakening tracked invalidation or adding stronger read-after-invalidation guarantees.
• Hiding the SoT failure from telemetry.

Decision status

Confirmed

• The feature is opt-in and default-off.
• Redis uses one value key, not parallel fresh and stale keys.
• The existing Redis-server created_at envelope timestamp determines logical age.
• Public/runtime configuration uses one absolute maximum-age scalar: staleOnErrorMaxAgeSec?: number.
• A positive staleOnErrorMaxAgeSec enables recovery; omission is off by default and inherits in an overlay; 0 explicitly disables an inherited policy.
• Enabled writes use physical Redis retention M = staleOnErrorMaxAgeSec.
• Normal Redis reads return a miss once age reaches F = ttlSec[CacheLayer.REMOTE].
• After the SoT rejects, DialCache may reread the same Redis key while age is below M.
• Stale recovery remains subject to the current invalidation watermark.
• A recovered value is never written back, refreshed, or otherwise promoted to fresh.
• Backward compatibility with readers that do not understand logical expiry is not required.

Confirmed for v1 implementation

• Initial scope uses Redis as the stale reservoir; process-local storage is not extended beyond F.
• Fresh and recovery reads use the invocation's once-resolved F/M policy snapshot.
• All SoT rejections qualify initially, including FallbackTimeoutError; no error predicate in v1.
• A recovered value may be memoized only within an already-enabled request-local scope.
• Add one bounded stale-recovery metric; do not add a new public error class or duration histogram.
• Reuse the current key/frame with a coordinated rollout: upgrade all readers before enabling writes with retention beyond F.

Proposed API

new DialCacheKeyConfig({
  ttlSec: {
    [CacheLayer.REMOTE]: 300, // logically fresh for 5 minutes
  },
  ramp: {
    [CacheLayer.REMOTE]: 100,
  },
  staleOnErrorMaxAgeSec: 3_600, // recoverable for up to 1 hour
});

Proposed contract:

• In a static/default config, omission means disabled.
• In a runtime overlay, omission inherits the default value.
• 0 explicitly disables an inherited stale-on-error policy.
• A positive safe integer opts in and supplies absolute M, measured from the frame's Redis-server created_at timestamp.
• After overlay resolution, enabled recovery requires 0 < F < M <= 31,536,000 seconds.
• Invalid stale-on-error policy disables recovery while preserving ordinary fresh caching and established config-error telemetry.
• DialCacheKeyConfig.disabled() explicitly sets staleOnErrorMaxAgeSec to 0.
• No separate boolean, stale duration map, stale ramp, or predicate hook in v1. Existing config-provider cohorting can return a positive value or 0.
• Recovery participates only when the remote serving layer itself resolves enabled. Ramp-zero/dark Redis, policy-disabled, malformed, or config-error paths cannot serve stale.

maxAgeSec is intentional terminology: M is an absolute age since creation. Redis TTL is the physical retention mechanism and may reflect the policy in effect when the value was written.

Freshness and retention model

For an opted-in successful remote write:

created_at             = Redis TIME at write
logical fresh-until    = created_at + F
logical recovery limit = created_at + M
physical Redis PX      = M

Reads use Redis server time:

age = Redis TIME - created_at

fresh read:    return payload only when age < F
recovery read: return payload only when age < M

At exact age F, the normal read is a miss. At exact age M, recovery is a miss. Missing, short, malformed, unsupported-version, invalid-encoding, or deserialization-failing values never qualify.

Both resolved ages remain within the current 365-day duration ceiling:

0 < F < M <= 31,536,000 seconds

Unlike fixed 2× retention, this does not halve the supported fresh TTL range or require an internal two-year duration. Static invalid combinations fail fast; invalid runtime policy follows the established runtime config fail-open behavior for recovery while ordinary fresh caching remains available.

With the current-policy timestamp model:

• Lowering F immediately makes older retained values stale sooner.
• Increasing F can reclassify an existing retained value as fresh while it remains physically present.
• Lowering M immediately makes values at or beyond the new limit ineligible, although Redis may retain an older write until its physical expiry.
• Increasing M cannot extend or resurrect an existing key beyond the physical TTL chosen at write time; the next successful write receives the longer retention.
• Enabling affects future writes; existing PX F values may disappear before recovery is needed.
• Disabling stops recovery immediately, although earlier PX M keys remain until physical expiry.

M is an upper bound, not a guarantee: invalidation, eviction, or other Redis removal may make a value unavailable earlier.

Execution flow

flowchart TD
  A[Fresh Redis read: maxAge F] -->|fresh hit| B[Return fresh]
  A -->|logical or physical miss| C[Call SoT]
  A -->|Redis error or timeout| D[Call SoT; recovery forbidden]
  C -->|success| E[Publish normally; Redis PX is M]
  C -->|rejection| F[Recovery Redis read: maxAge M]
  F -->|eligible and valid| G[Return retained value without publication]
  F -->|miss, timeout, error, or decode failure| H[Throw original SoT rejection]
  D -->|rejection| H

Loading

Detailed semantics:

• Fresh hits never call the SoT.
• A logical stale value plus SoT success publishes and returns the new SoT value normally; no stale reread occurs.
• The recovery read happens at failure time. A slow failure cannot retain an earlier candidate past M, and an invalidation during the SoT attempt is rechecked.
• A concurrent writer may refresh the key before recovery. Returning that now-fresh cached value is valid.
• If recovery fails in any way, throw the exact original SoT rejection object/value. Redis, timeout, protocol, or serializer failures must not replace it.
• A recovered value does not write Redis, extend TTL, populate process-local cache, or enter a shadow-fill/publication path.
• The next independent invocation retries the SoT immediately. There is no recovery cooldown.

Why reread after failure

Holding a stale payload from the first lookup would transfer and deserialize it on every logical miss even when the SoT succeeds. It could also cross the hard-age boundary or be invalidated while the SoT is running. Rereading only after rejection keeps the healthy path lean and rechecks current Redis state.

Deliberate nil tradeoff

The normal Lua result remains payload-or-nil. A logical stale value, a cold miss, an invalid frame, and an invalidated value are not distinguished in core. Consequently every qualifying SoT rejection after a definitive normal Redis miss performs one recovery read, even when no stale value exists.

A tri-state fresh | stale_candidate | miss result could avoid cold-miss rereads but would expand the semantic client/reply contract. Keep payload-or-nil initially unless outage measurements show that extra miss traffic warrants the additional state.

Lua and Redis protocol

Use two semantic read modes against the same Redis data key:

fresh:          maxAgeMs = F
stale recovery: maxAgeMs = M

They may be separate registered scripts or one parameterized implementation; that packaging is internal. The correctness contract is:

local max_age_ms = validate_max_age(ARGV[1])
local value_or_header = redis.call(...)
-- validate frame version and created_at
local redis_time = redis.call("TIME")
local now_ms = tonumber(redis_time[1]) * 1000
  + math.floor(tonumber(redis_time[2]) / 1000)
local created_at = struct.unpack(">I8", ...)

if now_ms - created_at >= max_age_ms then
  return false
end

-- tracked mode must require a valid watermark and created_at > watermark
return payload

Required properties:

• maxAgeMs is validated as a positive finite integer within the existing one-year ceiling.
• Redis server time, not caller wall time, decides age.
• Tracked reads atomically enforce both age and watermark eligibility and remain primary-routed.
• Missing or malformed tracked watermarks remain misses.
• Value and watermark keys retain their Redis Cluster hash-tag colocation.
• Recovery never uses GETEX, PEXPIRE, or another TTL-extending command.
• UTF-8, binary, and cached undefined behavior remains unchanged.

Header-first optimization

A naive logical-age check uses GET and materializes the whole retained payload merely to return a miss. For larger payloads, prefer inspecting the first nine frame bytes with GETRANGE 0 8, then fetching the full frame only when the age is eligible. A full read must still validate the complete minimum frame, encoding, and payload before returning it.

For tracked normal reads, age may be checked before fetching the watermark because an age-ineligible value cannot serve. Every payload that is actually returned—fresh or recovered—must validate the watermark.

Semantic client and adapters

The semantic Redis boundary must require adapters to enforce logical age atomically. A minimal shape is:

interface RedisReadRequest {
  readonly valueKey: string;
  readonly watermarkKey?: string;
  readonly maxAgeMs: number;
}

DialCacheRedisClient.read() can continue returning RedisCachePayload | null; core knows whether it requested F or M. The write request continues carrying the physical TTL, which is F or M according to the resolved policy.

Adapter requirements:

• node-redis and Valkey GLIDE pass and validate maxAgeMs in their registered scripts.
• Tracked reads preserve primary routing and atomic watermark enforcement.
• Existing AbortSignal, native finite-budget, reply validation, binary ownership, and script lifecycle guarantees remain.
• Custom semantic clients must implement the new required age contract. Compatibility with pre-feature clients is not promised.
• Public redis-protocol exports and fake/test clients advance together.

An adapter that ignores maxAgeMs would treat the retained stale window as fresh, so making this argument optional is unsafe.

Invalidation

Tracked fresh and recovery reads may return a payload only when:

age < requested maxAge
and the watermark exists and parses
and created_at > watermark

Consequences:

• A watermark equal to or newer than created_at blocks both normal and recovery reads.
• Missing or malformed watermarks block recovery.
• An invalidation between the initial miss and the recovery read blocks recovery.
• A future watermark blocks recovery while it blocks ordinary tracked reads and writes.
• The tracked write already derives watermark retention from the physical value TTL plus the existing 60-second margin; passing M therefore retains it for approximately M + 60s.
• Recovery cannot strengthen the existing invalidation contract into a synchronous recall of a response after the recovery script has already returned.

Liveness and failure semantics

• The initial Redis read retains its effective remoteReadTimeoutMs.
• The SoT retains its existing fallback deadline.
• A permitted recovery read receives one independent effective remoteReadTimeoutMs budget.
• Worst-case waiting may therefore include the initial read budget, the fallback budget, and one recovery-read budget.
• A recovery timeout may settle later at the adapter/server; late settlement is consumed and cannot publish.
• If the initial Redis read itself failed or timed out, do not perform another Redis operation after a later SoT rejection.
• If the initial read returned a definitive miss but recovery later fails, rethrow the original SoT rejection.
• If FallbackTimeoutError qualifies, its late SoT settlement remains ignored and unpublished.

Coalescing and request scope

• The existing leader performs the SoT call and at most one recovery read.
• Process/request followers share the same recovered value/reference; reads and metrics are not follower-multiplied.
• The flight clears after recovery succeeds or the original error propagates.
• The next independent flight retries the SoT.
• The leader's once-resolved config snapshot governs its followers, consistent with existing coalescing.

Open decision: request-local caching currently memoizes every successful lower-chain result. If a recovered value is memoized, “next independent invocation” means outside that outer enabled request scope. Shared Redis/process-local TTLs are never refreshed.

Local and shadow interactions

Recommended initial scope: Redis is the only stale reservoir.

• Process-local entries keep their existing physical F lifetime.
• After a local miss/expiry, the normal Redis/SoT/recovery flow can still recover the retained remote value.
• Local-only use cases cannot use stale-on-error in v1.
• A recovered remote value is not inserted into process-local storage.
• Remote ramp 0 and dark/shadow Redis reads remain observational and cannot serve stale.
• A normal fresh-hit detached shadow source_error remains unchanged and does not invoke stale recovery.
• A recovered value is not accepted as shadow SoT S and does not schedule a fill or comparison.

Extending process-local entries to M would require dual-expiry LRU semantics, memory accounting, and tracked invalidation safety. Defer it unless a concrete Redis-free use case needs it.

Observability

Existing telemetry must remain truthful:

• The normal logical-expiry read records a remote miss.
• The SoT rejection still records error="fallback", inFallback=true exactly once, even if the caller ultimately receives stale data.
• The existing fallback duration remains recorded.
• Recovery request, get-duration, miss, read-timeout/read-error, and deserialization error accounting is defined explicitly.

Add one bounded signal for recovery, preferably an optional adapter hook for source compatibility:

type StaleRecoveryOutcome =
  | "served"
  | "miss"
  | "read_error"
  | "read_timeout"
  | "deserialization_error";

staleRecovery?(labels: {
  cacheNamespace: string;
  useCase: string;
  keyType: string;
  outcome: StaleRecoveryOutcome;
}): void;

Record one terminal outcome per attempted recovery. Do not include key IDs, values, payloads, native error names, or other unbounded labels. Avoid a warning for every served request because an SoT outage could turn it into a log storm.

CPU, network, and memory

Directional local Redis 8.8 benchmarks compared the exact current scripts with timestamp-aware scripts using warm EVALSHA, 50 clients, pipeline 64, one hot key, loopback networking, and persistence disabled:

Path	Current server CPU	Timestamp-aware CPU	Change
64-byte untracked fresh hit	1.57 µs	2.29 µs	+46%
64-byte tracked fresh hit	2.17 µs	2.77 µs	+28%
4 KiB untracked fresh hit	12.44 µs	13.09 µs	+5%

The age check adds roughly 0.6–0.7 µs of fixed Redis CPU per present read in this synthetic setup. At 100k present reads/s, that is approximately 0.06–0.07 CPU core. Treat these as directional, not production capacity promises.

For a logically stale 4 KiB value:

Initial normal-read behavior	Server CPU
Today's physical miss	1.25 µs
Naive full GET then age rejection	7.27 µs
Nine-byte GETRANGE header then age rejection	2.10 µs

Header-first reduced stale-check CPU by about 71% in this setup while adding roughly 2% versus the naive timestamp-aware implementation on a fresh 4 KiB hit. Benchmark the final script across supported servers and payloads before locking this optimization.

Recovery traffic is approximately:

extra reads/sec = Redis-reaching lookups
                × normal miss rate
                × qualifying SoT error rate

Because the proposed payload-or-nil result does not expose stale-candidate status, this includes cold and otherwise-ineligible misses. Under healthy error rates it should be tiny. During a synchronized expiry plus SoT outage, read command count can approach 2×, and the recovery read returns another full payload.

Changing PX F to PX M does not increase write command count or add a second value. Longer retention can increase resident key/value memory; under a steady one-shot or high-cardinality workload without other limits, the upper directional multiplier is approximately M / F. It also lengthens tracked-watermark residency and can indirectly add eviction, allocator, and active-expiry pressure. Hot keys refreshed before F see much less growth. Stale availability remains best-effort under eviction.

Rollout and compatibility

An old DialCache reader treats physical Redis presence as freshness. If a new writer stores the same v1 key through age M, an old reader can serve ages F..M as ordinary fresh hits.

Because backward compatibility is explicitly out of scope, the simplest rollout is coordinated:

1. Ship readers/adapters that enforce logical age while staleOnErrorMaxAgeSec remains omitted or 0.
2. Upgrade the entire reader fleet.
3. Only then enable selected use cases so writes begin using physical retention M.
4. Monitor Redis CPU, memory, evictions, source errors, and recovery outcomes.
5. Disabling recovery stops it immediately; retained keys expire naturally.

If mixed-version rollout must be safe, use a new value-key/frame version instead. Do not add dual-read/dual-write migration unless that requirement changes. Tracked generations must continue sharing the authoritative watermark identity.

Alternatives considered

Parallel fresh and stale keys

Rejected for the initial design. It keeps age logic off healthy reads but duplicates payload writes/storage, creates divergence and partial-write cases, complicates invalidation, and expands Cluster/script handling.

Hold a stale candidate across the SoT call

Rejected. It transfers/deserializes stale data even when the SoT succeeds and risks crossing the hard boundary or missing an intervening invalidation.

Tri-state initial read

Deferred. It can avoid a useless recovery lookup for cold/invalid misses, but adds a new semantic result and adapter/reply contract. Measure the simple payload-or-nil design first.

Process-local stale retention

Deferred. It requires soft/hard LRU expiry plus explicit tracked invalidation safety. Redis already provides the cross-process stale reservoir.

Acceptance criteria

Configuration and boundaries

• Default-off and feature-off behavior is byte/TTL/flow compatible with current behavior.
• Static/runtime positive staleOnErrorMaxAgeSec, explicit runtime 0, overlay inheritance, and DialCacheKeyConfig.disabled() behave as documented.
• Malformed runtime configuration follows the established fail-open/config telemetry contract without disabling valid fresh caching.
• Resolved 0 < F < M <= 365 days is validated, including exact maximum, equality, negative, fractional, unsafe-integer, and overflow cases.
• Exact-age tests cover F-1, F, M-1, and M.
• Feature-off writes use PX F; feature-on writes use PX M.
• Lowering M takes effect immediately; increasing M does not resurrect or extend values written with a shorter physical TTL.

Core behavior

• Fresh hit returns without SoT or recovery.
• Logical stale plus SoT success publishes/returns the SoT value and performs no recovery read.
• Logical stale plus qualifying synchronous throw or async rejection performs exactly one bounded recovery read and can return retained data.
• Cold, malformed, unsupported-version, invalid-encoding, deserialization-failing, missing-watermark, malformed-watermark, and invalidated values never qualify.
• Recovery miss/error/timeout/decode failure rethrows the identical original SoT rejection.
• Initial Redis read error/timeout never triggers a second Redis operation.
• Recovery never writes Redis, extends TTL, populates process-local cache, or shadow-fills.
• The next independent flight retries the SoT.
• One leader performs the SoT and recovery; all followers share the result/reference without multiplying reads or metrics.
• Late source or Redis settlement cannot publish or cause an unhandled rejection.

Invalidation, shadow, and rollout

• Tracked fresh and recovery reads require a present valid watermark and created_at > watermark.
• Invalidation during the SoT attempt prevents stale recovery.
• Tracked reads remain primary-routed and Cluster-colocated.
• Watermark retention covers M plus the existing margin.
• Ramp-zero/dark and all-disabled paths cannot serve stale.
• Fresh-hit detached shadow behavior remains unchanged; recovered stale never becomes accepted shadow source data.
• Coordinated rollout or an isolated value-key version prevents old readers from misclassifying M values.

Adapters, tests, and performance

• node-redis, Valkey GLIDE, fake clients, custom-client type fixtures, and public Lua exports implement the required age contract.
• Unit coverage includes config, boundaries, rejection identity, coalescing, publication suppression, request-local decision, and observer failure isolation.
• Live coverage passes across Redis 6.2, Valkey 8, both bundled adapters, and the existing three-primary Redis 7 Cluster.
• Packed ESM/CJS/TypeScript consumers validate the semantic-client and config surface.
• Benchmarks report fresh-hit CPU, logical-stale miss cost, outage recovery QPS/bytes, coalesced fanout, MEMORY USAGE, watermark residency, expiration, and eviction behavior without an arbitrary timing threshold.

Confirmed implementation decisions

• Confirm all SoT rejections versus exclusions for FallbackTimeoutError, abort/cancellation-shaped errors, or arbitrary rejection values.
• Confirm request-local memoization of a recovered value.
• Confirm Redis-only stale retention and no process-local/local-only recovery in v1.
• Confirm current-policy F/M age checks using the invocation's once-resolved configuration.
• Confirm a separate recovery-read remoteReadTimeoutMs budget.
• Confirm one bounded staleRecovery metric and no per-request success warning.
• Confirm coordinated same-key rollout; otherwise choose a new value-key/frame version.

Implementation

• feat: add stale-on-error Redis recovery #121 — draft implementation and validation evidence.

Related work

• Design optional cross-process cache stampede protection #7 — cross-process stampede protection; mentions stale-while-revalidate only as a different weaker-consistency strategy.
• Add opt-in Redis Pub/Sub coherence for tracked process-local caching #106 and feat: coordinate process-local invalidation #107 — optional coordinated process-local invalidation; relevant if local stale retention is later added.
• Validate TTL and invalidation duration arithmetic consistently #33 and fix: cap cache durations at one year #96 — shared duration-bound enforcement.