diff --git a/README.md b/README.md index 2b8ec06..680e525 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ Fine-grained TypeScript caching with explicit enabled contexts, request-local me - [Keys, ids, and extra dimensions](#keys-ids-and-extra-dimensions) - [Runtime config and ramp controls](#runtime-config-and-ramp-controls) - [Cache layers](#cache-layers) - - [Request-local cache](#request-local-cache) · [Process-local cache](#process-local-cache) · [Redis-backed TTL cache](#redis-backed-ttl-cache) · [Remote read deadlines](#remote-read-deadlines-and-async-liveness) · [Serialization](#serialization) · [Shadow validation](#shadow-validation) + - [Request-local cache](#request-local-cache) · [Process-local cache](#process-local-cache) · [Redis-backed TTL cache](#redis-backed-ttl-cache) · [Stale-on-error recovery](#stale-on-error-recovery) · [Remote read deadlines](#remote-read-deadlines-and-async-liveness) · [Serialization](#serialization) · [Shadow validation](#shadow-validation) - [Cached-value ownership](#cached-value-ownership) - [Targeted invalidation and watermarks](#targeted-invalidation-and-watermarks) - [Request coalescing](#request-coalescing) @@ -81,6 +81,7 @@ request-local cache -> process-local cache -> Redis cache -> fallback function - Process-local hits return immediately. - Process-local misses try Redis and populate the process-local cache on a Redis hit. - Redis misses call the fallback and attempt to populate Redis and, when active, the process-local cache. Tracked invalidation may suppress both publications. +- An opted-in Redis use case can retain a value beyond its logical freshness age and return it only when the source of truth rejects; see [Stale-on-error recovery](#stale-on-error-recovery). - Selected tracked Redis keys can execute non-serving [shadow work](#shadow-validation) that validates hits and fills clean misses, even before Redis is allowed to serve callers. - Redis read failures and timeouts are logged, counted in metrics, and fail open without attempting a second Redis operation. Redis write failures also fail open. `invalidateRemote` requires a configured Redis client; missing configuration and Redis failures are logged, counted, and rethrown so callers do not assume invalidation succeeded. - Cache-key construction and config-provider failures also fail open and run the fallback uncached. @@ -207,21 +208,21 @@ Instance-wide behavior is set through the `DialCache` constructor: | `metrics` | disabled | A `DialCacheMetricsAdapter` (see [Metrics](#metrics)). | | `logger` | `console` | Receives operational cache failures and opted-in confirmed shadow mismatch warnings (`debug`, `warn`, `error`). Synchronous throws and rejections from returned promises or thenables are isolated without being awaited. | -Per-invocation cache policy is a `DialCacheKeyConfig`: per-layer `ttlSec` and `ramp` maps keyed by `CacheLayer.LOCAL` (process-local) and `CacheLayer.REMOTE` (Redis), a `requestLocal` boolean, an optional `remoteReadTimeoutMs`, and an optional `shadow` group. `ShadowConfig` contains the independent shadow `ramp` percentage plus the default-off `logMismatches` control. +Per-invocation cache policy is a `DialCacheKeyConfig`: per-layer `ttlSec` and `ramp` maps keyed by `CacheLayer.LOCAL` (process-local) and `CacheLayer.REMOTE` (Redis), a `requestLocal` boolean, optional `remoteReadTimeoutMs` and `staleOnErrorMaxAgeSec` scalars, and an optional `shadow` group. `ShadowConfig` contains the independent shadow `ramp` percentage plus the default-off `logMismatches` control. Every cached definition or `getOrLoad()` invocation can provide an optional per-use-case `defaultConfig`. It is the baseline policy, and the `cacheConfigProvider` result is a sparse field-level overlay on that baseline. For cache enablement fields, precedence is runtime config, then `defaultConfig`, then DialCache's disabled baseline. For the remote-read deadline, precedence is runtime `remoteReadTimeoutMs`, `defaultConfig.remoteReadTimeoutMs`, `redis.readTimeoutMs`, then the 50 ms library default. -The disabled baseline sets `requestLocal` to false, leaves the process-local and Redis TTLs unset, and sets `shadow.ramp` to 0% with mismatch logging false. A shared layer with no effective TTL is disabled by policy. When a shared layer has an effective TTL but no effective ramp, its ramp defaults to 100%. Shadow work remains disabled unless `shadow.ramp` is explicitly greater than zero. +The disabled baseline sets `requestLocal` to false, leaves the process-local and Redis TTLs unset, explicitly sets `staleOnErrorMaxAgeSec` to `0`, and sets `shadow.ramp` to 0% with mismatch logging false. A shared layer with no effective TTL is disabled by policy. When a shared layer has an effective TTL but no effective ramp, its ramp defaults to 100%. Shadow work remains disabled unless `shadow.ramp` is explicitly greater than zero. `DialCacheKeyConfig` preserves an omitted `requestLocal` as `undefined` so the overlay can distinguish omission from an explicit `false`; the effective value still defaults to false after resolution. -A provider result of `null` (or defensive `undefined`) applies no overrides. An empty `DialCacheKeyConfig` and omitted runtime fields also inherit the baseline. Top-level fields, cache-layer leaves, and leaves inside `shadow` merge independently; an explicit `false` logging flag overrides an inherited `true`. Use explicit values to override inherited policy: `requestLocal: false` disables request-local caching and a layer ramp of `0` disables that shared layer. `DialCacheKeyConfig.disabled()` is the complete new-cache-invocation kill switch in one call: request-local and shadow work off, shadow logging off, and both shared layers ramped to 0. It does not cancel already-admitted work, and explicit maintenance operations such as `invalidateRemote()` remain available. To stop new cache-invocation Redis reads and fills while preserving other runtime settings, explicitly set both `ramp.remote` and `shadow.ramp` to `0`; the remote ramp alone stops serving but does not override an inherited nonzero shadow ramp. +A provider result of `null` (or defensive `undefined`) applies no overrides. An empty `DialCacheKeyConfig` and omitted runtime fields also inherit the baseline. Top-level fields, cache-layer leaves, and leaves inside `shadow` merge independently; an explicit `false` logging flag overrides an inherited `true`. Use explicit values to override inherited policy: `requestLocal: false` disables request-local caching, `staleOnErrorMaxAgeSec: 0` disables inherited stale recovery, and a layer ramp of `0` disables that shared layer. `DialCacheKeyConfig.disabled()` is the complete new-cache-invocation kill switch in one call: request-local and stale recovery off, shadow work and logging off, and both shared layers ramped to 0. It does not cancel already-admitted work, and explicit maintenance operations such as `invalidateRemote()` remain available. To stop new cache-invocation Redis reads and fills while preserving other runtime settings, explicitly set both `ramp.remote` and `shadow.ramp` to `0`; the remote ramp alone stops serving but does not override an inherited nonzero shadow ramp. -DialCache validates `defaultConfig` when `cached()` registers a definition and whenever `getOrLoad()` is invoked: TTLs must be positive safe integers no greater than 31,536,000 seconds (a fixed 365-day duration), remote-read deadlines must be positive safe integers within their documented limit, layer and shadow ramps must be finite percentages from 0 to 100, layer maps and `shadow` must be objects, and `requestLocal` and `shadow.logMismatches` must be booleans when present. Invalid defaults are rejected immediately. +DialCache validates `defaultConfig` when `cached()` registers a definition and whenever `getOrLoad()` is invoked: TTLs must be positive safe integers no greater than 31,536,000 seconds (a fixed 365-day duration), remote-read deadlines must be positive safe integers within their documented limit, layer and shadow ramps must be finite percentages from 0 to 100, layer maps and `shadow` must be objects, and `requestLocal` and `shadow.logMismatches` must be booleans when present. A positive static `staleOnErrorMaxAgeSec` must be a safe integer greater than the remote TTL and no greater than the same 365-day limit; `0` is valid and disables recovery. Invalid defaults are rejected immediately. Each registration or one-shot invocation captures an immutable internal snapshot of `defaultConfig`; mutating the supplied config or its maps later does not change that operation's baseline. Runtime policy changes belong in the provider's returned overlay. -Runtime TTL and ramp leaves are used as supplied instead of falling back to valid default leaves. A TTL outside the same 1-to-31,536,000-second range disables that layer with `invalid_ttl`; a nonnumeric, non-finite, or out-of-range ramp disables it with `invalid_ramp`. Valid ramps include both `0` and `100`. Other layers can still run, and invalid leaves also record a `config_resolution` error so provider garbage is alertable separately from intentional ramp-downs. A malformed runtime config object, layer-map shape, `requestLocal` value, explicit `remoteReadTimeoutMs`, or removed top-level `shadowRamp` fails config resolution for the invocation, records `config_error`, and executes the fallback uncached without attempting Redis. The public `DialCacheKeyConfig` constructor and static defaults likewise reject `shadowRamp` immediately; migrate it to `shadow.ramp`. +Runtime TTL and ramp leaves are used as supplied instead of falling back to valid default leaves. A TTL outside the same 1-to-31,536,000-second range disables that layer with `invalid_ttl`; a nonnumeric, non-finite, or out-of-range ramp disables it with `invalid_ramp`. Valid ramps include both `0` and `100`. Other layers can still run, and invalid leaves also record a `config_resolution` error so provider garbage is alertable separately from intentional ramp-downs. An invalid runtime `staleOnErrorMaxAgeSec`, including a positive value less than or equal to the resolved remote TTL, disables only stale recovery and records one remote `config_resolution` error; ordinary fresh Redis caching continues with its resolved TTL. A malformed runtime config object, layer-map shape, `requestLocal` value, explicit `remoteReadTimeoutMs`, or removed top-level `shadowRamp` fails config resolution for the invocation, records `config_error`, and executes the fallback uncached without attempting Redis. The public `DialCacheKeyConfig` constructor and static defaults likewise reject `shadowRamp` immediately; migrate it to `shadow.ramp`. An invalid runtime `shadow.ramp` does not affect the cache result or disable an otherwise valid Redis policy. If normal traversal reaches an otherwise shadow-eligible Redis path, DialCache skips shadow work and records a `config_resolution` error. An invalid runtime `shadow.logMismatches` likewise preserves the cache result, Redis policy, shadow result, and shadow metric while suppressing the warning. DialCache validates this diagnostic leaf only after the metrics hook, exact-key cohort, and capacity gates admit shadow work, then records one remote `config_resolution` error for that admitted resolution. @@ -398,6 +399,46 @@ The node-redis adapter owns no additional resources, so the application closes t Node-redis computes each script's SHA, uses `EVALSHA`, and retries with `EVAL` after `NOSCRIPT`. Its cluster client routes scripts by their first key and performs that fallback on the selected shard. The GLIDE adapter uses GLIDE's native `Script` lifecycle and byte decoder; GLIDE routes scripts from their declared keys. Tracked reads are deliberately routed to primaries so a lagging replica cannot hide an invalidation watermark. +#### Stale-on-error recovery + +Stale-on-error is an opt-in Redis availability policy. Let `F` be the resolved remote `ttlSec` and `M` be `staleOnErrorMaxAgeSec`: + +```text +age < F fresh +F <= age < M stale; recoverable only after the source rejects +age >= M unavailable +``` + +```ts +import { CacheLayer, DialCacheKeyConfig } from "dialcache"; + +const getUser = dialcache.cached( + (userId: string) => db.fetchUser(userId), + { + keyType: "user_id", + useCase: "GetUserWithStaleRecovery", + cacheKey: (userId) => userId, + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 300 }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: 3_600, + }), + }, +); +``` + +Positive `M` enables recovery and must satisfy `0 < F < M <= 31_536_000` seconds. Omission is off by default and inherits through a runtime overlay; `0` explicitly disables inherited recovery. Redis stores one existing v1 frame with physical `PX M`, while every normal read atomically checks Redis server time and returns it only below `F`. After a definitive normal miss, DialCache calls the source. A source success publishes normally; a source rejection performs one independent, bounded reread of the same key using `M`. A qualifying reread returns the retained value without extending its TTL or publishing it anywhere. All source rejections qualify, including `FallbackTimeoutError` and arbitrary rejection values. + +If the recovery read misses, times out, fails, or cannot deserialize the payload, DialCache throws the exact original source rejection. An initial Redis error or timeout forbids the recovery reread, so the existing fail-open path never retries Redis after an uncertain initial read. The recovery read uses a fresh effective `remoteReadTimeoutMs` budget; worst-case caller wait can therefore include the initial Redis budget, the fallback budget, and one recovery budget. + +Redis is the only shared stale reservoir. Recovered values do not populate process-local storage, start shadow comparison/fill work, refresh Redis, or create a cooldown. Existing single-flight behavior means one leader performs the source attempt and at most one recovery read. When request-local caching is enabled, a recovered value is memoized for the remainder of that already-enabled outer scope, consistent with request-local caching's existing no-TTL semantics; the next independent scope retries the source. + +Tracked recovery atomically rechecks both age and the current invalidation watermark. An invalidation during the source attempt therefore blocks recovery, and missing or malformed watermarks remain misses. Shadow-only and confirmation reads always use `F`; a successful shadow clean-miss fill uses physical retention `M` when the resolved policy enables it, but shadow work can never serve stale data. + +Changing policy applies current read-time `F` and `M` to frames that still exist. Lowering either boundary takes effect immediately. Raising `M` cannot extend or resurrect a key written with a shorter physical TTL; only a later successful write receives the longer retention. `M` is an upper bound rather than a guarantee because invalidation, eviction, expiry, or external removal may make the frame unavailable sooner. + +The v1 frame and value key are deliberately reused. Older DialCache readers do not enforce logical age and would misclassify a value retained through `M` as fresh. Roll this out readers-first: deploy the age-aware library and adapters to the entire reader fleet while the option remains omitted or `0`, then enable positive `staleOnErrorMaxAgeSec` values. A downgrade has the same barrier in reverse: setting the policy to `0` stops new extended-retention writes but does not remove existing ones, so do not restore an old reader until at least the largest previously enabled `M` has elapsed since the last `PX M` write (or those value keys have been isolated or removed). If mixed-version readers or immediate rollback must remain safe, use a new key/frame version instead. Monitor Redis CPU, memory, evictions, source failures, and bounded stale-recovery outcomes; retaining high-cardinality values longer can increase resident memory toward a workload-dependent `M / F` upper bound. + #### Remote read deadlines and async liveness DialCache bounds every active Redis read. The effective timeout is resolved per use case and per invocation: runtime `remoteReadTimeoutMs`, then `defaultConfig.remoteReadTimeoutMs`, then optional instance `redis.readTimeoutMs`, then 50 ms. Values must be positive safe integers no greater than 2,147,483,647. There is no unbounded escape hatch for remote reads. @@ -412,7 +453,7 @@ Writes, invalidations, async `cacheConfigProvider` calls, and custom serializer #### Serialization -The core Redis boundary is the client-agnostic `DialCacheRedisClient` interface. It exchanges serialized values as `string | Buffer` and does not expose client commands or wire encodings. Distinct untracked/tracked read and write Lua sources, the invalidation source, and wire constants are available from `dialcache/redis-protocol`. Custom adapters can throw the root-exported `DialCacheRedisPayloadError`, `DialCacheRedisPayloadEncodingError`, and `DialCacheRedisProtocolError` classes to distinguish malformed payloads, unsupported encodings, and Lua reply-domain violations in logs. DialCache records bounded `cache_read`, `cache_write`, or `invalidation` metrics by failure site. +The core Redis boundary is the client-agnostic `DialCacheRedisClient` interface. It exchanges serialized values as `string | Buffer` and does not expose client commands or wire encodings. Every `RedisReadRequest` carries a required positive `maxAgeMs`; custom adapters must atomically enforce that Redis-server age bound and, for tracked requests, the watermark. Ignoring it can expose physically retained stale data as a normal hit. Custom clients must therefore declare `enforcesMaxAge: true`; DialCache rejects clients without that capability marker at construction, including older compiled JavaScript adapters. Distinct untracked/tracked read and write Lua sources, the invalidation source, and wire constants are available from `dialcache/redis-protocol`. Custom adapters can throw the root-exported `DialCacheRedisPayloadError`, `DialCacheRedisPayloadEncodingError`, and `DialCacheRedisProtocolError` classes to distinguish malformed payloads, unsupported encodings, and Lua reply-domain violations in logs. DialCache records bounded `cache_read`, `cache_write`, or `invalidation` metrics by failure site. Redis values use a compact binary frame: @@ -423,11 +464,11 @@ byte 10 payload encoding (0 = UTF-8, 1 = raw binary) bytes 11... serialized payload ``` -Redis's Lua `struct` library packs and unpacks the timestamp. Redis TTL is authoritative, so expiry metadata is not duplicated in the frame. `payload` is produced by the operation's serializer, or by `JsonSerializer` by default. Custom serializers can return either `string` or `Buffer`; strings are stored as UTF-8 and Buffers are stored byte-for-byte without base64 expansion. Adapters restore the same representation before calling `serializer.load`. +Redis's Lua `struct` library packs and unpacks the timestamp. Reads compare that timestamp with Redis server time and the invocation's requested maximum age; the key TTL is the physical retention bound and is not duplicated in the frame. `payload` is produced by the operation's serializer, or by `JsonSerializer` by default. Custom serializers can return either `string` or `Buffer`; strings are stored as UTF-8 and Buffers are stored byte-for-byte without base64 expansion. Adapters restore the same representation before calling `serializer.load`. DialCache uses native `JSON.stringify` and `JSON.parse` by default. There is no runtime validation pass, so the default adds no traversal beyond JSON serialization itself. A top-level `undefined` result is supported with an internal sentinel. -When `serializer.load` rejects a Redis payload, DialCache records a `serialization_load` error, counts the read as a remote cache miss, runs the fallback, and attempts to replace the rejected payload. A validating custom serializer can therefore treat an incompatible cached value as a refreshable miss without adding a schema version to the cache key. +When `serializer.load` rejects a normal Redis payload, DialCache records a `serialization_load` error, counts the read as a remote cache miss, runs the fallback, and attempts to replace the rejected payload. That failed payload is not eligible for a stale recovery retry. A validating custom serializer can therefore treat an incompatible cached value as a refreshable miss without adding a schema version to the cache key. `JsonSerializer` validates JSON syntax only. It cannot detect that a structurally valid payload came from an incompatible application value schema. Applications that keep the same `useCase` across deployments must keep default-JSON values backward compatible. For an incompatible change, either provide a serializer whose `load` method validates and rejects the old shape, or change `useCase` to isolate the new cache entries. On the caller-serving Redis path, mutually incompatible validating serializers in a mixed deployment can repeatedly reject and replace each other's values; correctness is preserved, but expect additional fallback and Redis-write load until the rollout converges. Shadow work reports a non-null payload that fails `load` as `deserialization_error` and never replaces it. @@ -614,7 +655,7 @@ Invalidation writes a Redis watermark at `{encodedNamespace:encodedKeyType:encod The internal `:dialcache-frame-v1` suffix identifies values written with DialCache's binary protocol. Watermarks are stored as decimal timestamps. -A cached Redis value whose Redis-created timestamp is older than or equal to the watermark is treated as stale and refreshed through fallback. `invalidateRemote(keyType, id, futureBufferMs)` sets the watermark to the greater of its existing value and Redis's current time plus the buffer. While that future window is active, an invocation that reaches the tracked Redis read treats the covered value as a miss. If its fallback then reaches the tracked Redis write, Redis rejects the write and DialCache also suppresses the corresponding process-local population; the fallback value still returns to its caller. Request-local memoization remains unconditional. A ramped-out invocation without shadow work does not consult the watermark; a selected shadow path does consult it for `C0` and any clean-miss fill, although caller-path request-local/process-local publication remains independent. +A tracked Redis value is usable only while its Redis-created age is below the read's requested `F` or recovery `M` boundary and its timestamp is newer than the watermark. `invalidateRemote(keyType, id, futureBufferMs)` sets the watermark to the greater of its existing value and Redis's current time plus the buffer. While that future window is active, an invocation that reaches the tracked Redis read treats the covered value as a miss; a stale-on-error recovery reread rechecks the same watermark after the source rejection. If a fallback then reaches the tracked Redis write, Redis rejects the write and DialCache also suppresses the corresponding process-local population; the fallback value still returns to its caller. Request-local memoization remains unconditional. A ramped-out invocation without shadow work does not consult the watermark; a selected shadow path does consult it for `C0` and any clean-miss fill, although caller-path request-local/process-local publication remains independent. The bundled timestamp protocol assumes that system clocks are synchronized across every Redis node eligible for primary promotion. Redis does not guarantee that `TIME` is monotonic across nodes, and DialCache does not detect or compensate for cross-node clock skew. If this deployment assumption is violated, failover can temporarily suppress tracked cache fills or allow a pre-invalidation value to remain readable until it expires or a later invalidation advances the watermark past its timestamp. @@ -644,7 +685,7 @@ await dialcache.enable(async () => { }); ``` -With Redis configured, an instance-scoped leader that misses the process-local cache runs one bounded Redis read and, on a normal miss, the fallback/cache write; followers share its remaining read budget and await the same result. On a shadow-selected served Redis hit, only that leader can schedule detached validation, so followers do not multiply source reads. Process-local-only misses share the leader's fallback/cache write. This protects Redis and the source of truth from a thundering herd on hot keys. +With Redis configured, an instance-scoped leader that misses the process-local cache runs one bounded Redis read and, on a normal miss, the fallback/cache write; followers share its remaining read budget and await the same result. If stale-on-error is enabled and the source rejects, that same leader performs at most one recovery read and followers share the recovered reference or identical rejection. On a shadow-selected served Redis hit, only that leader can schedule detached validation, so followers do not multiply source reads. Process-local-only misses share the leader's fallback/cache write. This protects Redis and the source of truth from a thundering herd on hot keys. Coalescing only applies when at least one cache layer is active. Calls outside `enable()` are true pass-through. Calls where request-local, process-local, and Redis serving are all disabled are uncached and uncoalesced, even if shadowing independently schedules detached Redis work; same-key shadow deduplication drops duplicate jobs but does not combine caller fallbacks. Because these calls were initially enabled, the fallback deadline below still applies. @@ -748,6 +789,7 @@ The Prometheus adapter emits: | `dialcache_invalidation_counter` | Counter | `cache_namespace`, `key_type`, `layer` | Invalidation calls for the layers touched | | `dialcache_coalesced_counter` | Counter | `cache_namespace`, `use_case`, `key_type`, `scope` | Coalesced requests split by `request_local` or `process` scope | | `dialcache_shadow_validation_counter` | Counter | `cache_namespace`, `use_case`, `key_type`, `outcome` | Sampled Redis shadow-job outcomes | +| `dialcache_stale_recovery_counter` | Counter | `cache_namespace`, `use_case`, `key_type`, `outcome` | Attempted stale-on-error recoveries by bounded terminal outcome | | `dialcache_get_timer` | Histogram | `cache_namespace`, `use_case`, `key_type`, `layer` | Cache get latency in seconds | | `dialcache_fallback_timer` | Histogram | `cache_namespace`, `use_case`, `key_type`, `layer` | Elapsed time until the underlying function settles or timeout rejection is delivered | | `dialcache_serialization_timer` | Histogram | `cache_namespace`, `use_case`, `key_type`, `layer`, `operation` | Redis serializer dump/load latency | @@ -755,7 +797,7 @@ The Prometheus adapter emits: `policy_disabled` means that a process-local or Redis layer has no effective TTL after runtime overlays are applied. It is an intentional policy outcome, including the default when `defaultConfig` is omitted, rather than a configuration-loading failure. -Every metric carries `cache_namespace`, including disabled-context, key-construction, coalescing, shadow-validation, and invalidation paths that do not have a constructed key. Its value is `DialCacheConfig.namespace`, defaulting to `urn`. The `layer` label is `request_local`, `local` (process-local), `remote` (caller-serving Redis), or `remote_shadow` (detached, non-serving Redis work); `noop` means no cache layer was reached. Detached reads, serializer work, payload sizes, and Redis read/write errors use `remote_shadow`, while the dedicated bounded shadow `outcome` records the terminal job result. The bounded `scope` label on `dialcache_coalesced_counter` distinguishes request-local from instance-scoped single-flight work. `scope="process"` coordinates calls only within one `DialCache` instance; separate instances in the same process do not share in-flight state. +Every metric carries `cache_namespace`, including disabled-context, key-construction, coalescing, shadow-validation, stale-recovery, and invalidation paths that do not have a constructed key. Its value is `DialCacheConfig.namespace`, defaulting to `urn`. The `layer` label is `request_local`, `local` (process-local), `remote` (caller-serving Redis), or `remote_shadow` (detached, non-serving Redis work); `noop` means no cache layer was reached. Detached reads, serializer work, payload sizes, and Redis read/write errors use `remote_shadow`, while the dedicated bounded shadow `outcome` records the terminal job result. Stale recovery remains caller-serving `remote` traffic and records exactly one of `served`, `miss`, `read_error`, `read_timeout`, or `deserialization_error` per attempted recovery; the original source rejection still records the existing fallback error and duration. The bounded `scope` label on `dialcache_coalesced_counter` distinguishes request-local from instance-scoped single-flight work. `scope="process"` coordinates calls only within one `DialCache` instance; separate instances in the same process do not share in-flight state. ### Datadog @@ -806,6 +848,7 @@ The Datadog adapter emits exact increments of `1` for counters and preserves sec | `dialcache.invalidation.count` | Count | `cache_namespace`, `key_type`, `layer` | Invalidation calls for the layers touched | | `dialcache.coalesced.count` | Count | `cache_namespace`, `use_case`, `key_type`, `scope` | Coalesced requests by sharing scope | | `dialcache.shadow.count` | Count | `cache_namespace`, `use_case`, `key_type`, `outcome` | Sampled Redis shadow-job outcomes | +| `dialcache.stale_recovery.count` | Count | `cache_namespace`, `use_case`, `key_type`, `outcome` | Attempted stale-on-error recoveries by bounded terminal outcome | | `dialcache.get.duration` | Distribution or histogram | `cache_namespace`, `use_case`, `key_type`, `layer` | Cache get latency in seconds | | `dialcache.fallback.duration` | Distribution or histogram | `cache_namespace`, `use_case`, `key_type`, `layer` | Elapsed time until the underlying function settles or timeout rejection is delivered | | `dialcache.serialization.duration` | Distribution or histogram | `cache_namespace`, `use_case`, `key_type`, `layer`, `operation` | Redis serializer dump/load latency in seconds | @@ -834,7 +877,7 @@ These values are defined by the backend-neutral core and are identical for every ### Custom adapters -For other telemetry backends, implement `DialCacheMetricsAdapter` and pass the adapter through `new DialCache({ metrics })`. Every backend-neutral label object exposes the logical namespace as camel-case `cacheNamespace`; adapters should map it to their backend's `cache_namespace` label/tag. This field is present even when no key or cache layer was reached. Implement the optional `shadowValidation` method to enable shadow work as well as record its outcomes; omitting it leaves all shadow work disabled even when `shadow.ramp` is nonzero or mismatch logging is enabled. Every metrics callback is fire-and-forget: DialCache isolates synchronous throws and consumes rejections from returned promises or thenables, but never awaits or drains observer work. Omit `metrics` to disable metrics. +For other telemetry backends, implement `DialCacheMetricsAdapter` and pass the adapter through `new DialCache({ metrics })`. Every backend-neutral label object exposes the logical namespace as camel-case `cacheNamespace`; adapters should map it to their backend's `cache_namespace` label/tag. This field is present even when no key or cache layer was reached. Implement the optional `shadowValidation` method to enable shadow work as well as record its outcomes; omitting it leaves all shadow work disabled even when `shadow.ramp` is nonzero or mismatch logging is enabled. The optional `staleRecovery` method observes recovery outcomes but does not enable or disable the configured recovery policy. Every metrics callback is fire-and-forget: DialCache isolates synchronous throws and consumes rejections from returned promises or thenables, but never awaits or drains observer work. Omit `metrics` to disable metrics. ## Maintainers @@ -848,6 +891,18 @@ pnpm benchmark:request-local The command builds `dist` before reporting ten scenarios: sequential request-local hits, sequential process-local hits, enabled bounded fallbacks, request-local coalescing fan-out, process coalescing fan-out, remote-read-deadline coalescing fan-out, tracked Redis hits with shadow omitted, tracked Redis hits deterministically outside a partial shadow ramp, a ramped-down warm-hit confirmation, and a ramped-down clean-miss fill. Both shadow scenarios prove that the caller completes before detached Redis work. The benchmark is a maintainer tool and is not included in the published package. It asserts fallback counts, Redis behavior, coalescing state, timer cleanup, returned values, exactly-once SoT reuse, and conditional confirmation/fill without applying a timing threshold. Override its work sizes with `DIALCACHE_BENCH_ITERATIONS` and `DIALCACHE_BENCH_FANOUT`. +### Stale-on-error Redis benchmark + +Run the stale-on-error resource benchmark against a dedicated Redis-compatible endpoint: + +```bash +DIALCACHE_BENCH_REDIS_URL=redis://127.0.0.1:6379 pnpm benchmark:stale-on-error +``` + +The command uses the node-redis semantic adapter and a random, self-cleaning namespace. It reports fresh-hit and logical-stale-miss Redis CPU from `INFO commandstats`, sequential outage-recovery throughput and `total_net_input_bytes` / `total_net_output_bytes` deltas, 100-way coalescing, `MEMORY USAGE`, tracked value/watermark retention, physical expiration, and current eviction policy/residency. Semantic counts are asserted; CPU, throughput, latency, bytes, and memory remain informational with no timing threshold. Run the matrix against each supported server and set `DIALCACHE_BENCH_PAYLOAD_BYTES` to representative sizes such as `64` and `4096`; override work with `DIALCACHE_BENCH_ITERATIONS` and `DIALCACHE_BENCH_FANOUT`. + +Actual eviction pressure is opt-in because the target may be shared. On a dedicated endpoint already configured with `maxmemory` and an eviction policy, set `DIALCACHE_BENCH_EVICTION_KEYS` to create that many random benchmark-owned 16 KiB expiring keys. Override their size with `DIALCACHE_BENCH_EVICTION_VALUE_BYTES`. The harness never runs `CONFIG`, `FLUSHDB`, or `FLUSHALL`; it reports the pressure-local `evicted_keys` delta, sentinel residency, and whether the retained untracked value still qualifies after pressure. If `DIALCACHE_BENCH_REDIS_URL` is omitted, the command exits with an explicit JSON skip instead of connecting to a default server. A 120-second hard watchdog bounds stalled endpoints; override it with `DIALCACHE_BENCH_TIMEOUT_MS`. Normal cleanup and client shutdown each receive their own two-second best-effort bound, and every owned key also has a finite Redis TTL. + ### Releasing Publishing starts by manually running the `Release` workflow from current `main`. After the package checks pass, Semantic Release selects the next version from Conventional Commits since the highest stable `vX.Y.Z` tag. Breaking changes bump major, `feat` bumps minor, and every other normal PR-title type (`fix`, `perf`, `docs`, `style`, `refactor`, `test`, `build`, `chore`, `ci`, and `revert`) bumps patch. The highest required bump wins. diff --git a/package.json b/package.json index c219b21..e52ce13 100644 --- a/package.json +++ b/package.json @@ -77,6 +77,7 @@ ], "scripts": { "benchmark:request-local": "pnpm build && node scripts/benchmark-request-local.mjs", + "benchmark:stale-on-error": "pnpm build && node scripts/benchmark-stale-on-error.mjs", "build": "tsup src/index.ts src/datadog.ts src/node-redis.ts src/prometheus.ts src/redis-protocol.ts src/valkey-glide.ts --format esm,cjs --dts --clean", "check": "pnpm typecheck && pnpm test && pnpm build && pnpm test:package", "typecheck": "tsc --noEmit", diff --git a/scripts/benchmark-request-local.mjs b/scripts/benchmark-request-local.mjs index 8a0c3b9..73daa95 100644 --- a/scripts/benchmark-request-local.mjs +++ b/scripts/benchmark-request-local.mjs @@ -244,6 +244,7 @@ async function benchmarkRedisReadDeadlineCoalescing(fanout) { const originalSetTimeout = globalThis.setTimeout; const originalClearTimeout = globalThis.clearTimeout; const redisClient = { + enforcesMaxAge: true, async read() { redisReadCalls += 1; started.resolve(); @@ -319,6 +320,7 @@ async function benchmarkSequentialTrackedRedisHits(iterations, { scenario, useCa let redisWriteCalls = 0; let redisInvalidationCalls = 0; const redisClient = { + enforcesMaxAge: true, async read({ watermarkKey }) { assert.equal(typeof watermarkKey, "string", "the benchmark must exercise tracked Redis reads"); redisReadCalls += 1; @@ -392,6 +394,7 @@ async function benchmarkDarkShadowDetachment() { const cachedValue = { source: "redis" }; const sourceValue = { source: "truth" }; const redisClient = { + enforcesMaxAge: true, async read({ watermarkKey }) { assert.equal(typeof watermarkKey, "string", "dark shadow reads must remain tracked"); redisReadCalls += 1; @@ -476,6 +479,7 @@ async function benchmarkDarkShadowFillDetachment() { let redisWriteCalls = 0; const sourceValue = { source: "truth" }; const redisClient = { + enforcesMaxAge: true, async read({ watermarkKey }) { assert.equal(typeof watermarkKey, "string", "dark shadow reads must remain tracked"); redisReadCalls += 1; diff --git a/scripts/benchmark-stale-on-error.mjs b/scripts/benchmark-stale-on-error.mjs new file mode 100644 index 0000000..1decb7a --- /dev/null +++ b/scripts/benchmark-stale-on-error.mjs @@ -0,0 +1,1111 @@ +import assert from "node:assert/strict"; +import { randomUUID } from "node:crypto"; +import { performance } from "node:perf_hooks"; +import { setImmediate as nextTurn, setTimeout as delay } from "node:timers/promises"; + +async function runBenchmark(url) { + const iterations = readPositiveInteger("DIALCACHE_BENCH_ITERATIONS", 2_000); + const fanout = readPositiveInteger("DIALCACHE_BENCH_FANOUT", 100); + const targetPayloadBytes = readPositiveInteger("DIALCACHE_BENCH_PAYLOAD_BYTES", 1_024); + const evictionPressureKeys = readNonnegativeInteger("DIALCACHE_BENCH_EVICTION_KEYS", 0); + const evictionValueBytes = readPositiveInteger("DIALCACHE_BENCH_EVICTION_VALUE_BYTES", 16_384); + const freshTtlSec = 60; + const staleMaxAgeSec = 300; + const staleAgeMs = 120_000; + const physicalTtlMs = staleMaxAgeSec * 1_000; + const namespace = `dialcache-stale-bench-${process.pid}-${Date.now()}-${randomUUID().slice(0, 8)}`; + const ownedKeys = new Set(); + + const [redisModule, dialcacheModule, nodeRedisModule] = await Promise.all([ + import("redis"), + import("../dist/index.js"), + import("../dist/node-redis.js"), + ]); + const { commandOptions, createClient } = redisModule; + const { + CacheLayer, + DialCache, + DialCacheKey, + DialCacheKeyConfig, + invalidationPrefix, + redisClusterHashTag, + } = dialcacheModule; + const { createNodeRedisDialCacheClient, dialcacheRedisScripts } = nodeRedisModule; + + const admin = createClient({ + url, + scripts: dialcacheRedisScripts, + disableOfflineQueue: true, + socket: { + connectTimeout: 5_000, + reconnectStrategy: false, + }, + }); + let lastClientError = null; + admin.on("error", (error) => { + lastClientError = error; + }); + + try { + try { + await admin.connect(); + await admin.ping(); + } catch (error) { + const cause = lastClientError ?? error; + throw new Error( + `Could not connect to DIALCACHE_BENCH_REDIS_URL (${safeEndpoint(url)}): ${errorMessage(cause)}`, + { cause }, + ); + } + + const adapter = createNodeRedisDialCacheClient(admin); + const server = await readServerReport(admin, url); + const evictedKeysBefore = await readEvictedKeys(admin); + const payload = makeJsonPayload(targetPayloadBytes); + const payloadBytes = Buffer.byteLength(payload.encoded); + + const freshKey = redisKeys({ + DialCacheKey, + invalidationPrefix, + redisClusterHashTag, + namespace, + keyType: "benchmark_id", + id: "redis-cpu", + useCase: "BenchmarkStaleRedisCpu", + tracked: false, + }); + own(ownedKeys, freshKey); + assert.equal( + await adapter.write({ valueKey: freshKey.valueKey, cacheTtlMs: physicalTtlMs, value: payload.encoded }), + true, + "fresh CPU fixture should be written", + ); + + const freshCpu = await benchmarkRedisReads({ + admin, + adapter, + iterations, + valueKey: freshKey.valueKey, + maxAgeMs: freshTtlSec * 1_000, + expected: payload.encoded, + scenario: "fresh_hit", + }); + + await markFrameStale({ + admin, + commandOptions, + valueKey: freshKey.valueKey, + ageMs: staleAgeMs, + }); + assert.equal( + await adapter.read({ valueKey: freshKey.valueKey, maxAgeMs: freshTtlSec * 1_000 }), + null, + "the CPU fixture should be logically stale at the fresh TTL", + ); + assert.equal( + await adapter.read({ valueKey: freshKey.valueKey, maxAgeMs: physicalTtlMs }), + payload.encoded, + "the CPU fixture should remain readable at the stale maximum age", + ); + + const logicalStaleCpu = await benchmarkRedisReads({ + admin, + adapter, + iterations, + valueKey: freshKey.valueKey, + maxAgeMs: freshTtlSec * 1_000, + expected: null, + scenario: "logical_stale_miss", + }); + + const outageKey = redisKeys({ + DialCacheKey, + invalidationPrefix, + redisClusterHashTag, + namespace, + keyType: "benchmark_id", + id: "outage", + useCase: "BenchmarkStaleOutage", + tracked: false, + }); + own(ownedKeys, outageKey); + await writeLogicallyStaleFixture({ + admin, + adapter, + commandOptions, + key: outageKey, + value: payload.encoded, + physicalTtlMs, + staleAgeMs, + freshMaxAgeMs: freshTtlSec * 1_000, + }); + const outageNetworkBefore = await readNetworkBytes(admin); + const outageRecoveryResult = await benchmarkOutageRecovery({ + CacheLayer, + DialCache, + DialCacheKeyConfig, + adapter, + iterations, + namespace, + payload: payload.value, + payloadBytes, + freshTtlSec, + staleMaxAgeSec, + }); + const outageNetworkAfter = await readNetworkBytes(admin); + const outageRecovery = { + ...outageRecoveryResult, + redisNetwork: networkByteDelta(outageNetworkBefore, outageNetworkAfter, iterations), + }; + + const coalescedKey = redisKeys({ + DialCacheKey, + invalidationPrefix, + redisClusterHashTag, + namespace, + keyType: "benchmark_id", + id: "coalesced", + useCase: "BenchmarkStaleCoalescing", + tracked: false, + }); + own(ownedKeys, coalescedKey); + await writeLogicallyStaleFixture({ + admin, + adapter, + commandOptions, + key: coalescedKey, + value: payload.encoded, + physicalTtlMs, + staleAgeMs, + freshMaxAgeMs: freshTtlSec * 1_000, + }); + const coalescing = await benchmarkCoalescedRecovery({ + CacheLayer, + DialCache, + DialCacheKeyConfig, + adapter, + fanout, + namespace, + payload: payload.value, + freshTtlSec, + staleMaxAgeSec, + }); + + const trackedKey = redisKeys({ + DialCacheKey, + invalidationPrefix, + redisClusterHashTag, + namespace, + keyType: "benchmark_id", + id: "tracked-residency", + useCase: "BenchmarkTrackedResidency", + tracked: true, + }); + own(ownedKeys, trackedKey); + const trackedResidency = await inspectTrackedResidency({ + admin, + adapter, + key: trackedKey, + value: payload.encoded, + physicalTtlMs, + }); + + const expirationKey = redisKeys({ + DialCacheKey, + invalidationPrefix, + redisClusterHashTag, + namespace, + keyType: "benchmark_id", + id: "expiration", + useCase: "BenchmarkTrackedExpiration", + tracked: true, + }); + own(ownedKeys, expirationKey); + const expiration = await inspectExpiration({ + admin, + adapter, + key: expirationKey, + value: payload.encoded, + }); + + const memoryUsage = { + logicalStaleValueBytes: await requiredMemoryUsage(admin, freshKey.valueKey, "logical-stale value"), + outageValueBytes: await requiredMemoryUsage(admin, outageKey.valueKey, "outage recovery value"), + trackedValueBytes: trackedResidency.valueMemoryBytes, + trackedWatermarkBytes: trackedResidency.watermarkMemoryBytes, + expirationWatermarkBytesAfterValueExpiry: expiration.watermarkMemoryBytesAfterValueExpiry, + }; + + const sentinelKeys = { + logicalStaleValue: freshKey.valueKey, + outageValue: outageKey.valueKey, + coalescedValue: coalescedKey.valueKey, + trackedValue: trackedKey.valueKey, + trackedWatermark: trackedKey.watermarkKey, + expirationWatermark: expirationKey.watermarkKey, + }; + const evictedKeysBeforePressure = await readEvictedKeys(admin); + const evictionPressure = await applyEvictionPressure({ + admin, + adapter, + namespace, + ownedKeys, + server, + keyCount: evictionPressureKeys, + valueBytes: evictionValueBytes, + recoveryValueKey: outageKey.valueKey, + recoveryMaxAgeMs: physicalTtlMs, + }); + const evictedKeysAfter = await readEvictedKeys(admin); + const residency = Object.fromEntries( + await Promise.all( + Object.entries(sentinelKeys).map(async ([name, key]) => [name, (await admin.exists(key)) === 1]), + ), + ); + const eviction = evictionReport({ + server, + evictedKeysBefore, + evictedKeysBeforePressure, + evictedKeysAfter, + residency, + pressure: evictionPressure, + }); + + return { + status: "passed", + generatedAt: new Date().toISOString(), + server, + settings: { + adapter: "node-redis", + iterations, + fanout, + requestedPayloadBytes: targetPayloadBytes, + actualJsonPayloadBytes: payloadBytes, + freshTtlSec, + staleOnErrorMaxAgeSec: staleMaxAgeSec, + logicalStaleAgeMs: staleAgeMs, + evictionPressureKeys, + evictionValueBytes, + }, + redisCpu: { + freshHit: freshCpu, + logicalStaleMiss: logicalStaleCpu, + logicalStaleMinusFreshUsecPerOperation: round( + logicalStaleCpu.redisUsecPerOperation - freshCpu.redisUsecPerOperation, + 4, + ), + measurementNote: + "Redis CPU is the INFO commandstats delta for top-level EVAL/EVALSHA-family commands. Use an isolated server to avoid unrelated script traffic.", + }, + outageRecovery, + coalescing, + memoryUsage, + trackedResidency, + expiration, + eviction, + assertions: "passed", + thresholdPolicy: "No latency, throughput, CPU, memory, or eviction-count threshold is enforced.", + }; + } finally { + if (admin.isOpen) { + if (ownedKeys.size > 0) { + try { + await withSafetyTimeout(admin.del([...ownedKeys]), 2_000, "benchmark key cleanup"); + } catch { + // All owned keys have finite TTLs, so failed best-effort cleanup remains bounded in Redis. + } + } + try { + await withSafetyTimeout(admin.quit(), 2_000, "Redis client shutdown"); + } catch { + if (admin.isOpen) { + admin.disconnect(); + } + } + } + } +} + +async function benchmarkRedisReads({ + admin, + adapter, + iterations, + valueKey, + maxAgeMs, + expected, + scenario, +}) { + assert.equal(await adapter.read({ valueKey, maxAgeMs }), expected, `${scenario} warm-up read should match`); + const before = await readScriptCommandStats(admin); + const start = performance.now(); + let payloadResponseBytes = 0; + for (let index = 0; index < iterations; index += 1) { + const result = await adapter.read({ valueKey, maxAgeMs }); + if (result !== expected) { + throw new assert.AssertionError({ + message: `${scenario} read ${index} returned an unexpected result`, + actual: result, + expected, + operator: "strictEqual", + }); + } + if (result !== null) { + payloadResponseBytes += Buffer.isBuffer(result) ? result.byteLength : Buffer.byteLength(result); + } + } + const elapsedMs = performance.now() - start; + const after = await readScriptCommandStats(admin); + const commandstats = subtractCommandStats(before, after); + assert.ok( + commandstats.calls >= iterations, + `${scenario} should add at least ${iterations} EVAL/EVALSHA-family command calls; observed ${commandstats.calls}`, + ); + return { + operations: iterations, + elapsedMs: round(elapsedMs, 3), + operationsPerSecond: round((iterations / elapsedMs) * 1_000, 1), + redisCommandCalls: commandstats.calls, + redisUsec: commandstats.usec, + redisUsecPerOperation: round(commandstats.usec / iterations, 4), + applicationPayloadResponseBytes: payloadResponseBytes, + commandstatsByCommand: commandstats.byCommand, + }; +} + +async function benchmarkOutageRecovery({ + CacheLayer, + DialCache, + DialCacheKeyConfig, + adapter, + iterations, + namespace, + payload, + payloadBytes, + freshTtlSec, + staleMaxAgeSec, +}) { + const counters = { reads: 0, sourceCalls: 0, staleServed: 0 }; + const client = countingClient(adapter, counters); + const metrics = recordingMetrics({ + staleRecovery({ outcome }) { + if (outcome === "served") { + counters.staleServed += 1; + } + }, + }); + const dialcache = new DialCache({ + namespace, + redis: { client, readTimeoutMs: 5_000 }, + metrics, + logger: silentLogger, + }); + const sourceError = new Error("simulated source outage"); + const getValue = dialcache.cached( + async () => { + counters.sourceCalls += 1; + throw sourceError; + }, + { + keyType: "benchmark_id", + useCase: "BenchmarkStaleOutage", + cacheKey: () => "outage", + defaultConfig: staleConfig(CacheLayer, DialCacheKeyConfig, freshTtlSec, staleMaxAgeSec), + }, + ); + + let bodyBytes = 0; + const start = performance.now(); + for (let index = 0; index < iterations; index += 1) { + const recovered = await dialcache.enable(async () => await getValue()); + assert.equal(recovered.kind, payload.kind); + bodyBytes += recovered.body.length; + } + const elapsedMs = performance.now() - start; + assert.equal(counters.sourceCalls, iterations, "each sequential logical miss should call the failing source once"); + assert.equal(counters.reads, iterations * 2, "each recovery should perform one fresh read and one stale reread"); + assert.equal(counters.staleServed, iterations, "every simulated outage should serve the retained value"); + assert.equal(bodyBytes, payload.body.length * iterations, "all recovered response bodies should be intact"); + + return { + operations: iterations, + elapsedMs: round(elapsedMs, 3), + operationsPerSecond: round((iterations / elapsedMs) * 1_000, 1), + sourceRejections: counters.sourceCalls, + redisReads: counters.reads, + staleResponsesServed: counters.staleServed, + redisPayloadResponseBytesPerOperation: payloadBytes, + redisPayloadResponseBytesTotal: payloadBytes * iterations, + responseByteNote: "Payload bytes exclude RESP framing and the logical-miss null reply.", + }; +} + +async function benchmarkCoalescedRecovery({ + CacheLayer, + DialCache, + DialCacheKeyConfig, + adapter, + fanout, + namespace, + payload, + freshTtlSec, + staleMaxAgeSec, +}) { + const sourceGate = deferred(); + const sourceStarted = deferred(); + const counters = { reads: 0, sourceCalls: 0, staleServed: 0, processFollowers: 0 }; + const client = countingClient(adapter, counters); + const metrics = recordingMetrics({ + coalesced({ scope }) { + if (scope === "process") { + counters.processFollowers += 1; + } + }, + staleRecovery({ outcome }) { + if (outcome === "served") { + counters.staleServed += 1; + } + }, + }); + const dialcache = new DialCache({ + namespace, + redis: { client, readTimeoutMs: 5_000 }, + metrics, + logger: silentLogger, + }); + const sourceError = new Error("simulated coalesced source outage"); + const getValue = dialcache.cached( + async () => { + counters.sourceCalls += 1; + sourceStarted.resolve(); + await sourceGate.promise; + throw sourceError; + }, + { + keyType: "benchmark_id", + useCase: "BenchmarkStaleCoalescing", + cacheKey: () => "coalesced", + defaultConfig: staleConfig(CacheLayer, DialCacheKeyConfig, freshTtlSec, staleMaxAgeSec), + }, + ); + + const start = performance.now(); + const calls = Array.from( + { length: fanout }, + () => dialcache.enable(async () => await getValue()), + ); + await withSafetyTimeout(sourceStarted.promise, 5_000, "coalesced source start"); + let activeState = dialcache.getCoalescingState().process; + for (let turn = 0; turn < 20 && activeState.activeFollowers < fanout - 1; turn += 1) { + await nextTurn(); + activeState = dialcache.getCoalescingState().process; + } + sourceGate.resolve(); + const values = await Promise.all(calls); + const elapsedMs = performance.now() - start; + + assert.equal(activeState.activeLeaders, 1, "the fanout should have one process leader while the source is pending"); + assert.equal(activeState.activeFollowers, fanout - 1, "all remaining callers should be process followers"); + assert.equal(counters.processFollowers, fanout - 1, "coalescing telemetry should count every follower"); + assert.equal(counters.sourceCalls, 1, "coalescing should invoke the failing source once"); + assert.equal(counters.reads, 2, "the leader should perform one fresh read and one stale reread"); + assert.equal(counters.staleServed, 1, "the leader should record one stale recovery"); + assert.equal(values.length, fanout); + assert.ok(values.every((value) => value.kind === payload.kind && value.body.length === payload.body.length)); + assert.deepEqual(dialcache.getCoalescingState().process, { + activeLeaders: 0, + activeFollowers: 0, + oldestLeaderAgeMs: null, + }); + + return { + callers: fanout, + elapsedMs: round(elapsedMs, 3), + callersPerSecond: round((fanout / elapsedMs) * 1_000, 1), + activeLeadersAtSource: activeState.activeLeaders, + activeFollowersAtSource: activeState.activeFollowers, + processFollowerMetrics: counters.processFollowers, + sourceRejections: counters.sourceCalls, + redisReads: counters.reads, + staleRecoveryMetrics: counters.staleServed, + }; +} + +async function inspectTrackedResidency({ admin, adapter, key, value, physicalTtlMs }) { + assert.equal( + await adapter.write({ + valueKey: key.valueKey, + watermarkKey: key.watermarkKey, + cacheTtlMs: physicalTtlMs, + value, + }), + true, + "tracked residency fixture should be written", + ); + const valueTtlMs = await admin.pTTL(key.valueKey); + const watermarkTtlMs = await admin.pTTL(key.watermarkKey); + assert.ok(valueTtlMs > 0, "tracked value should have a positive TTL"); + assert.ok(watermarkTtlMs > valueTtlMs, "tracked watermark should outlive its value"); + const valueMemoryBytes = await requiredMemoryUsage(admin, key.valueKey, "tracked value"); + const watermarkMemoryBytes = await requiredMemoryUsage(admin, key.watermarkKey, "tracked watermark"); + + await adapter.invalidate({ watermarkKey: key.watermarkKey, futureBufferMs: 0 }); + assert.equal( + await adapter.read({ valueKey: key.valueKey, watermarkKey: key.watermarkKey, maxAgeMs: physicalTtlMs }), + null, + "the resident tracked watermark should fence the older value", + ); + const watermarkTtlAfterInvalidationMs = await admin.pTTL(key.watermarkKey); + assert.ok(watermarkTtlAfterInvalidationMs > 0, "invalidation should retain a positive watermark TTL"); + + return { + valueTtlMs, + watermarkTtlMs, + watermarkRetentionLeadMs: watermarkTtlMs - valueTtlMs, + watermarkTtlAfterInvalidationMs, + valueMemoryBytes, + watermarkMemoryBytes, + invalidatedTrackedRead: "miss", + }; +} + +async function inspectExpiration({ admin, adapter, key, value }) { + const physicalTtlMs = 1_000; + const startedAt = performance.now(); + assert.equal( + await adapter.write({ + valueKey: key.valueKey, + watermarkKey: key.watermarkKey, + cacheTtlMs: physicalTtlMs, + value, + }), + true, + "expiration fixture should be written", + ); + const valueInitialTtlMs = await admin.pTTL(key.valueKey); + const watermarkInitialTtlMs = await admin.pTTL(key.watermarkKey); + await waitFor( + async () => (await admin.exists(key.valueKey)) === 0, + 5_000, + "tracked value physical expiration", + ); + const expirationWaitMs = performance.now() - startedAt; + const watermarkResidentAfterValueExpiry = (await admin.exists(key.watermarkKey)) === 1; + const watermarkTtlAfterValueExpiryMs = await admin.pTTL(key.watermarkKey); + assert.equal(watermarkResidentAfterValueExpiry, true, "watermark should remain after its tracked value expires"); + assert.ok(watermarkTtlAfterValueExpiryMs > 0, "remaining watermark should still have a positive TTL"); + assert.equal( + await adapter.read({ valueKey: key.valueKey, watermarkKey: key.watermarkKey, maxAgeMs: physicalTtlMs }), + null, + "an expired tracked value should read as a miss", + ); + + return { + configuredValueTtlMs: physicalTtlMs, + valueInitialTtlMs, + watermarkInitialTtlMs, + expirationWaitMs: round(expirationWaitMs, 3), + valueExpired: true, + watermarkResidentAfterValueExpiry, + watermarkTtlAfterValueExpiryMs, + watermarkMemoryBytesAfterValueExpiry: await requiredMemoryUsage( + admin, + key.watermarkKey, + "expiration watermark", + ), + }; +} + +async function writeLogicallyStaleFixture({ + admin, + adapter, + commandOptions, + key, + value, + physicalTtlMs, + staleAgeMs, + freshMaxAgeMs, +}) { + assert.equal( + await adapter.write({ valueKey: key.valueKey, cacheTtlMs: physicalTtlMs, value }), + true, + "logical-stale fixture should be written", + ); + await markFrameStale({ admin, commandOptions, valueKey: key.valueKey, ageMs: staleAgeMs }); + assert.equal( + await adapter.read({ valueKey: key.valueKey, maxAgeMs: freshMaxAgeMs }), + null, + "logical-stale fixture should miss at the fresh age", + ); + assert.equal( + await adapter.read({ valueKey: key.valueKey, maxAgeMs: physicalTtlMs }), + value, + "logical-stale fixture should hit at the maximum age", + ); +} + +async function markFrameStale({ admin, commandOptions, valueKey, ageMs }) { + const frame = await admin.get(commandOptions({ returnBuffers: true }), valueKey); + assert.ok(Buffer.isBuffer(frame) && frame.length >= 10, "fixture should contain a DialCache frame"); + const remainingTtlMs = await admin.pTTL(valueKey); + assert.ok(remainingTtlMs > 0, "fixture should have positive physical retention"); + const serverNowMs = (await admin.time()).getTime(); + const staleFrame = Buffer.from(frame); + staleFrame.writeBigUInt64BE(BigInt(serverNowMs - ageMs), 1); + await admin.set(valueKey, staleFrame, { PX: remainingTtlMs }); +} + +async function readScriptCommandStats(admin) { + const parsed = parseCommandStats(await admin.info("commandstats")); + return Object.fromEntries( + Object.entries(parsed).filter(([command]) => /^(eval|eval_ro|evalsha|evalsha_ro)$/.test(command)), + ); +} + +function subtractCommandStats(before, after) { + const commands = new Set([...Object.keys(before), ...Object.keys(after)]); + const byCommand = {}; + let calls = 0; + let usec = 0; + for (const command of [...commands].sort()) { + const commandCalls = (after[command]?.calls ?? 0) - (before[command]?.calls ?? 0); + const commandUsec = (after[command]?.usec ?? 0) - (before[command]?.usec ?? 0); + if (commandCalls !== 0 || commandUsec !== 0) { + byCommand[command] = { calls: commandCalls, usec: commandUsec }; + calls += commandCalls; + usec += commandUsec; + } + } + return { calls, usec, byCommand }; +} + +function parseCommandStats(info) { + const result = {}; + for (const line of info.split(/\r?\n/)) { + if (!line.startsWith("cmdstat_")) { + continue; + } + const colon = line.indexOf(":"); + const command = line.slice("cmdstat_".length, colon); + const fields = Object.fromEntries( + line.slice(colon + 1).split(",").map((field) => { + const equals = field.indexOf("="); + return [field.slice(0, equals), Number(field.slice(equals + 1))]; + }), + ); + result[command] = { + calls: Number.isFinite(fields.calls) ? fields.calls : 0, + usec: Number.isFinite(fields.usec) ? fields.usec : 0, + }; + } + return result; +} + +async function readServerReport(admin, url) { + const serverInfo = parseInfo(await admin.info("server")); + const memoryInfo = parseInfo(await admin.info("memory")); + return { + endpoint: safeEndpoint(url), + engine: serverInfo.server_name ?? (serverInfo.redis_version === undefined ? "unknown" : "Redis-compatible"), + version: serverInfo.valkey_version ?? serverInfo.redis_version ?? "unknown", + mode: serverInfo.redis_mode ?? "unknown", + maxmemoryBytes: readInfoNumber(memoryInfo, "maxmemory"), + maxmemoryHuman: memoryInfo.maxmemory_human ?? "unknown", + maxmemoryPolicy: memoryInfo.maxmemory_policy ?? "unknown", + }; +} + +async function readEvictedKeys(admin) { + return readInfoNumber(parseInfo(await admin.info("stats")), "evicted_keys"); +} + +async function readNetworkBytes(admin) { + const stats = parseInfo(await admin.info("stats")); + return { + inputBytes: readInfoNumber(stats, "total_net_input_bytes"), + outputBytes: readInfoNumber(stats, "total_net_output_bytes"), + }; +} + +function networkByteDelta(before, after, operations) { + const inputBytes = after.inputBytes - before.inputBytes; + const outputBytes = after.outputBytes - before.outputBytes; + assert.ok(inputBytes >= 0 && outputBytes >= 0, "Redis network byte counters must be monotonic"); + return { + inputBytes, + outputBytes, + inputBytesPerOperation: round(inputBytes / operations, 2), + outputBytesPerOperation: round(outputBytes / operations, 2), + measurementNote: + "INFO total_net_* deltas include RESP framing and benchmark commands on this endpoint; use an isolated server to exclude unrelated traffic.", + }; +} + +function parseInfo(info) { + const result = {}; + for (const line of info.split(/\r?\n/)) { + if (line.length === 0 || line.startsWith("#")) { + continue; + } + const colon = line.indexOf(":"); + if (colon > 0) { + result[line.slice(0, colon)] = line.slice(colon + 1); + } + } + return result; +} + +function readInfoNumber(info, key) { + const value = Number(info[key]); + if (!Number.isFinite(value)) { + throw new Error(`Redis INFO did not include a finite ${key}`); + } + return value; +} + +async function applyEvictionPressure({ + admin, + adapter, + namespace, + ownedKeys, + server, + keyCount, + valueBytes, + recoveryValueKey, + recoveryMaxAgeMs, +}) { + if (keyCount === 0) { + return { + attempted: false, + keyCount: 0, + valueBytes, + staleRecoveryAfterPressure: "not_measured", + note: "Set DIALCACHE_BENCH_EVICTION_KEYS on a dedicated maxmemory endpoint to exercise eviction.", + }; + } + if (server.maxmemoryBytes <= 0 || server.maxmemoryPolicy === "noeviction") { + throw new Error( + "DIALCACHE_BENCH_EVICTION_KEYS requires a dedicated endpoint with maxmemory and an eviction policy", + ); + } + + const value = Buffer.alloc(valueBytes, 0x78); + for (let index = 0; index < keyCount; index += 1) { + const key = `${namespace}:eviction-pressure:${index}`; + ownedKeys.add(key); + await admin.set(key, value, { PX: 300_000 }); + } + const recovery = await adapter.read({ valueKey: recoveryValueKey, maxAgeMs: recoveryMaxAgeMs }); + return { + attempted: true, + keyCount, + valueBytes, + requestedValueBytes: keyCount * valueBytes, + staleRecoveryAfterPressure: recovery === null ? "miss" : "served", + note: "Pressure uses only random benchmark-owned keys and never changes Redis CONFIG.", + }; +} + +function evictionReport({ + server, + evictedKeysBefore, + evictedKeysBeforePressure, + evictedKeysAfter, + residency, + pressure, +}) { + const evictedKeysTotalDelta = evictedKeysAfter - evictedKeysBefore; + const evictedKeysPressureDelta = evictedKeysAfter - evictedKeysBeforePressure; + if (pressure.attempted) { + assert.ok( + evictedKeysPressureDelta > 0, + "requested eviction pressure did not increase evicted_keys; increase DIALCACHE_BENCH_EVICTION_KEYS", + ); + } + const allSentinelsResident = Object.values(residency).every(Boolean); + const evictionConfigured = server.maxmemoryBytes > 0 && server.maxmemoryPolicy !== "noeviction"; + let assessment; + if (!allSentinelsResident) { + assessment = + "At least one benchmark sentinel disappeared. Missing values remove stale recovery capacity; missing watermarks force tracked reads to miss."; + } else if (evictionConfigured) { + assessment = + "Eviction is configured, but all benchmark sentinels remained resident. DialCache does not pin values or watermarks; either can be evicted under the configured policy."; + } else { + assessment = + "Redis eviction is not currently configured. DialCache still does not pin values or watermarks, and external deletion remains observable as a cache miss."; + } + return { + configured: evictionConfigured, + maxmemoryBytes: server.maxmemoryBytes, + maxmemoryPolicy: server.maxmemoryPolicy, + evictedKeysBefore, + evictedKeysBeforePressure, + evictedKeysAfter, + evictedKeysTotalDelta, + evictedKeysPressureDelta, + pressure, + benchmarkSentinelResidency: residency, + allSentinelsResident, + assessment, + attributionNote: + "The pressure delta is sampled immediately around owned pressure writes; global counters can still include concurrent shared-server traffic.", + }; +} + +function redisKeys({ + DialCacheKey, + invalidationPrefix, + redisClusterHashTag, + namespace, + keyType, + id, + useCase, + tracked, +}) { + const key = new DialCacheKey({ namespace, keyType, id, useCase, trackForInvalidation: tracked }); + return { + valueKey: `${key.urn}:dialcache-frame-v1`, + ...(tracked + ? { watermarkKey: `${redisClusterHashTag(invalidationPrefix(namespace, keyType, id))}#watermark` } + : {}), + }; +} + +function own(ownedKeys, key) { + ownedKeys.add(key.valueKey); + if (key.watermarkKey !== undefined) { + ownedKeys.add(key.watermarkKey); + } +} + +function countingClient(adapter, counters) { + return { + enforcesMaxAge: true, + async read(request, context) { + counters.reads += 1; + return await adapter.read(request, context); + }, + async write(request) { + return await adapter.write(request); + }, + async invalidate(request) { + return await adapter.invalidate(request); + }, + }; +} + +function staleConfig(CacheLayer, DialCacheKeyConfig, freshTtlSec, staleMaxAgeSec) { + return new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: freshTtlSec }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: staleMaxAgeSec, + }); +} + +function recordingMetrics(overrides = {}) { + return { + request() {}, + miss() {}, + disabled() {}, + error() {}, + invalidation() {}, + coalesced() {}, + shadowValidation() {}, + staleRecovery() {}, + observeGet() {}, + observeFallback() {}, + observeSerialization() {}, + observeSize() {}, + ...overrides, + }; +} + +const silentLogger = { + debug() {}, + error() {}, + warn() {}, +}; + +function makeJsonPayload(targetBytes) { + const emptyValue = { kind: "dialcache-stale-benchmark", body: "" }; + const overheadBytes = Buffer.byteLength(JSON.stringify(emptyValue)); + const value = { + ...emptyValue, + body: "x".repeat(Math.max(0, targetBytes - overheadBytes)), + }; + return { value, encoded: JSON.stringify(value) }; +} + +async function requiredMemoryUsage(admin, key, label) { + const bytes = await admin.memoryUsage(key); + assert.equal(typeof bytes, "number", `${label} should be resident for MEMORY USAGE`); + return bytes; +} + +async function waitFor(predicate, timeoutMs, label) { + const deadline = performance.now() + timeoutMs; + while (performance.now() < deadline) { + if (await predicate()) { + return; + } + await delay(25); + } + throw new Error(`Timed out after ${timeoutMs}ms waiting for ${label}`); +} + +async function withSafetyTimeout(promise, timeoutMs, label) { + let timeout; + const deadline = new Promise((_, reject) => { + timeout = globalThis.setTimeout(() => { + reject(new Error(`Timed out after ${timeoutMs}ms waiting for ${label}`)); + }, timeoutMs); + timeout.unref?.(); + }); + try { + return await Promise.race([promise, deadline]); + } finally { + globalThis.clearTimeout(timeout); + } +} + +function deferred() { + let resolve; + const promise = new Promise((resolvePromise) => { + resolve = resolvePromise; + }); + return { promise, resolve }; +} + +function readPositiveInteger(name, fallback) { + const raw = process.env[name]; + if (raw === undefined) { + return fallback; + } + const value = Number(raw); + if (!Number.isSafeInteger(value) || value <= 0) { + throw new RangeError(`${name} must be a positive safe integer`); + } + return value; +} + +function readNonnegativeInteger(name, fallback) { + const raw = process.env[name]; + if (raw === undefined) { + return fallback; + } + const value = Number(raw); + if (!Number.isSafeInteger(value) || value < 0) { + throw new RangeError(`${name} must be a nonnegative safe integer`); + } + return value; +} + +function safeEndpoint(url) { + try { + const parsed = new URL(url); + const port = parsed.port.length > 0 ? `:${parsed.port}` : ""; + return `${parsed.protocol}//${parsed.hostname}${port}`; + } catch { + return ""; + } +} + +function errorMessage(error) { + return error instanceof Error ? error.message : String(error); +} + +function round(value, digits) { + const factor = 10 ** digits; + return Math.round(value * factor) / factor; +} + +function printReport(report) { + console.log("\nStale-on-error Redis CPU and throughput"); + console.table([ + { + scenario: "fresh Redis hit", + operations: report.redisCpu.freshHit.operations, + "elapsed ms": report.redisCpu.freshHit.elapsedMs, + "ops/sec": report.redisCpu.freshHit.operationsPerSecond, + "Redis usec": report.redisCpu.freshHit.redisUsec, + "Redis usec/op": report.redisCpu.freshHit.redisUsecPerOperation, + "payload bytes/op": report.settings.actualJsonPayloadBytes, + }, + { + scenario: "logical-stale Redis miss", + operations: report.redisCpu.logicalStaleMiss.operations, + "elapsed ms": report.redisCpu.logicalStaleMiss.elapsedMs, + "ops/sec": report.redisCpu.logicalStaleMiss.operationsPerSecond, + "Redis usec": report.redisCpu.logicalStaleMiss.redisUsec, + "Redis usec/op": report.redisCpu.logicalStaleMiss.redisUsecPerOperation, + "payload bytes/op": 0, + }, + { + scenario: "SoT outage stale recovery", + operations: report.outageRecovery.operations, + "elapsed ms": report.outageRecovery.elapsedMs, + "ops/sec": report.outageRecovery.operationsPerSecond, + "Redis usec": "not isolated", + "Redis usec/op": "not isolated", + "payload bytes/op": report.outageRecovery.redisPayloadResponseBytesPerOperation, + "wire input bytes/op": report.outageRecovery.redisNetwork.inputBytesPerOperation, + "wire output bytes/op": report.outageRecovery.redisNetwork.outputBytesPerOperation, + }, + ]); + + console.log("\nMemory, retention, and eviction"); + console.table([ + { item: "logical-stale value", bytes: report.memoryUsage.logicalStaleValueBytes }, + { item: "outage value", bytes: report.memoryUsage.outageValueBytes }, + { item: "tracked value", bytes: report.memoryUsage.trackedValueBytes }, + { item: "tracked watermark", bytes: report.memoryUsage.trackedWatermarkBytes }, + { + item: "watermark after value expiry", + bytes: report.memoryUsage.expirationWatermarkBytesAfterValueExpiry, + }, + ]); + console.log( + `Coalesced fanout: ${report.coalescing.callers} callers, ${report.coalescing.sourceRejections} source rejection, ${report.coalescing.redisReads} Redis reads, ${report.coalescing.staleRecoveryMetrics} stale-recovery metric.`, + ); + console.log( + `Tracked expiration: value expired=${report.expiration.valueExpired}; watermark remained=${report.expiration.watermarkResidentAfterValueExpiry}; remaining watermark PTTL=${report.expiration.watermarkTtlAfterValueExpiryMs}ms.`, + ); + console.log(`Eviction: ${report.eviction.assessment}`); + if (report.eviction.pressure.attempted) { + console.log( + `Eviction pressure: ${report.eviction.pressure.keyCount} owned keys; evicted_keys +${report.eviction.evictedKeysPressureDelta}; stale recovery after pressure=${report.eviction.pressure.staleRecoveryAfterPressure}.`, + ); + } + console.log("Semantic assertions passed; measurements are informational and have no pass/fail timing threshold."); + console.log("\nJSON report (pasteable):"); + console.log(JSON.stringify(report, null, 2)); +} + +const redisUrl = process.env.DIALCACHE_BENCH_REDIS_URL; + +if (redisUrl === undefined || redisUrl.length === 0) { + const skipped = { + status: "skipped", + reason: "DIALCACHE_BENCH_REDIS_URL is not set", + example: "DIALCACHE_BENCH_REDIS_URL=redis://127.0.0.1:6379 corepack pnpm benchmark:stale-on-error", + }; + console.log("Stale-on-error Redis benchmark skipped: set DIALCACHE_BENCH_REDIS_URL to a dedicated Redis or Valkey endpoint."); + console.log(JSON.stringify(skipped, null, 2)); +} else { + const benchmarkTimeoutMs = readPositiveInteger("DIALCACHE_BENCH_TIMEOUT_MS", 120_000); + const watchdog = globalThis.setTimeout(() => { + console.error(`Stale-on-error Redis benchmark exceeded its ${benchmarkTimeoutMs}ms watchdog.`); + process.exit(1); + }, benchmarkTimeoutMs); + watchdog.unref?.(); + try { + const report = await runBenchmark(redisUrl); + printReport(report); + } catch (error) { + console.error("Stale-on-error Redis benchmark failed."); + console.error(error instanceof Error ? error.stack ?? error.message : error); + process.exitCode = 1; + } finally { + globalThis.clearTimeout(watchdog); + } +} diff --git a/scripts/test-package.mjs b/scripts/test-package.mjs index b2a9c58..2b24953 100644 --- a/scripts/test-package.mjs +++ b/scripts/test-package.mjs @@ -39,12 +39,15 @@ const rootConsumer = `import { type RedisConfig, type RedisInvalidationRequest, type RedisReadContext, + type RedisReadRequest, type RedisWriteRequest, type Serializer, type ShadowComparator, type ShadowConfig, type ShadowValidationMetricLabels, type ShadowValidationOutcome, + type StaleRecoveryMetricLabels, + type StaleRecoveryOutcome, } from "dialcache"; // @ts-expect-error The unused MissingKeyConfigError class was removed instead of deprecated. import { MissingKeyConfigError } from "dialcache"; @@ -86,6 +89,13 @@ const shadowMetrics: DialCacheMetricsAdapter = { void outcome; }, }; +const staleRecoveryMetrics: DialCacheMetricsAdapter = { + ...metrics, + staleRecovery: (labels: StaleRecoveryMetricLabels) => { + const outcome: StaleRecoveryOutcome = labels.outcome; + void outcome; + }, +}; const shadowOutcomes: Readonly> = { match: true, mismatch: true, @@ -102,6 +112,14 @@ const shadowOutcomes: Readonly> = { dropped: true, }; void shadowOutcomes; +const staleRecoveryOutcomes: Readonly> = { + served: true, + miss: true, + read_error: true, + read_timeout: true, + deserialization_error: true, +}; +void staleRecoveryOutcomes; const metricLayers: Readonly> = { [CacheLayer.LOCAL]: true, [CacheLayer.REMOTE]: true, @@ -153,6 +171,7 @@ const load = cache.cached(async (id: string) => id, { defaultConfig: new DialCacheKeyConfig({ ttlSec: { [CacheLayer.LOCAL]: 60, [CacheLayer.REMOTE]: 60 }, ramp: { [CacheLayer.LOCAL]: 100, [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: 3_600, remoteReadTimeoutMs: 100, }), }); @@ -319,17 +338,39 @@ const metricErrorKinds: Readonly> = { const unboundedErrorKind: MetricErrorKind = "Tenant123Error"; const customRedisClient: DialCacheRedisClient = { - // The optional second read argument preserves one-argument custom clients. - read: async () => Buffer.from([0, 255]), + enforcesMaxAge: true, + // Implementations may ignore the optional context, but every request carries a required age bound. + read: async ({ maxAgeMs }) => maxAgeMs > 0 ? Buffer.from([0, 255]) : null, write: async ({ value }) => typeof value === "string" || Buffer.isBuffer(value), invalidate: async () => undefined, }; +// @ts-expect-error Legacy clients must explicitly attest Redis-server max-age enforcement. +const legacyCustomRedisClient: DialCacheRedisClient = { + read: async () => null, + write: async () => true, + invalidate: async () => undefined, +}; +const untrackedRedisReadRequest: RedisReadRequest = { + valueKey: "plain:value", + maxAgeMs: 60_000, +}; +const trackedRedisReadRequest: RedisReadRequest = { + valueKey: "tracked:{id}:value", + watermarkKey: "tracked:{id}:watermark", + maxAgeMs: 60_000, +}; +// @ts-expect-error Logical age is a required part of the semantic read contract. +const legacyRedisReadRequest: RedisReadRequest = { valueKey: "legacy:value" }; const redisClientMethods: Readonly> = { + enforcesMaxAge: true, read: true, write: true, invalidate: true, }; void redisClientMethods; +void untrackedRedisReadRequest; +void trackedRedisReadRequest; +void legacyRedisReadRequest; const cacheHasNoFlushAll: "flushAll" extends keyof DialCache ? false : true = true; const cacheHasNoClose: "close" extends keyof DialCache ? false : true = true; const clientHasNoFlushAll: "flushAll" extends keyof DialCacheRedisClient ? false : true = true; @@ -413,6 +454,7 @@ void missingInlineDateSerializer; void requestLocalConfig; void structuralConfigProvider; void shadowCache; +void staleRecoveryMetrics; void shadowKeyConfig; void requestLocalCoalescingLabels; void cacheMetricLabels; @@ -436,6 +478,7 @@ void unboundedErrorKind; void createNodeRedisDialCacheClient; void READ_CACHE_SCRIPT; void customRedisClient; +void legacyCustomRedisClient; const globalSerializer: Serializer = { dump: () => "global", load: () => ({ source: "global" }), @@ -607,6 +650,19 @@ const idleCoalescingState = { process: { activeLeaders: 0, activeFollowers: 0, o if (JSON.stringify(coalescingState) !== JSON.stringify(idleCoalescingState)) { throw new Error("The root ESM coalescing snapshot export is invalid"); } +const legacyRedisClient = { + read: async () => null, + write: async () => true, + invalidate: async () => undefined, +}; +try { + new root.DialCache({ redis: { client: legacyRedisClient } }); + throw new Error("Expected the packed ESM runtime to reject a legacy Redis client"); +} catch (error) { + if (!(error instanceof TypeError) || error.message !== "DialCache Redis client must declare enforcesMaxAge: true") { + throw error; + } +} const timeoutCache = new root.DialCache(); const neverSettles = timeoutCache.cached(async () => await new Promise(() => undefined), { keyType: "id", @@ -638,6 +694,7 @@ if ("MissingKeyConfigError" in root) { const esmDisabledOverlay = root.DialCacheKeyConfig.disabled(); if ( esmDisabledOverlay.requestLocal !== false + || esmDisabledOverlay.staleOnErrorMaxAgeSec !== 0 || esmDisabledOverlay.shadow?.ramp !== 0 || esmDisabledOverlay.shadow.logMismatches !== false || esmDisabledOverlay.ramp[root.CacheLayer.LOCAL] !== 0 @@ -745,7 +802,11 @@ console.log("${observerIsolationMarker}");`, let payload = Buffer.alloc(4 * 1024 * 1024, 1); const payloadReference = new WeakRef(payload); const redis = { - read: async () => payload, + enforcesMaxAge: true, + read: async ({ maxAgeMs }) => { + if (!(maxAgeMs > 0)) throw new Error("missing Redis read age"); + return payload; + }, write: async () => true, invalidate: async () => undefined, }; @@ -843,6 +904,19 @@ const idleCoalescingState = { process: { activeLeaders: 0, activeFollowers: 0, o if (JSON.stringify(coalescingState) !== JSON.stringify(idleCoalescingState)) { throw new Error("The root CommonJS coalescing snapshot export is invalid"); } +const legacyRedisClient = { + read: async () => null, + write: async () => true, + invalidate: async () => undefined, +}; +try { + new root.DialCache({ redis: { client: legacyRedisClient } }); + throw new Error("Expected the packed CommonJS runtime to reject a legacy Redis client"); +} catch (error) { + if (!(error instanceof TypeError) || error.message !== "DialCache Redis client must declare enforcesMaxAge: true") { + throw error; + } +} const timeoutCache = new root.DialCache(); const neverSettles = timeoutCache.cached(async () => await new Promise(() => undefined), { keyType: "id", @@ -876,6 +950,7 @@ if ("MissingKeyConfigError" in root) { const cjsDisabledOverlay = root.DialCacheKeyConfig.disabled(); if ( cjsDisabledOverlay.requestLocal !== false + || cjsDisabledOverlay.staleOnErrorMaxAgeSec !== 0 || cjsDisabledOverlay.shadow?.ramp !== 0 || cjsDisabledOverlay.shadow.logMismatches !== false || cjsDisabledOverlay.ramp[root.CacheLayer.LOCAL] !== 0 diff --git a/src/config.ts b/src/config.ts index b40c24f..acd358a 100644 --- a/src/config.ts +++ b/src/config.ts @@ -33,6 +33,14 @@ export class DialCacheKeyConfig { * Request-local caching is disabled by default and has no TTL or ramp. */ readonly requestLocal?: boolean; + /** + * Absolute Redis-frame age in seconds through which a retained value may be + * returned after the source of truth rejects. Omission disables recovery by + * default and inherits in runtime overlays; zero explicitly disables an + * inherited policy. A positive value requires a smaller positive remote TTL + * and may not exceed 31,536,000 seconds (365 days). + */ + readonly staleOnErrorMaxAgeSec?: number; /** * Maximum time DialCache waits for a remote read before failing open to the * source of truth. Overrides the instance default for this use case. @@ -44,6 +52,7 @@ export class DialCacheKeyConfig { ramp?: LayerConfig; shadow?: ShadowConfig; requestLocal?: boolean; + staleOnErrorMaxAgeSec?: number; remoteReadTimeoutMs?: number; }) { if (config === null || typeof config !== "object" || Array.isArray(config)) { @@ -64,6 +73,11 @@ export class DialCacheKeyConfig { if (config.requestLocal !== undefined) { this.requestLocal = config.requestLocal; } + // Like ttlSec/ramp leaves, validation is deferred to static-default capture + // or runtime resolution so malformed runtime policy can fail open narrowly. + if (config.staleOnErrorMaxAgeSec !== undefined) { + this.staleOnErrorMaxAgeSec = config.staleOnErrorMaxAgeSec; + } if (config.remoteReadTimeoutMs !== undefined) { assertValidDeadlineMs(config.remoteReadTimeoutMs, "DialCache remoteReadTimeoutMs"); this.remoteReadTimeoutMs = config.remoteReadTimeoutMs; @@ -93,6 +107,7 @@ export class DialCacheKeyConfig { static disabled(): DialCacheKeyConfig { return new DialCacheKeyConfig({ requestLocal: false, + staleOnErrorMaxAgeSec: 0, shadow: { ramp: 0, logMismatches: false, diff --git a/src/datadog.ts b/src/datadog.ts index 3f58607..7d549e6 100644 --- a/src/datadog.ts +++ b/src/datadog.ts @@ -7,6 +7,7 @@ import type { InvalidationMetricLabels, SerializationMetricLabels, ShadowValidationMetricLabels, + StaleRecoveryMetricLabels, } from "./metrics.js"; export type DatadogObservationMetricType = "histogram" | "distribution"; @@ -46,6 +47,7 @@ const METRIC_SUFFIXES = { invalidation: "invalidation.count", coalesced: "coalesced.count", shadowValidation: "shadow.count", + staleRecovery: "stale_recovery.count", get: "get.duration", fallback: "fallback.duration", serialization: "serialization.duration", @@ -122,6 +124,15 @@ export class DatadogDialCacheMetrics implements DialCacheMetricsAdapter { }); } + staleRecovery(labels: StaleRecoveryMetricLabels): void { + return this.increment(this.metricNames.staleRecovery, { + cache_namespace: labels.cacheNamespace, + use_case: labels.useCase, + key_type: labels.keyType, + outcome: labels.outcome, + }); + } + observeGet(labels: CacheMetricLabels, seconds: number): void { this.observe(this.metricNames.get, seconds, cacheTags(labels)); } diff --git a/src/dialcache.ts b/src/dialcache.ts index 4c670c1..52e77a0 100644 --- a/src/dialcache.ts +++ b/src/dialcache.ts @@ -38,8 +38,10 @@ import { RedisCache } from "./internal/redis-cache.js"; import { fetchKeyConfig, resolveLayerConfigResult, + resolveRemoteLayerConfigResult, type LayerConfigResolution, type ResolvedLayerConfig, + type ResolvedRemoteLayerConfig, } from "./internal/runtime-config.js"; import { shadowMismatchLogDetails } from "./internal/shadow-log-json.js"; @@ -229,23 +231,27 @@ interface ShadowMismatchDetails { } type ShadowValidationStart = - | { readonly kind: "retained"; readonly payload: RedisCachePayload } + | { + readonly kind: "retained"; + readonly payload: RedisCachePayload; + readonly remoteConfig: ResolvedRemoteLayerConfig; + } | { readonly kind: "redis"; /** The caller-owned, fallback-deadline-bounded SoT operation. */ readonly source: Promise; /** Valid remote policy retained even though its serving ramp excluded this key. */ - readonly remoteConfig: ResolvedLayerConfig; + readonly remoteConfig: ResolvedRemoteLayerConfig; /** Includes synchronous SoT work that ran before shadow admission. */ readonly startedAtMs: number | null; }; type ShadowValidationRunStart = - | { readonly kind: "retained" } + | { readonly kind: "retained"; readonly remoteConfig: ResolvedRemoteLayerConfig } | { readonly kind: "redis"; readonly source: Promise; - readonly remoteConfig: ResolvedLayerConfig; + readonly remoteConfig: ResolvedRemoteLayerConfig; }; const DEFAULT_LOCAL_MAX_SIZE = 10_000; @@ -712,7 +718,7 @@ export class DialCache { key: DialCacheKey, keyConfig: DialCacheKeyConfig | null, local: CacheGetResult | null, - remoteConfig: ResolvedLayerConfig, + remoteConfig: ResolvedRemoteLayerConfig, fallbackLabels: CacheMetricLabels, fallback: () => Promise, shadowValidation: ShadowValidationPlan, @@ -745,20 +751,22 @@ export class DialCache { remote: RemoteCacheGetResult, fallback: () => Promise, shadowValidation: ShadowValidationPlan, - resolvedRemoteConfig?: ResolvedLayerConfig, + resolvedRemoteConfig?: ResolvedRemoteLayerConfig, ): Promise { if (remote.status === "hit") { if (local.status === "miss") { await this.putLocalFailOpen(key, remote.value, local.config); } - this.scheduleShadowValidation( - redisCache, - key, - keyConfig, - { kind: "retained", payload: remote.payload }, - shadowValidation, - keyConfig?.remoteReadTimeoutMs ?? redisCache.readTimeoutMs, - ); + if (resolvedRemoteConfig !== undefined) { + this.scheduleShadowValidation( + redisCache, + key, + keyConfig, + { kind: "retained", payload: remote.payload, remoteConfig: resolvedRemoteConfig }, + shadowValidation, + keyConfig?.remoteReadTimeoutMs ?? redisCache.readTimeoutMs, + ); + } return remote.value; } @@ -771,9 +779,37 @@ export class DialCache { } const remoteErrored = remote.status === "disabled" && remote.reason === "config_error"; - const remoteWriteConfig = remote.status === "miss" ? remote.config : remoteErrored ? resolvedRemoteConfig : undefined; + const remoteWriteConfig = remote.status === "miss" || remoteErrored ? resolvedRemoteConfig : undefined; const fallbackLayer = remote.status === "miss" || remoteErrored ? CacheLayer.REMOTE : CacheLayer.LOCAL; - const value = await this.callFallback(labelsFor(key, fallbackLayer), fallback); + let value: T; + try { + value = await this.callFallback(labelsFor(key, fallbackLayer), fallback); + } catch (fallbackError) { + if ( + remote.status === "miss" + && remote.skipStaleRecovery !== true + && resolvedRemoteConfig !== undefined + && resolvedRemoteConfig.staleOnErrorMaxAgeSec !== null + ) { + try { + const recovered = await redisCache.recoverWithResolvedConfig( + key, + resolvedRemoteConfig, + keyConfig?.remoteReadTimeoutMs ?? redisCache.readTimeoutMs, + ); + if (recovered.status === "hit") { + return recovered.value; + } + if (recovered.status === "error") { + this.logger.warn("Error getting value from Redis cache during stale recovery", recovered.error); + } + } catch (recoveryError) { + // Recovery is subordinate to the source rejection and must never replace it. + this.logger.warn("Error getting value from Redis cache during stale recovery", recoveryError); + } + } + throw fallbackError; + } const skipCacheWrite = (remote.status === "miss" || remote.status === "disabled") && remote.skipCacheWrite === true; let suppressCacheWrite = skipCacheWrite; if (!suppressCacheWrite && remoteWriteConfig !== undefined) { @@ -850,7 +886,7 @@ export class DialCache { : performance.now(), }; const runStart: ShadowValidationRunStart = start.kind === "retained" - ? { kind: "retained" } + ? { kind: "retained", remoteConfig: start.remoteConfig } : { kind: "redis", source: start.source, remoteConfig: start.remoteConfig }; this.shadowFlights.set(key.urn, flight); this.deferShadowValidation( @@ -938,7 +974,11 @@ export class DialCache { maybeRelease(); }; const readShadowPayload = (): Promise => { - const read = redisCache.startTrackedPayloadReadForShadow(key, readTimeoutMs); + const read = redisCache.startTrackedPayloadReadForShadow( + key, + start.remoteConfig.ttlSec, + readTimeoutMs, + ); pendingRedisReads.add(read.settled); void read.settled.then(() => { pendingRedisReads.delete(read.settled); @@ -975,7 +1015,7 @@ export class DialCache { return "timeout"; } - let shadowFillConfig: ResolvedLayerConfig | null = null; + let shadowFillConfig: ResolvedRemoteLayerConfig | null = null; if (start.kind === "redis") { let payload: RedisCachePayload | null; try { @@ -1191,11 +1231,13 @@ export class DialCache { private async resolveRemoteLayerConfig(key: DialCacheKey, keyConfig: DialCacheKeyConfig | null) { try { - const result = resolveLayerConfigResult({ + const result = resolveRemoteLayerConfigResult({ config: keyConfig, key, - layer: CacheLayer.REMOTE, }); + if (result.staleOnErrorConfigError === true) { + this.recordError(key, CacheLayer.REMOTE, "config_resolution"); + } if (result.status === "disabled") { this.metrics?.disabled({ ...labelsFor(key, CacheLayer.REMOTE), reason: result.reason }); this.recordInvalidLeaf(key, CacheLayer.REMOTE, result.reason); @@ -1212,7 +1254,7 @@ export class DialCache { private async readRemoteWithResolvedConfig( redisCache: RedisCache, key: DialCacheKey, - layerConfig: ResolvedLayerConfig, + layerConfig: ResolvedRemoteLayerConfig, readTimeoutMs: number, ): Promise> { try { @@ -1387,6 +1429,7 @@ function snapshotDefaultConfig(config: DialCacheKeyConfig | null | undefined): D const rampConfig = config.ramp; const shadowConfig = config.shadow; const requestLocal = config.requestLocal; + const staleOnErrorMaxAgeSec = config.staleOnErrorMaxAgeSec; const remoteReadTimeoutMs = config.remoteReadTimeoutMs; if (requestLocal !== undefined && typeof requestLocal !== "boolean") { throw new TypeError("DialCache defaultConfig requestLocal must be a boolean"); @@ -1400,6 +1443,7 @@ function snapshotDefaultConfig(config: DialCacheKeyConfig | null | undefined): D ttlSec: ttlSecConfig, ramp: rampConfig, ...(requestLocal === undefined ? {} : { requestLocal }), + ...(staleOnErrorMaxAgeSec === undefined ? {} : { staleOnErrorMaxAgeSec }), ...(remoteReadTimeoutMs === undefined ? {} : { remoteReadTimeoutMs }), ...(shadowConfig === undefined ? {} : { shadow: shadowConfig }), }); @@ -1428,6 +1472,31 @@ function snapshotDefaultConfig(config: DialCacheKeyConfig | null | undefined): D } } + if (snapshot.staleOnErrorMaxAgeSec !== undefined) { + const maxAgeSec = snapshot.staleOnErrorMaxAgeSec; + if (typeof maxAgeSec !== "number") { + throw new TypeError("DialCache defaultConfig staleOnErrorMaxAgeSec must be a number"); + } + if (!Number.isSafeInteger(maxAgeSec) || maxAgeSec < 0 || maxAgeSec > MAX_CACHE_TTL_SEC) { + throw new RangeError( + `DialCache defaultConfig staleOnErrorMaxAgeSec must be a nonnegative safe integer no greater than ${MAX_CACHE_TTL_SEC}`, + ); + } + if (maxAgeSec > 0) { + const remoteTtlSec = snapshot.ttlSec[CacheLayer.REMOTE]; + if (remoteTtlSec === undefined) { + throw new RangeError( + "DialCache defaultConfig staleOnErrorMaxAgeSec requires ttlSec.remote", + ); + } + if (maxAgeSec <= remoteTtlSec) { + throw new RangeError( + "DialCache defaultConfig staleOnErrorMaxAgeSec must be greater than ttlSec.remote", + ); + } + } + } + if (snapshot.shadow !== undefined) { if (snapshot.shadow.ramp !== undefined) { if (typeof snapshot.shadow.ramp !== "number") { @@ -1525,6 +1594,12 @@ function safeMetrics(metrics: DialCacheMetricsAdapter | null): DialCacheMetricsA callObserver(() => metrics.shadowValidation!(labels)), } : {}), + ...(typeof metrics.staleRecovery === "function" + ? { + staleRecovery: (labels) => + callObserver(() => metrics.staleRecovery!(labels)), + } + : {}), observeGet: (labels, seconds) => callObserver(() => metrics.observeGet(labels, seconds)), observeFallback: (labels, seconds) => callObserver(() => metrics.observeFallback(labels, seconds)), observeSerialization: (labels, seconds) => callObserver(() => metrics.observeSerialization(labels, seconds)), diff --git a/src/index.ts b/src/index.ts index bfa5a63..0fd50fd 100644 --- a/src/index.ts +++ b/src/index.ts @@ -15,6 +15,8 @@ export type { SerializationMetricLabels, ShadowValidationMetricLabels, ShadowValidationOutcome, + StaleRecoveryMetricLabels, + StaleRecoveryOutcome, } from "./metrics.js"; export { DialCacheError, diff --git a/src/internal/cache-result.ts b/src/internal/cache-result.ts index 85b8188..420005d 100644 --- a/src/internal/cache-result.ts +++ b/src/internal/cache-result.ts @@ -13,7 +13,11 @@ export type CacheGetResult = export type RedisCacheGetResult = | { readonly status: "hit"; readonly value: T; readonly payload: RedisCachePayload } - | Exclude, { readonly status: "hit" }>; + | (Extract, { readonly status: "miss" }> & { + /** The present payload failed normal deserialization and cannot later qualify as stale. */ + readonly skipStaleRecovery?: boolean; + }) + | Extract, { readonly status: "disabled" }>; export type RemoteCacheGetResult = | RedisCacheGetResult diff --git a/src/internal/redis-cache.ts b/src/internal/redis-cache.ts index 3241a5d..445bc15 100644 --- a/src/internal/redis-cache.ts +++ b/src/internal/redis-cache.ts @@ -9,13 +9,18 @@ import { type DialCacheMetricsAdapter, type MetricErrorKind, type MetricLayer, + type StaleRecoveryOutcome, } from "../metrics.js"; import type { DialCacheRedisClient, RedisCachePayload } from "../redis-client.js"; import { JsonSerializer, type Serializer } from "../serializer.js"; import type { RedisCacheGetResult } from "./cache-result.js"; import { assertValidDeadlineMs, withMonotonicDeadline } from "./deadline.js"; import { cacheTtlSecToMs } from "./duration.js"; -import { fetchKeyConfig, resolveLayerConfigResult, type ResolvedLayerConfig } from "./runtime-config.js"; +import { + fetchKeyConfig, + resolveRemoteLayerConfigResult, + type ResolvedRemoteLayerConfig, +} from "./runtime-config.js"; export interface RedisConfig { /** @@ -44,6 +49,11 @@ interface StartedRedisRead { readonly settled: Promise; } +type RedisStaleRecoveryResult = + | { readonly status: "hit"; readonly value: T } + | { readonly status: "miss" } + | { readonly status: "error"; readonly error: unknown }; + const defaultSerializer = new JsonSerializer(); const REDIS_FRAME_KEY_SUFFIX = ":dialcache-frame-v1"; const DEFAULT_REMOTE_READ_TIMEOUT_MS = 50; @@ -79,6 +89,9 @@ export class RedisCache { if (options.redis.client === undefined) { throw new TypeError("Redis config requires client"); } + if (options.redis.client.enforcesMaxAge !== true) { + throw new TypeError("DialCache Redis client must declare enforcesMaxAge: true"); + } this.client = options.redis.client; } @@ -103,7 +116,7 @@ export class RedisCache { async getWithResolvedConfig( key: DialCacheKey, - layerConfig: ResolvedLayerConfig, + layerConfig: ResolvedRemoteLayerConfig, readTimeoutMs = this.readTimeoutMs, ): Promise> { const metricLayer = CacheLayer.REMOTE; @@ -112,7 +125,12 @@ export class RedisCache { try { let payload: RedisCachePayload | null; try { - payload = await this.startPayloadRead(key, readTimeoutMs, false).result; + payload = await this.startPayloadRead( + key, + cacheTtlSecToMs(layerConfig.ttlSec), + readTimeoutMs, + false, + ).result; } catch (error) { this.recordError( key, @@ -131,7 +149,7 @@ export class RedisCache { return { status: "hit", value, payload }; } catch { this.recordMetric((metrics) => metrics.miss(labelsFor(key, metricLayer))); - return { status: "miss", config: layerConfig }; + return { status: "miss", config: layerConfig, skipStaleRecovery: true }; } } finally { // Preserve the established caller-serving boundary: Redis read plus load. @@ -139,6 +157,64 @@ export class RedisCache { } } + /** + * Reread a definitive normal miss after the source rejects, using the + * configured absolute recovery age. Every failure is contained so it cannot + * replace the original source rejection held by the caller. + */ + async recoverWithResolvedConfig( + key: DialCacheKey, + layerConfig: ResolvedRemoteLayerConfig, + readTimeoutMs: number, + ): Promise> { + const metricLayer = CacheLayer.REMOTE; + const maxAgeSec = layerConfig.staleOnErrorMaxAgeSec; + if (maxAgeSec === null) { + throw new Error("DialCache stale recovery requires an enabled maximum age"); + } + + const start = performance.now(); + this.recordMetric((metrics) => metrics.request(labelsFor(key, metricLayer))); + try { + let payload: RedisCachePayload | null; + try { + payload = await this.startPayloadRead( + key, + cacheTtlSecToMs(maxAgeSec), + readTimeoutMs, + false, + ).result; + } catch (error) { + const outcome = error instanceof RedisReadTimeoutError ? "read_timeout" : "read_error"; + this.recordError( + key, + metricLayer, + error instanceof RedisReadTimeoutError ? "cache_read_timeout" : "cache_read", + ); + this.recordStaleRecovery(key, outcome); + return { status: "error", error }; + } + + if (payload === null) { + this.recordMetric((metrics) => metrics.miss(labelsFor(key, metricLayer))); + this.recordStaleRecovery(key, "miss"); + return { status: "miss" }; + } + + try { + const value = await this.deserializePayload(key, payload, metricLayer); + this.recordStaleRecovery(key, "served"); + return { status: "hit", value }; + } catch { + this.recordMetric((metrics) => metrics.miss(labelsFor(key, metricLayer))); + this.recordStaleRecovery(key, "deserialization_error"); + return { status: "miss" }; + } + } finally { + this.recordMetric((metrics) => metrics.observeGet(labelsFor(key, metricLayer), elapsedSeconds(start))); + } + } + /** * Decode the retained Redis payload again for detached semantic comparison, * recording it separately from caller-serving Redis work. @@ -155,6 +231,7 @@ export class RedisCache { */ startTrackedPayloadReadForShadow( key: DialCacheKey, + maxAgeSec: number, readTimeoutMs: number, ): StartedRedisRead { if (!key.trackForInvalidation) { @@ -162,25 +239,28 @@ export class RedisCache { } return this.startMeasuredPayloadRead( key, + cacheTtlSecToMs(maxAgeSec), readTimeoutMs, REMOTE_SHADOW_CACHE_LAYER, true, ); } - async put(key: DialCacheKey, value: T, config?: { readonly ttlSec: number }): Promise { - const ttlSec = config?.ttlSec ?? await this.resolveRemoteTtlSec(key); - if (ttlSec === null) { + async put(key: DialCacheKey, value: T, config?: ResolvedRemoteLayerConfig): Promise { + const retentionTtlSec = config === undefined + ? await this.resolveRemoteRetentionTtlSec(key) + : retentionTtlSecFor(config); + if (retentionTtlSec === null) { return true; } - return await this.putWithLayer(key, value, ttlSec, CacheLayer.REMOTE); + return await this.putWithLayer(key, value, retentionTtlSec, CacheLayer.REMOTE); } /** Populate a definitive detached tracked miss using the caller's resolved policy snapshot. */ async putForShadow( key: DialCacheKey, value: T, - config: { readonly ttlSec: number }, + config: ResolvedRemoteLayerConfig, shouldWrite: () => boolean, ): Promise { if (!key.trackForInvalidation) { @@ -189,7 +269,7 @@ export class RedisCache { return await this.putWithLayer( key, value, - config.ttlSec, + retentionTtlSecFor(config), REMOTE_SHADOW_CACHE_LAYER, shouldWrite, ); @@ -271,6 +351,7 @@ export class RedisCache { private startPayloadRead( key: DialCacheKey, + maxAgeMs: number, readTimeoutMs: number, unrefTimer: boolean, ): StartedRedisRead { @@ -280,6 +361,7 @@ export class RedisCache { { valueKey: this.redisKey(key), ...(key.trackForInvalidation ? { watermarkKey: this.redisWatermarkKeyFromKey(key) } : {}), + maxAgeMs, }, { timeoutMs: readTimeoutMs, signal: abortController.signal }, ) @@ -302,13 +384,14 @@ export class RedisCache { private startMeasuredPayloadRead( key: DialCacheKey, + maxAgeMs: number, readTimeoutMs: number, metricLayer: MetricLayer, unrefTimer: boolean, ): StartedRedisRead { const start = performance.now(); this.recordMetric((metrics) => metrics.request(labelsFor(key, metricLayer))); - const read = this.startPayloadRead(key, readTimeoutMs, unrefTimer); + const read = this.startPayloadRead(key, maxAgeMs, readTimeoutMs, unrefTimer); const result = read.result.then( (payload) => { if (payload === null) { @@ -355,16 +438,15 @@ export class RedisCache { private async resolveRemoteLayerConfig(key: DialCacheKey, keyConfig?: DialCacheKeyConfig | null) { const config = keyConfig === undefined ? await fetchKeyConfig(this.configProvider, key) : keyConfig; - return resolveLayerConfigResult({ + return resolveRemoteLayerConfigResult({ config, key, - layer: CacheLayer.REMOTE, }); } - private async resolveRemoteTtlSec(key: DialCacheKey): Promise { + private async resolveRemoteRetentionTtlSec(key: DialCacheKey): Promise { const layerConfig = await this.resolveRemoteLayerConfig(key); - return layerConfig.status === "enabled" ? layerConfig.config.ttlSec : null; + return layerConfig.status === "enabled" ? retentionTtlSecFor(layerConfig.config) : null; } private recordMetric(record: (metrics: DialCacheMetricsAdapter) => void): void { @@ -381,6 +463,19 @@ export class RedisCache { private recordError(key: DialCacheKey, layer: MetricLayer, kind: MetricErrorKind): void { this.recordMetric((metrics) => metrics.error({ ...labelsFor(key, layer), error: kind, inFallback: false })); } + + private recordStaleRecovery(key: DialCacheKey, outcome: StaleRecoveryOutcome): void { + this.recordMetric((metrics) => metrics.staleRecovery?.({ + cacheNamespace: key.namespace, + useCase: key.useCase, + keyType: key.keyType, + outcome, + })); + } +} + +function retentionTtlSecFor(config: ResolvedRemoteLayerConfig): number { + return config.staleOnErrorMaxAgeSec ?? config.ttlSec; } function payloadSize(payload: string | Buffer): number { diff --git a/src/internal/redis-scripts.ts b/src/internal/redis-scripts.ts index 891c0ac..d980a11 100644 --- a/src/internal/redis-scripts.ts +++ b/src/internal/redis-scripts.ts @@ -34,6 +34,18 @@ if string.byte(value, 1) ~= ${REDIS_FRAME_VERSION} then return false end`; +const VALIDATE_READ_ARGUMENTS_LUA = String.raw`local max_age_ms = tonumber(ARGV[1]) +if not max_age_ms + or max_age_ms ~= max_age_ms + or max_age_ms >= math.huge + or max_age_ms <= -math.huge + or max_age_ms ~= math.floor(max_age_ms) + or max_age_ms <= 0 + or max_age_ms > ${MAX_SUPPORTED_DURATION_MS} +then + return redis.error_reply("ERR invalid DialCache max age") +end`; + const RETURN_PAYLOAD_LUA = String.raw`return string.sub(value, 10)`; const VALIDATE_WRITE_ARGUMENTS_LUA = String.raw`local cache_ttl_ms = ceil_finite_number(ARGV[1]) @@ -48,17 +60,31 @@ end`; const REDIS_TIME_LUA = String.raw`local redis_time = redis.call("TIME") local now_ms = tonumber(redis_time[1]) * 1000 + math.floor(tonumber(redis_time[2]) / 1000)`; +const VALIDATE_FRAME_AGE_LUA = String.raw`local created_at = struct.unpack(">I8", string.sub(value, 2, 9)) +if now_ms - created_at >= max_age_ms then + return false +end`; + const WRITE_FRAME_LUA = String.raw`local frame = string.char(${REDIS_FRAME_VERSION}) .. struct.pack(">I8", now_ms) .. string.char(encoding) .. ARGV[3] redis.call("SET", KEYS[1], frame, "PX", cache_ttl_ms)`; -export const READ_CACHE_SCRIPT = [READ_FRAME_LUA, RETURN_PAYLOAD_LUA].join("\n\n"); +export const READ_CACHE_SCRIPT = [ + VALIDATE_READ_ARGUMENTS_LUA, + READ_FRAME_LUA, + REDIS_TIME_LUA, + VALIDATE_FRAME_AGE_LUA, + RETURN_PAYLOAD_LUA, +].join("\n\n"); export const READ_TRACKED_CACHE_SCRIPT = [ PARSE_WATERMARK_LUA, + VALIDATE_READ_ARGUMENTS_LUA, READ_FRAME_LUA, + REDIS_TIME_LUA, + VALIDATE_FRAME_AGE_LUA, String.raw`local raw_watermark = redis.call("GET", KEYS[2]) if not raw_watermark then return false @@ -69,7 +95,6 @@ if not watermark then return false end -local created_at = struct.unpack(">I8", string.sub(value, 2, 9)) if created_at <= watermark then return false end`, diff --git a/src/internal/runtime-config.ts b/src/internal/runtime-config.ts index fec7c4e..15d56af 100644 --- a/src/internal/runtime-config.ts +++ b/src/internal/runtime-config.ts @@ -15,25 +15,43 @@ export interface ResolvedLayerConfig { readonly ramp: number; } -export type LayerConfigResolution = - | { readonly status: "enabled"; readonly config: ResolvedLayerConfig } +/** Remote-only policy resolved against the same invocation snapshot as its TTL. */ +export interface ResolvedRemoteLayerConfig extends ResolvedLayerConfig { + readonly staleOnErrorMaxAgeSec: number | null; +} + +export type LayerConfigResolution = + | { readonly status: "enabled"; readonly config: Config } | { readonly status: "disabled"; readonly reason: "ramped_down"; /** Valid policy retained even though its ramp excluded this key. */ - readonly config: ResolvedLayerConfig; + readonly config: Config; } | { readonly status: "disabled"; readonly reason: Exclude; }; +/** + * A malformed optional stale policy is diagnostic-only: the valid remote layer + * remains available with recovery disabled. + */ +export type RemoteLayerConfigResolution = LayerConfigResolution & { + readonly staleOnErrorConfigError?: true; +}; + interface ResolveLayerConfigOptions { readonly config: DialCacheKeyConfig | null; readonly key: DialCacheKey; readonly layer: CacheLayer; } +interface ResolveRemoteLayerConfigOptions { + readonly config: DialCacheKeyConfig | null; + readonly key: DialCacheKey; +} + export async function fetchKeyConfig( configProvider: CacheConfigProvider, key: DialCacheKey, @@ -91,6 +109,48 @@ export function resolveLayerConfigResult(options: ResolveLayerConfigOptions): La : { status: "disabled", reason: "ramped_down", config: { ttlSec, ramp } }; } +export function resolveRemoteLayerConfigResult( + options: ResolveRemoteLayerConfigOptions, +): RemoteLayerConfigResolution { + const resolution = resolveLayerConfigResult({ + ...options, + layer: CacheLayer.REMOTE, + }); + const configuredMaxAge: unknown = options.config?.staleOnErrorMaxAgeSec; + if (!("config" in resolution)) { + if ( + resolution.reason === "policy_disabled" + && configuredMaxAge !== undefined + && configuredMaxAge !== 0 + ) { + return { ...resolution, staleOnErrorConfigError: true }; + } + return resolution; + } + + if (configuredMaxAge === undefined || configuredMaxAge === 0) { + return { + ...resolution, + config: { ...resolution.config, staleOnErrorMaxAgeSec: null }, + }; + } + if ( + !isSupportedCacheTtlSec(configuredMaxAge) + || configuredMaxAge <= resolution.config.ttlSec + ) { + return { + ...resolution, + config: { ...resolution.config, staleOnErrorMaxAgeSec: null }, + staleOnErrorConfigError: true, + }; + } + + return { + ...resolution, + config: { ...resolution.config, staleOnErrorMaxAgeSec: configuredMaxAge }, + }; +} + function mergeKeyConfig( defaultConfig: DialCacheKeyConfig | null, runtimeConfig: DialCacheKeyConfig | null | undefined, @@ -108,12 +168,16 @@ function mergeKeyConfig( const remoteReadTimeoutMs = overlay?.remoteReadTimeoutMs !== undefined ? overlay.remoteReadTimeoutMs : defaultConfig?.remoteReadTimeoutMs; + const staleOnErrorMaxAgeSec = overlay?.staleOnErrorMaxAgeSec !== undefined + ? overlay.staleOnErrorMaxAgeSec + : defaultConfig?.staleOnErrorMaxAgeSec; const shadow = mergeShadowConfig(defaultConfig?.shadow, overlay?.shadow); return new DialCacheKeyConfig({ ttlSec: mergeLayerConfig(defaultConfig?.ttlSec, overlay?.ttlSec, "ttlSec"), ramp: mergeLayerConfig(defaultConfig?.ramp, overlay?.ramp, "ramp"), requestLocal, + ...(staleOnErrorMaxAgeSec === undefined ? {} : { staleOnErrorMaxAgeSec }), ...(remoteReadTimeoutMs === undefined ? {} : { remoteReadTimeoutMs }), ...(shadow === undefined ? {} : { shadow }), }); diff --git a/src/metrics.ts b/src/metrics.ts index 3b4638a..2fb5dd6 100644 --- a/src/metrics.ts +++ b/src/metrics.ts @@ -25,6 +25,13 @@ export type ShadowValidationOutcome = | "confirmation_error" | "timeout" | "dropped"; +/** Bounded terminal outcomes for an attempted stale-on-error Redis recovery. */ +export type StaleRecoveryOutcome = + | "served" + | "miss" + | "read_error" + | "read_timeout" + | "deserialization_error"; /** Bounded reasons for skipping cache work; policy_disabled means a shared layer has no effective TTL. */ export type DisabledReason = "context" | "policy_disabled" | "invalid_ttl" | "invalid_ramp" | "ramped_down" | "config_error"; /** Stable failure sites used instead of backend- or application-defined error names. */ @@ -81,6 +88,13 @@ export interface ShadowValidationMetricLabels { readonly outcome: ShadowValidationOutcome; } +export interface StaleRecoveryMetricLabels { + readonly cacheNamespace: string; + readonly useCase: string; + readonly keyType: string; + readonly outcome: StaleRecoveryOutcome; +} + export interface DialCacheMetricsAdapter { request(labels: CacheMetricLabels): void; miss(labels: CacheMetricLabels): void; @@ -91,6 +105,8 @@ export interface DialCacheMetricsAdapter { coalesced?(labels: CoalescedMetricLabels): void; // Optional so existing custom adapters keep compiling without changes. shadowValidation?(labels: ShadowValidationMetricLabels): void; + // Optional so existing custom adapters keep compiling without changes. + staleRecovery?(labels: StaleRecoveryMetricLabels): void; observeGet(labels: CacheMetricLabels, seconds: number): void; observeFallback(labels: CacheMetricLabels, seconds: number): void; observeSerialization(labels: SerializationMetricLabels, seconds: number): void; diff --git a/src/node-redis.ts b/src/node-redis.ts index 2093b8f..61d3af9 100644 --- a/src/node-redis.ts +++ b/src/node-redis.ts @@ -46,8 +46,11 @@ function defineDialCacheScript, Reply>( } export type DialCacheNodeRedisScripts = { - readonly dialcacheRead: NodeRedisScript<[valueKey: string], string | null>; - readonly dialcacheReadTracked: NodeRedisScript<[valueKey: string, watermarkKey: string], string | null>; + readonly dialcacheRead: NodeRedisScript<[valueKey: string, maxAgeMs: number], string | null>; + readonly dialcacheReadTracked: NodeRedisScript< + [valueKey: string, watermarkKey: string, maxAgeMs: number], + string | null + >; readonly dialcacheWrite: NodeRedisScript< [valueKey: string, cacheTtlMs: number, encoding: number, payload: string | Buffer], number @@ -74,8 +77,8 @@ export const dialcacheRedisScripts: DialCacheNodeRedisScripts = { NUMBER_OF_KEYS: 1, FIRST_KEY_INDEX: 0, IS_READ_ONLY: true, - transformArguments(valueKey: string): Array { - return [valueKey]; + transformArguments(valueKey: string, maxAgeMs: number): Array { + return [valueKey, String(maxAgeMs)]; }, transformReply: readReply, }), @@ -85,8 +88,8 @@ export const dialcacheRedisScripts: DialCacheNodeRedisScripts = { FIRST_KEY_INDEX: 0, // Replica lag must not hide a newly-written invalidation watermark. IS_READ_ONLY: false, - transformArguments(valueKey: string, watermarkKey: string): Array { - return [valueKey, watermarkKey]; + transformArguments(valueKey: string, watermarkKey: string, maxAgeMs: number): Array { + return [valueKey, watermarkKey, String(maxAgeMs)]; }, transformReply: readReply, }), @@ -134,11 +137,12 @@ export const dialcacheRedisScripts: DialCacheNodeRedisScripts = { }; interface NodeRedisScriptClient { - dialcacheRead(options: BufferReplyOptions, valueKey: string): Promise; + dialcacheRead(options: BufferReplyOptions, valueKey: string, maxAgeMs: number): Promise; dialcacheReadTracked( options: BufferReplyOptions, valueKey: string, watermarkKey: string, + maxAgeMs: number, ): Promise; dialcacheWrite(valueKey: string, cacheTtlMs: number, encoding: number, payload: string | Buffer): Promise; dialcacheWriteTracked( @@ -160,13 +164,14 @@ interface NodeRedisScriptClient { */ export function createNodeRedisDialCacheClient(client: NodeRedisScriptClient): DialCacheRedisClient { return { - async read({ valueKey, watermarkKey }, context) { + enforcesMaxAge: true, + async read({ valueKey, watermarkKey, maxAgeMs }, context) { const options: BufferReplyOptions = context === undefined ? bufferReplyOptions : commandOptions({ returnBuffers: true, signal: context.signal }); const raw = watermarkKey === undefined - ? await client.dialcacheRead(options, valueKey) - : await client.dialcacheReadTracked(options, valueKey, watermarkKey); + ? await client.dialcacheRead(options, valueKey, maxAgeMs) + : await client.dialcacheReadTracked(options, valueKey, watermarkKey, maxAgeMs); return raw === null ? null : decodeRedisPayload(raw); }, async write(request) { diff --git a/src/prometheus.ts b/src/prometheus.ts index 8c7d7fd..f333ad1 100644 --- a/src/prometheus.ts +++ b/src/prometheus.ts @@ -9,6 +9,7 @@ import type { InvalidationMetricLabels, SerializationMetricLabels, ShadowValidationMetricLabels, + StaleRecoveryMetricLabels, } from "./metrics.js"; export interface PrometheusMetricsOptions { @@ -25,6 +26,7 @@ type SerializationLabels = CounterLabels | "operation"; type InvalidationLabels = "cache_namespace" | "key_type" | "layer"; type CoalescedLabels = "cache_namespace" | "use_case" | "key_type" | "scope"; type ShadowValidationLabels = "cache_namespace" | "use_case" | "key_type" | "outcome"; +type StaleRecoveryLabels = "cache_namespace" | "use_case" | "key_type" | "outcome"; interface BaseCollectorConfig { readonly name: string; @@ -63,6 +65,7 @@ export class PrometheusDialCacheMetrics implements DialCacheMetricsAdapter { private readonly invalidationCounter: Counter; private readonly coalescedCounter: Counter; private readonly shadowValidationCounter: Counter; + private readonly staleRecoveryCounter: Counter; private readonly getTimer: Histogram; private readonly fallbackTimer: Histogram; private readonly serializationTimer: Histogram; @@ -81,6 +84,7 @@ export class PrometheusDialCacheMetrics implements DialCacheMetricsAdapter { this.invalidationCounter = counter(registry, collectors.invalidationCounter); this.coalescedCounter = counter(registry, collectors.coalescedCounter); this.shadowValidationCounter = counter(registry, collectors.shadowValidationCounter); + this.staleRecoveryCounter = counter(registry, collectors.staleRecoveryCounter); this.getTimer = histogram(registry, collectors.getTimer); this.fallbackTimer = histogram(registry, collectors.fallbackTimer); this.serializationTimer = histogram(registry, collectors.serializationTimer); @@ -133,6 +137,15 @@ export class PrometheusDialCacheMetrics implements DialCacheMetricsAdapter { }); } + staleRecovery(labels: StaleRecoveryMetricLabels): void { + this.staleRecoveryCounter.inc({ + cache_namespace: labels.cacheNamespace, + use_case: labels.useCase, + key_type: labels.keyType, + outcome: labels.outcome, + }); + } + observeGet(labels: CacheMetricLabels, seconds: number): void { this.getTimer.observe(cacheLabels(labels), seconds); } @@ -207,6 +220,12 @@ function collectorConfigs(prefix: string) { help: "Sampled DialCache Redis shadow-validation outcomes.", labelNames: ["cache_namespace", "use_case", "key_type", "outcome"], }, + staleRecoveryCounter: { + type: "counter", + name: `${prefix}dialcache_stale_recovery_counter`, + help: "DialCache stale-on-error Redis recovery outcomes.", + labelNames: ["cache_namespace", "use_case", "key_type", "outcome"], + }, getTimer: { type: "histogram", name: `${prefix}dialcache_get_timer`, diff --git a/src/redis-client.ts b/src/redis-client.ts index 2bf18d6..7f90136 100644 --- a/src/redis-client.ts +++ b/src/redis-client.ts @@ -41,15 +41,20 @@ interface RedisValueRequest { readonly valueKey: string; } -interface TrackedRedisValueRequest extends RedisValueRequest { +interface RedisReadBase extends RedisValueRequest { + /** Positive integer no greater than 31,536,000,000 (365 days). */ + readonly maxAgeMs: number; +} + +interface TrackedRedisReadRequest extends RedisReadBase { readonly watermarkKey: string; } -interface UntrackedRedisValueRequest extends RedisValueRequest { +interface UntrackedRedisReadRequest extends RedisReadBase { readonly watermarkKey?: never; } -export type RedisReadRequest = TrackedRedisValueRequest | UntrackedRedisValueRequest; +export type RedisReadRequest = TrackedRedisReadRequest | UntrackedRedisReadRequest; /** * Per-use-case read policy supplied by DialCache. Adapters may use the signal @@ -61,13 +66,17 @@ export interface RedisReadContext { } interface RedisWriteBase extends RedisValueRequest { - /** Positive integer no greater than 31,536,000,000 (365 days). */ + /** + * Physical key retention in milliseconds. This is the fresh TTL when stale + * recovery is disabled and its maximum age when enabled. Positive integer no + * greater than 31,536,000,000 (365 days). + */ readonly cacheTtlMs: number; readonly value: RedisCachePayload; } -type TrackedRedisWriteRequest = RedisWriteBase & TrackedRedisValueRequest; -type UntrackedRedisWriteRequest = RedisWriteBase & UntrackedRedisValueRequest; +type TrackedRedisWriteRequest = RedisWriteBase & { readonly watermarkKey: string }; +type UntrackedRedisWriteRequest = RedisWriteBase & { readonly watermarkKey?: never }; export type RedisWriteRequest = TrackedRedisWriteRequest | UntrackedRedisWriteRequest; @@ -94,7 +103,14 @@ export interface RedisInvalidationRequest { */ export interface DialCacheRedisClient { /** - * Atomically read and validate a value against its watermark when tracked. + * Safety capability marker. Custom clients must explicitly attest that every + * read atomically enforces `RedisReadRequest.maxAgeMs` using Redis server + * time. DialCache also checks this marker at runtime for JavaScript clients. + */ + readonly enforcesMaxAge: true; + /** + * Atomically read a value whose Redis-server age is strictly less than + * `maxAgeMs`, and validate it against its watermark when tracked. * * A non-null payload is transferred to DialCache. A returned Buffer must * remain stable and must not be mutated, pooled, or reused after this method diff --git a/src/valkey-glide.ts b/src/valkey-glide.ts index c63485c..ef80118 100644 --- a/src/valkey-glide.ts +++ b/src/valkey-glide.ts @@ -93,10 +93,11 @@ export function createValkeyGlideDialCacheClient dropped: true, }; const shadowValidationOutcomes = Object.keys(SHADOW_VALIDATION_OUTCOMES) as ShadowValidationOutcome[]; +const STALE_RECOVERY_OUTCOMES: Readonly> = { + served: true, + miss: true, + read_error: true, + read_timeout: true, + deserialization_error: true, +}; +const staleRecoveryOutcomes = Object.keys(STALE_RECOVERY_OUTCOMES) as StaleRecoveryOutcome[]; const metricLayers: readonly MetricLayer[] = [ CacheLayer.LOCAL, CacheLayer.REMOTE, @@ -147,6 +156,12 @@ describe("Datadog metrics adapter", () => { keyType: "user_id", outcome: "match", }); + metrics.staleRecovery({ + cacheNamespace: cacheLabels.cacheNamespace, + useCase: "LoadUser", + keyType: "user_id", + outcome: "served", + }); metrics.observeGet(cacheLabels, 0.125); metrics.observeFallback(cacheLabels, 0.5); metrics.observeSerialization({ ...cacheLabels, operation: "dump" }, 0.25); @@ -186,6 +201,12 @@ describe("Datadog metrics adapter", () => { value: 1, tags: { cache_namespace: "users", use_case: "LoadUser", key_type: "user_id", outcome: "match" }, }, + { + method: "increment", + name: "dialcache.stale_recovery.count", + value: 1, + tags: { cache_namespace: "users", use_case: "LoadUser", key_type: "user_id", outcome: "served" }, + }, { method: "distribution", name: "dialcache.get.duration", value: 0.125, tags: baseTags }, { method: "distribution", name: "dialcache.fallback.duration", value: 0.5, tags: baseTags }, { @@ -256,6 +277,14 @@ describe("Datadog metrics adapter", () => { outcome, }); } + for (const outcome of staleRecoveryOutcomes) { + metrics.staleRecovery({ + cacheNamespace: cacheLabels.cacheNamespace, + useCase: cacheLabels.useCase, + keyType: cacheLabels.keyType, + outcome, + }); + } expect(client.calls.slice(0, metricLayers.length).map(({ tags }) => tags.layer)).toEqual(metricLayers); expect( @@ -290,6 +319,18 @@ describe("Datadog metrics adapter", () => { outcome, })), ); + expect( + client.calls + .filter(({ name }) => name === "dialcache.stale_recovery.count") + .map(({ tags }) => tags), + ).toEqual( + staleRecoveryOutcomes.map((outcome) => ({ + cache_namespace: cacheLabels.cacheNamespace, + use_case: cacheLabels.useCase, + key_type: cacheLabels.keyType, + outcome, + })), + ); }); it("accepts hot-shots directly and emits DogStatsD distribution datagrams", () => { @@ -385,6 +426,7 @@ describe("Datadog metrics adapter", () => { const rawErrorMessage = "Redis failed for a private cache key"; let redisValueKey = ""; const redis: DialCacheRedisClient = { + enforcesMaxAge: true, read: async ({ valueKey }) => { redisValueKey = valueKey; const error = new Error(`${rawErrorMessage}: ${valueKey}`); diff --git a/test/dialcache-config-ramp.test.ts b/test/dialcache-config-ramp.test.ts index 0a1d9c3..31db506 100644 --- a/test/dialcache-config-ramp.test.ts +++ b/test/dialcache-config-ramp.test.ts @@ -9,6 +9,7 @@ import { type Serializer, } from "../src/index.js"; import { deterministicRampSample, deterministicShadowRampSample } from "../src/internal/ramp.js"; +import { fetchKeyConfig } from "../src/internal/runtime-config.js"; import { FakeRedis } from "./fake-redis.js"; const configFor = (ttlSec: Partial>, ramp: Partial>) => @@ -38,6 +39,12 @@ describe("DialCache runtime config and ramp controls", () => { expect(new DialCacheKeyConfig({ requestLocal: false }).requestLocal).toBe(false); }); + it("preserves stale-on-error omission and explicit disable values", () => { + expect(new DialCacheKeyConfig({}).staleOnErrorMaxAgeSec).toBeUndefined(); + expect(new DialCacheKeyConfig({ staleOnErrorMaxAgeSec: 0 }).staleOnErrorMaxAgeSec).toBe(0); + expect(new DialCacheKeyConfig({ staleOnErrorMaxAgeSec: 3_600 }).staleOnErrorMaxAgeSec).toBe(3_600); + }); + it("preserves shadow omission and explicit kill-switch values", () => { expect(new DialCacheKeyConfig({}).shadow).toBeUndefined(); expect(new DialCacheKeyConfig({ shadow: {} }).shadow).toEqual({}); @@ -71,12 +78,13 @@ describe("DialCache runtime config and ramp controls", () => { it("captures an immutable default policy snapshot when the use case is registered", async () => { const suppliedDefault = new DialCacheKeyConfig({ - ttlSec: { [CacheLayer.LOCAL]: 60 }, - ramp: { [CacheLayer.LOCAL]: 100 }, + ttlSec: { [CacheLayer.LOCAL]: 60, [CacheLayer.REMOTE]: 60 }, + ramp: { [CacheLayer.LOCAL]: 100, [CacheLayer.REMOTE]: 100 }, shadow: { ramp: 25, logMismatches: true, }, + staleOnErrorMaxAgeSec: 3_600, }); const observedDefaults: Array = []; const dialcache = new DialCache({ @@ -96,6 +104,7 @@ describe("DialCache runtime config and ramp controls", () => { suppliedDefault.ttlSec[CacheLayer.LOCAL] = 0; suppliedDefault.ramp[CacheLayer.LOCAL] = 0; + (suppliedDefault as { staleOnErrorMaxAgeSec?: number }).staleOnErrorMaxAgeSec = 0; const mutableShadow = suppliedDefault.shadow as { ramp?: number; logMismatches?: boolean; @@ -112,6 +121,7 @@ describe("DialCache runtime config and ramp controls", () => { expect(observedDefaults[1]).toBe(observedDefaults[0]); expect(observedDefaults[0]?.ttlSec[CacheLayer.LOCAL]).toBe(60); expect(observedDefaults[0]?.ramp[CacheLayer.LOCAL]).toBe(100); + expect(observedDefaults[0]?.staleOnErrorMaxAgeSec).toBe(3_600); expect(observedDefaults[0]?.shadow).toEqual({ ramp: 25, logMismatches: true, @@ -347,6 +357,75 @@ describe("DialCache runtime config and ramp controls", () => { RangeError, `no greater than ${MAX_CACHE_TTL_SEC}`, ], + [ + "negative stale-on-error max age", + new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: -1, + }), + RangeError, + "nonnegative safe integer", + ], + [ + "fractional stale-on-error max age", + new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: 60.5, + }), + RangeError, + "nonnegative safe integer", + ], + [ + "non-finite stale-on-error max age", + new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: Number.NaN, + }), + RangeError, + "nonnegative safe integer", + ], + [ + "unsafe stale-on-error max age", + new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: Number.MAX_SAFE_INTEGER + 1, + }), + RangeError, + "nonnegative safe integer", + ], + [ + "over-maximum stale-on-error max age", + new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: MAX_CACHE_TTL_SEC + 1, + }), + RangeError, + `no greater than ${MAX_CACHE_TTL_SEC}`, + ], + [ + "stale-on-error max age equal to the remote TTL", + new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: 60, + }), + RangeError, + "must be greater than ttlSec.remote", + ], + [ + "positive stale-on-error max age without a remote TTL", + new DialCacheKeyConfig({ staleOnErrorMaxAgeSec: 3_600 }), + RangeError, + "requires ttlSec.remote", + ], + [ + "stale-on-error max age below the remote TTL", + new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: 30, + }), + RangeError, + "must be greater than ttlSec.remote", + ], ["negative ramp", new DialCacheKeyConfig({ ramp: { [CacheLayer.LOCAL]: -1 } }), RangeError, "between 0 and 100"], ["over-100 ramp", new DialCacheKeyConfig({ ramp: { [CacheLayer.LOCAL]: 101 } }), RangeError, "between 0 and 100"], ["non-finite ramp", new DialCacheKeyConfig({ ramp: { [CacheLayer.LOCAL]: Number.POSITIVE_INFINITY } }), RangeError, "between 0 and 100"], @@ -392,6 +471,12 @@ describe("DialCache runtime config and ramp controls", () => { TypeError, "must be a boolean", ], + [ + "wrong-type stale-on-error max age", + new DialCacheKeyConfig({ staleOnErrorMaxAgeSec: "3600" as unknown as number }), + TypeError, + "must be a number", + ], ["primitive config", 42 as unknown as DialCacheKeyConfig, TypeError, "must be an object"], ["array config", [] as unknown as DialCacheKeyConfig, TypeError, "must be an object"], [ @@ -438,6 +523,35 @@ describe("DialCache runtime config and ramp controls", () => { })).not.toThrow(); }); + it("accepts disabled and exact-maximum static stale-on-error policy", () => { + const dialcache = new DialCache(); + + expect(() => dialcache.cached(async () => "value", { + keyType: "item_id", + useCase: "DisabledStaticStaleOnErrorWithoutRemote", + cacheKey: () => "000", + defaultConfig: new DialCacheKeyConfig({ staleOnErrorMaxAgeSec: 0 }), + })).not.toThrow(); + expect(() => dialcache.cached(async () => "value", { + keyType: "item_id", + useCase: "DisabledStaticStaleOnError", + cacheKey: () => "123", + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: 0, + }), + })).not.toThrow(); + expect(() => dialcache.cached(async () => "value", { + keyType: "item_id", + useCase: "MaximumStaticStaleOnError", + cacheKey: () => "456", + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: MAX_CACHE_TTL_SEC - 1 }, + staleOnErrorMaxAgeSec: MAX_CACHE_TTL_SEC, + }), + })).not.toThrow(); + }); + it.each([ ["a primitive", 42], ["an array", []], @@ -467,6 +581,7 @@ describe("DialCache runtime config and ramp controls", () => { it("returns the explicit kill-switch overlay from DialCacheKeyConfig.disabled()", () => { expect(DialCacheKeyConfig.disabled()).toEqual(new DialCacheKeyConfig({ requestLocal: false, + staleOnErrorMaxAgeSec: 0, shadow: { ramp: 0, logMismatches: false, @@ -552,6 +667,135 @@ describe("DialCache runtime config and ramp controls", () => { expect(second.calls).toBe(2); }); + it.each([ + ["null", null], + ["negative", -1], + ["fractional", 60.5], + ["NaN", Number.NaN], + ["infinite", Number.POSITIVE_INFINITY], + ["far over maximum", Number.MAX_SAFE_INTEGER], + ["unsafe", Number.MAX_SAFE_INTEGER + 1], + ["over maximum", MAX_CACHE_TTL_SEC + 1], + ["equal to fresh TTL", 60], + ["below fresh TTL", 30], + ["wrong type", "3600"], + ] as const)( + "disables only stale recovery for an invalid runtime max age ($0)", + async (_name, configuredMaxAge) => { + const redis = new FakeRedis(); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + cacheConfigProvider: async () => new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: configuredMaxAge as unknown as number, + }), + }); + let calls = 0; + const getUser = dialcache.cached(async (userId: string) => ({ userId, calls: ++calls }), { + keyType: "user_id", + useCase: `InvalidRuntimeStaleMaxAge${String(_name)}`, + cacheKey: (userId) => userId, + }); + + const first = await dialcache.enable(async () => await getUser("123")); + const second = await dialcache.enable(async () => await getUser("123")); + + expect(second).toEqual(first); + expect(calls).toBe(1); + expect(redis.getCalls).toBe(2); + expect(redis.setCalls).toBe(1); + }, + ); + + it("inherits, overrides, and explicitly disables stale-on-error runtime policy", async () => { + const defaultConfig = new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: 3_600, + }); + const key = new DialCacheKey({ + keyType: "user_id", + id: "123", + useCase: "StaleOnErrorOverlay", + defaultConfig, + }); + + await expect(fetchKeyConfig(async () => new DialCacheKeyConfig({}), key)).resolves.toMatchObject({ + staleOnErrorMaxAgeSec: 3_600, + }); + await expect( + fetchKeyConfig(async () => new DialCacheKeyConfig({ staleOnErrorMaxAgeSec: 7_200 }), key), + ).resolves.toMatchObject({ staleOnErrorMaxAgeSec: 7_200 }); + await expect( + fetchKeyConfig(async () => new DialCacheKeyConfig({ staleOnErrorMaxAgeSec: 0 }), key), + ).resolves.toMatchObject({ staleOnErrorMaxAgeSec: 0 }); + }); + + it.each([ + [ + "an explicit stale-on-error zero", + () => new DialCacheKeyConfig({ staleOnErrorMaxAgeSec: 0 }), + 1, + ], + [ + "an invalid stale-on-error maximum", + () => new DialCacheKeyConfig({ staleOnErrorMaxAgeSec: 1 }), + 1, + ], + ["the complete disabled overlay", () => DialCacheKeyConfig.disabled(), 0], + ] as const)( + "does not recover a retained stale value after $0", + async (_name, disabledOverlay, expectedRemoteReads) => { + vi.useFakeTimers(); + vi.setSystemTime(new Date("2026-08-02T12:00:00.000Z")); + try { + const useCase = `RetainedStaleRuntimeDisable${expectedRemoteReads}`; + const redis = new FakeRedis(); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const source = vi.fn<() => Promise<{ readonly id: string; readonly version: number }>>() + .mockResolvedValueOnce({ id: "123", version: 1 }) + .mockRejectedValueOnce(sourceError); + let runtimeConfig = new DialCacheKeyConfig({}); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + cacheConfigProvider: async () => runtimeConfig, + }); + const getUser = dialcache.cached(source, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 1 }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: 10, + }), + }); + const valueKey = `${new DialCacheKey({ + keyType: "user_id", + id: "123", + useCase, + }).urn}:dialcache-frame-v1`; + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual({ + id: "123", + version: 1, + }); + expect(redis.ttlMs(valueKey)).toBe(10_000); + await vi.advanceTimersByTimeAsync(2_000); + runtimeConfig = disabledOverlay(); + const readsBeforeDisabledCall = redis.getCalls; + + await expect(dialcache.enable(async () => await getUser())).rejects.toBe(sourceError); + + expect(redis.getCalls - readsBeforeDisabledCall).toBe(expectedRemoteReads); + expect(redis.setCalls).toBe(1); + expect(redis.ttlMs(valueKey)).toBe(8_000); + expect(source).toHaveBeenCalledTimes(2); + } finally { + vi.useRealTimers(); + } + }, + ); + it("applies runtime config changes to subsequent calls", async () => { // Given a provider whose config can change without redeploying the cached function. let runtimeConfig: DialCacheKeyConfig | null = DialCacheKeyConfig.enabled(60); diff --git a/test/dialcache-liveness.test.ts b/test/dialcache-liveness.test.ts index 788160a..81a2830 100644 --- a/test/dialcache-liveness.test.ts +++ b/test/dialcache-liveness.test.ts @@ -485,6 +485,7 @@ describe("DialCache fallback liveness", () => { const fallback = vi.fn(async () => "value"); const setTimeoutSpy = vi.spyOn(globalThis, "setTimeout"); const redis: DialCacheRedisClient = { + enforcesMaxAge: true, read: async () => { readStarted.resolve(); return await readGate.promise; @@ -530,6 +531,7 @@ describe("DialCache fallback liveness", () => { }, }; const redis: DialCacheRedisClient = { + enforcesMaxAge: true, read: async () => "stored", write: async () => true, invalidate: async () => undefined, @@ -574,6 +576,7 @@ describe("DialCache fallback liveness", () => { load: (value) => value.toString(), }; const redis: DialCacheRedisClient = { + enforcesMaxAge: true, read: async () => null, write: async () => { writeStarted.resolve(); diff --git a/test/dialcache-logger.test.ts b/test/dialcache-logger.test.ts index a7df3e1..06cc93e 100644 --- a/test/dialcache-logger.test.ts +++ b/test/dialcache-logger.test.ts @@ -159,6 +159,7 @@ describe("DialCache logger isolation", () => { const logger = throwingLogger(); const invalidationError = new Error("invalidation failed"); const redis = { + enforcesMaxAge: true, read: vi.fn(async () => null), write: vi.fn(async () => true), invalidate: vi.fn(async () => { diff --git a/test/dialcache-metrics.test.ts b/test/dialcache-metrics.test.ts index 15f5fa1..09c2da7 100644 --- a/test/dialcache-metrics.test.ts +++ b/test/dialcache-metrics.test.ts @@ -3,6 +3,7 @@ import { describe, expect, it, vi } from "vitest"; import { CacheLayer, DialCache, + DialCacheKey, DialCacheKeyConfig, type CacheMetricLabels, type CoalescedMetricLabels, @@ -14,8 +15,9 @@ import { type SerializationMetricLabels, type Serializer, type ShadowValidationMetricLabels, + type StaleRecoveryMetricLabels, } from "../src/index.js"; -import { FakeRedis } from "./fake-redis.js"; +import { encodeFrame, FakeRedis } from "./fake-redis.js"; class RecordingMetrics implements DialCacheMetricsAdapter { readonly events: Array<{ readonly name: string; readonly labels: Record; readonly value?: number }> = []; @@ -44,6 +46,10 @@ class RecordingMetrics implements DialCacheMetricsAdapter { this.record("coalesced", labels); } + staleRecovery(labels: StaleRecoveryMetricLabels): void { + this.record("staleRecovery", labels); + } + observeGet(labels: CacheMetricLabels, seconds: number): void { this.record("get", labels, seconds); } @@ -96,6 +102,7 @@ describe("DialCache observability metrics", () => { invalidation: vi.fn(() => thenable), coalesced: vi.fn(() => thenable), shadowValidation: vi.fn(() => thenable), + staleRecovery: vi.fn(() => thenable), observeGet: vi.fn(() => thenable), observeFallback: vi.fn(() => thenable), observeSerialization: vi.fn(() => thenable), @@ -133,6 +140,12 @@ describe("DialCache observability metrics", () => { keyType: "user_id", outcome: "match", } satisfies ShadowValidationMetricLabels); + isolatedMetrics.staleRecovery?.({ + cacheNamespace: "urn", + useCase: "RejectingMetricsThenable", + keyType: "user_id", + outcome: "served", + } satisfies StaleRecoveryMetricLabels); isolatedMetrics.observeGet(labels, 0); isolatedMetrics.observeFallback(labels, 0); isolatedMetrics.observeSerialization({ ...labels, operation: "dump" }, 0); @@ -140,7 +153,7 @@ describe("DialCache observability metrics", () => { expect(then).not.toHaveBeenCalled(); await tick(); - expect(then).toHaveBeenCalledTimes(11); + expect(then).toHaveBeenCalledTimes(12); }); it("includes the configured cache namespace on every metric path", async () => { @@ -410,6 +423,43 @@ describe("DialCache observability metrics", () => { expect(events(metrics, "error", { useCase: "DisabledByPolicy" })).toHaveLength(0); }); + it("reports invalid stale-on-error policy without disabling fresh Redis", async () => { + const metrics = new RecordingMetrics(); + const redis = new FakeRedis(); + const dialcache = new DialCache({ + metrics, + redis: { client: redis, readTimeoutMs: 1_000 }, + cacheConfigProvider: async () => new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: 60, + }), + }); + let calls = 0; + const getUser = dialcache.cached(async (userId: string) => ({ userId, calls: ++calls }), { + keyType: "user_id", + useCase: "InvalidStaleOnErrorPolicy", + cacheKey: (userId) => userId, + }); + + const first = await dialcache.enable(async () => await getUser("123")); + const second = await dialcache.enable(async () => await getUser("123")); + + expect(second).toEqual(first); + expect(calls).toBe(1); + expect(redis.getCalls).toBe(2); + expect(redis.setCalls).toBe(1); + expect(events(metrics, "error", { + useCase: "InvalidStaleOnErrorPolicy", + layer: CacheLayer.REMOTE, + error: "config_resolution", + inFallback: false, + })).toHaveLength(2); + expect(events(metrics, "disabled", { + useCase: "InvalidStaleOnErrorPolicy", + layer: CacheLayer.REMOTE, + })).toHaveLength(0); + }); + it("labels cache errors separately from fallback errors", async () => { // Given cache and fallback errors carry caller-defined names containing dynamic identifiers. const metrics = new RecordingMetrics(); @@ -417,6 +467,7 @@ describe("DialCache observability metrics", () => { const cacheError = new Error("redis key urn:user_id:tenant-123 failed"); cacheError.name = "Tenant123RedisError"; const failingRedis: DialCacheRedisClient = { + enforcesMaxAge: true, read: vi.fn(async () => { throw cacheError; }), @@ -468,6 +519,87 @@ describe("DialCache observability metrics", () => { ); }); + it("records one complete telemetry trail when stale recovery serves a retained value", async () => { + const metrics = new RecordingMetrics(); + const redis = new FakeRedis(); + const useCase = "StaleRecoveryServedMetrics"; + const staleValue = { userId: "123", version: 1 }; + const key = new DialCacheKey({ keyType: "user_id", id: "123", useCase }); + redis.setRaw( + `${key.urn}:dialcache-frame-v1`, + encodeFrame(staleValue, Date.now() - 2_000), + 10_000, + ); + const source = vi.fn(async () => { + throw new Error("source unavailable"); + }); + const logger = { debug: vi.fn(), warn: vi.fn(), error: vi.fn() }; + const dialcache = new DialCache({ + metrics, + redis: { client: redis, readTimeoutMs: 1_000 }, + logger, + }); + const getUser = dialcache.cached(source, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 1 }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: 10, + }), + }); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual(staleValue); + + expect(source).toHaveBeenCalledOnce(); + expect(events(metrics, "error", { useCase })).toEqual([ + { + name: "error", + labels: { + cacheNamespace: "urn", + useCase, + keyType: "user_id", + layer: CacheLayer.REMOTE, + error: "fallback", + inFallback: true, + }, + }, + ]); + const remoteLabels = { + cacheNamespace: "urn", + useCase, + keyType: "user_id", + layer: CacheLayer.REMOTE, + }; + expect(events(metrics, "fallback", { useCase })).toEqual([ + { name: "fallback", labels: remoteLabels, value: expect.any(Number) }, + ]); + expect(events(metrics, "miss", { useCase })).toEqual([ + { name: "miss", labels: remoteLabels }, + ]); + expect(events(metrics, "request", { useCase })).toEqual([ + { name: "request", labels: remoteLabels }, + { name: "request", labels: remoteLabels }, + ]); + expect(events(metrics, "get", { useCase })).toEqual([ + { name: "get", labels: remoteLabels, value: expect.any(Number) }, + { name: "get", labels: remoteLabels, value: expect.any(Number) }, + ]); + expect(events(metrics, "staleRecovery", { useCase })).toEqual([ + { + name: "staleRecovery", + labels: { + cacheNamespace: "urn", + useCase, + keyType: "user_id", + outcome: "served", + }, + }, + ]); + expect(logger.warn).not.toHaveBeenCalled(); + }); + it("classifies config, Redis write, and serializer failures by stable operation", async () => { const metrics = new RecordingMetrics(); const logger = { debug: vi.fn(), warn: vi.fn(), error: vi.fn() }; diff --git a/test/dialcache-observability-internals.test.ts b/test/dialcache-observability-internals.test.ts index 96915e4..bb2c442 100644 --- a/test/dialcache-observability-internals.test.ts +++ b/test/dialcache-observability-internals.test.ts @@ -3,7 +3,11 @@ import { describe, expect, it } from "vitest"; import { CacheLayer, DialCacheKey, DialCacheKeyConfig } from "../src/index.js"; import { LocalCache } from "../src/internal/local-cache.js"; import { RedisCache } from "../src/internal/redis-cache.js"; -import { fetchKeyConfig, resolveLayerConfig } from "../src/internal/runtime-config.js"; +import { + fetchKeyConfig, + resolveLayerConfig, + resolveRemoteLayerConfigResult, +} from "../src/internal/runtime-config.js"; import { encodeFrame, FakeRedis } from "./fake-redis.js"; const key = (defaultConfig: DialCacheKeyConfig | null = DialCacheKeyConfig.enabled(60)) => @@ -19,6 +23,7 @@ describe("DialCache observability internal compatibility paths", () => { ramp: 20, logMismatches: true, }, + staleOnErrorMaxAgeSec: 3_600, }); const cases = [ { @@ -36,6 +41,7 @@ describe("DialCache observability internal compatibility paths", () => { ramp: 80, logMismatches: true, }, + staleOnErrorMaxAgeSec: 3_600, }), }, { @@ -45,6 +51,7 @@ describe("DialCache observability internal compatibility paths", () => { shadow: { logMismatches: false, }, + staleOnErrorMaxAgeSec: 0, }), expected: new DialCacheKeyConfig({ requestLocal: true, @@ -54,10 +61,11 @@ describe("DialCache observability internal compatibility paths", () => { ramp: 20, logMismatches: false, }, + staleOnErrorMaxAgeSec: 0, }), }, { - runtime: new DialCacheKeyConfig({ shadow: {} }), + runtime: new DialCacheKeyConfig({ shadow: {}, staleOnErrorMaxAgeSec: 7_200 }), expected: new DialCacheKeyConfig({ requestLocal: true, ttlSec: { [CacheLayer.LOCAL]: 60, [CacheLayer.REMOTE]: 120 }, @@ -66,6 +74,7 @@ describe("DialCache observability internal compatibility paths", () => { ramp: 20, logMismatches: true, }, + staleOnErrorMaxAgeSec: 7_200, }), }, ]; @@ -143,4 +152,48 @@ describe("DialCache observability internal compatibility paths", () => { expect(noConfig).toBeNull(); expect(noRamp).toEqual({ ttlSec: 60, ramp: 100 }); }); + + it("keeps invalid stale-on-error policy diagnostic-only in remote resolution", () => { + const remoteKey = key(); + + expect(resolveRemoteLayerConfigResult({ + config: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: 3_600, + }), + key: remoteKey, + })).toEqual({ + status: "enabled", + config: { ttlSec: 60, ramp: 100, staleOnErrorMaxAgeSec: 3_600 }, + }); + expect(resolveRemoteLayerConfigResult({ + config: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: 60, + }), + key: remoteKey, + })).toEqual({ + status: "enabled", + config: { ttlSec: 60, ramp: 100, staleOnErrorMaxAgeSec: null }, + staleOnErrorConfigError: true, + }); + expect(resolveRemoteLayerConfigResult({ + config: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + staleOnErrorMaxAgeSec: 0, + }), + key: remoteKey, + })).toEqual({ + status: "enabled", + config: { ttlSec: 60, ramp: 100, staleOnErrorMaxAgeSec: null }, + }); + expect(resolveRemoteLayerConfigResult({ + config: new DialCacheKeyConfig({ staleOnErrorMaxAgeSec: 3_600 }), + key: remoteKey, + })).toEqual({ + status: "disabled", + reason: "policy_disabled", + staleOnErrorConfigError: true, + }); + }); }); diff --git a/test/dialcache-redis-read-deadline.test.ts b/test/dialcache-redis-read-deadline.test.ts index 6c9eea3..a6410c4 100644 --- a/test/dialcache-redis-read-deadline.test.ts +++ b/test/dialcache-redis-read-deadline.test.ts @@ -65,6 +65,7 @@ function redisClient(read: DialCacheRedisClient["read"]): { const write = vi.fn(async () => true); return { client: { + enforcesMaxAge: true, read: readMock, write, invalidate: async () => undefined, @@ -180,6 +181,20 @@ describe("DialCache Redis read deadlines", () => { ).not.toThrow(); }); + it("rejects legacy semantic clients that do not attest max-age enforcement", () => { + const legacyClient = { + read: async () => null, + write: async () => true, + invalidate: async () => undefined, + }; + + expect( + () => new DialCache({ + redis: { client: legacyClient as unknown as DialCacheRedisClient }, + }), + ).toThrow(new TypeError("DialCache Redis client must declare enforcesMaxAge: true")); + }); + it("rejects invalid static use-case overrides before reserving the use-case name", () => { const client = redisClient(async () => null).client; const invalidValues: readonly unknown[] = [ diff --git a/test/dialcache-redis.test.ts b/test/dialcache-redis.test.ts index b59432d..f90206c 100644 --- a/test/dialcache-redis.test.ts +++ b/test/dialcache-redis.test.ts @@ -300,13 +300,57 @@ describe("DialCache Redis TTL layer", () => { const payload = Buffer.from([0, 1, 2, 0xff]); await redis.write({ valueKey, cacheTtlMs: 60_000, value: payload }); - const firstRead = await redis.read({ valueKey }); + const firstRead = await redis.read({ valueKey, maxAgeMs: 60_000 }); if (!Buffer.isBuffer(firstRead)) { throw new Error("Expected a binary Redis payload"); } firstRead[0] = 0xff; - expect(await redis.read({ valueKey })).toEqual(payload); + expect(await redis.read({ valueKey, maxAgeMs: 60_000 })).toEqual(payload); + }); + + it("enforces the fresh and stale maximum-age boundaries exactly", async () => { + vi.useFakeTimers(); + vi.setSystemTime(new Date("2026-08-02T12:00:00.000Z")); + const redis = new FakeRedis(); + const freshBeforeKey = "age-boundary:{item:f-minus-one}:value"; + const freshAtKey = "age-boundary:{item:f}:value"; + const staleBeforeKey = "age-boundary:{item:m-minus-one}:value"; + const staleAtKey = "age-boundary:{item:m}:value"; + const nowMs = Date.now(); + const freshMaxAgeMs = 1_000; + const staleMaxAgeMs = 3_000; + + redis.setRaw(freshBeforeKey, encodeFrame("F-1", nowMs - (freshMaxAgeMs - 1))); + redis.setRaw(freshAtKey, encodeFrame("F", nowMs - freshMaxAgeMs)); + redis.setRaw(staleBeforeKey, encodeFrame("M-1", nowMs - (staleMaxAgeMs - 1))); + redis.setRaw(staleAtKey, encodeFrame("M", nowMs - staleMaxAgeMs)); + + await expect(redis.read({ valueKey: freshBeforeKey, maxAgeMs: freshMaxAgeMs })).resolves.toBe("F-1"); + await expect(redis.read({ valueKey: freshAtKey, maxAgeMs: freshMaxAgeMs })).resolves.toBeNull(); + await expect(redis.read({ valueKey: staleBeforeKey, maxAgeMs: staleMaxAgeMs })).resolves.toBe("M-1"); + await expect(redis.read({ valueKey: staleAtKey, maxAgeMs: staleMaxAgeMs })).resolves.toBeNull(); + }); + + it("validates FakeRedis maximum ages before reading", async () => { + const redis = new FakeRedis(); + const valueKey = "age-validation:{item:1}:value"; + await redis.write({ valueKey, cacheTtlMs: 60_000, value: "value" }); + + await expect(redis.read({ valueKey, maxAgeMs: 31_536_000_000 })).resolves.toBe("value"); + for (const maxAgeMs of [ + 0, + -1, + 1.5, + Number.NaN, + Number.POSITIVE_INFINITY, + Number.NEGATIVE_INFINITY, + 31_536_000_001, + Number.MAX_SAFE_INTEGER, + ]) { + await expect(redis.read({ valueKey, maxAgeMs })).rejects.toThrow("Invalid DialCache Redis maxAgeMs"); + } + expect(redis.getCalls).toBe(1); }); it("fails open when Redis serializer dump fails", async () => { @@ -421,6 +465,7 @@ describe("DialCache Redis TTL layer", () => { it("records a distinct metric label when a Redis adapter reports invalid payload encoding", async () => { const redisClient: DialCacheRedisClient = { + enforcesMaxAge: true, read: vi.fn(async () => { throw new DialCacheRedisPayloadEncodingError("Invalid DialCache Redis payload encoding"); }), diff --git a/test/dialcache-shadow-confirmation.test.ts b/test/dialcache-shadow-confirmation.test.ts index 95e7e45..f0820f3 100644 --- a/test/dialcache-shadow-confirmation.test.ts +++ b/test/dialcache-shadow-confirmation.test.ts @@ -24,6 +24,7 @@ import { type SerializationMetricLabels, type Serializer, type ShadowValidationMetricLabels, + type StaleRecoveryMetricLabels, } from "../src/index.js"; import { deterministicRampSample, @@ -52,9 +53,12 @@ function deferred(): Deferred { return { promise, resolve, reject }; } -type ReadStep = () => RedisCachePayload | null | Promise; +type ReadStep = ( + request: RedisReadRequest, +) => RedisCachePayload | null | Promise; class ScriptedRedis implements DialCacheRedisClient { + readonly enforcesMaxAge = true as const; readonly requests: RedisReadRequest[] = []; readonly contexts: Array = []; readonly write = vi.fn(async (_request: RedisWriteRequest): Promise => true); @@ -69,7 +73,7 @@ class ScriptedRedis implements DialCacheRedisClient { if (step === undefined) { throw new Error("Unexpected Redis read"); } - return await step(); + return await step(request); } } @@ -93,6 +97,7 @@ interface OrdinaryMetricEvent { class RecordingMetrics implements DialCacheMetricsAdapter { readonly ordinaryEvents: OrdinaryMetricEvent[] = []; readonly shadowEvents: ShadowValidationMetricLabels[] = []; + readonly staleRecoveryEvents: StaleRecoveryMetricLabels[] = []; request(labels: CacheMetricLabels): void { this.record("request", labels); @@ -122,6 +127,10 @@ class RecordingMetrics implements DialCacheMetricsAdapter { this.shadowEvents.push({ ...labels }); } + staleRecovery(labels: StaleRecoveryMetricLabels): void { + this.staleRecoveryEvents.push({ ...labels }); + } + observeGet(labels: CacheMetricLabels, _seconds: number): void { this.record("get", labels); } @@ -217,10 +226,11 @@ async function waitForShadowEvents(metrics: RecordingMetrics, count: number): Pr function expectTrackedReads( redis: ScriptedRedis, count: number, - options: { readonly singleWatermark?: boolean } = { singleWatermark: true }, + options: { readonly singleWatermark?: boolean; readonly maxAgeMs?: number } = { singleWatermark: true }, ): void { expect(redis.requests).toHaveLength(count); expect(redis.requests.every(({ watermarkKey }) => typeof watermarkKey === "string")).toBe(true); + expect(redis.requests.every(({ maxAgeMs }) => maxAgeMs === (options.maxAgeMs ?? 60_000))).toBe(true); if (options.singleWatermark !== false) { expect(new Set(redis.requests.map(({ watermarkKey }) => watermarkKey)).size).toBe(1); } @@ -782,6 +792,55 @@ describe("DialCache Redis shadow confirmation", () => { expectTrackedReads(redis, 1); }); + it("never serves retained stale data when remote serving is ramped down", async () => { + const stalePayload = JSON.stringify({ id: "123", source: "stale-cache" }); + const redis = new ScriptedRedis([ + ({ maxAgeMs }) => maxAgeMs === 60_000 ? null : stalePayload, + ({ maxAgeMs }) => maxAgeMs === 3_600_000 ? stalePayload : null, + ]); + const metrics = new RecordingMetrics(); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const source = vi.fn(async () => { + throw sourceError; + }); + const dialcache = createCache(redis, metrics); + const getUser = dialcache.cached(source, { + ...trackedOptions("ShadowDarkRetainedStale", new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + ramp: { [CacheLayer.REMOTE]: 0 }, + staleOnErrorMaxAgeSec: 3_600, + shadow: { ramp: 100 }, + })), + cacheKey: () => "123", + }); + + await expect(dialcache.enable(async () => await getUser())).rejects.toBe(sourceError); + await waitForShadowEvents(metrics, 1); + + expect(source).toHaveBeenCalledOnce(); + expectTrackedReads(redis, 1, { maxAgeMs: 60_000 }); + expect(metrics.staleRecoveryEvents).toHaveLength(0); + expect(redis.write).not.toHaveBeenCalled(); + expect(redis.invalidate).not.toHaveBeenCalled(); + expect(metrics.shadowEvents).toEqual([{ + cacheNamespace: "urn", + useCase: "ShadowDarkRetainedStale", + keyType: "user_id", + outcome: "source_error", + }]); + expect(metrics.ordinaryEvents.filter(({ name, labels }) => + name === "request" && labels.layer === REMOTE_SHADOW_CACHE_LAYER + )).toHaveLength(1); + expect(metrics.ordinaryEvents.filter(({ name, labels }) => + name === "get" && labels.layer === REMOTE_SHADOW_CACHE_LAYER + )).toHaveLength(1); + expect(metrics.ordinaryEvents.filter(({ name, labels }) => + name === "disabled" + && labels.layer === CacheLayer.REMOTE + && labels.reason === "ramped_down" + )).toHaveLength(1); + }); + it("does not misclassify a source-propagated FallbackTimeoutError as its own timeout", async () => { const redis = new ScriptedRedis([() => JSON.stringify({ id: "123", source: "cache" })]); const metrics = new RecordingMetrics(); @@ -955,6 +1014,31 @@ describe("DialCache Redis shadow confirmation", () => { )).toHaveLength(1); }); + it("uses the fresh age for a dark read and maximum retention for its stale-enabled fill", async () => { + const redis = new ScriptedRedis([() => null]); + const metrics = new RecordingMetrics(); + const dialcache = createCache(redis, metrics); + const getUser = dialcache.cached(async () => ({ id: "123" }), { + ...trackedOptions("ShadowDarkStaleRetention", new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 60 }, + ramp: { [CacheLayer.REMOTE]: 0 }, + staleOnErrorMaxAgeSec: 3_600, + shadow: { ramp: 100 }, + })), + cacheKey: () => "123", + }); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual({ id: "123" }); + await waitForShadowEvents(metrics, 1); + + expectTrackedReads(redis, 1, { maxAgeMs: 60_000 }); + expect(redis.write).toHaveBeenCalledWith(expect.objectContaining({ + cacheTtlMs: 3_600_000, + watermarkKey: expect.any(String), + })); + expect(metrics.shadowEvents.map(({ outcome }) => outcome)).toEqual(["filled"]); + }); + it("reports fill_blocked when tracked invalidation rejects a detached fill", async () => { const redis = new ScriptedRedis([() => null]); redis.write.mockImplementationOnce(async () => false); diff --git a/test/dialcache-stale-on-error.test.ts b/test/dialcache-stale-on-error.test.ts new file mode 100644 index 0000000..cbb16f6 --- /dev/null +++ b/test/dialcache-stale-on-error.test.ts @@ -0,0 +1,861 @@ +import { performance } from "node:perf_hooks"; + +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +import { + CacheLayer, + DialCache, + DialCacheKey, + DialCacheKeyConfig, + type DialCacheMetricsAdapter, + type RedisCachePayload, + type RedisReadContext, + type RedisReadRequest, + type Serializer, +} from "../src/index.js"; +import { decodeFrame, encodeFrame, FakeRedis } from "./fake-redis.js"; + +const FRESH_TTL_SEC = 1; +const MAX_AGE_SEC = 10; + +class RecordingRedis extends FakeRedis { + readonly readRequests: RedisReadRequest[] = []; + readonly readContexts: Array = []; + + override async read( + request: RedisReadRequest, + context?: RedisReadContext, + ): Promise { + this.readRequests.push(request); + this.readContexts.push(context); + return await super.read(request); + } +} + +class HangingReadRedis extends FakeRedis { + readonly readRequests: RedisReadRequest[] = []; + readonly readContexts: Array = []; + + constructor(private readonly hangOnCall: number) { + super(); + } + + override async read( + request: RedisReadRequest, + context?: RedisReadContext, + ): Promise { + this.readRequests.push(request); + this.readContexts.push(context); + if (this.readRequests.length === this.hangOnCall) { + return await new Promise(() => undefined); + } + return await super.read(request); + } +} + +interface Deferred { + readonly promise: Promise; + resolve(value: T): void; + reject(error: unknown): void; +} + +function deferred(): Deferred { + let resolve!: (value: T) => void; + let reject!: (error: unknown) => void; + const promise = new Promise((resolvePromise, rejectPromise) => { + resolve = resolvePromise; + reject = rejectPromise; + }); + return { promise, resolve, reject }; +} + +function staleConfig(options: { readonly local?: boolean; readonly requestLocal?: boolean } = {}): DialCacheKeyConfig { + return new DialCacheKeyConfig({ + ttlSec: { + ...(options.local ? { [CacheLayer.LOCAL]: 60 } : {}), + [CacheLayer.REMOTE]: FRESH_TTL_SEC, + }, + ramp: { + ...(options.local ? { [CacheLayer.LOCAL]: 100 } : {}), + [CacheLayer.REMOTE]: 100, + }, + ...(options.requestLocal ? { requestLocal: true } : {}), + staleOnErrorMaxAgeSec: MAX_AGE_SEC, + }); +} + +function redisValueKey(useCase: string, id = "123", trackForInvalidation = false): string { + const key = new DialCacheKey({ keyType: "user_id", id, useCase, trackForInvalidation }); + return `${key.urn}:dialcache-frame-v1`; +} + +function watermarkKey(id = "123"): string { + return `{urn:user_id:${id}}#watermark`; +} + +function seedStale(redis: FakeRedis, useCase: string, value: unknown, trackForInvalidation = false): void { + redis.setRaw( + redisValueKey(useCase, "123", trackForInvalidation), + encodeFrame(value, Date.now() - 2_000), + MAX_AGE_SEC * 1_000, + ); +} + +function recordingMetrics(): { + readonly metrics: DialCacheMetricsAdapter; + readonly staleRecovery: ReturnType; + readonly shadowValidation: ReturnType; +} { + const staleRecovery = vi.fn(); + const shadowValidation = vi.fn(); + return { + staleRecovery, + shadowValidation, + metrics: { + request: vi.fn(), + miss: vi.fn(), + disabled: vi.fn(), + error: vi.fn(), + invalidation: vi.fn(), + shadowValidation, + staleRecovery, + observeGet: vi.fn(), + observeFallback: vi.fn(), + observeSerialization: vi.fn(), + observeSize: vi.fn(), + }, + }; +} + +function rejectionReason(result: PromiseSettledResult): unknown { + if (result.status !== "rejected") { + throw new Error("Expected rejection"); + } + return result.reason; +} + +describe("DialCache stale-on-error recovery", () => { + beforeEach(() => { + vi.useFakeTimers(); + vi.setSystemTime(new Date("2026-08-02T12:00:00.000Z")); + const clockOriginMs = Date.now(); + vi.spyOn(performance, "now").mockImplementation(() => Date.now() - clockOriginMs); + }); + + afterEach(() => { + vi.useRealTimers(); + vi.restoreAllMocks(); + }); + + it("rereads a logical miss at the maximum age and serves it without publication", async () => { + const useCase = "StaleRecoveryServed"; + const redis = new RecordingRedis(); + const staleValue = { id: "123", version: 1 }; + seedStale(redis, useCase, staleValue); + const ttlBefore = redis.ttlMs(redisValueKey(useCase)); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const source = vi.fn((): { readonly id: string; readonly version: number } => { + throw sourceError; + }); + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ redis: { client: redis, readTimeoutMs: 1_000 }, metrics }); + const getUser = dialcache.cached(source, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + }); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual(staleValue); + + expect(source).toHaveBeenCalledOnce(); + expect(redis.readRequests.map(({ maxAgeMs }) => maxAgeMs)).toEqual([1_000, 10_000]); + expect(redis.setCalls).toBe(0); + expect(redis.ttlMs(redisValueKey(useCase))).toBe(ttlBefore); + expect(staleRecovery).toHaveBeenCalledOnce(); + expect(staleRecovery).toHaveBeenCalledWith({ + cacheNamespace: "urn", + useCase, + keyType: "user_id", + outcome: "served", + }); + }); + + it("returns a logically fresh Redis hit without calling the source or recovery", async () => { + const useCase = "StaleRecoveryFreshHit"; + const redis = new RecordingRedis(); + redis.setRaw( + redisValueKey(useCase), + encodeFrame({ id: "123", version: 1 }, Date.now()), + MAX_AGE_SEC * 1_000, + ); + const source = vi.fn(async () => ({ id: "123", version: 2 })); + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ redis: { client: redis, readTimeoutMs: 1_000 }, metrics }); + const getUser = dialcache.cached(source, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + }); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual({ id: "123", version: 1 }); + + expect(source).not.toHaveBeenCalled(); + expect(redis.readRequests.map(({ maxAgeMs }) => maxAgeMs)).toEqual([1_000]); + expect(staleRecovery).not.toHaveBeenCalled(); + }); + + it("keeps feature-off writes at the fresh TTL and never performs a recovery read", async () => { + const useCase = "StaleRecoveryFeatureOff"; + const redis = new RecordingRedis(); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const source = vi.fn<() => Promise<{ readonly id: string; readonly version: number }>>() + .mockResolvedValueOnce({ id: "123", version: 1 }) + .mockRejectedValueOnce(sourceError); + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ redis: { client: redis, readTimeoutMs: 1_000 }, metrics }); + const getUser = dialcache.cached(source, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: FRESH_TTL_SEC }, + ramp: { [CacheLayer.REMOTE]: 100 }, + }), + }); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual({ id: "123", version: 1 }); + expect(redis.ttlMs(redisValueKey(useCase))).toBe(1_000); + await vi.advanceTimersByTimeAsync(1_000); + const [settled] = await Promise.allSettled([dialcache.enable(async () => await getUser())]); + + expect(rejectionReason(settled!)).toBe(sourceError); + expect(redis.readRequests.map(({ maxAgeMs }) => maxAgeMs)).toEqual([1_000, 1_000]); + expect(redis.setCalls).toBe(1); + expect(staleRecovery).not.toHaveBeenCalled(); + }); + + it("refreshes a retained logically stale frame without a recovery reread and writes with maximum retention", async () => { + const useCase = "StaleRecoverySourceSuccess"; + const redis = new RecordingRedis(); + seedStale(redis, useCase, { id: "123", version: 1 }); + const { metrics, staleRecovery } = recordingMetrics(); + const sourceValue = { id: "123", version: 2 }; + const source = vi.fn(async () => sourceValue); + const dialcache = new DialCache({ redis: { client: redis, readTimeoutMs: 1_000 }, metrics }); + const getUser = dialcache.cached(source, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + }); + + await expect(dialcache.enable(async () => await getUser())).resolves.toBe(sourceValue); + + expect(source).toHaveBeenCalledOnce(); + expect(redis.readRequests.map(({ maxAgeMs }) => maxAgeMs)).toEqual([1_000]); + expect(redis.setCalls).toBe(1); + expect(redis.ttlMs(redisValueKey(useCase))).toBe(10_000); + const refreshedFrame = decodeFrame(redis.raw(redisValueKey(useCase))); + expect(refreshedFrame.createdAtMs).toBe(Date.now()); + expect(JSON.parse(refreshedFrame.payload as string)).toEqual(sourceValue); + expect(staleRecovery).not.toHaveBeenCalled(); + }); + + it("rejects with the original source error when a retained frame reaches the exact maximum age", async () => { + const useCase = "StaleRecoveryCrossesMaximumDuringSource"; + const redis = new RecordingRedis(); + const valueKey = redisValueKey(useCase); + redis.setRaw( + valueKey, + encodeFrame({ id: "123", version: 1 }, Date.now() - 9_000), + 20_000, + ); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const sourceStarted = deferred(); + const sourceGate = deferred<{ readonly id: string; readonly version: number }>(); + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ redis: { client: redis, readTimeoutMs: 1_000 }, metrics }); + const getUser = dialcache.cached(async () => { + sourceStarted.resolve(); + return await sourceGate.promise; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + }); + + const result = Promise.allSettled([dialcache.enable(async () => await getUser())]); + await sourceStarted.promise; + await vi.advanceTimersByTimeAsync(1_000); + sourceGate.reject(sourceError); + const [settled] = await result; + + expect(rejectionReason(settled!)).toBe(sourceError); + expect(redis.readRequests.map(({ maxAgeMs }) => maxAgeMs)).toEqual([1_000, 10_000]); + expect(redis.ttlMs(valueKey)).toBe(19_000); + expect(redis.setCalls).toBe(0); + expect(staleRecovery).toHaveBeenCalledOnce(); + expect(staleRecovery).toHaveBeenCalledWith(expect.objectContaining({ outcome: "miss" })); + }); + + it("uses one runtime policy snapshot for the initial read and delayed recovery", async () => { + const useCase = "StaleRecoveryRuntimePolicySnapshot"; + const redis = new RecordingRedis(); + const staleValue = { id: "123", version: 1 }; + seedStale(redis, useCase, staleValue); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const sourceStarted = deferred(); + const sourceGate = deferred<{ readonly id: string; readonly version: number }>(); + let freshTtlSec = 1; + let maxAgeSec = 3; + let remoteReadTimeoutMs = 25; + const cacheConfigProvider = vi.fn(async () => new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: freshTtlSec }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: maxAgeSec, + remoteReadTimeoutMs, + })); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + cacheConfigProvider, + }); + const getUser = dialcache.cached(async () => { + sourceStarted.resolve(); + return await sourceGate.promise; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + }); + + const result = Promise.allSettled([dialcache.enable(async () => await getUser())]); + await sourceStarted.promise; + freshTtlSec = 4; + maxAgeSec = 20; + remoteReadTimeoutMs = 75; + sourceGate.reject(sourceError); + const [settled] = await result; + + expect(settled).toEqual({ status: "fulfilled", value: staleValue }); + expect(cacheConfigProvider).toHaveBeenCalledOnce(); + expect(redis.readRequests.map(({ maxAgeMs }) => maxAgeMs)).toEqual([1_000, 3_000]); + expect(redis.readContexts.map((context) => context?.timeoutMs)).toEqual([25, 25]); + }); + + it("applies the current runtime fresh age to an existing retained frame", async () => { + const useCase = "StaleRecoveryRuntimeFreshAge"; + const redis = new RecordingRedis(); + const retainedValue = { id: "123", version: 1 }; + redis.setRaw( + redisValueKey(useCase), + encodeFrame(retainedValue, Date.now() - 3_000), + MAX_AGE_SEC * 1_000, + ); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const source = vi.fn(async () => { + throw sourceError; + }); + let freshTtlSec = 4; + const cacheConfigProvider = vi.fn(async () => new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: freshTtlSec }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: MAX_AGE_SEC, + })); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + cacheConfigProvider, + }); + const getUser = dialcache.cached(source, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + }); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual(retainedValue); + expect(source).not.toHaveBeenCalled(); + + freshTtlSec = 2; + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual(retainedValue); + expect(source).toHaveBeenCalledOnce(); + + freshTtlSec = 4; + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual(retainedValue); + + expect(source).toHaveBeenCalledOnce(); + expect(cacheConfigProvider).toHaveBeenCalledTimes(3); + expect(redis.readRequests.map(({ maxAgeMs }) => maxAgeMs)).toEqual([ + 4_000, + 2_000, + 10_000, + 4_000, + ]); + }); + + it("retains a tracked watermark through the maximum age plus its existing margin", async () => { + const useCase = "StaleRecoveryTrackedRetention"; + const redis = new RecordingRedis(); + const dialcache = new DialCache({ redis: { client: redis, readTimeoutMs: 1_000 } }); + const getUser = dialcache.cached(async () => ({ id: "123" }), { + keyType: "user_id", + useCase, + cacheKey: () => "123", + trackForInvalidation: true, + defaultConfig: staleConfig(), + }); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual({ id: "123" }); + + expect(redis.ttlMs(redisValueKey(useCase, "123", true))).toBe(10_000); + expect(redis.ttlMs(watermarkKey())).toBe(70_000); + expect(redis.readRequests).toEqual([ + expect.objectContaining({ maxAgeMs: 1_000, watermarkKey: watermarkKey() }), + ]); + }); + + it.each([ + ["object", Object.freeze({ code: "SOURCE_OBJECT" })], + ["null", null], + ["undefined", undefined], + ] as const)("preserves an arbitrary %s rejection when recovery misses", async (_name, sourceError) => { + const useCase = `StaleRecoveryIdentity${_name}`; + const redis = new RecordingRedis(); + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ redis: { client: redis, readTimeoutMs: 1_000 }, metrics }); + const getUser = dialcache.cached(async () => { + throw sourceError; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + }); + + const [settled] = await Promise.allSettled([dialcache.enable(async () => await getUser())]); + + expect(rejectionReason(settled!)).toBe(sourceError); + expect(redis.readRequests.map(({ maxAgeMs }) => maxAgeMs)).toEqual([1_000, 10_000]); + expect(staleRecovery).toHaveBeenCalledOnce(); + expect(staleRecovery).toHaveBeenCalledWith(expect.objectContaining({ outcome: "miss" })); + }); + + it("never attempts recovery after the initial Redis read fails", async () => { + const useCase = "StaleRecoveryInitialReadError"; + const redis = new RecordingRedis(); + redis.failGet = true; + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + metrics, + logger: { debug: vi.fn(), error: vi.fn(), warn: vi.fn() }, + }); + const getUser = dialcache.cached(async () => { + throw sourceError; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + }); + + const [settled] = await Promise.allSettled([dialcache.enable(async () => await getUser())]); + + expect(rejectionReason(settled!)).toBe(sourceError); + expect(redis.readRequests).toHaveLength(1); + expect(staleRecovery).not.toHaveBeenCalled(); + }); + + it("preserves the source rejection when the recovery read fails", async () => { + const useCase = "StaleRecoveryReadError"; + const redis = new RecordingRedis(); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + metrics, + logger: { debug: vi.fn(), error: vi.fn(), warn: vi.fn() }, + }); + const getUser = dialcache.cached(async () => { + redis.failGet = true; + throw sourceError; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + }); + + const [settled] = await Promise.allSettled([dialcache.enable(async () => await getUser())]); + + expect(rejectionReason(settled!)).toBe(sourceError); + expect(redis.readRequests).toHaveLength(2); + expect(staleRecovery).toHaveBeenCalledOnce(); + expect(staleRecovery).toHaveBeenCalledWith(expect.objectContaining({ outcome: "read_error" })); + }); + + it("gives recovery an independent read deadline and preserves the source rejection on timeout", async () => { + const useCase = "StaleRecoveryReadTimeout"; + const redis = new HangingReadRedis(2); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 10 }, + metrics, + logger: { debug: vi.fn(), error: vi.fn(), warn: vi.fn() }, + }); + const getUser = dialcache.cached(async () => { + throw sourceError; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + }); + + const result = Promise.allSettled([dialcache.enable(async () => await getUser())]); + await vi.advanceTimersByTimeAsync(0); + expect(redis.readRequests).toHaveLength(2); + expect(redis.readContexts[1]?.signal.aborted).toBe(false); + await vi.advanceTimersByTimeAsync(10); + const [settled] = await result; + + expect(rejectionReason(settled!)).toBe(sourceError); + expect(redis.readContexts[1]?.timeoutMs).toBe(10); + expect(redis.readContexts[1]?.signal.aborted).toBe(true); + expect(staleRecovery).toHaveBeenCalledOnce(); + expect(staleRecovery).toHaveBeenCalledWith(expect.objectContaining({ outcome: "read_timeout" })); + }); + + it("does not retry Redis when the initial read times out", async () => { + const useCase = "StaleRecoveryInitialReadTimeout"; + const redis = new HangingReadRedis(1); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 10 }, + metrics, + logger: { debug: vi.fn(), error: vi.fn(), warn: vi.fn() }, + }); + const getUser = dialcache.cached(async () => { + throw sourceError; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + }); + + const result = Promise.allSettled([dialcache.enable(async () => await getUser())]); + await vi.advanceTimersByTimeAsync(10); + const [settled] = await result; + + expect(rejectionReason(settled!)).toBe(sourceError); + expect(redis.readRequests).toHaveLength(1); + expect(redis.readContexts[0]?.signal.aborted).toBe(true); + expect(staleRecovery).not.toHaveBeenCalled(); + }); + + it("serves stale after the fallback deadline and ignores the late source result", async () => { + const useCase = "StaleRecoveryFallbackTimeout"; + const redis = new RecordingRedis(); + seedStale(redis, useCase, { id: "123", version: 1 }); + const sourceGate = deferred<{ readonly id: string; readonly version: number }>(); + const sourceStarted = deferred(); + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ redis: { client: redis, readTimeoutMs: 100 }, metrics }); + const getUser = dialcache.cached(async () => { + sourceStarted.resolve(); + return await sourceGate.promise; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + fallbackTimeoutMs: 10, + defaultConfig: staleConfig(), + }); + + const result = dialcache.enable(async () => await getUser()); + await sourceStarted.promise; + await vi.advanceTimersByTimeAsync(10); + + await expect(result).resolves.toEqual({ id: "123", version: 1 }); + expect(redis.readRequests).toHaveLength(2); + expect(redis.setCalls).toBe(0); + expect(staleRecovery).toHaveBeenCalledOnce(); + expect(staleRecovery).toHaveBeenCalledWith(expect.objectContaining({ outcome: "served" })); + + sourceGate.resolve({ id: "123", version: 2 }); + await vi.advanceTimersByTimeAsync(0); + expect(redis.setCalls).toBe(0); + }); + + it("classifies recovery deserialization failure and never retries a normal deserialization miss", async () => { + const recoveryUseCase = "StaleRecoveryDeserializeError"; + const normalUseCase = "StaleRecoveryInitialDeserializeError"; + const redis = new RecordingRedis(); + seedStale(redis, recoveryUseCase, { id: "123" }); + redis.setRaw( + redisValueKey(normalUseCase), + encodeFrame({ id: "123" }, Date.now()), + MAX_AGE_SEC * 1_000, + ); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const serializer: Serializer<{ readonly id: string }> = { + dump: vi.fn(async (value) => JSON.stringify(value)), + load: vi.fn(async () => { + throw new Error("cannot decode"); + }), + }; + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ redis: { client: redis, readTimeoutMs: 1_000 }, metrics }); + const recover = dialcache.cached(async (): Promise<{ readonly id: string }> => { + throw sourceError; + }, { + keyType: "user_id", + useCase: recoveryUseCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + serializer, + }); + const initialFailure = dialcache.cached(async (): Promise<{ readonly id: string }> => { + throw sourceError; + }, { + keyType: "user_id", + useCase: normalUseCase, + cacheKey: () => "123", + defaultConfig: staleConfig(), + serializer, + }); + + const [recoverySettled] = await Promise.allSettled([dialcache.enable(async () => await recover())]); + expect(rejectionReason(recoverySettled!)).toBe(sourceError); + expect(staleRecovery).toHaveBeenCalledOnce(); + expect(staleRecovery).toHaveBeenCalledWith(expect.objectContaining({ + useCase: recoveryUseCase, + outcome: "deserialization_error", + })); + + const readsBeforeInitialFailure = redis.readRequests.length; + const [initialSettled] = await Promise.allSettled([ + dialcache.enable(async () => await initialFailure()), + ]); + expect(rejectionReason(initialSettled!)).toBe(sourceError); + expect(redis.readRequests).toHaveLength(readsBeforeInitialFailure + 1); + expect(staleRecovery).toHaveBeenCalledTimes(1); + expect(staleRecovery).not.toHaveBeenCalledWith(expect.objectContaining({ useCase: normalUseCase })); + }); + + it("rechecks tracked invalidation after the source attempt and blocks recovery", async () => { + const useCase = "StaleRecoveryInvalidatedDuringSource"; + const redis = new RecordingRedis(); + seedStale(redis, useCase, { id: "123", version: 1 }, true); + redis.setRaw(watermarkKey(), "0", MAX_AGE_SEC * 1_000 + 60_000); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const sourceStarted = deferred(); + const sourceGate = deferred<{ readonly id: string; readonly version: number }>(); + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ redis: { client: redis, readTimeoutMs: 1_000 }, metrics }); + const getUser = dialcache.cached(async () => { + sourceStarted.resolve(); + return await sourceGate.promise; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + trackForInvalidation: true, + defaultConfig: staleConfig(), + }); + + const result = Promise.allSettled([dialcache.enable(async () => await getUser())]); + await sourceStarted.promise; + await dialcache.invalidateRemote("user_id", "123"); + sourceGate.reject(sourceError); + const [settled] = await result; + + expect(rejectionReason(settled!)).toBe(sourceError); + expect(redis.readRequests).toHaveLength(2); + expect(redis.readRequests.every(({ watermarkKey: key }) => key === watermarkKey())).toBe(true); + expect(staleRecovery).toHaveBeenCalledOnce(); + expect(staleRecovery).toHaveBeenCalledWith(expect.objectContaining({ outcome: "miss" })); + }); + + it("recovers cached undefined without starting shadow validation", async () => { + const useCase = "StaleRecoveryUndefinedNoShadow"; + const redis = new RecordingRedis(); + redis.setRaw( + redisValueKey(useCase, "123", true), + encodeFrame("__dialcache_json_undefined_v1__", Date.now() - 2_000), + MAX_AGE_SEC * 1_000, + ); + redis.setRaw(watermarkKey(), "0", MAX_AGE_SEC * 1_000 + 60_000); + const { metrics, staleRecovery, shadowValidation } = recordingMetrics(); + const dialcache = new DialCache({ redis: { client: redis, readTimeoutMs: 1_000 }, metrics }); + const getOptional = dialcache.cached(async (): Promise => { + throw new Error("source unavailable"); + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + trackForInvalidation: true, + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: FRESH_TTL_SEC }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: MAX_AGE_SEC, + shadow: { ramp: 100 }, + }), + }); + + await expect(dialcache.enable(async () => await getOptional())).resolves.toBeUndefined(); + await vi.advanceTimersByTimeAsync(0); + + expect(redis.readRequests).toHaveLength(2); + expect(redis.setCalls).toBe(0); + expect(staleRecovery).toHaveBeenCalledWith(expect.objectContaining({ outcome: "served" })); + expect(shadowValidation).not.toHaveBeenCalled(); + }); + + it("applies a lowered runtime recovery maximum to an existing retained frame immediately", async () => { + const useCase = "StaleRecoveryLoweredRuntimeMaximum"; + const redis = new RecordingRedis(); + redis.setRaw( + redisValueKey(useCase), + encodeFrame({ id: "123", version: 1 }, Date.now() - 5_000), + 10_000, + ); + let maxAgeSec = 10; + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + cacheConfigProvider: async () => new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: FRESH_TTL_SEC }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: maxAgeSec, + }), + }); + const getUser = dialcache.cached(async (): Promise<{ readonly id: string; readonly version: number }> => { + throw sourceError; + }, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + }); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual({ id: "123", version: 1 }); + maxAgeSec = 3; + const [settled] = await Promise.allSettled([dialcache.enable(async () => await getUser())]); + + expect(rejectionReason(settled!)).toBe(sourceError); + expect(redis.readRequests.map(({ maxAgeMs }) => maxAgeMs)).toEqual([ + 1_000, + 10_000, + 1_000, + 3_000, + ]); + }); + + it("does not resurrect or extend a frame after raising the runtime recovery maximum", async () => { + const useCase = "StaleRecoveryRaisedRuntimeMaximum"; + const redis = new RecordingRedis(); + let maxAgeSec = 3; + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + let sourceCalls = 0; + const source = vi.fn(async (): Promise<{ readonly id: string; readonly version: number }> => { + if (++sourceCalls === 1) { + return { id: "123", version: 1 }; + } + throw sourceError; + }); + const dialcache = new DialCache({ + redis: { client: redis, readTimeoutMs: 1_000 }, + cacheConfigProvider: async () => new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: FRESH_TTL_SEC }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: maxAgeSec, + }), + }); + const getUser = dialcache.cached(source, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + }); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual({ id: "123", version: 1 }); + expect(redis.ttlMs(redisValueKey(useCase))).toBe(3_000); + await vi.advanceTimersByTimeAsync(3_000); + maxAgeSec = 10; + const [settled] = await Promise.allSettled([dialcache.enable(async () => await getUser())]); + + expect(rejectionReason(settled!)).toBe(sourceError); + expect(redis.readRequests.map(({ maxAgeMs }) => maxAgeMs)).toEqual([1_000, 1_000, 10_000]); + expect(redis.setCalls).toBe(1); + }); + + it("coalesces recovery and does not populate process-local cache", async () => { + const useCase = "StaleRecoveryProcessCoalescing"; + const redis = new RecordingRedis(); + seedStale(redis, useCase, { id: "123", version: 1 }); + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + const source = vi.fn(async () => { + throw sourceError; + }); + const { metrics, staleRecovery } = recordingMetrics(); + const dialcache = new DialCache({ redis: { client: redis, readTimeoutMs: 1_000 }, metrics }); + const localCache = (dialcache as unknown as { + readonly localCache: { put: (...args: unknown[]) => Promise }; + }).localCache; + const localPut = vi.spyOn(localCache, "put"); + const getUser = dialcache.cached(source, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig({ local: true }), + }); + + const values = await dialcache.enable(async () => await Promise.all([getUser(), getUser(), getUser()])); + + expect(source).toHaveBeenCalledOnce(); + expect(redis.readRequests).toHaveLength(2); + expect(values[1]).toBe(values[0]); + expect(values[2]).toBe(values[0]); + expect(localPut).not.toHaveBeenCalled(); + expect(staleRecovery).toHaveBeenCalledTimes(1); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual(values[0]); + expect(source).toHaveBeenCalledTimes(2); + expect(redis.readRequests).toHaveLength(4); + expect(localPut).not.toHaveBeenCalled(); + }); + + it("memoizes a recovered reference only within the active request-local scope", async () => { + const useCase = "StaleRecoveryRequestLocal"; + const redis = new RecordingRedis(); + seedStale(redis, useCase, { id: "123", version: 1 }); + const source = vi.fn(async () => { + throw new Error("source unavailable"); + }); + const dialcache = new DialCache({ redis: { client: redis, readTimeoutMs: 1_000 } }); + const getUser = dialcache.cached(source, { + keyType: "user_id", + useCase, + cacheKey: () => "123", + defaultConfig: staleConfig({ requestLocal: true }), + }); + + const [first, second] = await dialcache.enable(async () => { + const firstValue = await getUser(); + const secondValue = await getUser(); + return [firstValue, secondValue] as const; + }); + + expect(second).toBe(first); + expect(source).toHaveBeenCalledOnce(); + expect(redis.readRequests).toHaveLength(2); + + await expect(dialcache.enable(async () => await getUser())).resolves.toEqual(first); + expect(source).toHaveBeenCalledTimes(2); + expect(redis.readRequests).toHaveLength(4); + }); +}); diff --git a/test/fake-redis.ts b/test/fake-redis.ts index 93f5102..914c065 100644 --- a/test/fake-redis.ts +++ b/test/fake-redis.ts @@ -11,6 +11,7 @@ const FRAME_VERSION = 1; const ENCODING_OFFSET = 9; const PAYLOAD_OFFSET = 10; const WATERMARK_TTL_MARGIN_MS = 60_000; +const MAX_SUPPORTED_DURATION_MS = 365 * 24 * 60 * 60 * 1_000; interface StoredValue { value: Buffer; @@ -18,6 +19,7 @@ interface StoredValue { } export class FakeRedis implements DialCacheRedisClient { + readonly enforcesMaxAge = true as const; readonly values = new Map(); getCalls = 0; mGetCalls = 0; @@ -27,7 +29,8 @@ export class FakeRedis implements DialCacheRedisClient { failWatermarkGet = false; getGate: Promise | null = null; - async read({ valueKey, watermarkKey }: RedisReadRequest): Promise { + async read({ valueKey, watermarkKey, maxAgeMs }: RedisReadRequest): Promise { + assertValidMaxAgeMs(maxAgeMs); if (watermarkKey === undefined) { this.getCalls += 1; } else { @@ -35,7 +38,7 @@ export class FakeRedis implements DialCacheRedisClient { } await this.waitForRead(); this.throwIfReadFails(watermarkKey !== undefined); - return this.readPayload(valueKey, watermarkKey ?? null); + return this.readPayload(valueKey, watermarkKey ?? null, maxAgeMs); } async write({ @@ -122,12 +125,17 @@ export class FakeRedis implements DialCacheRedisClient { } } - private readPayload(valueKey: string, watermarkKey: string | null): RedisCachePayload | null { + private readPayload(valueKey: string, watermarkKey: string | null, maxAgeMs: number): RedisCachePayload | null { const raw = this.readRaw(valueKey); if (raw === null || raw.length < PAYLOAD_OFFSET || raw[0] !== FRAME_VERSION) { return null; } + const createdAtMs = Number(readTimestamp(raw)); + if (Date.now() - createdAtMs >= maxAgeMs) { + return null; + } + if (watermarkKey !== null) { let watermark: number | null; try { @@ -135,7 +143,7 @@ export class FakeRedis implements DialCacheRedisClient { } catch { return null; } - if (watermark === null || Number(readTimestamp(raw)) <= watermark) { + if (watermark === null || createdAtMs <= watermark) { return null; } } @@ -205,6 +213,17 @@ export class FakeRedis implements DialCacheRedisClient { } } +function assertValidMaxAgeMs(maxAgeMs: unknown): asserts maxAgeMs is number { + if ( + typeof maxAgeMs !== "number" + || !Number.isSafeInteger(maxAgeMs) + || maxAgeMs <= 0 + || maxAgeMs > MAX_SUPPORTED_DURATION_MS + ) { + throw new RangeError("Invalid DialCache Redis maxAgeMs"); + } +} + export function encodeFrame(value: unknown, createdAtMs = Date.now(), encoding = 0): Buffer { const timestamp = Buffer.alloc(8); timestamp.writeBigUInt64BE(BigInt(createdAtMs)); diff --git a/test/node-redis.test.ts b/test/node-redis.test.ts index 0447429..49d601e 100644 --- a/test/node-redis.test.ts +++ b/test/node-redis.test.ts @@ -56,13 +56,17 @@ describe("node-redis adapter", () => { it("provides the expected arguments for every bundled script", () => { const binary = Buffer.from([0, 0xff]); - expect(dialcacheRedisScripts.dialcacheRead.transformArguments("plain:value")).toEqual(["plain:value"]); + expect(dialcacheRedisScripts.dialcacheRead.transformArguments("plain:value", 60_000)).toEqual([ + "plain:value", + "60000", + ]); expect( dialcacheRedisScripts.dialcacheReadTracked.transformArguments( "tracked:{id}:value", "tracked:{id}:watermark", + 120_000, ), - ).toEqual(["tracked:{id}:value", "tracked:{id}:watermark"]); + ).toEqual(["tracked:{id}:value", "tracked:{id}:watermark", "120000"]); expect(dialcacheRedisScripts.dialcacheWrite.transformArguments("plain:value", 1_000, 0, "plain")).toEqual([ "plain:value", "1000", @@ -93,9 +97,14 @@ describe("node-redis adapter", () => { }); const adapter = createNodeRedisDialCacheClient(client as never); - await expect(adapter.read({ valueKey: "plain:value" })).resolves.toBe("plain"); + expect(adapter.enforcesMaxAge).toBe(true); + await expect(adapter.read({ valueKey: "plain:value", maxAgeMs: 60_000 })).resolves.toBe("plain"); await expect( - adapter.read({ valueKey: "tracked:{id}:value", watermarkKey: "tracked:{id}:watermark" }), + adapter.read({ + valueKey: "tracked:{id}:value", + watermarkKey: "tracked:{id}:watermark", + maxAgeMs: 120_000, + }), ).resolves.toEqual(Buffer.from([0, 0xff])); await expect( adapter.write({ valueKey: "plain:value", cacheTtlMs: 1_000, value: "plain" }), @@ -119,20 +128,26 @@ describe("node-redis adapter", () => { const controller = new AbortController(); const context = { timeoutMs: 25, signal: controller.signal } as const; - await adapter.read({ valueKey: "plain:value" }, context); + await adapter.read({ valueKey: "plain:value", maxAgeMs: 60_000 }, context); await adapter.read( - { valueKey: "tracked:{id}:value", watermarkKey: "tracked:{id}:watermark" }, + { + valueKey: "tracked:{id}:value", + watermarkKey: "tracked:{id}:watermark", + maxAgeMs: 120_000, + }, context, ); expect(client.dialcacheRead).toHaveBeenCalledWith( expect.objectContaining({ returnBuffers: true, signal: controller.signal }), "plain:value", + 60_000, ); expect(client.dialcacheReadTracked).toHaveBeenCalledWith( expect.objectContaining({ returnBuffers: true, signal: controller.signal }), "tracked:{id}:value", "tracked:{id}:watermark", + 120_000, ); }); diff --git a/test/prometheus.test.ts b/test/prometheus.test.ts index f45e24b..2ee20e2 100644 --- a/test/prometheus.test.ts +++ b/test/prometheus.test.ts @@ -15,6 +15,7 @@ import { type DisabledReason, type MetricErrorKind, type ShadowValidationOutcome, + type StaleRecoveryOutcome, } from "../src/index.js"; import { PrometheusDialCacheMetrics, createPrometheusDialCacheMetrics } from "../src/prometheus.js"; import { FakeRedis } from "./fake-redis.js"; @@ -71,6 +72,13 @@ const SHADOW_VALIDATION_OUTCOMES: Readonly timeout: true, dropped: true, }; +const STALE_RECOVERY_OUTCOMES: Readonly> = { + served: true, + miss: true, + read_error: true, + read_timeout: true, + deserialization_error: true, +}; interface IncompatibleCollectorCase { readonly schemaPart: string; @@ -171,6 +179,12 @@ describe("Prometheus metrics adapter", () => { keyType: labels.keyType, outcome: "match", }); + metrics.staleRecovery({ + cacheNamespace: labels.cacheNamespace, + useCase: labels.useCase, + keyType: labels.keyType, + outcome: "served", + }); metrics.observeGet(labels, 0.05); metrics.observeFallback(labels, 0.05); metrics.observeSerialization({ ...labels, operation: "dump" }, 0.05); @@ -205,6 +219,10 @@ describe("Prometheus metrics adapter", () => { ["cache_namespace", "use_case", "key_type", "outcome"], ), histogramSchema("schema_dialcache_size_histogram", ["cache_namespace", "use_case", "key_type", "layer"], SIZE_BUCKETS), + counterSchema( + "schema_dialcache_stale_recovery_counter", + ["cache_namespace", "use_case", "key_type", "outcome"], + ), ]); const serialization = families.find(({ name }) => name === "schema_dialcache_serialization_timer"); @@ -319,6 +337,39 @@ describe("Prometheus metrics adapter", () => { ); }); + it("exports every bounded stale-recovery outcome without adding cache identity or layer labels", async () => { + const registry = new Registry(); + const metrics = new PrometheusDialCacheMetrics({ registry, prefix: "stale_" }); + const labels = { + cacheNamespace: "users", + useCase: "PrometheusStaleRecovery", + keyType: "user_id", + } as const; + const outcomes = Object.keys(STALE_RECOVERY_OUTCOMES) as StaleRecoveryOutcome[]; + + for (const outcome of outcomes) { + metrics.staleRecovery({ ...labels, outcome }); + } + + for (const outcome of outcomes) { + await expect( + sumMetric(registry, "stale_dialcache_stale_recovery_counter", { + cache_namespace: labels.cacheNamespace, + use_case: labels.useCase, + key_type: labels.keyType, + outcome, + }), + ).resolves.toBe(1); + } + + const family = ((await registry.getMetricsAsJSON()) as unknown as MetricFamily[]).find( + ({ name }) => name === "stale_dialcache_stale_recovery_counter", + ); + expect(family?.values.map(({ labels: emitted }) => Object.keys(emitted))).toEqual( + outcomes.map(() => ["cache_namespace", "use_case", "key_type", "outcome"]), + ); + }); + it("uses the existing layer label for detached Redis telemetry", async () => { const registry = new Registry(); const metrics = new PrometheusDialCacheMetrics({ registry, prefix: "shadow_layer_" }); diff --git a/test/redis-cluster.integration.test.ts b/test/redis-cluster.integration.test.ts index 1a5f721..2763291 100644 --- a/test/redis-cluster.integration.test.ts +++ b/test/redis-cluster.integration.test.ts @@ -187,7 +187,64 @@ describe("DialCache Lua protocol on Redis Cluster", () => { expect(before).toEqual({ id: "123", version: 1 }); expect(after).toEqual({ id: "123", version: 2 }); - await expect(cluster.dialcacheReadTracked("{slot-a}:value", "{slot-b}:watermark")).rejects.toThrow(/CROSSSLOT/); + await expect(cluster.dialcacheReadTracked("{slot-a}:value", "{slot-b}:watermark", 60_000)).rejects.toThrow( + /CROSSSLOT/, + ); + }); + + it("recovers a logically stale tracked value through Cluster routing", async () => { + if (cluster === undefined) { + throw new Error("Redis Cluster did not start"); + } + const activeCluster = cluster; + const namespace = "cluster-stale"; + const useCase = "ClusterTrackedStaleOnError"; + const id = "123"; + const valueKey = `{${namespace}:user_id:${id}}#${useCase}:dialcache-frame-v1`; + const watermarkKey = `{${namespace}:user_id:${id}}#watermark`; + const scriptClient = createNodeRedisDialCacheClient(activeCluster); + const dialcache = new DialCache({ + namespace, + redis: { client: scriptClient, readTimeoutMs: 10_000 }, + }); + const sourceError = new Error("cluster source unavailable"); + let sourceCalls = 0; + let sourceAvailable = true; + const getUser = dialcache.cached(async () => { + sourceCalls += 1; + if (!sourceAvailable) { + throw sourceError; + } + return { id, version: 1 }; + }, { + keyType: "user_id", + useCase, + cacheKey: () => id, + trackForInvalidation: true, + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 1 }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: 60, + }), + }); + + const fresh = await dialcache.enable(async () => await getUser()); + const stored = await activeCluster.get(commandOptions({ returnBuffers: true }), valueKey); + if (stored === null) { + throw new Error("Tracked Cluster value was not written"); + } + const logicallyStale = Buffer.from(stored); + logicallyStale.writeBigUInt64BE(BigInt(Date.now() - 5_000), 1); + await activeCluster.set(valueKey, logicallyStale, { PX: 60_000 }); + const ttlBeforeRecovery = await activeCluster.pTTL(valueKey); + + sourceAvailable = false; + const recovered = await dialcache.enable(async () => await getUser()); + + expect(recovered).toEqual(fresh); + expect(sourceCalls).toBe(2); + expect(await activeCluster.get(watermarkKey)).toBe("0"); + expect(await activeCluster.pTTL(valueKey)).toBeLessThanOrEqual(ttlBeforeRecovery); }); it("round-trips binary payloads through cluster script routing", async () => { @@ -199,7 +256,7 @@ describe("DialCache Lua protocol on Redis Cluster", () => { const payload = Buffer.from(Array.from({ length: 256 }, (_, index) => index)); expect(await scriptClient.write({ valueKey, cacheTtlMs: 60_000, value: payload })).toBe(true); - expect(await scriptClient.read({ valueKey })).toEqual(payload); + expect(await scriptClient.read({ valueKey, maxAgeMs: 60_000 })).toEqual(payload); const stored = await cluster.get(commandOptions({ returnBuffers: true }), valueKey); expect(stored?.length).toBe(10 + payload.length); @@ -217,6 +274,8 @@ describe("DialCache Lua protocol on Redis Cluster", () => { value: trackedPayload, }), ).toBe(true); - expect(await scriptClient.read({ valueKey: trackedValueKey, watermarkKey })).toEqual(trackedPayload); + expect(await scriptClient.read({ valueKey: trackedValueKey, watermarkKey, maxAgeMs: 60_000 })).toEqual( + trackedPayload, + ); }); }); diff --git a/test/redis-real.integration.test.ts b/test/redis-real.integration.test.ts index 4ca5df9..9d9a917 100644 --- a/test/redis-real.integration.test.ts +++ b/test/redis-real.integration.test.ts @@ -13,6 +13,7 @@ import { } from "../src/index.js"; import { INVALIDATE_CACHE_SCRIPT, + READ_CACHE_SCRIPT, WRITE_CACHE_SCRIPT, WRITE_TRACKED_CACHE_SCRIPT, } from "../src/internal/redis-scripts.js"; @@ -253,6 +254,75 @@ describe.each(engines)("DialCache Lua protocol on $name", ({ image }) => { expect(inlineCalls).toBe(1); }); + it("writes physical maximum retention and recovers a logically stale value after source rejection", async () => { + if (client === undefined || admin === undefined) { + throw new Error("Redis test clients did not start"); + } + const namespace = `real-stale-${kind}`; + const useCase = "RealStaleOnError"; + const valueKey = `${namespace}:item_id:123#${useCase}:dialcache-frame-v1`; + const sourceValue = { id: "123", version: 1 }; + const sourceError = Object.freeze({ code: "SOURCE_UNAVAILABLE" }); + let sourceCalls = 0; + const source = vi.fn(async (): Promise => { + if (++sourceCalls === 1) { + return sourceValue; + } + throw sourceError; + }); + const staleRecovery = vi.fn(); + const metrics: DialCacheMetricsAdapter = { + request: vi.fn(), + miss: vi.fn(), + disabled: vi.fn(), + error: vi.fn(), + invalidation: vi.fn(), + staleRecovery, + observeGet: vi.fn(), + observeFallback: vi.fn(), + observeSerialization: vi.fn(), + observeSize: vi.fn(), + }; + const dialcache = new DialCache({ + namespace, + redis: { client: client.adapter, readTimeoutMs: 10_000 }, + metrics, + }); + const getItem = dialcache.cached(source, { + keyType: "item_id", + useCase, + cacheKey: () => "123", + defaultConfig: new DialCacheKeyConfig({ + ttlSec: { [CacheLayer.REMOTE]: 1 }, + ramp: { [CacheLayer.REMOTE]: 100 }, + staleOnErrorMaxAgeSec: 60, + }), + }); + + await expect(dialcache.enable(async () => await getItem())).resolves.toBe(sourceValue); + expect(await admin.pTTL(valueKey)).toBeGreaterThan(55_000); + + const redisNowMs = (await admin.time()).getTime(); + await admin.set( + valueKey, + encodeFrame(JSON.stringify(sourceValue), 0, redisNowMs - 2_000), + { PX: 60_000 }, + ); + const ttlBeforeRecovery = await admin.pTTL(valueKey); + + await expect(dialcache.enable(async () => await getItem())).resolves.toEqual(sourceValue); + + expect(source).toHaveBeenCalledTimes(2); + expect(staleRecovery).toHaveBeenCalledOnce(); + expect(staleRecovery).toHaveBeenCalledWith({ + cacheNamespace: namespace, + useCase, + keyType: "item_id", + outcome: "served", + }); + expect(await admin.pTTL(valueKey)).toBeLessThanOrEqual(ttlBeforeRecovery); + }); + it("stores arbitrary binary payloads without base64 expansion", async () => { if (client === undefined || admin === undefined) { throw new Error("Redis test clients did not start"); @@ -268,7 +338,7 @@ describe.each(engines)("DialCache Lua protocol on $name", ({ image }) => { const valueKey = `binary-raw:{item:${index}}:value`; expect(await scriptClient.write({ valueKey, cacheTtlMs: 60_000, value: payload })).toBe(true); - const roundTrip = await scriptClient.read({ valueKey }); + const roundTrip = await scriptClient.read({ valueKey, maxAgeMs: 60_000 }); const stored = await admin.get(commandOptions({ returnBuffers: true }), valueKey); expect(Buffer.isBuffer(roundTrip)).toBe(true); @@ -291,7 +361,9 @@ describe.each(engines)("DialCache Lua protocol on $name", ({ image }) => { value: trackedPayload, }), ).toBe(true); - expect(await scriptClient.read({ valueKey: trackedValueKey, watermarkKey })).toEqual(trackedPayload); + expect(await scriptClient.read({ valueKey: trackedValueKey, watermarkKey, maxAgeMs: 60_000 })).toEqual( + trackedPayload, + ); }); it("shadow-validates the deserialized tracked value without repairing a mismatch", async () => { @@ -650,7 +722,9 @@ describe.each(engines)("DialCache Lua protocol on $name", ({ image }) => { cacheTtlMs: 60_000, value: JSON.stringify(sourceValue), }); - expect(await client.adapter.read({ valueKey, watermarkKey })).toBe(JSON.stringify(sourceValue)); + expect(await client.adapter.read({ valueKey, watermarkKey, maxAgeMs: 60_000 })).toBe( + JSON.stringify(sourceValue), + ); expect(await admin.get(watermarkKey)).toBe("0"); expect(await admin.pTTL(valueKey)).toBeGreaterThan(55_000); expect(await admin.pTTL(valueKey)).toBeLessThanOrEqual(60_000); @@ -779,7 +853,7 @@ describe.each(engines)("DialCache Lua protocol on $name", ({ image }) => { await admin.scriptFlush(); expect(await scriptClient.write({ valueKey, cacheTtlMs: 60_000, value: "untracked" })).toBe(true); await admin.scriptFlush(); - expect(await scriptClient.read({ valueKey })).toBe("untracked"); + expect(await scriptClient.read({ valueKey, maxAgeMs: 60_000 })).toBe("untracked"); const trackedValueKey = "script-recovery:{item:tracked}:value"; const watermarkKey = "script-recovery:{item:tracked}:watermark"; @@ -793,7 +867,7 @@ describe.each(engines)("DialCache Lua protocol on $name", ({ image }) => { }), ).toBe(true); await admin.scriptFlush(); - expect(await scriptClient.read({ valueKey: trackedValueKey, watermarkKey })).toBe("tracked"); + expect(await scriptClient.read({ valueKey: trackedValueKey, watermarkKey, maxAgeMs: 60_000 })).toBe("tracked"); await admin.scriptFlush(); await expect( scriptClient.invalidate({ @@ -810,29 +884,115 @@ describe.each(engines)("DialCache Lua protocol on $name", ({ image }) => { const scriptClient = client.adapter; const valueKey = "read-paths:{item:read}:value"; const watermarkKey = "read-paths:{item:read}:watermark"; + const createdAtMs = Date.now(); - expect(await scriptClient.read({ valueKey })).toBeNull(); + expect(await scriptClient.read({ valueKey, maxAgeMs: 60_000 })).toBeNull(); await admin.set(valueKey, Buffer.alloc(9)); - expect(await scriptClient.read({ valueKey })).toBeNull(); + expect(await scriptClient.read({ valueKey, maxAgeMs: 60_000 })).toBeNull(); await admin.set(valueKey, encodeFrame("wrong-version", 0, 1_000, 2)); - expect(await scriptClient.read({ valueKey })).toBeNull(); + expect(await scriptClient.read({ valueKey, maxAgeMs: 60_000 })).toBeNull(); - await admin.set(valueKey, encodeFrame("tracked", 0, 1_000)); - expect(await scriptClient.read({ valueKey, watermarkKey })).toBeNull(); + await admin.set(valueKey, encodeFrame("tracked", 0, createdAtMs)); + expect(await scriptClient.read({ valueKey, watermarkKey, maxAgeMs: 60_000 })).toBeNull(); await admin.set(watermarkKey, "not-a-watermark"); - expect(await scriptClient.read({ valueKey, watermarkKey })).toBeNull(); + expect(await scriptClient.read({ valueKey, watermarkKey, maxAgeMs: 60_000 })).toBeNull(); await admin.set(watermarkKey, "9".repeat(400)); - expect(await scriptClient.read({ valueKey, watermarkKey })).toBeNull(); + expect(await scriptClient.read({ valueKey, watermarkKey, maxAgeMs: 60_000 })).toBeNull(); + + await admin.set(watermarkKey, String(createdAtMs)); + expect(await scriptClient.read({ valueKey, watermarkKey, maxAgeMs: 60_000 })).toBeNull(); + + await admin.set(watermarkKey, String(createdAtMs - 0.5)); + expect(await scriptClient.read({ valueKey, watermarkKey, maxAgeMs: 60_000 })).toBe("tracked"); + }); + + it("enforces bounded maximum age before tracked watermark lookup", async () => { + if (client === undefined || admin === undefined) { + throw new Error("Redis test clients did not start"); + } + const scriptClient = client.adapter; + const freshKey = "read-age:{item:fresh}:value"; + const expiredKey = "read-age:{item:expired}:value"; + const trackedExpiredKey = "read-age:{item:tracked}:value"; + const watermarkKey = "read-age:{item:tracked}:watermark"; + const nowMs = (await admin.time()).getTime(); + + // A future timestamp avoids a timing-sensitive positive-boundary assertion. + await admin.set(freshKey, encodeFrame("fresh", 0, nowMs + 5_000), { PX: 60_000 }); + await admin.set(expiredKey, encodeFrame("expired", 0, nowMs - 5_000), { PX: 60_000 }); + await admin.set(trackedExpiredKey, encodeFrame("tracked", 0, nowMs - 5_000), { PX: 60_000 }); + await admin.hSet(watermarkKey, "malformed", "state"); + + expect(await scriptClient.read({ valueKey: freshKey, maxAgeMs: 1 })).toBe("fresh"); + expect(await scriptClient.read({ valueKey: expiredKey, maxAgeMs: 1_000 })).toBeNull(); + // Age rejection precedes the tracked watermark GET, so expired data misses without surfacing WRONGTYPE. + expect( + await scriptClient.read({ + valueKey: trackedExpiredKey, + watermarkKey, + maxAgeMs: 1_000, + }), + ).toBeNull(); + expect( + await scriptClient.read({ valueKey: freshKey, maxAgeMs: MAX_SUPPORTED_DURATION_MS }), + ).toBe("fresh"); + + for (const maxAgeMs of [ + 0, + -1, + 1.5, + Number.NaN, + Number.POSITIVE_INFINITY, + Number.NEGATIVE_INFINITY, + MAX_SUPPORTED_DURATION_MS + 1, + Number.MAX_SAFE_INTEGER, + ]) { + await expect(scriptClient.read({ valueKey: freshKey, maxAgeMs })).rejects.toThrow( + "invalid DialCache max age", + ); + } + }); - await admin.set(watermarkKey, "1000"); - expect(await scriptClient.read({ valueKey, watermarkKey })).toBeNull(); + it("enforces the exact max-age boundary in the real Lua engine", async () => { + if (admin === undefined) { + throw new Error("Redis test clients did not start"); + } + const activeAdmin = admin; + const redisTimePrelude = String.raw`local redis_time = redis.call("TIME") +local now_ms = tonumber(redis_time[1]) * 1000 + math.floor(tonumber(redis_time[2]) / 1000)`; + const deterministicTimePrelude = "local now_ms = tonumber(ARGV[2])"; + const deterministicReadScript = READ_CACHE_SCRIPT.replace(redisTimePrelude, deterministicTimePrelude); + expect(deterministicReadScript).not.toBe(READ_CACHE_SCRIPT); + expect(deterministicReadScript).not.toContain(redisTimePrelude); + + const fixedNowMs = 2_000_000_000_000; + const maxAgeMs = 1_000; + const beforeKey = "read-age-exact:{item:before}:value"; + const atKey = "read-age-exact:{item:at}:value"; + const afterKey = "read-age-exact:{item:after}:value"; + await admin.set(beforeKey, encodeFrame("before", 0, fixedNowMs - maxAgeMs + 1)); + await admin.set(atKey, encodeFrame("at", 0, fixedNowMs - maxAgeMs)); + await admin.set(afterKey, encodeFrame("after", 0, fixedNowMs - maxAgeMs - 1)); + + const readAtFixedTime = async (valueKey: string): Promise => { + const result = await activeAdmin.eval(deterministicReadScript, { + keys: [valueKey], + arguments: [String(maxAgeMs), String(fixedNowMs)], + }); + if (result === null || typeof result === "string") { + return result; + } + throw new Error("Unexpected deterministic DialCache read reply"); + }; - await admin.set(watermarkKey, "999.5"); - expect(await scriptClient.read({ valueKey, watermarkKey })).toBe("tracked"); + // The raw script reply includes the encoding byte; semantic adapters decode it. + await expect(readAtFixedTime(beforeKey)).resolves.toBe("\0before"); + await expect(readAtFixedTime(atKey)).resolves.toBeNull(); + await expect(readAtFixedTime(afterKey)).resolves.toBeNull(); }); it("rejects invalid raw script arguments before mutating Redis", async () => { @@ -1029,7 +1189,7 @@ describe.each(engines)("DialCache Lua protocol on $name", ({ image }) => { await expect(client.raw.writeTracked(valueKey, watermarkKey, 60_000, 0, "replacement")).rejects.toThrow( "invalid DialCache watermark", ); - expect(await scriptClient.read({ valueKey })).toBe("original"); + expect(await scriptClient.read({ valueKey, maxAgeMs: 60_000 })).toBe("original"); } }); @@ -1180,7 +1340,7 @@ describe.each(engines)("DialCache Lua protocol on $name", ({ image }) => { expect(wrote).toBe(true); expect(await admin.get(watermarkKey)).toBe("1.75"); expect(await admin.pTTL(watermarkKey)).toBeGreaterThanOrEqual(61_000); - expect(await scriptClient.read({ valueKey, watermarkKey })).toBe("cached"); + expect(await scriptClient.read({ valueKey, watermarkKey, maxAgeMs: 60_000 })).toBe("cached"); }); it("does not rewrite sufficient or persistent watermarks on tracked writes", async () => { @@ -1242,16 +1402,16 @@ describe.each(engines)("DialCache Lua protocol on $name", ({ image }) => { expect(ttlAfterWrite).toBeGreaterThanOrEqual(61_000); await scriptClient.invalidate({ watermarkKey, futureBufferMs: 100 }); - expect(await scriptClient.read({ valueKey, watermarkKey })).toBeNull(); + expect(await scriptClient.read({ valueKey, watermarkKey, maxAgeMs: 60_000 })).toBeNull(); expect(await scriptClient.write({ ...writeRequest, value: "blocked" })).toBe(false); - expect(await scriptClient.read({ valueKey })).toBe("cached"); + expect(await scriptClient.read({ valueKey, maxAgeMs: 60_000 })).toBe("cached"); const ttlBeforeRead = await admin.pTTL(watermarkKey); - await scriptClient.read({ valueKey, watermarkKey }); + await scriptClient.read({ valueKey, watermarkKey, maxAgeMs: 60_000 }); expect(await admin.pTTL(watermarkKey)).toBeLessThanOrEqual(ttlBeforeRead); await new Promise((resolve) => setTimeout(resolve, 110)); expect(await scriptClient.write({ ...writeRequest, value: "fresh" })).toBe(true); - expect(await scriptClient.read({ valueKey, watermarkKey })).toBe("fresh"); + expect(await scriptClient.read({ valueKey, watermarkKey, maxAgeMs: 60_000 })).toBe("fresh"); }); it("documents that losing a watermark removes its publication fence", async () => { @@ -1275,7 +1435,7 @@ describe.each(engines)("DialCache Lua protocol on $name", ({ image }) => { expect(await scriptClient.write(staleWrite)).toBe(true); expect(await admin.get(watermarkKey)).toBe("0"); - expect(await scriptClient.read({ valueKey, watermarkKey })).toBe("stale"); + expect(await scriptClient.read({ valueKey, watermarkKey, maxAgeMs: 60_000 })).toBe("stale"); }); }); @@ -1289,10 +1449,12 @@ describe.each(engines)("DialCache Lua protocol on $name", ({ image }) => { const binary = Buffer.from([0, 0xff, 0xc3, 0x28, 0x80]); await nodeRedis.write({ valueKey: "interop:node-to-glide", cacheTtlMs: 60_000, value: binary }); - await expect(valkeyGlide.read({ valueKey: "interop:node-to-glide" })).resolves.toEqual(binary); + await expect( + valkeyGlide.read({ valueKey: "interop:node-to-glide", maxAgeMs: 60_000 }), + ).resolves.toEqual(binary); await valkeyGlide.write({ valueKey: "interop:glide-to-node", cacheTtlMs: 60_000, value: "hello" }); - await expect(nodeRedis.read({ valueKey: "interop:glide-to-node" })).resolves.toBe("hello"); + await expect(nodeRedis.read({ valueKey: "interop:glide-to-node", maxAgeMs: 60_000 })).resolves.toBe("hello"); const nodeTrackedValueKey = "interop:{node-tracked}:value"; const nodeTrackedWatermarkKey = "interop:{node-tracked}:watermark"; @@ -1306,6 +1468,7 @@ describe.each(engines)("DialCache Lua protocol on $name", ({ image }) => { valkeyGlide.read({ valueKey: nodeTrackedValueKey, watermarkKey: nodeTrackedWatermarkKey, + maxAgeMs: 60_000, }), ).resolves.toEqual(binary); @@ -1321,6 +1484,7 @@ describe.each(engines)("DialCache Lua protocol on $name", ({ image }) => { nodeRedis.read({ valueKey: glideTrackedValueKey, watermarkKey: glideTrackedWatermarkKey, + maxAgeMs: 60_000, }), ).resolves.toBe("tracked"); }); diff --git a/test/valkey-glide.test.ts b/test/valkey-glide.test.ts index cb7d405..b573fb1 100644 --- a/test/valkey-glide.test.ts +++ b/test/valkey-glide.test.ts @@ -70,23 +70,28 @@ describe("Valkey GLIDE adapter", () => { const client = fakeClient(Buffer.from([0, ...Buffer.from("plain")]), Buffer.from([1, 0, 0xff]), null); const adapter = createValkeyGlideDialCacheClient(client, mockGlide); - await expect(adapter.read({ valueKey: "plain:value" })).resolves.toBe("plain"); + expect(adapter.enforcesMaxAge).toBe(true); + await expect(adapter.read({ valueKey: "plain:value", maxAgeMs: 60_000 })).resolves.toBe("plain"); await expect( - adapter.read({ valueKey: "tracked:{id}:value", watermarkKey: "tracked:{id}:watermark" }), + adapter.read({ + valueKey: "tracked:{id}:value", + watermarkKey: "tracked:{id}:watermark", + maxAgeMs: 120_000, + }), ).resolves.toEqual(Buffer.from([0, 0xff])); - await expect(adapter.read({ valueKey: "missing:value" })).resolves.toBeNull(); + await expect(adapter.read({ valueKey: "missing:value", maxAgeMs: 60_000 })).resolves.toBeNull(); expect(client.invokeScript).toHaveBeenNthCalledWith( 1, expect.any(MockScript), - { keys: ["plain:value"], args: [], decoder: decoderBytes }, + { keys: ["plain:value"], args: ["60000"], decoder: decoderBytes }, ); expect(client.invokeScript).toHaveBeenNthCalledWith( 2, expect.any(MockScript), { keys: ["tracked:{id}:value", "tracked:{id}:watermark"], - args: [], + args: ["120000"], decoder: decoderBytes, }, ); @@ -99,13 +104,13 @@ describe("Valkey GLIDE adapter", () => { const controller = new AbortController(); await adapter.read( - { valueKey: "plain:value" }, + { valueKey: "plain:value", maxAgeMs: 60_000 }, { timeoutMs: 25, signal: controller.signal }, ); expect(client.invokeScript).toHaveBeenCalledWith( expect.any(MockScript), - { keys: ["plain:value"], args: [], decoder: decoderBytes }, + { keys: ["plain:value"], args: ["60000"], decoder: decoderBytes }, ); adapter.dispose(); }); @@ -155,9 +160,13 @@ describe("Valkey GLIDE adapter", () => { const client = fakeClient("not-bytes", Buffer.alloc(0), Buffer.from([2, 1]), "not-an-integer", null); const adapter = createValkeyGlideDialCacheClient(client, mockGlide); - await expect(adapter.read({ valueKey: "wrong-type" })).rejects.toBeInstanceOf(DialCacheRedisPayloadError); - await expect(adapter.read({ valueKey: "empty" })).rejects.toBeInstanceOf(DialCacheRedisPayloadError); - await expect(adapter.read({ valueKey: "wrong-encoding" })).rejects.toBeInstanceOf( + await expect(adapter.read({ valueKey: "wrong-type", maxAgeMs: 60_000 })).rejects.toBeInstanceOf( + DialCacheRedisPayloadError, + ); + await expect(adapter.read({ valueKey: "empty", maxAgeMs: 60_000 })).rejects.toBeInstanceOf( + DialCacheRedisPayloadError, + ); + await expect(adapter.read({ valueKey: "wrong-encoding", maxAgeMs: 60_000 })).rejects.toBeInstanceOf( DialCacheRedisPayloadEncodingError, ); await expectProtocolError( @@ -221,7 +230,9 @@ describe("Valkey GLIDE adapter", () => { for (const script of scriptInstances) { expect(script.release).toHaveBeenCalledTimes(1); } - await expect(adapter.read({ valueKey: "disposed" })).rejects.toThrow("Valkey GLIDE DialCache client is disposed"); + await expect(adapter.read({ valueKey: "disposed", maxAgeMs: 60_000 })).rejects.toThrow( + "Valkey GLIDE DialCache client is disposed", + ); expect(client.invokeScript).not.toHaveBeenCalled(); }); @@ -235,7 +246,7 @@ describe("Valkey GLIDE adapter", () => { ); const adapter = createValkeyGlideDialCacheClient(client, mockGlide); - const read = adapter.read({ valueKey: "in-flight" }); + const read = adapter.read({ valueKey: "in-flight", maxAgeMs: 60_000 }); expect(() => adapter.dispose()).toThrow( "Cannot dispose Valkey GLIDE DialCache client while operations are in flight", ); @@ -258,7 +269,7 @@ describe("Valkey GLIDE adapter", () => { const client = fakeClient(null); const adapter = createValkeyGlideDialCacheClient(client, mockGlide); - await adapter.read({ valueKey: "module-instance" }); + await adapter.read({ valueKey: "module-instance", maxAgeMs: 60_000 }); const [script, options] = client.invokeScript.mock.calls[0] ?? []; expect(script).toBeInstanceOf(MockScript);