Skip to content

Establish a local-hit performance budget and remove avoidable hot-path work #43

Description

@lan17

Priority

P3 — Low — benchmark-led hot-path work after the comparable harness exists; no public API change is required.

v0.14.0 grooming verification (2026-08-01)

The candidate costs remain, but the v0.11.0 timings below are now historical: v0.12.0v0.14.0 added shadow planning, detached validation/fill work, and observer wrappers. Reprofile exact current main through #35 before changing the hot path. Closed issue #103 is no longer separate work; its observer-isolation guarantee must simply be preserved. Keep P3, sequenced after correctness issues #101 and #97.

Historical baseline and current candidates

At the v0.11.0 baseline, the following enabled-path costs were identified:

  • every enabled invocation constructs a key;
  • every invocation crosses runtime-config resolution, including a synthetic () => null provider when no custom provider exists;
  • measured cache and fallback paths take clocks even when metrics are absent;
  • request-local and process-local hits pass through single-flight bookkeeping;
  • metric label objects are reconstructed around individual events.

Current evidence:

  • Invocation/config path:

    DialCache/src/dialcache.ts

    Lines 302 to 364 in e06d833

    private async executeCacheOperation<Value>(
    load: () => Awaitable<Value>,
    selectKey: () => CacheKeySpec,
    options: CacheOperationOptions<Value>,
    defaultConfig: DialCacheKeyConfig | null,
    fallbackTimeoutMs: number | null,
    ): Promise<Value> {
    const rawFallback = async (): Promise<Value> => await load();
    const noLayerLabels = {
    cacheNamespace: this.namespace,
    useCase: options.useCase,
    keyType: options.keyType,
    layer: NO_CACHE_LAYER,
    } as const;
    if (!this.isEnabled()) {
    this.metrics?.disabled({ ...noLayerLabels, reason: "context" });
    return await rawFallback();
    }
    const fallback = (): Promise<Value> => withFallbackTimeout(rawFallback, options.useCase, fallbackTimeoutMs);
    let key: DialCacheKey;
    try {
    key = this.buildKey(options, selectKey(), defaultConfig);
    } catch (error) {
    this.logger.error("Could not construct DialCache key", error);
    this.metrics?.error({
    ...noLayerLabels,
    error: "key_construction",
    inFallback: false,
    });
    return await this.callFallback(noLayerLabels, fallback);
    }
    let keyConfig: DialCacheKeyConfig | null;
    try {
    keyConfig = await fetchKeyConfig(this.configProvider, key);
    } catch (error) {
    // Provider failure: fail open and run uncached, mirroring the per-layer config_error path.
    this.logger.warn("Could not resolve DialCache key config", error);
    this.recordError(key, NO_CACHE_LAYER, "config_resolution");
    this.metrics?.disabled({ ...noLayerLabels, reason: "config_error" });
    return await this.callFallback(noLayerLabels, fallback);
    }
    // An unawaited child can inherit the async store after its outer enable()
    // callback settles. The closed holder turns that detached work back into
    // pass-through instead of allowing it to repopulate request state.
    if (!this.isEnabled()) {
    this.metrics?.disabled({ ...noLayerLabels, reason: "context" });
    return await this.callFallback(noLayerLabels, fallback);
    }
    if (keyConfig?.requestLocal === true) {
    const requestLocalCache = getOrCreateRequestLocalCache(this.context);
    if (requestLocalCache !== null) {
    return await this.getThroughRequestLocal(requestLocalCache, key, keyConfig, fallback);
    }
    }
    return await this.getThroughSharedLayers(key, keyConfig, fallback, CacheLayer.LOCAL);
    }
  • Request-local path:

    DialCache/src/dialcache.ts

    Lines 428 to 447 in e06d833

    private async getThroughRequestLocal<T>(
    requestLocalCache: RequestLocalCache,
    key: DialCacheKey,
    keyConfig: DialCacheKeyConfig,
    fallback: () => Promise<T>,
    ): Promise<T> {
    return await this.singleFlightRequestLocal(requestLocalCache.inFlight, key, async () => {
    const start = performance.now();
    const result = requestLocalCache.read<T>(key.urn);
    this.metrics?.request(labelsFor(key, REQUEST_LOCAL_CACHE_LAYER));
    this.metrics?.observeGet(labelsFor(key, REQUEST_LOCAL_CACHE_LAYER), elapsedSeconds(start));
    if (result.status === "hit") {
    return result.value;
    }
    this.metrics?.miss(labelsFor(key, REQUEST_LOCAL_CACHE_LAYER));
    const value = await this.getThroughSharedLayers(key, keyConfig, fallback, REQUEST_LOCAL_CACHE_LAYER);
    requestLocalCache.set(key.urn, value);
    return value;
    });
  • Shared/local path:

    DialCache/src/dialcache.ts

    Lines 450 to 513 in e06d833

    private async getThroughSharedLayers<T>(
    key: DialCacheKey,
    keyConfig: DialCacheKeyConfig | null,
    fallback: () => Promise<T>,
    fallbackMetricLayer: MetricLayer,
    ): Promise<T> {
    const localLayer = await this.resolveLocalLayerConfig(key, keyConfig);
    if (localLayer.status === "enabled") {
    return await this.singleFlightProcess(key, async () =>
    await this.getThroughActiveLocal(key, keyConfig, localLayer.config, fallback),
    );
    }
    const redisCache = this.redisCache;
    if (redisCache === null) {
    return await this.callFallback(labelsFor(key, fallbackMetricLayer), fallback);
    }
    const remoteLayer = await this.resolveRemoteLayerConfig(key, keyConfig);
    if (remoteLayer.status === "disabled") {
    const fallbackLayer = remoteLayer.reason === "config_error" ? CacheLayer.REMOTE : fallbackMetricLayer;
    return await this.callFallback(labelsFor(key, fallbackLayer), fallback);
    }
    return await this.singleFlightProcess(key, async () => {
    const remote = await this.readRemoteWithResolvedConfig<T>(
    redisCache,
    key,
    remoteLayer.config,
    keyConfig?.remoteReadTimeoutMs ?? redisCache.readTimeoutMs,
    );
    return await this.finishRedisChain(redisCache, key, localLayer, remote, fallback, remoteLayer.config);
    });
    }
    private async getThroughActiveLocal<T>(
    key: DialCacheKey,
    keyConfig: DialCacheKeyConfig | null,
    localConfig: ResolvedLayerConfig,
    fallback: () => Promise<T>,
    ): Promise<T> {
    const local = this.readLocalWithResolvedConfig<T>(key, localConfig);
    if (local.status === "hit") {
    return local.value;
    }
    const redisCache = this.redisCache;
    if (redisCache === null) {
    return await this.finishLocalOnly(key, local, fallback);
    }
    const remoteLayer = await this.resolveRemoteLayerConfig(key, keyConfig);
    if (remoteLayer.status === "disabled") {
    return await this.finishRedisChain(redisCache, key, local, remoteLayer, fallback);
    }
    const remote = await this.readRemoteWithResolvedConfig<T>(
    redisCache,
    key,
    remoteLayer.config,
    keyConfig?.remoteReadTimeoutMs ?? redisCache.readTimeoutMs,
    );
    return await this.finishRedisChain(redisCache, key, local, remote, fallback, remoteLayer.config);
    }
  • Local/fallback clocks:

    DialCache/src/dialcache.ts

    Lines 567 to 665 in e06d833

    private async resolveLocalLayerConfig(
    key: DialCacheKey,
    keyConfig: DialCacheKeyConfig | null,
    ): Promise<LayerConfigResolution> {
    try {
    const result = await this.localCache.resolveLayerConfig(key, keyConfig);
    if (result.status === "disabled") {
    this.metrics?.disabled({ ...labelsFor(key, CacheLayer.LOCAL), reason: result.reason });
    this.recordInvalidLeaf(key, CacheLayer.LOCAL, result.reason);
    }
    return result;
    } catch (error) {
    this.logger.error("Error resolving local cache config", error);
    this.recordError(key, CacheLayer.LOCAL, "config_resolution");
    this.metrics?.disabled({ ...labelsFor(key, CacheLayer.LOCAL), reason: "config_error" });
    return { status: "disabled", reason: "config_error" };
    }
    }
    private readLocalWithResolvedConfig<T>(key: DialCacheKey, layerConfig: ResolvedLayerConfig): CacheGetResult<T> {
    const start = performance.now();
    try {
    const result = this.localCache.getWithResolvedConfig<T>(key, layerConfig);
    this.metrics?.request(labelsFor(key, CacheLayer.LOCAL));
    this.metrics?.observeGet(labelsFor(key, CacheLayer.LOCAL), elapsedSeconds(start));
    if (result.status === "miss") {
    this.metrics?.miss(labelsFor(key, CacheLayer.LOCAL));
    }
    return result;
    } catch (error) {
    this.logger.error("Error getting value from local cache", error);
    this.recordError(key, CacheLayer.LOCAL, "cache_read");
    this.metrics?.disabled({ ...labelsFor(key, CacheLayer.LOCAL), reason: "config_error" });
    return { status: "disabled", reason: "config_error" } as const;
    }
    }
    private async resolveRemoteLayerConfig(key: DialCacheKey, keyConfig: DialCacheKeyConfig | null) {
    try {
    const result = resolveLayerConfigResult({
    config: keyConfig,
    key,
    layer: CacheLayer.REMOTE,
    });
    if (result.status === "disabled") {
    this.metrics?.disabled({ ...labelsFor(key, CacheLayer.REMOTE), reason: result.reason });
    this.recordInvalidLeaf(key, CacheLayer.REMOTE, result.reason);
    }
    return result;
    } catch (error) {
    this.logger.warn("Error resolving Redis cache config", error);
    this.recordError(key, CacheLayer.REMOTE, "config_resolution");
    this.metrics?.disabled({ ...labelsFor(key, CacheLayer.REMOTE), reason: "config_error" });
    return { status: "disabled", reason: "config_error", ...(key.trackForInvalidation ? { skipCacheWrite: true } : {}) } as const;
    }
    }
    private async readRemoteWithResolvedConfig<T>(
    redisCache: RedisCache,
    key: DialCacheKey,
    layerConfig: ResolvedLayerConfig,
    readTimeoutMs: number,
    ): Promise<RemoteCacheGetResult<T>> {
    const start = performance.now();
    this.metrics?.request(labelsFor(key, CacheLayer.REMOTE));
    try {
    const result = await redisCache.getWithResolvedConfig<T>(key, layerConfig, readTimeoutMs);
    if (result.status === "miss") {
    this.metrics?.miss(labelsFor(key, CacheLayer.REMOTE));
    }
    return result;
    } catch (error) {
    this.logger.warn("Error getting value from Redis cache", error);
    return { status: "error", operation: "read" };
    } finally {
    this.metrics?.observeGet(labelsFor(key, CacheLayer.REMOTE), elapsedSeconds(start));
    }
    }
    private async putLocalFailOpen<T>(key: DialCacheKey, value: T, config?: { readonly ttlSec: number }): Promise<void> {
    try {
    await this.localCache.put(key, value, config);
    } catch (error) {
    this.logger.warn("Error putting value in local cache", error);
    this.recordError(key, CacheLayer.LOCAL, "cache_write");
    }
    }
    private async callFallback<T>(labels: CacheMetricLabels, fallback: () => Promise<T>): Promise<T> {
    const start = performance.now();
    try {
    return await fallback();
    } catch (error) {
    this.metrics?.error({ ...labels, error: "fallback", inFallback: true });
    throw error;
    } finally {
    this.metrics?.observeFallback(labels, elapsedSeconds(start));
    }
    }
  • Key construction:

    DialCache/src/dialcache.ts

    Lines 697 to 713 in e06d833

    private buildKey<Value>(
    options: CacheOperationOptions<Value>,
    cacheKey: CacheKeySpec,
    defaultConfig: DialCacheKeyConfig | null,
    ): DialCacheKey {
    const spec = typeof cacheKey === "object" ? cacheKey : { id: cacheKey };
    return new DialCacheKey({
    keyType: options.keyType,
    id: String(spec.id),
    useCase: options.useCase,
    args: normalizeArgs(spec.args ?? {}),
    namespace: this.namespace,
    defaultConfig,
    serializer: (options.serializer as Serializer<unknown> | null | undefined) ?? null,
    trackForInvalidation: options.trackForInvalidation ?? false,
    });
    }
  • Single-flight bookkeeping:

    DialCache/src/dialcache.ts

    Lines 715 to 783 in e06d833

    private singleFlightRequestLocal<T>(
    inFlight: Map<string, Promise<unknown>>,
    key: DialCacheKey,
    run: () => Promise<T>,
    ): Promise<T> {
    const existing = inFlight.get(key.urn);
    if (existing !== undefined) {
    this.metrics?.coalesced?.({
    cacheNamespace: key.namespace,
    useCase: key.useCase,
    keyType: key.keyType,
    scope: "request_local",
    });
    return existing as Promise<T>;
    }
    const promise = run();
    inFlight.set(key.urn, promise);
    const clear = (): void => {
    if (inFlight.get(key.urn) === promise) {
    inFlight.delete(key.urn);
    }
    };
    void promise.then(clear, clear);
    return promise;
    }
    private singleFlightProcess<T>(key: DialCacheKey, run: () => Promise<T>): Promise<T> {
    const existing = this.processFlights.get(key.urn);
    if (existing !== undefined) {
    if (existing.promise === null) {
    throw new Error("DialCache process flight was joined before initialization");
    }
    existing.followers += 1;
    this.activeProcessFollowers += 1;
    this.metrics?.coalesced?.({
    cacheNamespace: key.namespace,
    useCase: key.useCase,
    keyType: key.keyType,
    scope: "process",
    });
    return existing.promise as Promise<T>;
    }
    const flight: ProcessFlight = {
    promise: null,
    startedAtMs: performance.now(),
    followers: 0,
    };
    this.processFlights.set(key.urn, flight);
    let promise: Promise<T>;
    try {
    promise = run();
    } catch (error) {
    if (this.processFlights.get(key.urn) === flight) {
    this.processFlights.delete(key.urn);
    }
    throw error;
    }
    flight.promise = promise;
    const clear = (): void => {
    if (this.processFlights.get(key.urn) === flight) {
    this.activeProcessFollowers -= flight.followers;
    this.processFlights.delete(key.urn);
    }
    };
    void promise.then(clear, clear);
    return promise;
    }
  • Benchmark harness: https://github.com/lan17/DialCache/blob/e06d833ba245706a499d66056fdad15dc1210b68/scripts/benchmark-request-local.mjs

#39 remains consolidated here: calls without a custom provider still invoke and await a synthetic provider before request-local lookup.

Fresh directional profile

Node.js 22.22.0 repeated probes against exact v0.11.0 measured:

Component or path Median
Full process-local hit 1,358.6 ns/op
Full request-local hit 978.3 ns/op
Direct resolved LRU read 36.7 ns/op
Key construction 224.3 ns/op
Resolved local-config async boundary 208.3 ns/op
Process-flight sequential bookkeeping 127.1 ns/op
Request-flight sequential bookkeeping 100.5 ns/op
performance.now() 16.8 ns/call

A deterministic no-op metrics adapter added approximately 1.5% to process-local hits and 3% to request-local hits. Real backend cost will depend on the adapter.

Tracked key construction measured approximately 420.2 ns/op versus 222.8 ns/op untracked. The tracked constructor currently builds an untracked prefix and then validates and encodes the same namespace/type/ID again for the Redis hash tag:

  • DialCache/src/key.ts

    Lines 40 to 45 in e06d833

    assertValidNamespace(this.namespace);
    const rawPrefix = joinUrnComponents(this.namespace, this.keyType, this.id);
    this.prefix = this.trackForInvalidation ? redisClusterHashTag(invalidationPrefix(this.namespace, this.keyType, this.id)) : rawPrefix;
    const args = this.args.length > 0 ? `?${this.args.map(([name, value]) => `${encodeComponent(name)}=${encodeComponent(value)}`).join("&")}` : "";
    this.urn = `${this.prefix}${args}#${encodeComponent(this.useCase)}`;
  • DialCache/src/key.ts

    Lines 60 to 65 in e06d833

    export function invalidationPrefix(namespace: string, keyType: string, id: string): string {
    assertValidNamespace(namespace);
    assertRedisHashTagComponent("keyType", keyType);
    assertRedisHashTagComponent("id", id);
    return joinUrnComponents(namespace, keyType, id);
    }

The internal cache layers also retain provider-owning compatibility paths that production does not call directly:

  • constructor(
    private readonly configProvider: CacheConfigProvider,
    maxSize: number,
    ) {
    this.cache =
    maxSize === 0
    ? null
    : new LRUCache({
    // Weight every entry as one so large configured limits remain sparse
    // instead of eagerly preallocating max-sized storage arrays.
    maxSize,
    // Read a fresh monotonic integer clock and avoid lru-cache's zero
    // timestamp sentinel when the process or a fake clock starts at 0.
    perf: { now: () => Math.floor(performance.now()) + 1 },
    ttlResolution: 0,
    });
    }
    async get<T>(key: DialCacheKey, fallback: Fallback<T>): Promise<T> {
    const result = await this.getIfPresentResult<T>(key);
    if (result.status === "hit") {
    return result.value;
    }
    const value = await fallback();
    if (result.status === "miss") {
    await this.put(key, value, result.config);
    }
    return value;
    }
    async getIfPresent<T>(key: DialCacheKey): Promise<T | undefined> {
    const result = await this.getIfPresentResult<T>(key);
    return result.status === "hit" ? result.value : undefined;
    }
    async getIfPresentResult<T>(key: DialCacheKey, keyConfig?: DialCacheKeyConfig | null): Promise<CacheGetResult<T>> {
    const layerConfig = await this.resolveLayerConfig(key, keyConfig);
    if (layerConfig.status === "disabled") {
    return layerConfig;
    }
    return this.getWithResolvedConfig<T>(key, layerConfig.config);
    }
    getWithResolvedConfig<T>(key: DialCacheKey, layerConfig: ResolvedLayerConfig): CacheGetResult<T> {
    const hit = this.cache?.get(key.urn) as LocalEntry<T> | undefined;
    if (hit !== undefined) {
    return { status: "hit", value: hit.value };
    }
    return { status: "miss", config: layerConfig };
    }
    async resolveLayerConfig(
    key: DialCacheKey,
    keyConfig?: DialCacheKeyConfig | null,
    ): Promise<LayerConfigResolution> {
    // Chain callers pass the once-resolved config; standalone callers omit it and we fetch.
    const config = keyConfig === undefined ? await fetchKeyConfig(this.configProvider, key) : keyConfig;
    return resolveLayerConfigResult({
    config,
    key,
    layer: CacheLayer.LOCAL,
    });
    }
    async put<T>(key: DialCacheKey, value: T, config?: { readonly ttlSec: number }): Promise<void> {
    const ttlSec = config?.ttlSec ?? await this.resolveLocalTtlSec(key);
    if (ttlSec === null) {
    return;
    }
    // lru-cache expires when age > ttl, while DialCache historically expired
    // when its integer-millisecond clock reached the configured boundary.
    const ttlMs = ttlSec * 1000 - 1;
    this.cache?.set(key.urn, { value }, { size: 1, ttl: ttlMs });
    }
    private async resolveLocalTtlSec(key: DialCacheKey): Promise<number | null> {
    const layerConfig = await this.resolveLayerConfig(key);
    return layerConfig.status === "enabled" ? layerConfig.config.ttlSec : null;
    }
  • interface RedisCacheOptions {
    readonly configProvider: CacheConfigProvider;
    readonly redis: RedisConfig;
    readonly metrics: DialCacheMetricsAdapter | null;
    }
    const defaultSerializer = new JsonSerializer<unknown>();
    const REDIS_FRAME_KEY_SUFFIX = ":dialcache-frame-v1";
    const DEFAULT_REMOTE_READ_TIMEOUT_MS = 50;
    export class RedisCache {
    private readonly configProvider: CacheConfigProvider;
    private readonly defaultSerializer: Serializer<unknown>;
    private readonly client: DialCacheRedisClient;
    private readonly metrics: DialCacheMetricsAdapter | null;
    readonly readTimeoutMs: number;
    constructor(options: RedisCacheOptions) {
    if (Object.hasOwn(options.redis, "keyPrefix")) {
    throw new TypeError("RedisConfig.keyPrefix was removed; use DialCacheConfig.namespace for cache identity");
    }
    if (Object.hasOwn(options.redis, "createClient")) {
    throw new TypeError(
    "RedisConfig.createClient was removed; create and connect a client, then pass it as RedisConfig.client",
    );
    }
    if (Object.hasOwn(options.redis, "watermarkTtlSec")) {
    throw new TypeError("RedisConfig.watermarkTtlSec was removed; watermark lifetime is derived by DialCache");
    }
    this.configProvider = options.configProvider;
    this.defaultSerializer = options.redis.serializer ?? defaultSerializer;
    this.metrics = options.metrics;
    this.readTimeoutMs = options.redis.readTimeoutMs === undefined
    ? DEFAULT_REMOTE_READ_TIMEOUT_MS
    : options.redis.readTimeoutMs;
    assertValidDeadlineMs(this.readTimeoutMs, "Redis readTimeoutMs");
    if (options.redis.client === undefined) {
    throw new TypeError("Redis config requires client");
    }
    this.client = options.redis.client;
    }
    async get<T>(key: DialCacheKey): Promise<T | undefined> {
    const result = await this.getResult<T>(key);
    return result.status === "hit" ? result.value : undefined;
    }
    async getResult<T>(key: DialCacheKey, keyConfig?: DialCacheKeyConfig | null): Promise<CacheGetResult<T>> {
    const layerConfig = await this.resolveRemoteLayerConfig(key, keyConfig);
    if (layerConfig.status === "disabled") {
    return layerConfig;
    }
    return await this.getWithResolvedConfig(
    key,
    layerConfig.config,
    keyConfig?.remoteReadTimeoutMs ?? this.readTimeoutMs,
    );
    }
  • private async resolveRemoteLayerConfig(key: DialCacheKey, keyConfig?: DialCacheKeyConfig | null) {
    const config = keyConfig === undefined ? await fetchKeyConfig(this.configProvider, key) : keyConfig;
    return resolveLayerConfigResult({
    config,
    key,
    layer: CacheLayer.REMOTE,
    });
    }
    private async resolveRemoteTtlSec(key: DialCacheKey): Promise<number | null> {
    const layerConfig = await this.resolveRemoteLayerConfig(key);
    return layerConfig.status === "enabled" ? layerConfig.config.ttlSec : null;
    }

These figures are motivating evidence, not accepted budgets. Reproduce them through #35 before implementation decisions.

Candidate scope after #35

Profile first, then take only supported changes:

  • Skip timing clocks when metrics are absent.
  • Add an internal no-custom-provider path that uses the captured definition default directly.
  • Preserve exactly one provider resolution per enabled invocation when a custom dynamic provider exists.
  • Build tracked key prefixes with one encoding pass while preserving exact key identity.
  • Consider making DialCache/runtime-config the sole policy owner and reducing LocalCache/RedisCache to resolved storage/protocol operations.
  • Remove production-unused internal compatibility methods only after repository and packed-package confirmation that they are not supported exports.
  • Precompute immutable definition-level metric metadata only where allocation profiles justify it.
  • Evaluate a hit path that checks an existing flight before synchronous storage lookup and installs a leader only on miss.

Any flight-order change must preserve synchronous leader registration before user fallback work and exact follower observability.

Out of scope

  • Public cache API or configuration knobs.
  • Cache-key identity or Redis wire-format changes.
  • Dynamic-provider memoization, refresh, staleness, or library-owned deadlines.
  • Counter sampling or metric semantic changes.
  • Vendor-client encoding work inside core.
  • Optimizations without warmup, repeated samples, captured environment, and CPU/allocation profiles.
  • Regressing the asynchronous logger/metrics rejection isolation shipped through Consume rejected promises from logger and metrics adapters #103.

Acceptance criteria

  • Reproducible before/after evidence through Add a repeatable performance and scale benchmark suite #35.
  • CPU and allocation profiles identify every changed cost.
  • Metrics-disabled paths avoid unnecessary clocks.
  • No-custom-provider calls avoid the synthetic provider call/await.
  • Dynamic providers retain exactly-once per-invocation semantics and application-owned settlement/freshness policy.
  • Tracked and untracked cache keys remain byte-for-byte compatible.
  • Request-local and process-local single-flight semantics remain exact under concurrent and synchronous-prefix tests.
  • Metrics-enabled counters/timers and observer isolation remain exact.
  • Internal simplification does not create a new public abstraction.
  • No public API or Redis wire change.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions