Characterize and guard watermark cardinality and Redis Cluster hot slots

[lan17]

Priority

P2 — Medium — measure and document tracked-cache scale limits before high-cardinality or hot-ID adoption; the former fixed-retention P1 risk was removed.

v0.14.0 grooming verification (2026-08-01)

The scale risk remains on main@34bd022: each tracked identity owns a watermark, all variants share one cluster slot, and tracked node-redis reads remain primary-routed. The v0.14.0 shadow path adds tracked primary reads/fills. Batch invalidation PR #114 improves dispatch but does not remove per-ID hot-slot concentration; proposed stale-on-error #117 would extend value/watermark residency. Keep P2 and gather evidence through #35 before adding controls.

Shipped mitigation in v0.10.0

#88 removed the four-hour watermark floor and its public configuration knobs.

Watermark lifetime is now derived from active state:

• A tracked write retains the watermark for at least the written value TTL plus one minute.
• An invalidation retains it for at least the remaining future-buffer window plus one minute.
• Neither operation shortens a longer or persistent watermark.
• Reads do not extend watermark lifetime.

This removes the stale capacity model that assumed every new tracked ID remained for at least four hours.

Current evidence:

• Tracked write TTL derivation:

  DialCache/src/internal/redis-scripts.ts

  Lines 85 to 115 in 84341f2

  	export const WRITE_TRACKED_CACHE_SCRIPT = [
  	PARSE_WATERMARK_LUA,
  	CEIL_FINITE_NUMBER_LUA,
  	VALIDATE_WRITE_ARGUMENTS_LUA,
  	REDIS_TIME_LUA,
  	String.raw`local raw_watermark = redis.call("GET", KEYS[2])
  	local watermark = 0
  	if raw_watermark then
  	watermark = parse_watermark(raw_watermark)
  	if not watermark then
  	return redis.error_reply("ERR invalid DialCache watermark")
  	end
  	end
  	if watermark >= now_ms then
  	return 0
  	end`,
  	WRITE_FRAME_LUA,
  	String.raw`local desired_ttl_ms = cache_ttl_ms + ${WATERMARK_TTL_MARGIN_MS}
  	if not raw_watermark then
  	redis.call("SET", KEYS[2], "0", "PX", desired_ttl_ms)
  	else
  	local current_ttl_ms = redis.call("PTTL", KEYS[2])
  	if current_ttl_ms == -2 then
  	redis.call("SET", KEYS[2], raw_watermark, "PX", desired_ttl_ms)
  	elseif current_ttl_ms ~= -1 and current_ttl_ms < desired_ttl_ms then
  	redis.call("PEXPIRE", KEYS[2], desired_ttl_ms)
  	end
  	end`,
  	"return 1",
  	].join("\n\n");

• Invalidation TTL derivation:

  DialCache/src/internal/redis-scripts.ts

  Lines 117 to 156 in 84341f2

  	export const INVALIDATE_CACHE_SCRIPT = [
  	PARSE_WATERMARK_LUA,
  	CEIL_FINITE_NUMBER_LUA,
  	String.raw`local future_buffer_ms = ceil_finite_number(ARGV[1])
  	if not future_buffer_ms or future_buffer_ms < 0 then
  	return redis.error_reply("ERR invalid DialCache future buffer")
  	end`,
  	REDIS_TIME_LUA,
  	String.raw`local proposed_watermark = now_ms + future_buffer_ms
  	local raw_watermark = redis.call("GET", KEYS[1])
  	local current_watermark = 0
  	if raw_watermark then
  	local parsed_watermark = parse_watermark(raw_watermark)
  	if parsed_watermark then
  	current_watermark = parsed_watermark
  	end
  	end
  	local watermark = math.ceil(math.max(current_watermark, proposed_watermark))
  	local current_ttl_ms = -2
  	if raw_watermark then
  	current_ttl_ms = redis.call("PTTL", KEYS[1])
  	end
  	local desired_ttl_ms = math.max(
  	future_buffer_ms + ${WATERMARK_TTL_MARGIN_MS},
  	watermark - now_ms + ${WATERMARK_TTL_MARGIN_MS}
  	)
  	if current_ttl_ms > desired_ttl_ms then
  	desired_ttl_ms = current_ttl_ms
  	end
  	local encoded_watermark = string.format("%.0f", watermark)
  	if current_ttl_ms == -1 then
  	redis.call("SET", KEYS[1], encoded_watermark)
  	else
  	redis.call("SET", KEYS[1], encoded_watermark, "PX", desired_ttl_ms)
  	end`,
  	"return 1",
  	].join("\n\n");

• Real-Redis lifetime coverage:

  DialCache/test/redis-real.integration.test.ts

  Lines 629 to 720 in 84341f2

  	it("does not rewrite sufficient or persistent watermarks on tracked writes", async () => {
  	if (client === undefined || admin === undefined) {
  	throw new Error("Redis test clients did not start");
  	}
  	const scriptClient = client.adapter;
  	const sufficientValueKey = "write-sufficient:{item:sufficient}:value";
  	const sufficientWatermarkKey = "write-sufficient:{item:sufficient}:watermark";
  	await admin.set(sufficientWatermarkKey, "1.75", { PX: 120_000 });
  	const sufficientTtlBefore = await admin.pTTL(sufficientWatermarkKey);
  	expect(
  	await scriptClient.write({
  	valueKey: sufficientValueKey,
  	watermarkKey: sufficientWatermarkKey,
  	cacheTtlMs: 2_000,
  	value: "cached",
  	}),
  	).toBe(true);
  	expect(await admin.get(sufficientWatermarkKey)).toBe("1.75");
  	expect(await admin.pTTL(sufficientWatermarkKey)).toBeGreaterThan(sufficientTtlBefore - 1_000);
  	expect(await admin.pTTL(sufficientWatermarkKey)).toBeLessThanOrEqual(sufficientTtlBefore);
  	const persistentValueKey = "write-persistent:{item:persistent}:value";
  	const persistentWatermarkKey = "write-persistent:{item:persistent}:watermark";
  	await admin.set(persistentWatermarkKey, "2.25");
  	expect(
  	await scriptClient.write({
  	valueKey: persistentValueKey,
  	watermarkKey: persistentWatermarkKey,
  	cacheTtlMs: 2_000,
  	value: "cached",
  	}),
  	).toBe(true);
  	expect(await admin.get(persistentWatermarkKey)).toBe("2.25");
  	expect(await admin.pTTL(persistentWatermarkKey)).toBe(-1);
  	});
  	it("atomically blocks writes during the buffer and extends watermark TTL", async () => {
  	if (client === undefined || admin === undefined) {
  	throw new Error("Redis test clients did not start");
  	}
  	const scriptClient = client.adapter;
  	const valueKey = "protocol:{item:ttl}:value";
  	const watermarkKey = "protocol:{item:ttl}:watermark";
  	const writeRequest = {
  	valueKey,
  	watermarkKey,
  	cacheTtlMs: 2_000,
  	value: "cached",
  	};
  	expect(await scriptClient.write(writeRequest)).toBe(true);
  	expect(await admin.get(watermarkKey)).toBe("0");
  	const ttlAfterWrite = await admin.pTTL(watermarkKey);
  	expect(ttlAfterWrite).toBeGreaterThanOrEqual(61_000);
  	await scriptClient.invalidate({ watermarkKey, futureBufferMs: 100 });
  	expect(await scriptClient.read({ valueKey, watermarkKey })).toBeNull();
  	expect(await scriptClient.write({ ...writeRequest, value: "blocked" })).toBe(false);
  	expect(await scriptClient.read({ valueKey })).toBe("cached");
  	const ttlBeforeRead = await admin.pTTL(watermarkKey);
  	await scriptClient.read({ valueKey, watermarkKey });
  	expect(await admin.pTTL(watermarkKey)).toBeLessThanOrEqual(ttlBeforeRead);
  	await new Promise((resolve) => setTimeout(resolve, 110));
  	expect(await scriptClient.write({ ...writeRequest, value: "fresh" })).toBe(true);
  	expect(await scriptClient.read({ valueKey, watermarkKey })).toBe("fresh");
  	});
  	it("documents that losing a watermark removes its publication fence", async () => {
  	if (client === undefined || admin === undefined) {
  	throw new Error("Redis test clients did not start");
  	}
  	const scriptClient = client.adapter;
  	const valueKey = "watermark-loss:{item:tracked}:value";
  	const watermarkKey = "watermark-loss:{item:tracked}:watermark";
  	const staleWrite = {
  	valueKey,
  	watermarkKey,
  	cacheTtlMs: 60_000,
  	value: "stale",
  	};
  	await scriptClient.invalidate({ watermarkKey, futureBufferMs: 60_000 });
  	expect(await scriptClient.write(staleWrite)).toBe(false);
  	await admin.del(watermarkKey);
  	expect(await scriptClient.write(staleWrite)).toBe(true);
  	expect(await admin.get(watermarkKey)).toBe("0");

Remaining scale risks

The residual risks are workload-dependent but real:

• Every active tracked keyType / id owns one watermark.
• All use-case/argument variants for one tracked ID share one Redis Cluster hash slot.
• Tracked node-redis reads stay on primaries so replica lag cannot hide invalidation.
• A high-cardinality workload can retain many marker keys for its effective value/future-window lifetime; a celebrity ID can concentrate all variants on one primary.

The relevant capacity model is now approximately:

active distinct tracked-ID rate × effective derived watermark lifetime

plus the workload's ID-skew distribution and tracked-read/write QPS.

Scope

• Extend Add a repeatable performance and scale benchmark suite #35 with tracked-write and invalidation scenarios.
• Measure memory/operations for high-cardinality tracked writes across representative value TTLs.
• Measure primary/cluster throughput for uniform and highly skewed IDs.
• Document the derived-lifetime capacity model, primary-only read constraint, and per-ID hot-slot limit.
• Use existing Redis/client telemetry first. Do not add public configuration, a new metric family, or sparse-watermark protocol complexity until measurements prove the need.

Acceptance criteria

• Reproducible memory and operation results for representative distinct-ID rates and TTLs.
• Reproducible cluster results for uniform and hot-ID distributions.
• Documentation gives operators the capacity model and concrete signals to monitor.
• Any proposed new guardrail is tied to measured failure pressure and preserves invalidation correctness.
• Logical clock correctness remains owned by Document the synchronized Redis clock assumption for tracked invalidation #32.