diff --git a/docs/impersonate-circuit-breaker.md b/docs/impersonate-circuit-breaker.md new file mode 100644 index 00000000..85233d5b --- /dev/null +++ b/docs/impersonate-circuit-breaker.md @@ -0,0 +1,186 @@ +# Impersonate Circuit Breaker + +## Overview + +`POST /api/admin/users/:address/impersonate` wraps all downstream work — JWT +signing and audit log writes — in a **circuit breaker**. When the downstream +is repeatedly failing the breaker trips to the OPEN state and the endpoint +immediately returns **HTTP 503** instead of waiting for a slow or failing call +to time out. + +## Why a circuit breaker? + +Without one, a transient DB outage or JWT service failure causes every +impersonate call to hang until it eventually times out, holding server +resources and leaving the admin client waiting. The circuit breaker detects +the fault quickly and short-circuits all subsequent calls until the downstream +recovers, giving a fast, predictable failure signal. + +## State machine + +``` + ┌─────────────────────────────────────────┐ + │ │ + ▼ failureThreshold consecutive fails │ + CLOSED ───────────────────────────────► OPEN + ▲ │ + │ halfOpenAfterMs elapses + │ │ + │ ▼ + │ successThreshold successes HALF_OPEN + └────────────────────────────────────┘ + │ + probe fails │ + ┌──────┘ + ▼ + OPEN +``` + +| State | Behaviour | +|---|---| +| **CLOSED** | Normal operation. Failures are counted. | +| **OPEN** | Fast-fail: every call throws `CircuitOpenError` immediately. No downstream calls are made. | +| **HALF_OPEN** | One probe call is allowed through. Success → CLOSED. Failure → OPEN. | + +## Default configuration + +| Option | Default | Description | +|---|---|---| +| `failureThreshold` | `5` | Consecutive failures that trip CLOSED → OPEN | +| `successThreshold` | `1` | Consecutive probe successes needed to reset OPEN → CLOSED | +| `halfOpenAfterMs` | `30 000` | Milliseconds in OPEN before a probe is allowed (HALF_OPEN) | + +These defaults are applied when the router is instantiated without overrides. + +## API behaviour + +### Happy path (CLOSED state) + +``` +POST /api/admin/users/:address/impersonate +Authorization: Bearer + +200 OK +{ + "data": { + "token": "" + } +} +``` + +### Circuit OPEN + +``` +POST /api/admin/users/:address/impersonate +Authorization: Bearer + +503 Service Unavailable +{ + "error": { + "code": "service_unavailable", + "message": "Impersonate service is temporarily unavailable. Please retry later.", + "retryAfterMs": 30000, + "requestId": "" + } +} +``` + +- `retryAfterMs` tells the caller how long before the breaker will allow a + probe (i.e. the configured `halfOpenAfterMs` value). +- When the circuit is OPEN, **no downstream calls are made** — the JWT service + and audit service are never invoked. + +### Other responses + +| Status | `error.code` | Cause | +|---|---|---| +| `400` | `validation_error` | `:address` param is blank / whitespace-only | +| `403` | `forbidden` | Missing, invalid, or non-admin JWT | +| `429` | `rate_limit_exceeded` | Rate limit: 60 req/min per admin token | + +## Implementation + +The circuit breaker is implemented in `src/lib/circuitBreaker.ts` as a +**generic, per-name** state machine. The registry is keyed by name so multiple +routes can each have their own isolated breaker. + +```ts +import { getCircuitBreaker, CircuitOpenError } from "../../../lib/circuitBreaker"; + +const breaker = getCircuitBreaker("impersonate", { + failureThreshold: 5, + halfOpenAfterMs: 30_000, +}); + +const token = await breaker.execute(async () => { + await createAuditLog(...); + await db.insert(adminAuditLog).values(...); + return signAccessToken({ sub: targetAddress, role: "user" }); +}); +``` + +The `CircuitOpenError` is caught in the route handler and mapped to HTTP 503. +All other errors are forwarded to the global error handler. + +## Testing + +### Unit tests — `tests/circuitBreaker.test.ts` + +Tests every state transition, the public API (`execute`, `snapshot`, `state`), +`CircuitOpenError`, and both test helpers. + +### Integration tests — `tests/impersonateCircuitBreaker.test.ts` + +Tests the route end-to-end against every circuit state: + +| Scenario | Expected | +|---|---| +| CLOSED + succeeding downstream | 200 + token | +| CLOSED + failing downstream (below threshold) | 500 propagated; counter incremented | +| CLOSED + failing downstream (at threshold) | next call → 503 | +| OPEN | 503, downstream not called, `retryAfterMs` present | +| HALF_OPEN + success | 200, breaker → CLOSED | +| HALF_OPEN + failure | 500, breaker → OPEN | +| OPEN after `halfOpenAfterMs` elapses | probe allowed, responds 200 | + +Auth and validation guards run **before** circuit evaluation, so a request +without a valid admin JWT always returns 403 regardless of circuit state. + +## Runbook + +### How to tell if the circuit has tripped + +Look for the `circuit_breaker_opened` log line in the application logs: + +```json +{ + "level": "warn", + "circuitName": "impersonate", + "failures": 5, + "openedAt": 1722175200000, + "msg": "circuit_breaker_opened" +} +``` + +### How to tell when it recovers + +```json +{ + "level": "info", + "circuitName": "impersonate", + "state": "CLOSED", + "msg": "circuit_breaker_closed" +} +``` + +### Forcing a fresh breaker (process restart) + +The circuit state is **in-memory only** and resets when the process restarts. +A rolling restart is the quickest way to reset a stuck-open breaker if the +downstream has been fixed but the `halfOpenAfterMs` window hasn't elapsed yet. + +### Adjusting thresholds + +Pass custom options when constructing the router — see +`src/routes/admin/users/impersonate.ts` for the `circuitBreaker` option on +`AdminImpersonateRouterOptions`. diff --git a/src/lib/circuitBreaker.ts b/src/lib/circuitBreaker.ts new file mode 100644 index 00000000..4b1265c6 --- /dev/null +++ b/src/lib/circuitBreaker.ts @@ -0,0 +1,299 @@ +/** + * circuitBreaker.ts + * + * Generic per-name circuit breaker implementing the classic three-state machine: + * + * CLOSED → normal operation; failures are counted. + * OPEN → fast-fail; all calls throw CircuitOpenError immediately. + * HALF_OPEN → one probe call allowed; success → CLOSED, failure → OPEN. + * + * Usage: + * + * const breaker = getCircuitBreaker("impersonate", { + * failureThreshold: 5, + * successThreshold: 1, + * halfOpenAfterMs: 30_000, + * }); + * + * // Wrap any async operation: + * const token = await breaker.execute(() => signAccessToken(...)); + * + * When the breaker is OPEN, `execute` throws a `CircuitOpenError` which + * the route layer converts to HTTP 503. + * + * State is stored in-memory and resets on process restart, which is + * correct for a transient fast-fail guard. Use `resetForTests()` in + * unit tests to isolate state between test cases. + */ + +import { logger } from "../config/logger"; + +// --------------------------------------------------------------------------- +// Public types +// --------------------------------------------------------------------------- + +export type CircuitState = "CLOSED" | "OPEN" | "HALF_OPEN"; + +export interface CircuitBreakerOptions { + /** + * Number of consecutive failures needed to trip the breaker from CLOSED + * to OPEN. + * @default 5 + */ + failureThreshold?: number; + + /** + * Number of consecutive successes in HALF_OPEN needed to reset to CLOSED. + * @default 1 + */ + successThreshold?: number; + + /** + * Milliseconds to wait in OPEN state before moving to HALF_OPEN. + * @default 30_000 + */ + halfOpenAfterMs?: number; +} + +export interface CircuitBreakerSnapshot { + name: string; + state: CircuitState; + failures: number; + successes: number; + openedAt: number | null; + failureThreshold: number; + successThreshold: number; + halfOpenAfterMs: number; +} + +// --------------------------------------------------------------------------- +// Errors +// --------------------------------------------------------------------------- + +/** + * Thrown when `execute` is called on an open circuit breaker. + * Route handlers should map this to HTTP 503. + */ +export class CircuitOpenError extends Error { + readonly name = "CircuitOpenError"; + readonly circuitName: string; + readonly openedAt: number; + + constructor(circuitName: string, openedAt: number) { + super(`Circuit '${circuitName}' is OPEN — downstream call rejected`); + this.circuitName = circuitName; + this.openedAt = openedAt; + Object.setPrototypeOf(this, CircuitOpenError.prototype); + } +} + +// --------------------------------------------------------------------------- +// Internal state +// --------------------------------------------------------------------------- + +interface InternalState { + state: CircuitState; + failures: number; + successes: number; + openedAt: number | null; + opts: Required; +} + +/** Registry of all named circuit breakers in this process. */ +const registry = new Map(); + +function defaultOpts( + opts: CircuitBreakerOptions = {}, +): Required { + return { + failureThreshold: opts.failureThreshold ?? 5, + successThreshold: opts.successThreshold ?? 1, + halfOpenAfterMs: opts.halfOpenAfterMs ?? 30_000, + }; +} + +function createState(opts: Required): InternalState { + return { + state: "CLOSED", + failures: 0, + successes: 0, + openedAt: null, + opts, + }; +} + +// --------------------------------------------------------------------------- +// Circuit breaker handle +// --------------------------------------------------------------------------- + +export interface CircuitBreaker { + /** Current state of this circuit breaker. */ + readonly state: CircuitState; + + /** Snapshot of full internal state (useful for observability). */ + snapshot(): CircuitBreakerSnapshot; + + /** + * Executes `fn` through the circuit breaker. + * + * - CLOSED: calls `fn`; on failure increments failure counter. + * - OPEN: throws `CircuitOpenError` immediately (fast-fail). + * - HALF_OPEN: lets one probe through; success resets to CLOSED, + * failure trips back to OPEN. + */ + execute(fn: () => Promise): Promise; +} + +// --------------------------------------------------------------------------- +// Core state-machine transitions +// --------------------------------------------------------------------------- + +function resolveEffectiveState(s: InternalState, now: number): CircuitState { + if ( + s.state === "OPEN" && + s.openedAt !== null && + now - s.openedAt >= s.opts.halfOpenAfterMs + ) { + s.state = "HALF_OPEN"; + s.failures = 0; + s.successes = 0; + logger.info({ circuit: s }, "circuit_breaker_half_open"); + } + return s.state; +} + +function onSuccess(name: string, s: InternalState): void { + if (s.state === "HALF_OPEN") { + s.successes += 1; + if (s.successes >= s.opts.successThreshold) { + s.state = "CLOSED"; + s.failures = 0; + s.successes = 0; + s.openedAt = null; + logger.info({ circuitName: name, state: s.state }, "circuit_breaker_closed"); + } + } else { + // CLOSED — any success resets the failure counter + s.failures = 0; + } +} + +function onFailure(name: string, s: InternalState, now: number): void { + s.failures += 1; + + if (s.state === "HALF_OPEN" || s.failures >= s.opts.failureThreshold) { + s.state = "OPEN"; + s.openedAt = now; + s.successes = 0; + logger.warn( + { circuitName: name, failures: s.failures, openedAt: s.openedAt }, + "circuit_breaker_opened", + ); + } +} + +// --------------------------------------------------------------------------- +// Public API +// --------------------------------------------------------------------------- + +/** + * Returns (and lazily creates) a named circuit breaker. + * + * If the breaker already exists, the existing instance is returned and + * `opts` is **ignored** — options are only applied at creation time. + * This ensures a single stable configuration across the application. + */ +export function getCircuitBreaker( + name: string, + opts: CircuitBreakerOptions = {}, +): CircuitBreaker { + if (!registry.has(name)) { + registry.set(name, createState(defaultOpts(opts))); + } + + const s = registry.get(name)!; + + return { + get state(): CircuitState { + return resolveEffectiveState(s, Date.now()); + }, + + snapshot(): CircuitBreakerSnapshot { + return { + name, + state: s.state, + failures: s.failures, + successes: s.successes, + openedAt: s.openedAt, + failureThreshold: s.opts.failureThreshold, + successThreshold: s.opts.successThreshold, + halfOpenAfterMs: s.opts.halfOpenAfterMs, + }; + }, + + async execute(fn: () => Promise): Promise { + const now = Date.now(); + const effective = resolveEffectiveState(s, now); + + if (effective === "OPEN") { + throw new CircuitOpenError(name, s.openedAt!); + } + + try { + const result = await fn(); + onSuccess(name, s); + return result; + } catch (err) { + if (err instanceof CircuitOpenError) { + // Propagate without counting — shouldn't happen, but be safe. + throw err; + } + onFailure(name, s, Date.now()); + throw err; + } + }, + }; +} + +/** + * Test-only: resets the registry so every test starts with a clean slate. + * Never call this in production code. + */ +export function resetCircuitBreakersForTests(): void { + registry.clear(); +} + +/** + * Test-only: forcibly sets the state of a named breaker so tests can drive + * specific scenarios without having to trigger the threshold conditions. + * + * If the breaker doesn't exist yet it is created. If it already exists its + * internal state object is **mutated in-place** so that any existing handles + * (returned from earlier `getCircuitBreaker` calls) immediately see the new + * state through their closure reference. + */ +export function forceCircuitStateForTests( + name: string, + state: CircuitState, + opts: CircuitBreakerOptions = {}, +): void { + if (!registry.has(name)) { + // Create the entry fresh and set the desired state. + const resolvedOpts = defaultOpts(opts); + const s = createState(resolvedOpts); + applyForcedState(s, state); + registry.set(name, s); + } else { + // Mutate in-place so handles that already hold a reference to `s` see + // the new state without needing to re-call getCircuitBreaker. + const s = registry.get(name)!; + applyForcedState(s, state); + } +} + +function applyForcedState(s: InternalState, state: CircuitState): void { + s.state = state; + s.failures = state === "OPEN" ? s.opts.failureThreshold : 0; + s.successes = 0; + s.openedAt = state === "OPEN" ? Date.now() : null; +} diff --git a/src/routes/admin/users/impersonate.ts b/src/routes/admin/users/impersonate.ts index c48329c4..b6e601af 100644 --- a/src/routes/admin/users/impersonate.ts +++ b/src/routes/admin/users/impersonate.ts @@ -1,3 +1,35 @@ +/** + * admin/users/impersonate.ts + * + * POST /api/admin/users/:address/impersonate + * + * Allows an admin to obtain a short-lived access token that impersonates + * the target Stellar wallet address. + * + * Security: + * - Requires a valid admin JWT (role: "admin") via requireAdmin middleware. + * - Rate-limited to 60 requests per minute per admin token. + * - Every call is double-audited: once in the global audit_logs table and + * once in the admin_audit_log table keyed by target address. + * + * Circuit breaker: + * - The downstream work (token signing + audit writes) is wrapped in a + * named circuit breaker ("impersonate"). + * - CLOSED (normal): calls execute and failures are counted. + * - OPEN (tripped): the breaker fast-fails immediately with HTTP 503 so + * the admin UI / caller gets a clear signal that the downstream is + * unhealthy rather than hanging for the full request timeout. + * - HALF_OPEN (recovery probe): one trial call is allowed; a success + * resets the breaker to CLOSED; a failure keeps it OPEN. + * + * HTTP status codes: + * - 200 OK impersonation succeeded; body: { data: { token } } + * - 400 Bad Request address param is blank / whitespace-only + * - 403 Forbidden missing/invalid/non-admin JWT + * - 429 Too Many Requests rate limit exceeded + * - 503 Service Unavailable circuit is OPEN; body: { error: { code: "service_unavailable", retryAfterMs } } + */ + import { Router } from "express"; import { rateLimit } from "express-rate-limit"; import { z } from "zod"; @@ -8,12 +40,22 @@ import { getRequestId } from "../../../lib/requestContext"; import { logger } from "../../../config/logger"; import { db } from "../../../db/client"; import { adminAuditLog } from "../../../db/schema"; +import { + getCircuitBreaker, + CircuitOpenError, + type CircuitBreakerOptions, +} from "../../../lib/circuitBreaker"; export interface AdminImpersonateRouterOptions { /** Requests per minute per admin token. Default: 60 */ rateLimitPerMinute?: number; + /** Circuit breaker configuration for downstream calls. */ + circuitBreaker?: CircuitBreakerOptions; } +/** Name used to identify the impersonate circuit breaker in logs and snapshots. */ +export const IMPERSONATE_CIRCUIT_NAME = "impersonate"; + const paramsSchema = z.object({ address: z.string().trim().min(1), }); @@ -24,6 +66,9 @@ export function createAdminImpersonateRouter( const router = Router(); const limit = opts.rateLimitPerMinute ?? 60; + // Lazily instantiated so tests can inject custom thresholds via opts. + const breaker = getCircuitBreaker(IMPERSONATE_CIRCUIT_NAME, opts.circuitBreaker); + router.use( rateLimit({ windowMs: 60_000, @@ -39,9 +84,10 @@ export function createAdminImpersonateRouter( router.use(requireAdmin); router.post("/:address/impersonate", async (req, res, next) => { + const reqId = getRequestId() ?? (req as { id?: string }).id ?? "unknown"; + try { const parsed = paramsSchema.safeParse(req.params); - const reqId = getRequestId() ?? (req as { id?: string }).id ?? "unknown"; if (!parsed.success) { res.status(400).json({ @@ -57,40 +103,61 @@ export function createAdminImpersonateRouter( const targetAddress = parsed.data.address; const adminAddress = req.adminAddress!; - // 1. Audit log in global auditLogs - await createAuditLog({ - action: "admin.impersonate", - walletAddress: adminAddress, - ip: req.ip ?? "unknown", - correlationId: reqId, - }); - - // 2. Audit log in adminAuditLog specific to target address - await db.insert(adminAuditLog).values({ - adminAddress, - action: "impersonate", - targetAddress, - }); + // Wrap all downstream I/O in the circuit breaker. + // If the breaker is OPEN this throws CircuitOpenError immediately + // (before any network or DB call is attempted). + const token = await breaker.execute(async () => { + // 1. Audit log in global audit_logs + await createAuditLog({ + action: "admin.impersonate", + walletAddress: adminAddress, + ip: req.ip ?? "unknown", + correlationId: reqId, + }); - // 3. Structured logging with correlation IDs - logger.info( - { + // 2. Audit log in admin_audit_log keyed by target address + await db.insert(adminAuditLog).values({ adminAddress, + action: "impersonate", targetAddress, - correlationId: reqId, - }, - "Admin impersonated user", - ); + }); - // 4. Generate token - const token = signAccessToken({ sub: targetAddress, role: "user" }); + // 3. Structured logging with correlation IDs + logger.info( + { + adminAddress, + targetAddress, + correlationId: reqId, + }, + "Admin impersonated user", + ); - res.status(200).json({ - data: { - token, - }, + // 4. Generate the impersonation token + return signAccessToken({ sub: targetAddress, role: "user" }); }); + + res.status(200).json({ data: { token } }); } catch (err) { + if (err instanceof CircuitOpenError) { + const retryAfterMs = breaker.snapshot().halfOpenAfterMs; + logger.warn( + { + circuitName: IMPERSONATE_CIRCUIT_NAME, + openedAt: err.openedAt, + reqId, + }, + "impersonate_circuit_open", + ); + res.status(503).json({ + error: { + code: "service_unavailable", + message: "Impersonate service is temporarily unavailable. Please retry later.", + retryAfterMs, + requestId: reqId, + }, + }); + return; + } next(err); } }); diff --git a/tests/adminImpersonate.test.ts b/tests/adminImpersonate.test.ts index 696d78b7..047d5890 100644 --- a/tests/adminImpersonate.test.ts +++ b/tests/adminImpersonate.test.ts @@ -14,7 +14,7 @@ import { createAuditLog } from "../src/services/auditService"; const mockSignAccessToken = signAccessToken as jest.MockedFunction; const mockCreateAuditLog = createAuditLog as jest.MockedFunction; -const SECRET = process.env.JWT_SECRET || "test-jwt-secret-at-least-32-bytes-long-000000"; +const SECRET = process.env.JWT_SECRET ?? "test-jwt-secret-that-is-at-least-32-chars!"; const ISSUER = process.env.JWT_ISSUER || "predictify"; const AUDIENCE = process.env.JWT_AUDIENCE || "predictify-app"; diff --git a/tests/circuitBreaker.test.ts b/tests/circuitBreaker.test.ts new file mode 100644 index 00000000..f5f39996 --- /dev/null +++ b/tests/circuitBreaker.test.ts @@ -0,0 +1,392 @@ +/** + * circuitBreaker.test.ts + * + * Unit tests for src/lib/circuitBreaker.ts + * + * Tests cover: + * - Default configuration + * - CLOSED state: successful calls and failure counting + * - OPEN state: fast-fail, CircuitOpenError, retryAfterMs + * - HALF_OPEN state: probe success → CLOSED, probe failure → OPEN + * - Automatic OPEN → HALF_OPEN transition after halfOpenAfterMs + * - successThreshold > 1 (requires multiple consecutive successes) + * - CircuitOpenError is not counted as a failure + * - resetCircuitBreakersForTests / forceCircuitStateForTests helpers + * - snapshot() accuracy + * - Registry isolation: two different names are independent + */ + +import { + getCircuitBreaker, + resetCircuitBreakersForTests, + forceCircuitStateForTests, + CircuitOpenError, + type CircuitBreakerSnapshot, +} from "../src/lib/circuitBreaker"; + +// --------------------------------------------------------------------------- +// Helpers +// --------------------------------------------------------------------------- + +const succeed = (value: T) => () => Promise.resolve(value); +const fail = (msg = "downstream failure") => () => Promise.reject(new Error(msg)); + +// --------------------------------------------------------------------------- +// Setup +// --------------------------------------------------------------------------- + +beforeEach(() => { + resetCircuitBreakersForTests(); +}); + +// --------------------------------------------------------------------------- +// CircuitOpenError +// --------------------------------------------------------------------------- + +describe("CircuitOpenError", () => { + it("has the correct name", () => { + const err = new CircuitOpenError("my-circuit", Date.now()); + expect(err.name).toBe("CircuitOpenError"); + }); + + it("carries the circuit name", () => { + const err = new CircuitOpenError("svc-a", 12345); + expect(err.circuitName).toBe("svc-a"); + }); + + it("carries openedAt timestamp", () => { + const now = Date.now(); + const err = new CircuitOpenError("svc-b", now); + expect(err.openedAt).toBe(now); + }); + + it("is instanceof Error", () => { + const err = new CircuitOpenError("x", 0); + expect(err).toBeInstanceOf(Error); + }); + + it("is instanceof CircuitOpenError", () => { + const err = new CircuitOpenError("x", 0); + expect(err).toBeInstanceOf(CircuitOpenError); + }); +}); + +// --------------------------------------------------------------------------- +// getCircuitBreaker — initial state +// --------------------------------------------------------------------------- + +describe("getCircuitBreaker — initial state", () => { + it("starts in CLOSED state", () => { + const cb = getCircuitBreaker("cb-test"); + expect(cb.state).toBe("CLOSED"); + }); + + it("snapshot reflects defaults", () => { + const cb = getCircuitBreaker("cb-defaults"); + const snap: CircuitBreakerSnapshot = cb.snapshot(); + expect(snap.state).toBe("CLOSED"); + expect(snap.failures).toBe(0); + expect(snap.successes).toBe(0); + expect(snap.openedAt).toBeNull(); + expect(snap.failureThreshold).toBe(5); + expect(snap.successThreshold).toBe(1); + expect(snap.halfOpenAfterMs).toBe(30_000); + }); + + it("snapshot reflects custom options", () => { + const cb = getCircuitBreaker("cb-custom", { + failureThreshold: 2, + successThreshold: 3, + halfOpenAfterMs: 5_000, + }); + const snap = cb.snapshot(); + expect(snap.failureThreshold).toBe(2); + expect(snap.successThreshold).toBe(3); + expect(snap.halfOpenAfterMs).toBe(5_000); + }); + + it("two handles for the same name share the same underlying state", async () => { + // Both handles wrap the same registry entry. Trip the breaker via one + // handle and confirm the other observes the same state. + const a = getCircuitBreaker("same-name", { failureThreshold: 1 }); + const b = getCircuitBreaker("same-name"); + + // Execute a failing call through `a` to trip the breaker. + await a.execute(fail()).catch(() => {}); + + expect(a.state).toBe("OPEN"); + expect(b.state).toBe("OPEN"); + }); + + it("different names have independent state", () => { + getCircuitBreaker("alpha"); + getCircuitBreaker("beta"); + forceCircuitStateForTests("alpha", "OPEN"); + expect(getCircuitBreaker("beta").state).toBe("CLOSED"); + }); +}); + +// --------------------------------------------------------------------------- +// CLOSED state +// --------------------------------------------------------------------------- + +describe("CLOSED state", () => { + it("execute resolves the return value", async () => { + const cb = getCircuitBreaker("closed-resolve"); + const result = await cb.execute(succeed(42)); + expect(result).toBe(42); + }); + + it("execute propagates rejection without tripping breaker below threshold", async () => { + const cb = getCircuitBreaker("closed-below-threshold", { failureThreshold: 5 }); + await expect(cb.execute(fail())).rejects.toThrow("downstream failure"); + expect(cb.state).toBe("CLOSED"); + expect(cb.snapshot().failures).toBe(1); + }); + + it("each failure increments the counter", async () => { + const cb = getCircuitBreaker("closed-counter", { failureThreshold: 10 }); + for (let i = 1; i <= 4; i++) { + await cb.execute(fail()).catch(() => {}); + expect(cb.snapshot().failures).toBe(i); + } + }); + + it("success resets the failure counter", async () => { + const cb = getCircuitBreaker("closed-reset", { failureThreshold: 5 }); + await cb.execute(fail()).catch(() => {}); + await cb.execute(fail()).catch(() => {}); + expect(cb.snapshot().failures).toBe(2); + + await cb.execute(succeed("ok")); + expect(cb.snapshot().failures).toBe(0); + }); + + it("trips to OPEN after failureThreshold failures", async () => { + const cb = getCircuitBreaker("closed-trip", { failureThreshold: 3 }); + await cb.execute(fail()).catch(() => {}); + await cb.execute(fail()).catch(() => {}); + expect(cb.state).toBe("CLOSED"); + await cb.execute(fail()).catch(() => {}); + expect(cb.state).toBe("OPEN"); + }); + + it("openedAt is null while CLOSED", () => { + const cb = getCircuitBreaker("closed-openat"); + expect(cb.snapshot().openedAt).toBeNull(); + }); +}); + +// --------------------------------------------------------------------------- +// OPEN state +// --------------------------------------------------------------------------- + +describe("OPEN state", () => { + it("execute throws CircuitOpenError immediately", async () => { + forceCircuitStateForTests("open-fast", "OPEN"); + const cb = getCircuitBreaker("open-fast"); + await expect(cb.execute(succeed("x"))).rejects.toBeInstanceOf(CircuitOpenError); + }); + + it("the wrapped function is never called when OPEN", async () => { + forceCircuitStateForTests("open-noop", "OPEN"); + const cb = getCircuitBreaker("open-noop"); + const fn = jest.fn().mockResolvedValue("result"); + await cb.execute(fn).catch(() => {}); + expect(fn).not.toHaveBeenCalled(); + }); + + it("CircuitOpenError carries the circuit name", async () => { + forceCircuitStateForTests("open-name", "OPEN"); + const cb = getCircuitBreaker("open-name"); + const err = await cb.execute(succeed("x")).catch((e) => e); + expect(err).toBeInstanceOf(CircuitOpenError); + expect((err as CircuitOpenError).circuitName).toBe("open-name"); + }); + + it("openedAt is set in the snapshot", () => { + const before = Date.now(); + forceCircuitStateForTests("open-ts", "OPEN"); + const snap = getCircuitBreaker("open-ts").snapshot(); + expect(snap.openedAt).not.toBeNull(); + expect(snap.openedAt!).toBeGreaterThanOrEqual(before); + }); + + it("naturally tripped: openedAt is recorded", async () => { + const before = Date.now(); + const cb = getCircuitBreaker("open-natural", { failureThreshold: 1 }); + await cb.execute(fail()).catch(() => {}); + const snap = cb.snapshot(); + expect(snap.state).toBe("OPEN"); + expect(snap.openedAt).not.toBeNull(); + expect(snap.openedAt!).toBeGreaterThanOrEqual(before); + }); +}); + +// --------------------------------------------------------------------------- +// OPEN → HALF_OPEN transition +// --------------------------------------------------------------------------- + +describe("OPEN → HALF_OPEN transition", () => { + it("transitions to HALF_OPEN after halfOpenAfterMs elapses", async () => { + jest.useFakeTimers(); + try { + forceCircuitStateForTests("half-time", "OPEN", { halfOpenAfterMs: 500 }); + const cb = getCircuitBreaker("half-time", { halfOpenAfterMs: 500 }); + + expect(cb.state).toBe("OPEN"); + + jest.advanceTimersByTime(600); + + // Accessing state triggers the time-based transition + expect(cb.state).toBe("HALF_OPEN"); + } finally { + jest.useRealTimers(); + } + }); + + it("stays OPEN before halfOpenAfterMs elapses", () => { + jest.useFakeTimers(); + try { + forceCircuitStateForTests("half-wait", "OPEN", { halfOpenAfterMs: 1_000 }); + const cb = getCircuitBreaker("half-wait", { halfOpenAfterMs: 1_000 }); + + jest.advanceTimersByTime(500); + expect(cb.state).toBe("OPEN"); + } finally { + jest.useRealTimers(); + } + }); +}); + +// --------------------------------------------------------------------------- +// HALF_OPEN state +// --------------------------------------------------------------------------- + +describe("HALF_OPEN state", () => { + it("probe success resets to CLOSED (successThreshold = 1)", async () => { + forceCircuitStateForTests("half-success", "HALF_OPEN"); + const cb = getCircuitBreaker("half-success"); + await cb.execute(succeed("ok")); + expect(cb.state).toBe("CLOSED"); + }); + + it("probe success resets failure and success counters", async () => { + forceCircuitStateForTests("half-counters", "HALF_OPEN"); + const cb = getCircuitBreaker("half-counters"); + await cb.execute(succeed("ok")); + const snap = cb.snapshot(); + expect(snap.failures).toBe(0); + expect(snap.successes).toBe(0); + expect(snap.openedAt).toBeNull(); + }); + + it("probe failure trips back to OPEN", async () => { + forceCircuitStateForTests("half-fail", "HALF_OPEN"); + const cb = getCircuitBreaker("half-fail"); + await cb.execute(fail()).catch(() => {}); + expect(cb.state).toBe("OPEN"); + }); + + it("probe failure updates openedAt", async () => { + const before = Date.now(); + forceCircuitStateForTests("half-openat", "HALF_OPEN"); + const cb = getCircuitBreaker("half-openat"); + await cb.execute(fail()).catch(() => {}); + const snap = cb.snapshot(); + expect(snap.openedAt).not.toBeNull(); + expect(snap.openedAt!).toBeGreaterThanOrEqual(before); + }); + + it("requires successThreshold successes before closing (threshold = 3)", async () => { + forceCircuitStateForTests("half-thresh", "HALF_OPEN", { successThreshold: 3 }); + const cb = getCircuitBreaker("half-thresh", { successThreshold: 3 }); + + await cb.execute(succeed(1)); + expect(cb.state).toBe("HALF_OPEN"); + await cb.execute(succeed(2)); + expect(cb.state).toBe("HALF_OPEN"); + await cb.execute(succeed(3)); + expect(cb.state).toBe("CLOSED"); + }); +}); + +// --------------------------------------------------------------------------- +// CircuitOpenError not counted as failure +// --------------------------------------------------------------------------- + +describe("CircuitOpenError passthrough", () => { + it("does not increment failure counter when OPEN throws CircuitOpenError", async () => { + forceCircuitStateForTests("open-no-count", "OPEN"); + const cb = getCircuitBreaker("open-no-count"); + const snap1 = cb.snapshot(); + + await cb.execute(succeed("x")).catch(() => {}); + + const snap2 = cb.snapshot(); + expect(snap2.failures).toBe(snap1.failures); + }); +}); + +// --------------------------------------------------------------------------- +// resetCircuitBreakersForTests +// --------------------------------------------------------------------------- + +describe("resetCircuitBreakersForTests", () => { + it("clears all registered breakers", () => { + forceCircuitStateForTests("reset-a", "OPEN"); + forceCircuitStateForTests("reset-b", "OPEN"); + resetCircuitBreakersForTests(); + + // After reset, a fresh getCircuitBreaker should start in CLOSED + expect(getCircuitBreaker("reset-a").state).toBe("CLOSED"); + expect(getCircuitBreaker("reset-b").state).toBe("CLOSED"); + }); +}); + +// --------------------------------------------------------------------------- +// forceCircuitStateForTests +// --------------------------------------------------------------------------- + +describe("forceCircuitStateForTests", () => { + it("can force CLOSED", () => { + forceCircuitStateForTests("force-closed", "CLOSED"); + expect(getCircuitBreaker("force-closed").state).toBe("CLOSED"); + }); + + it("can force OPEN", () => { + forceCircuitStateForTests("force-open", "OPEN"); + expect(getCircuitBreaker("force-open").state).toBe("OPEN"); + }); + + it("can force HALF_OPEN", () => { + forceCircuitStateForTests("force-half", "HALF_OPEN"); + expect(getCircuitBreaker("force-half").state).toBe("HALF_OPEN"); + }); + + it("sets openedAt when forcing OPEN", () => { + const before = Date.now(); + forceCircuitStateForTests("force-openat", "OPEN"); + const snap = getCircuitBreaker("force-openat").snapshot(); + expect(snap.openedAt).not.toBeNull(); + expect(snap.openedAt!).toBeGreaterThanOrEqual(before); + }); + + it("clears openedAt when forcing CLOSED", () => { + forceCircuitStateForTests("force-clear-openat", "OPEN"); + forceCircuitStateForTests("force-clear-openat", "CLOSED"); + const snap = getCircuitBreaker("force-clear-openat").snapshot(); + expect(snap.openedAt).toBeNull(); + }); +}); + +// --------------------------------------------------------------------------- +// snapshot() — name field +// --------------------------------------------------------------------------- + +describe("snapshot name field", () => { + it("includes the circuit name", () => { + const cb = getCircuitBreaker("named-circuit"); + expect(cb.snapshot().name).toBe("named-circuit"); + }); +}); diff --git a/tests/impersonateCircuitBreaker.test.ts b/tests/impersonateCircuitBreaker.test.ts new file mode 100644 index 00000000..e452ab91 --- /dev/null +++ b/tests/impersonateCircuitBreaker.test.ts @@ -0,0 +1,315 @@ +/** + * impersonateCircuitBreaker.test.ts + * + * Focused tests for the circuit breaker behaviour in + * POST /api/admin/users/:address/impersonate. + * + * Covers: + * - CLOSED (normal) state: request succeeds, returns 200 + * - OPEN state: request fast-fails with 503, includes retryAfterMs + * - HALF_OPEN state: successful probe resets breaker back to CLOSED + * - HALF_OPEN state: failing probe trips breaker back to OPEN + * - Breaker trips after failureThreshold consecutive errors + * - Existing tests still pass (auth, validation guards) + */ + +jest.mock("../src/services/jwtService"); +jest.mock("../src/services/auditService"); +jest.mock("../src/db/client", () => ({ + db: { + insert: jest.fn().mockReturnValue({ values: jest.fn().mockResolvedValue({}) }), + }, +})); + +import request from "supertest"; +import express from "express"; +import jwt from "jsonwebtoken"; +import { createAdminImpersonateRouter, IMPERSONATE_CIRCUIT_NAME } from "../src/routes/admin/users/impersonate"; +import { + resetCircuitBreakersForTests, + forceCircuitStateForTests, + getCircuitBreaker, +} from "../src/lib/circuitBreaker"; +import { errorHandler } from "../src/middleware/errorHandler"; +import { signAccessToken } from "../src/services/jwtService"; +import { createAuditLog } from "../src/services/auditService"; + +const mockSignAccessToken = signAccessToken as jest.MockedFunction; +const mockCreateAuditLog = createAuditLog as jest.MockedFunction; + +// --------------------------------------------------------------------------- +// JWT helpers +// --------------------------------------------------------------------------- + +const SECRET = process.env.JWT_SECRET ?? "test-jwt-secret-that-is-at-least-32-chars!"; +const ISSUER = process.env.JWT_ISSUER || "predictify"; +const AUDIENCE = process.env.JWT_AUDIENCE || "predictify-app"; + +const ADMIN_ADDRESS = "GADMIN7777777777777777777777777777777777777777777777777777"; +const USER_ADDRESS = "GUSER88888888888888888888888888888888888888888888888888888"; + +function signJwt(payload: object): string { + return jwt.sign(payload, SECRET, { issuer: ISSUER, audience: AUDIENCE, expiresIn: "1h" }); +} + +const adminJwt = signJwt({ sub: ADMIN_ADDRESS, role: "admin" }); + +// --------------------------------------------------------------------------- +// App factory +// --------------------------------------------------------------------------- + +/** + * Creates a fresh Express app with a fresh circuit breaker per test. + * circuitOpts are passed directly to the router so we can use low thresholds + * in tests without polluting the global registry between suites. + */ +function makeApp(circuitOpts = {}) { + // Reset shared state before each app build so each `makeApp` gets + // a deterministic starting breaker state. + resetCircuitBreakersForTests(); + const app = express(); + app.use(express.json()); + app.use( + "/api/admin/users", + createAdminImpersonateRouter({ rateLimitPerMinute: 100, circuitBreaker: circuitOpts }), + ); + app.use(errorHandler); + return app; +} + +function authReq(app: express.Express) { + return request(app) + .post(`/api/admin/users/${USER_ADDRESS}/impersonate`) + .set("Authorization", `Bearer ${adminJwt}`) + .send({}); +} + +// --------------------------------------------------------------------------- +// Setup +// --------------------------------------------------------------------------- + +beforeEach(() => { + jest.clearAllMocks(); + resetCircuitBreakersForTests(); + mockSignAccessToken.mockReturnValue("mocked-token-xyz"); + mockCreateAuditLog.mockResolvedValue("corr-id-123"); +}); + +// --------------------------------------------------------------------------- +// CLOSED state (normal operation) +// --------------------------------------------------------------------------- + +describe("CLOSED state — normal operation", () => { + it("returns 200 with a token when downstream calls succeed", async () => { + const res = await authReq(makeApp()); + expect(res.status).toBe(200); + expect(res.body.data.token).toBe("mocked-token-xyz"); + }); + + it("calls signAccessToken with the target address", async () => { + await authReq(makeApp()); + expect(mockSignAccessToken).toHaveBeenCalledWith({ sub: USER_ADDRESS, role: "user" }); + }); + + it("calls createAuditLog with admin.impersonate action", async () => { + await authReq(makeApp()); + expect(mockCreateAuditLog).toHaveBeenCalledWith( + expect.objectContaining({ + action: "admin.impersonate", + walletAddress: ADMIN_ADDRESS, + }), + ); + }); + + it("failure counter resets after a success", async () => { + const app = makeApp({ failureThreshold: 3 }); + + // Simulate one failure then a success + mockSignAccessToken + .mockImplementationOnce(() => { throw new Error("transient"); }) + .mockReturnValue("ok-token"); + + const firstRes = await authReq(app); + expect(firstRes.status).toBe(500); // error propagated + + const secondRes = await authReq(app); + expect(secondRes.status).toBe(200); + + // Breaker should still be CLOSED — single failure didn't trip it + const snapshot = getCircuitBreaker(IMPERSONATE_CIRCUIT_NAME).snapshot(); + expect(snapshot.state).toBe("CLOSED"); + expect(snapshot.failures).toBe(0); // reset by the success + }); +}); + +// --------------------------------------------------------------------------- +// OPEN state — fast-fail +// --------------------------------------------------------------------------- + +describe("OPEN state — fast-fail 503", () => { + it("returns 503 when breaker is forced OPEN", async () => { + const app = makeApp(); + // Force the breaker open after the app has been created + forceCircuitStateForTests(IMPERSONATE_CIRCUIT_NAME, "OPEN", { halfOpenAfterMs: 30_000 }); + + const res = await authReq(app); + expect(res.status).toBe(503); + }); + + it("503 response includes service_unavailable code", async () => { + const app = makeApp(); + forceCircuitStateForTests(IMPERSONATE_CIRCUIT_NAME, "OPEN", { halfOpenAfterMs: 30_000 }); + + const res = await authReq(app); + expect(res.body.error.code).toBe("service_unavailable"); + }); + + it("503 response includes retryAfterMs", async () => { + const app = makeApp({ halfOpenAfterMs: 15_000 }); + forceCircuitStateForTests(IMPERSONATE_CIRCUIT_NAME, "OPEN", { halfOpenAfterMs: 15_000 }); + + const res = await authReq(app); + expect(typeof res.body.error.retryAfterMs).toBe("number"); + expect(res.body.error.retryAfterMs).toBeGreaterThan(0); + }); + + it("does NOT call signAccessToken when circuit is OPEN", async () => { + const app = makeApp(); + forceCircuitStateForTests(IMPERSONATE_CIRCUIT_NAME, "OPEN"); + + await authReq(app); + expect(mockSignAccessToken).not.toHaveBeenCalled(); + }); + + it("does NOT call createAuditLog when circuit is OPEN", async () => { + const app = makeApp(); + forceCircuitStateForTests(IMPERSONATE_CIRCUIT_NAME, "OPEN"); + + await authReq(app); + expect(mockCreateAuditLog).not.toHaveBeenCalled(); + }); + + it("trips to OPEN after failureThreshold consecutive failures", async () => { + const app = makeApp({ failureThreshold: 3 }); + mockSignAccessToken.mockImplementation(() => { + throw new Error("downstream error"); + }); + + // Three failures should trip the breaker + const r1 = await authReq(app); + const r2 = await authReq(app); + const r3 = await authReq(app); + + expect(r1.status).toBe(500); + expect(r2.status).toBe(500); + expect(r3.status).toBe(500); + + const snapshot = getCircuitBreaker(IMPERSONATE_CIRCUIT_NAME).snapshot(); + expect(snapshot.state).toBe("OPEN"); + + // Fourth call should be 503 (fast-fail) + mockSignAccessToken.mockReturnValue("token"); + const r4 = await authReq(app); + expect(r4.status).toBe(503); + expect(mockSignAccessToken).toHaveBeenCalledTimes(3); // never called on 4th + }); +}); + +// --------------------------------------------------------------------------- +// HALF_OPEN state — probe behaviour +// --------------------------------------------------------------------------- + +describe("HALF_OPEN state — probe behaviour", () => { + it("lets a probe through when in HALF_OPEN and resets to CLOSED on success", async () => { + const app = makeApp({ halfOpenAfterMs: 0 }); // instant transition for tests + forceCircuitStateForTests(IMPERSONATE_CIRCUIT_NAME, "HALF_OPEN"); + + // The probe call should succeed → breaker returns to CLOSED + const res = await authReq(app); + expect(res.status).toBe(200); + + const snapshot = getCircuitBreaker(IMPERSONATE_CIRCUIT_NAME).snapshot(); + expect(snapshot.state).toBe("CLOSED"); + }); + + it("trips back to OPEN when the HALF_OPEN probe fails", async () => { + const app = makeApp({ halfOpenAfterMs: 0 }); + forceCircuitStateForTests(IMPERSONATE_CIRCUIT_NAME, "HALF_OPEN"); + + mockSignAccessToken.mockImplementationOnce(() => { + throw new Error("probe failed"); + }); + + const res = await authReq(app); + expect(res.status).toBe(500); // probe failure propagated + + const snapshot = getCircuitBreaker(IMPERSONATE_CIRCUIT_NAME).snapshot(); + expect(snapshot.state).toBe("OPEN"); + }); + + it("transitions from OPEN to HALF_OPEN after halfOpenAfterMs elapses", async () => { + jest.useFakeTimers(); + try { + const app = makeApp({ halfOpenAfterMs: 1_000 }); + forceCircuitStateForTests(IMPERSONATE_CIRCUIT_NAME, "OPEN", { halfOpenAfterMs: 1_000 }); + + // Still OPEN before window expires + let res = await authReq(app); + expect(res.status).toBe(503); + + // Advance time past the half-open window + jest.advanceTimersByTime(1_100); + + // Now a probe is allowed + mockSignAccessToken.mockReturnValue("new-token"); + res = await authReq(app); + expect(res.status).toBe(200); + + const snapshot = getCircuitBreaker(IMPERSONATE_CIRCUIT_NAME).snapshot(); + expect(snapshot.state).toBe("CLOSED"); + } finally { + jest.useRealTimers(); + } + }); +}); + +// --------------------------------------------------------------------------- +// Auth / validation guards still work with breaker in place +// --------------------------------------------------------------------------- + +describe("Auth and validation guards (breaker transparent)", () => { + it("returns 403 with no Authorization header (breaker CLOSED)", async () => { + const res = await request(makeApp()) + .post(`/api/admin/users/${USER_ADDRESS}/impersonate`) + .send({}); + expect(res.status).toBe(403); + }); + + it("returns 403 with a non-admin JWT (breaker CLOSED)", async () => { + const userJwt = signJwt({ sub: USER_ADDRESS, role: "user" }); + const res = await request(makeApp()) + .post(`/api/admin/users/${USER_ADDRESS}/impersonate`) + .set("Authorization", `Bearer ${userJwt}`) + .send({}); + expect(res.status).toBe(403); + }); + + it("returns 400 for blank address (breaker CLOSED)", async () => { + const res = await request(makeApp()) + .post("/api/admin/users/ /impersonate") + .set("Authorization", `Bearer ${adminJwt}`) + .send({}); + expect(res.status).toBe(400); + expect(res.body.error.code).toBe("validation_error"); + }); + + it("returns 403 when breaker is OPEN — auth guard fires before circuit", async () => { + forceCircuitStateForTests(IMPERSONATE_CIRCUIT_NAME, "OPEN"); + + // No auth → should 403 before circuit even checked + const res = await request(makeApp()) + .post(`/api/admin/users/${USER_ADDRESS}/impersonate`) + .send({}); + expect(res.status).toBe(403); + }); +});