feat(metrics): expose DB pool saturation as autoscaling signal #524

[Ajibola6921]

Overview

This PR adds DB pool saturation as a horizontal autoscaling signal so the API workload scales on the real bottleneck (connection-pool contention) instead of CPU alone. It exports db.pool.waiters and db.pool.utilization as OpenMetrics from a new scrape-authenticated endpoint, and documents the HPA/KEDA guidance, alert rules, and runbook thresholds.

Related Issue

Closes #524

Changes

OpenMetrics endpoint (GET /metrics/db-pool)

• [ADD] src/db/pool.ts — getDbPoolSaturation, getPrimaryPoolSaturation, getReplicaPoolSaturation: synchronous snapshots from pg's in-process counters (waitingCount, totalCount, idleCount, options.max). Never issues queries, so a scrape cannot load the database it observes.
• [ADD] src/lib/metrics.ts — MetricsCollector.collectDbPoolSaturation: emits db.pool.waiters{pool="primary"|"replica"} and db.pool.utilization{pool=...} gauges (ratio clamped to [0,1]). Both series are set on every scrape, so they stay defined (value 0) when the pool is idle.
• [ADD] src/middleware/metricsMiddleware.ts — createDbPoolMetricsHandler: refreshes the gauges and serves OpenMetrics v1.0.0 filtered to the db.pool. families.
• [MODIFY] src/app.ts — mounts GET /metrics/db-pool behind the existing createMetricsAuthMiddleware() (METRICS_TOKEN bearer scrape auth), same guard as /metrics.
• [FIX] src/lib/metrics.ts — sanitizeName now preserves dots. Dots are valid OpenMetrics names (v1.0.0); the previous regex silently mangled the codebase's own intended names (db.replica.lag_ms → db_replica_lag_ms), which would have corrupted the new db.pool.* families.

Tests (targeted suites: 102/102 passing)

• [ADD] src/db/pool.test.ts — idle pool (defined 0), partial/saturated utilization, clamping (active > max, negative active), default max fallback, max: 0, replica-null case.
• [ADD] src/lib/metrics.test.ts — collectDbPoolSaturation (labels, idle-defined, multiple pools, disabled config, prefix filtering) + dotted-name sanitization regression test.
• [ADD] src/middleware/metricsMiddleware.test.ts — OpenMetrics content type, db.pool.* families present, idle-zero series, unrelated families excluded, error path → structured 500.

Documentation

• [ADD] docs/autoscaling-db-pool-signal.md — metric reference, endpoint + security assumptions, runbook target thresholds (utilization ≤ 0.7, waiters 0), HPA guidance (Prometheus adapter + KEDA manifests), alert rules example (DbPoolWaitersHigh, DbPoolUtilizationHigh), troubleshooting, related code.
• [MODIFY] docs/runbooks/README.md — mapping-table rows + runbook index entries for the two new alerts.
• [MODIFY] scripts/validate-alert-mappings.ts — registers the new alerts (CI gate).
• [MODIFY] docs/prometheus-metrics-endpoint.md — documents the new endpoint and changelog entry.

Verification Results

npx jest src/db/pool.test.ts src/lib/metrics.test.ts src/middleware/metricsMiddleware.test.ts
✅ Test Suites: 3 passed, 3 total
✅ Tests: 102 passed, 102 total

npm run validate:alert-mappings
✅ OK: All 24 known alerts have mapping entries.

Notes:

• npm test (full suite): 169 failed / 63 passed suites — identical class of failures on the untouched baseline (192 failed / 40 passed); all are environment-dependent integration tests (Postgres, Stellar RPC, rate-limit timing) unrelated to this change. CI (ci.yml) gates only audit:ci and validate:alert-mappings.
• npm run lint fails on the baseline too (ESLint 9 with a legacy .eslintrc.cjs and no eslint.config.js) — pre-existing repo-wide config issue, unchanged by this PR.

Acceptance Criteria	Status
Emit db.pool.waiters and db.pool.utilization	✅ Gauges with pool label, OpenMetrics v1.0.0
Metric stays defined even when pool is idle	✅ Explicit 0 on every scrape + unit test
Metrics guarded by scrape auth	✅ Mounted behind createMetricsAuthMiddleware (METRICS_TOKEN)
HPA guidance added to docs	✅ Prometheus adapter HPA + KEDA manifests with stabilization notes
Alert rules example included	✅ DbPoolWaitersHigh / DbPoolUtilizationHigh + runbook mapping
Tests + docs complete	✅ 102/102 targeted tests; runbook & endpoint docs updated

[drips-wave]

@Ajibola6921 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[Ajibola6921]

@thlpkee20-wq done closes #524