Accountable Proof of Solvency monitoring (3Jane first): alert CRITICAL below 100% collateral ratio

[spalen0]

Goal

Monitor Accountable Proof of Solvency data for 3Jane (DFID=100000026) and wire it into the existing protocols/3jane/main.py monitor.

Primary requirement: CRITICAL alert when the 3Jane collateral ratio falls below 100%.

Accountable data being unavailable, malformed, inconsistent, or stale must also alert — and must never interrupt the existing onchain 3Jane checks.

Delivered as a single PR. No phase gating.

Verified data-access path

GET https://accountable.3jane.xyz/dashboard returns the full report as JSON (~17.7 KB, application/json, no auth, no headers required). Verified live:

{
  "res": "ok",
  "data": {
    "collateralization": 1.000279,
    "net": 20948.95,
    "ts": "1785483254636",
    "reserves": {
      "verifiability": "100",
      "interval": "live",
      "total_reserves": {"value": 75020844.79, "name": "Total Reserves"},
      "total_supply":   {"value": 74999895.84, "name": "Total Supply"},
      "reserves_split": [...], "supply_split": [...],
      "loan_receivables": [...], "asset_allocation": {...}, "timeline": [...]
    },
    "dataSources": { "<name>": {"frequency": "...", "lastUpdated": "...", "type": "..."} },
    "attestations": {...}
  }
}

Also present: snapshot hashes/signatures, Merkle-root material, ZK proofs, TEE attestation data, and signed HTTP response headers (Content-Digest, Signature-Input, Signature).

The SPA falls back to a Weavechain plugin call (accountablePlugin.proofOfSolvency({"type":"three-jane"}), anonymous login to org accountable, scope *) only if /dashboard fails. weave-py-api is on PyPI (1.2.8) if that fallback is ever needed, but it pulls heavy transitive deps (paramiko, pynacl, java-random, websocket-client). Out of scope — the plain JSON path is the integration.

Live state that drives the design

Two findings from the live payload materially shape this work.

The margin is razor-thin. collateralization = 1.000279 — 2.8 basis points above the alert line, with net equity of ~$21k on a $75M book. This crosses 1.00 on routine noise, not only on a solvency event.

Two of four sources are already stale against their declared frequency:

Source	Type	Frequency	Observed age
Slope — Forward Flows	Document Report	DAILY	55.6h (2.3× cadence)
LendSwift — Warehouse Senior	Document Report	WEEKLY	175.6h (7.3d)
USD3 Minted Liabilities	ERC4626	15 MIN	0.1h
USD3 On-Chain Reserves	ERC4626	15 MIN	0.1h

And the stale ones are load-bearing: reserves_split is 100% "Morpho Credit" ($75.02M), of which loan_receivables attributes $58.28M to Slope — Forward Flows ("per outstanding"). So ~78% of the reserves side is off-chain loan receivables priced by a document report that is currently two days old. A 2.8bp solvency margin computed largely from a stale manual valuation is the substance of what we are alerting on — source freshness is not a side-check.

Design decisions

1. Non-dispatching in v1. HIGH/CRITICAL on 3jane invoke utils.dispatch, whose receiver zeroes market caps (emergency_config.json / forced_caps.json) — an automated action, not an informational ping. At a 2.8bp margin that is unacceptable for v1. Alert already separates dispatch keying from Telegram routing, so this needs no shared-infra change:

Alert(AlertSeverity.CRITICAL, msg, protocol="3jane-accountable", channel="3jane")

"3jane-accountable" is absent from DISPATCHABLE_PROTOCOLS → no dispatch. channel="3jane" → normal 3Jane Telegram channel at full CRITICAL severity. This also serves as burn-in: real alerts from day one, plus weeks of live data on the noise profile before deciding whether it should ever drive automated action.

2. Require 2 consecutive runs below 1.00 before firing CRITICAL, to filter a transient blip when a stale document source refreshes.

3. Derive the ratio from totals, not the reported field. collateralization is server-rounded to 6dp (75020844.79 / 74999895.84 = 1.00027932… → 1.000279). Near the boundary a true 0.9999996 presents as 1.0 and < 1.00 is False — a missed-insolvency path. Compute from total_reserves / total_supply at full precision and treat the reported field as a cross-check with tolerance ≥1e-6 (the rounding sets a floor on any tighter tolerance).

4. Do not block on Accountable's response. Questions below are sent async. Building against the observed contract now is strictly better than no monitoring; the typed-failure handling catches a contract change if one comes.

Client — utils/accountable.py

• AccountableFeedConfig(dfid, dashboard_url, dashboard_type) registry — the request is URL/type-based and neither sends nor echoes the DFID, so feed identity must be bound explicitly in config.
• Typed AccountableReport on success.
• Two typed failure outcomes, not five: Unavailable (network error, non-200, bad schema, failed consistency check — "don't trust this number") and Stale (valid report, but aggregate or a required source is too old).
• Separate connect/read timeouts; bounded retries on timeout/429/5xx; no retry on permanent 4xx or schema failure.
• Never returns a silent None.

Validation

• res == "ok", required objects/fields present.
• Numbers finite and within plausible ranges.
• Coerce numeric strings — the live payload ships ts, verifiability, and timeline[].point as strings despite the docs typing them as Number/Integer. Strict type rejection would reject valid data.
• collateralization ≈ total_reserves / total_supply within tolerance.
• net ≈ total_reserves − total_supply within tolerance. Assert fx == 1 rather than assume it: the docs define both against liabilities, and total_supply.fx is documented as "FX rate for the supply (1 if USD-pegged)". The identity holds for 3Jane but breaks silently on the first non-USD-pegged feed.
• Do not cross-check timeline[-1] against top-level totals — they are different snapshot instants (observed supply 74999831.58 vs 74999895.84) and would produce false mismatches.

Freshness

Grace periods keyed by source type, not one global value — Document Report needs days, ERC4626 needs minutes. A single threshold cannot cover both, and as specified with a naive global grace the monitor fires on day one. Check aggregate report age and per-source age. Do not rely on the dashboard's generic staleAfterDays: 3 alone.

Alert state model

Persist a band and alert on transitions, not on every worsening tick. should_alert_value_drop() is not suitable — it re-alerts whenever a sub-threshold value drops further, which at 2.8bp would be constant.

OK -> HIGH -> CRITICAL

Metric	Condition	Severity
Collateral ratio	< 1.00, 2 consecutive runs	CRITICAL
Collateral ratio	< 1.05 and >= 1.00 (exact value TBD)	HIGH
Report/source freshness	Exceeds per-type cadence + grace	MEDIUM
Feed unavailable/invalid	Consecutive failures exceed limit	MEDIUM

Recovery to a healthier band re-arms the lower band. Skipping a band (OK → CRITICAL directly) fires CRITICAL. Improving bands re-arm without alerting. Repeated alerts within one band require an explicit cooldown.

Persist: last successful retrieval time, last report timestamp, consecutive failure count, current band, current data-health state.

Deferred from v1: verifiability-drop and total-reserves-change alerts. Neither drives action, and reserve-size movement is partly covered by the existing TVL check.

Scope

Single PR covering:

• utils/accountable.py — client, validation, freshness, typed outcomes
• protocols/3jane/main.py — isolated Accountable check in its own failure boundary; all existing onchain checks must still run when Accountable fails. Missing config logs and emits a coverage signal rather than silently appearing healthy.
• protocols/3jane/README.md — monitor list + threshold table
• monitoring.yaml — new monitors under 3jane
• Tests with recorded, minimized fixtures: valid report, below-100%, missing/mistyped fields, numeric-string coercion, non-finite values, inconsistent ratio/net, stale aggregate, stale individual source, unavailable endpoint + retry behavior, OK → HIGH → CRITICAL → OK transitions, noise within a band, and Accountable failure not preventing onchain checks.

No automation/jobs.yaml change — rides the existing hourly 3jane task.

Acceptance criteria

• collateralization retrieved and strictly validated each hourly run, computed from totals at full precision
• CRITICAL fires on transition below 1.00 after 2 consecutive runs, deduped by band, re-arms on recovery
• Alert does not invoke emergency dispatch in v1, but routes to the 3jane Telegram channel
• Frozen aggregate report detected
• Fresh aggregate with a stale required source detected, using per-source-type grace
• Persistent endpoint/schema/consistency failures produce a data-health alert
• Accountable failures never prevent existing onchain 3Jane checks
• Feed config explicitly binds DFID, dashboard URL, and dashboard type
• README.md and monitoring.yaml updated; tests/test_monitoring_config.py passes

Async questions for Accountable (non-blocking)

1. Is GET /dashboard a supported integration endpoint, and what schema/versioning/availability guarantees apply?
2. Can the response carry a stable feed identifier so a misrouted response can be rejected?
3. What cadence and grace period are intended for the aggregate report and each source — particularly the Document Report inputs currently running past their declared frequency?
4. What is the supported verification procedure and key rotation contract for the HTTP signatures and report attestations? (Signature verification deferred to v2; TLS + consistency checks for v1.)

Related

• 3Jane: redefine junior-buffer alert around at-risk credit coverage #323 — 3Jane: redefine junior-buffer alert around at-risk credit coverage