feat(http): standardize the 401 response, with a proxy-config note

[rustyconover]

The problem

A 401 was whatever Falcon produced: the styled HTML page, served to every caller including the Arrow client, which then pasted the entire page into RpcError.error_message. The only machine-readable signal was the status code, so a client wanting to refresh a token and retry had to substring-match an English sentence.

Worse, the most common production cause was invisible. Schemes that read a proxy-injected header — mTLS as x-forwarded-client-cert, a proxy proof — fail identically whether the caller sent a bad certificate or the proxy was never configured to forward one. The second is far more likely during a deployment, and the response said nothing about it. Operators rotate credentials that were never the problem.

What changed

A reason code from a closed set (missing_credential, invalid_credential, expired_credential, insufficient_scope, proxy_required, unauthorized) on a VGI-Auth-Reason header and in a JSON envelope. Accept: text/html still gets the page; */* — what every RPC client sends — gets JSON.

HTTP/1.1 401 Unauthorized
VGI-Auth-Reason: proxy_required
VGI-Auth-Proxy-Required: true
Cache-Control: no-store

{"error":"unauthorized","reason":"proxy_required",
 "detail":"Missing x-forwarded-client-cert header",
 "proxy_hint":"This service only accepts requests that arrive through its
   configured reverse proxy, which must set the x-forwarded-client-cert
   header. A rejection here is as likely to be a proxy that is not forwarding
   that header — or a request that reached the service without passing through
   the proxy at all — as it is a bad credential. Check the proxy configuration
   before rotating credentials."}

The code names the stage, never a verifier's diagnosis. Every proxy-proof outcome — absent, malformed, unknown kid, expired, bad MAC, replayed — collapses onto proxy_required with an identical detail. That keeps docs/proxy-proof-spec.md §6's uniform-rejection rule intact rather than carving an exception into it; a new conformance test asserts the uniformity from the outside.

The proxy note is derived from server configuration, not from what failed on a given request. It is therefore identical on every 401 from a proxy-dependent service and discloses nothing a caller could not read off the capability headers. That is also what makes it right in the case it exists for: when the proxy isn't forwarding the header, everything 401s, and every one of those says so.

Header dependencies are discovered from the installed authenticators — mtls_authenticate* and the require-mode proxy_proof_gate declare their own via declare_proxy_headers, and chain_authenticate / require_all propagate — with make_wsgi_app(proxy_auth_headers=[...]) for a custom callback the framework can't introspect.

Client side: 401s raise AuthenticationError, an RpcError subclass with error_type still "AuthenticationError", so existing except RpcError / match="AuthenticationError" call sites are unaffected. It exposes reason / detail / proxy_hint, and appends the hint to the message because the place it actually gets read is a traceback in a deployment log. A body that isn't the envelope degrades rather than being dumped whole — a 401 can come from a gateway, WAF, or SSO portal the service never sees.

Contract and coverage

• docs/unauthorized-spec.md — new normative spec (added to the mkdocs nav).
• TestUnauthorized — 8 conformance tests, capability-gated on VGI-Auth-Reason so unported languages skip cleanly. Two reuse proof_worker_factory for the proxy-dependent cases.
• tests/test_unauthorized.py — 41 Python-specific tests: per-authenticator codes, chain/require-all composition, negotiation, HTML escaping, cache bounding, client parsing.
• docs/porting-guide.md, docs/WIRE_PROTOCOL.md (which still described 401 bodies as plain text), docs/proxy-proof-spec.md, CLAUDE.md, README/index updated.

Behaviour change to flag

bearer_authenticate used to raise ValueError("Missing or invalid Authorization header") for both an absent header and a non-Bearer scheme. It now splits them — missing_credential vs invalid_credential — with distinct messages, since telling a caller who sent Basic to "send a credential" is the wrong advice. One test in tests/test_bearer.py was updated accordingly.

Not touched: the access log gains no auth_reason field. That would be an access-log-spec.md change and is easy to add separately if wanted.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR standardizes HTTP 401 (Unauthorized) responses for vgi-rpc’s Falcon-based HTTP transport, providing a closed-set reason code plus an optional, configuration-derived reverse-proxy hint so both machine clients and operators can reliably interpret auth failures.

Changes:

• Introduces a cross-language 401 contract (VGI-Auth-Reason + JSON envelope by default; optional HTML when Accept requests it) and wires it into the Falcon error serializer.
• Adds a typed client-side AuthenticationError that parses the envelope and degrades safely for non-envelope 401 bodies.
• Adds proxy-dependency discovery (declare_proxy_headers / proxy_headers_of) to emit a static “check the reverse proxy config” note on every 401 for proxy-dependent services, plus conformance + Python tests and documentation updates.

Reviewed changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
vgi_rpc/http/server/_middleware.py	Classifies auth failures and stashes the reason on req.context for 401 serialization.
vgi_rpc/http/server/_factory.py	Plumbs proxy-header dependency discovery into app creation and installs the new error serializer.
vgi_rpc/http/server/_errors.py	Implements standardized 401 serialization (JSON by default, HTML for browsers), headers, and bounded caching.
vgi_rpc/http/_unauthorized.py	Adds the reference implementation for reason codes, typed exceptions, and proxy-header declarations.
vgi_rpc/http/_testing.py	Exposes new make_wsgi_app parameters through the sync test client helper.
vgi_rpc/http/_proof.py	Marks proxy-proof failures with the coarse proxy_required reason code and propagates proxy header dependency in require mode.
vgi_rpc/http/_oauth_pkce.py	Switches missing-cookie auth failures to the new AuthFailure/reason-code mechanism.
vgi_rpc/http/_oauth_jwt.py	Adds missing/invalid/expired reason-code separation for JWT auth failures.
vgi_rpc/http/_mtls.py	Adds proxy-required vs invalid-credential reason codes for mTLS header/cert parsing and declares proxy header dependency.
vgi_rpc/http/_common.py	Defines standardized 401 headers and adds CSS for reason/note display in the HTML page.
vgi_rpc/http/_client.py	Parses the standardized 401 envelope into AuthenticationError and truncates/normalizes foreign/HTML 401 bodies.
vgi_rpc/http/_bearer.py	Adds missing-vs-invalid bearer reason codes, reason aggregation for chains, and proxy-header propagation in compositions.
vgi_rpc/http/init.py	Exports the new public unauthorized/auth types and header constants.
vgi_rpc/conformance/_pytest_suite.py	Adds TestUnauthorized conformance group and closed-set reason assertions.
vgi_rpc/cli.py	Updates comments to reflect 401 bodies are JSON/HTML rather than plain text.
tests/test_unauthorized.py	Adds Python-specific tests for reason selection, proxy note discovery, negotiation, caching bounds, and client parsing.
tests/test_bearer.py	Updates bearer auth tests to assert reason-code behavior via AuthFailure.
README.md	Documents the new proxy_auth_headers option and links to the 401 spec.
mkdocs.yml	Adds the unauthorized response spec to the documentation nav.
docs/WIRE_PROTOCOL.md	Updates the wire protocol docs to reflect standardized 401 bodies.
docs/unauthorized-spec.md	Adds the new normative cross-language 401 response specification.
docs/proxy-proof-spec.md	Aligns proxy-proof rejection uniformity requirements with the new 401 contract.
docs/porting-guide.md	Documents how ports should implement the unauthorized response contract.
docs/index.md	Highlights standardized 401 behavior as a headline feature.
CLAUDE.md	Updates repository guidance to reflect the new 401 contract and implementation points.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov]

Codecov Report

❌ Patch coverage is 92.49012% with 19 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
vgi_rpc/conformance/_pytest_suite.py	88.88%	4 Missing and 5 partials ⚠️
vgi_rpc/http/server/_errors.py	89.13%	1 Missing and 4 partials ⚠️
vgi_rpc/http/_bearer.py	81.81%	2 Missing and 2 partials ⚠️
vgi_rpc/http/_mtls.py	92.85%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!