Refresh table metadata when psycodict announces schema changes

[roed-math]

Companion PR to roed314/psycodict#111 (LISTEN/NOTIFY), showing what the consumer side looks like: the website picks up schema changes without restarting every worker. Draft until psycodict 1.0 (with LMFDB#111) is released — but safe to merge early, since the refresher disables itself (with one log line) when psycodict lacks the API, and requirements.txt already installs psycodict from git master, so the feature lights up on its own once LMFDB#111 merges. To try it today: pip install "git+https://github.com/roed314/psycodict.git@listen-notify".

What it does. Each web worker owns a SchemaRefresher (new module lmfdb/schema_refresh.py), driven by a before_request hook. The refresher keeps a db.listener() subscribed to the psycodict_schema channel; each request does a non-blocking poll() — a socket read on an idle dedicated connection. When notifications have arrived, it calls db.refresh_tables() (psycodict#99, already merged) once per batch, so added/dropped/renamed columns and tables become visible before the request is handled. Log lines record each refresh with the table names from the notification payloads.

Policies this PR chooses — the things psycodict#111 deliberately leaves to the application:

• No background thread. Sync workers are single-threaded; polling at request boundaries avoids refresh-during-request races and costs nothing measurable. An idle worker lags until its next request, which is harmless: no requests means no queries to fail. A non-blocking lock makes the hook safe under threaded/gevent servers too.
• Reconnect with catch-up. On listener failure: drop it, back off 30s, resubscribe — and do a full refresh_tables() on every (re)subscription, because notifications sent while unsubscribed are lost (LISTEN only delivers what is sent after it). The same catch-up covers the window between worker start and first subscription, and a failed refresh drops the listener so the next resubscription retries it.
• Whole-catalog refresh. The payload (a table name) is used for logging and for collapsing bursts into one refresh; refresh_tables() re-reads everything anyway, which handles create/drop/rename uniformly. Per-table refresh via the payload is an easy later optimization if the full refresh ever gets expensive.
• Fork-safety. Under gunicorn --preload, an inherited listener is abandoned (not closed — its socket is shared with the parent) and each worker builds its own.

Cost. One extra idle PG connection per worker, one non-blocking poll per request, one catch-up refresh per worker per (re)subscription; refreshes otherwise happen only when the schema actually changes.

Tests. lmfdb/tests/test_schema_refresh.py unit-tests the control flow with stub db/listener objects (subscribe + catch-up, batch dedupe + channel filtering, subscription-failure backoff and recovery, lost-listener resubscribe, failed-refresh retry, hot-standby permanent-disable, fork handling) and passes against any psycodict version; the live NOTIFY→LISTEN round trip is covered by psycodict's own tests in LMFDB#111. Verified locally: 7 passed under sage -python -m pytest, and the site serves normally with today's psycodict (the hook logs the one-line "not available" notice exactly once and stays out of the way).

Deployment topology (verified live). devmirror is a physical hot standby (PG 18.1, pg_is_in_recovery() = true): a server in recovery refuses LISTEN outright (cannot execute LISTEN during recovery, SQLSTATE 25006), and NOTIFY is not WAL-logged, so notifications cannot reach websites pointing at it no matter what the sync does — no trigger or event trigger on devmirror can help (triggers don't fire during WAL replay, and a standby cannot NOTIFY). The refresher therefore treats SQLSTATE 25006 at subscribe time as permanent and disables itself for the life of the process, so development copies of the website (which read devmirror) keep today's behavior — restart to pick up schema changes — with a single info log line instead of a retry loop. Production/beta webservers, whose schema changes are applied by psycodict-driven processes on the primaries they read from, get the live refresh.

We've decided not to pursue automatic refresh for dev copies, since prod/beta don't need it. For the record, the workable design would be a replicated schema-generation marker: _notify_schema_change additionally bumps a one-row meta_schema_gen table in the same transaction on the primary. The row replicates through WAL in commit order — so by the time a website sees the bump, the schema change itself has already replayed on the standby (a guarantee primary-side LISTEN cannot give) — and a website that cannot LISTEN polls that single-row SELECT at most every few seconds. That is a small psycodict follow-up plus ~15 lines here; the refresh-on-UndefinedColumn fallback discussed in psycodict#99 remains a complementary option.

🤖 Generated with Claude Code