feat(cdc): durable per-table schema history in the position (DBZ-3 Area 2 step 2)

[devarismeroxa]

Problem

Step 1 (#322) detects schema drift by diffing successive RelationMessages held in an in-memory cache. That cache is empty on every process start, so a ALTER TABLE applied while the connector was down is invisible — the first RelationMessage after the restart has nothing to compare against and is accepted as ground truth.

pgoutput carries no DDL (unlike the MySQL binlog, which carries the original ALTER TABLE text), so a fresh RelationMessage is the only signal a shape changed. The restart gap is a real hole, not a theoretical one.

What this does

Carries a bounded record of observed shapes in the position itself, so continuity survives restarts the same way snapshot and CDC positions do.

Piece	Notes
Position.SchemaHistory	Additive, omitempty. Keyed by namespace.name — not RelationID, which is a pg_class OID: a drop-and-recreate would silently restart the history for what an operator calls the same table
HashColumnSet	Over (Name, DataType, TypeModifier) — the same identity diffRelations uses, so hash and diff cannot disagree. Sorted input (compared against a hash from a different process after a restart); separator can't occur in a Postgres identifier, ruling out the ("a|b","c") vs ("a","b|c") collision
RecordSchemaVersion	Pruned oldest-first to DefaultSchemaHistoryVersions (10)
CDCHandler.handleRelation	Returns driftKind: none / initial / inProcess / acrossRestart

Re-observing the current shape is a no-op. Postgres re-sends a RelationMessage on reconnect and when a new subscriber attaches. Appending on every sighting would fill the bounded history with duplicates of one shape and prune away the older ones carrying the drift signal — the mechanism would destroy its own evidence.

handleRelation returns the kind rather than only logging it so the decision is assertable in a test, and because it is the seam the halt|dlq|evolve policy attaches to in step 3.

Why no new storage backend

Riding in the existing position payload means invariant 5 (atomic state/checkpoint writes) is satisfied by the mechanism already in place, rather than by a second thing to keep crash-consistent. The per-table bound keeps the payload from growing without limit — which matters because positions are written on every batch.

Known limitation (deliberate)

Only a hash is retained, so drift across a restart reports that the schema changed, not which columns. Retaining full column sets for 10 versions per table would put roughly a kilobyte per table into a payload written on every batch. Detection is the requirement; the delta isn't worth that cost. The log line says so plainly rather than implying more.

Behavior change

None. This detects and reports. It does not halt, DLQ, or evolve. Making halt the default is step 3 — that's the breaking change that owes a migration note.

Adversarial self-review

Found and fixed while writing this:

1. previous_schema_hash logged the new hash. I read LastSchemaVersion after RecordSchemaVersion had already mutated it, so the "previous" field in the cross-restart warning was the current shape. An operator correlating a drift alert against their DDL history would have been handed the wrong hash. Fixed by capturing prev before recording.
2. Map aliasing into emitted positions. buildPosition hands SchemaHistory to Position by reference, and RecordSchemaVersion mutates it later. Verified safe: ToSDKPosition marshals to JSON eagerly, so no live reference escapes. Documented at the call site rather than left to be rediscovered.
3. Double-logging. In-process drift satisfies both diff.HasDrift() and "differs from durable history". Collapsed into one switch gated on whether a new version was actually recorded, so each RelationMessage produces at most one line.
4. First mutation run for the hash didn't apply (shell escaping ate \x1f), so DistinguishesShapes looked green under a mutation that never landed. Re-ran correctly — it does fail.

Tests — every one mutation-verified

Mutation	Tests killed
Drop the SchemaHistory carry-forward in buildPosition	DriftAcrossRestart, NoDriftAcrossCleanRestart, CarriesSchemaHistoryOnEveryRecord
Remove the same-shape dedupe	RepeatIsSilent, NoDriftAcrossCleanRestart, DedupesCurrentShape
Prune newest-first instead of oldest-first	StaysBounded, PrunesOldestFirst
Drop TypeModifier from the hash	DistinguishesShapes — this is varchar(10) -> varchar(20) passing as "same shape"

CarriesSchemaHistoryOnEveryRecord is the regression test for the failure mode buildPosition's own comment warns about, applied to the new field: positions are checkpointed per batch and a restart lands on whichever was last persisted, so a single position dropping the history loses drift detection intermittently.

What was actually run

• go build, go vet, golangci-lint — clean on the touched files (5 pre-existing hits elsewhere in the repo from a newer local linter; CI lint is green on main)
• Unit tests in source/position and source/logrepl — pass, and mutation-verified above
• Integration tests were not run locally: ports 5432/5433 are held by unrelated containers on this machine. They run here in CI on a clean runner.

Risk tier

1 — corrected. I first marked this Tier 2 on the reasoning that the format change is additive. That is the wrong test: CLAUDE.md puts serialization formats in Tier 1 regardless of whether a given change is additive, because "additive" is a claim about the change that has to be demonstrated, not asserted. So this needs DeVaris sign-off before merge; I am not admin-merging it.

Re-tiering also surfaced a gap. The rule is "never change a serialized format without a versioned migration path and an upgrade test", and I had only half of it:

Direction	Test	Status
N reads an N-1 position (upgrade)	LegacyPositionSeedsWithoutDrift	was present
N-1 reads an N position (rollback)	DowngradeIsSafe	was missing — added

The rollback direction is the one that matters more: a rollback is already an incident, and a position the older build cannot parse would turn it into a worse one. It passes because ParseSDKPosition uses a permissive json.Unmarshal. Mutation-verified by making the N-1 reader strict — it fails with unknown field "schema_history". Pinning it means someone adding DisallowUnknownFields later fails a test rather than silently making every rollback unrecoverable.

CurrentPositionVersion stays at 1: the field is additive and omitempty, and both directions are now covered by test, so there is nothing for a version bump to gate.

Failure-mode analysis (Tier 1)

Failure	Effect	Detection	Rollback
History dropped from one emitted position	Restart from that position resumes with no history; drift across that restart goes unreported. Intermittent	CarriesSchemaHistoryOnEveryRecord	n/a — caught in test
History grows without bound	Position payload inflates on a table churning schemas; written every batch	StaysBounded; bound is 10/table	Field is omitempty; older build ignores it
Hash unstable across builds	Every restart falsely reports drift; once step 3 lands, every restart halts	StableAcrossOrdering (sorted input, fixed separator)	Revert
Duplicate versions from re-sent RelationMessages	Bounded history fills with one shape, prunes away the drift signal	RepeatIsSilent, DedupesCurrentShape	Revert
Older build cannot parse a new position	Rollback bricks the pipeline	DowngradeIsSafe	n/a — caught in test

No data-path behavior changes: no ack, position-ordering, or checkpoint semantics are touched. buildPosition gains one carried field. Records are unaffected.

Roadmap

DBZ-3 (Debezium parity), Area 2 step 2. Follows #322.

Next: step 3 — the halt | dlq | evolve policy, where halt as the default is the breaking change requiring the migration note.

🤖 Generated with Claude Code

https://claude.ai/code/session_015GQFzakPShAYj8CcwajYDD