Skip to content

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#46

Draft
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-1d82
Draft

fix(rust_brain): preserve HLC on snapshot restore and gossip receive#46
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-investigation-1d82

Conversation

@cursor

@cursor cursor Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Bug and impact

Critical data-loss bug in disaster recovery and multi-node gossip.

After restore_from_file(), nodes were assigned fresh HLC timestamps instead of the values stored in the snapshot. Legitimate post-restore causal writes with successor HLCs were rejected as TimestampRegression, causing silent data loss.

Similarly, GossipProtocol.receive() ignored HLC timestamps from peers, allowing stale gossip events to overwrite fresher local state.

Concrete trigger scenario

  1. Node writes remember("k", "v1", hlc=(5000, 10, "nodeA"))
  2. Snapshot taken and restored after crash
  3. Restored node has HLC (wall, 0, uuid) instead of (5000, 10, "nodeA")
  4. Causal successor remember("k", "v2", hlc=(5000, 11, "nodeA")) raises TimestampRegression — write silently lost

Root cause

restore_from_file() reconstructed MemoryNode without passing the hlc field from the snapshot JSON. MemoryNode defaulted to _hlc.now(), producing unrelated timestamps. gossip.receive() also omitted HLC passthrough.

Fix

  • Add _parse_hlc() for snapshot/wire HLC normalisation (with legacy fallback for pre-v0.6.0 snapshots)
  • Restore HLC on snapshot load and advance global _hlc via _hlc.update()
  • Pass HLC through bulk_write and gossip.receive()
  • Reject stale gossip updates; skip HLC-less updates on existing keys
  • Hold lock during restore_from_file and gc_expired

Validation

  • Reproduced bug pre-fix: successor write failed with TimestampRegression
  • New regression tests: test_restore_preserves_hlc_for_replication, 4 gossip HLC tests
  • Full suite: 192 passed, 8 skipped
Open in Web View Automation 

After snapshot restore, nodes were assigned fresh HLC timestamps instead
of the stored values from the snapshot. Post-restore causal writes with
legitimate successor HLCs were rejected as TimestampRegression, causing
silent data loss in disaster-recovery and multi-node gossip scenarios.

- Add _parse_hlc() for snapshot/wire HLC normalisation
- Restore HLC on snapshot load and advance global _hlc
- Pass HLC through bulk_write and gossip receive
- Reject stale gossip updates; skip HLC-less updates on existing keys
- Add regression tests for restore round-trip and gossip ordering

Co-authored-by: Daniel <DJLougen@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant