Skip to content

Throttle background maintenance under pressure#1072

Open
bootjp wants to merge 1 commit into
mainfrom
fix/maintenance-pressure-followup
Open

Throttle background maintenance under pressure#1072
bootjp wants to merge 1 commit into
mainfrom
fix/maintenance-pressure-followup

Conversation

@bootjp

@bootjp bootjp commented Jul 11, 2026

Copy link
Copy Markdown
Owner

Summary

  • skip FSM compaction on multi-peer Raft leaders so best-effort MVCC GC does not contend with leader lease/propose work
  • add a per-group lock resolver scan cursor and budget backoff after timeout
  • treat work-context deadline expiry as resolver budget exhaustion even when the returned error is a gRPC DeadlineExceeded

Risk

  • expired orphan lock cleanup may be delayed after timeout, but foreground paths still resolve locks they encounter
  • FSM compaction still runs on followers and single-node leaders; multi-peer leaders avoid best-effort compaction to preserve availability

Tests

  • GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./kv -run 'Test(FSMCompactor|LockResolver)'\n- GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./adapter -run 'TestDeltaCompactor' -count=1 -timeout=2m\n- GOCACHE=$(pwd)/.cache GOTMPDIR=$(pwd)/.cache/tmp go test ./... (fails: existing TestRedis_StreamXReadLatencyIsConstant timeout; reproduced on clean origin/main with the same command subset)\n

Summary by CodeRabbit

  • 改善

    • ロック解決処理のスキャン位置を保持し、処理量超過時はグループごとにバックオフすることで、継続的な処理の効率と安定性を向上しました。
    • リーダーで複数ピアが存在する場合、FSMコンパクションを実行しないよう変更しました。
  • テスト

    • ロック解決のスキャン再開、範囲末尾での折り返し、バックオフ期限切れを検証するテストを追加しました。
    • マルチピア環境でのコンパクション抑制を検証するテストを更新しました。

@github-actions

Copy link
Copy Markdown
Contributor

TLA+ spec divergence review (auto-triggered)

This PR touches files that the TLA+ safety spec has an anchor on (per
docs/design/2026_05_28_implemented_tla_safety_spec.md §3),
so an AI review is requested below to verify the implementation has not drifted
from the model.

Anchored files changed in this PR head (a20162e):

  • kv/lock_resolver.go

What to check, by subsystem:

  • kv/hlc*.goNext() must respect the HLC-4 preconditions (i)/(ii)/(iii) from the design doc: bounded skew, logical-counter handoff on leader change (strategy (c) Observe(MaxAppliedHLC)), and the commit-time ceiling fence (fail-closed when wall_now >= physicalCeiling). Any change to the bit layout (48/16), the CAS loop, or the ceiling getter/setter is in scope.
  • kv/coordinator.go, kv/sharded_coordinator.goRunHLCLeaseRenewal, hlcRenewalInterval, hlcPhysicalWindowMs constants, and the new-term detection that calls Observe(fsm.MaxAppliedHLC()) (strategy (c)). Any change to renewal cadence, group selection, or fail-closed behaviour is in scope.
  • kv/transaction.go, kv/lock_resolver.go — OCC commit-ts assignment, lock-map encoding (key, lock_ts) -> start_ts, and the LockResolver action OCC-3 depends on. (M2 spec will land OCC-1..OCC-5; until then the spec doc §5.2 is the contract.)
  • kv/fsm.go — FSM apply of HLC lease entries (SetPhysicalCeiling), and any future MaxAppliedHLC() accessor that strategy (c) needs.
  • store/mvcc_store.go — version visibility, snapshot install, and the MVCC-1..MVCC-4 invariants (M3 scope).
  • distribution/** — route catalog versioning, SplitRange atomicity, and CatalogWatcher async fan-out (M4 scope).

If the change is correct but requires a spec update, edit tla/hlc/HLC.tla (or the corresponding M2..M5 module once landed) and the design doc in the same PR. The tla-check workflow runs the TLC model check on the same paths.


@claude review please verify TLA+ spec divergence per the checklist above.

@codex review please verify TLA+ spec divergence per the checklist above.

@coderabbitai

coderabbitai Bot commented Jul 11, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

マルチピアRaftリーダーのFSMコンパクションを抑制し、ロック解決器にグループ別スキャンカーソル、予算超過時のバックオフ、work context に基づく判定を追加しています。

Changes

FSMコンパクション制御

Layer / File(s) Summary
マルチピアリーダーのコンパクション抑制
kv/compactor.go, kv/compactor_test.go
リーダーかつピアが存在する場合にFSMコンパクションをスキップし、対応するテストを未実行の期待値へ変更します。

ロック解決制御

Layer / File(s) Summary
グループ状態と初期化
kv/lock_resolver.go
グループ別のスキャン再開位置とバックオフ期限を保持し、mutexで保護します。
カーソルベースのロックスキャン
kv/lock_resolver.go, kv/lock_resolver_test.go
再開位置からスキャンし、処理済みキー後へカーソルを進め、範囲末尾でリセットします。
予算超過バックオフと判定
kv/lock_resolver.go, kv/lock_resolver_test.go
予算超過時のバックオフと期限切れ処理を追加し、work context に基づく判定をテストします。

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant LockResolver
  participant resolveAllGroups
  participant resolveGroupLocks
  LockResolver->>resolveAllGroups: グループ解決を開始
  resolveAllGroups->>resolveGroupLocks: 再開位置からロックをスキャン
  resolveGroupLocks-->>resolveAllGroups: 次回スキャン位置を更新
  resolveAllGroups-->>LockResolver: 解決結果を返却
Loading

Possibly related PRs

  • bootjp/elastickv#1045: マルチピアRaftリーダーのFSMコンパクション動作を変更しています。
  • bootjp/elastickv#1069: ロック解決のRaft準備状態や予算超過時の処理を変更しています。
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed バックグラウンド保守の抑制というPRの主目的を端的に表しており、FSMコンパクション抑制とロック解決のバックオフ変更に合っています。
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request optimizes the background lock resolver and compactor. It updates the FSM compactor to skip compaction on multi-peer leaders and enhances the LockResolver by tracking the scan cursor progress and implementing a backoff mechanism when a group's resolution budget is exhausted. This prevents the resolver from repeatedly scanning from the beginning of the key range or hammering saturated paths. Unit tests have been added to verify these new behaviors. There are no review comments, so I have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
kv/lock_resolver.go (1)

144-159: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

raft未準備時はカーソルを進めない
advanceLockScanStartlockResolverRaftReady より先にあるため、ループ途中で raft が未準備になったときに未処理の kvp.Key までスキャン位置が進みます。advanceLockScanStartresolveScannedLock の直前へ移したいです。

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@kv/lock_resolver.go` around lines 144 - 159, Move the advanceLockScanStart
call in the lockKVs loop to immediately before resolveScannedLock, after the
lockResolverRaftReady check. Ensure the scan cursor is not advanced for the
current kvp.Key when Raft is unavailable, while preserving the existing
resolution and outcome-counting flow.
🧹 Nitpick comments (1)
kv/compactor_test.go (1)

163-184: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

単一ノードリーダーでコンパクションが維持されることを検証するテストケースを追加してください。

PR目的は「単一ノードリーダーではコンパクションを維持する」ことですが、そのケース(StateLeader + NumPeers が単一ノードを示す値)で compactCalledtrue になることを検証するテストがありません。マルチピアのスキップだけでなく、単一ノードリーダーの保持パスもテストでカバーすべきです。

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@kv/compactor_test.go` around lines 163 - 184, Add a test covering the
single-node leader path in FSM compaction, using StateLeader with NumPeers set
to the single-node value and the existing deadlineCapturingStore setup. Invoke
FSMCompactor.SyncOnce and assert that compactCalled is true, while preserving
the existing multi-peer skip test.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@kv/lock_resolver.go`:
- Around line 144-159: Move the advanceLockScanStart call in the lockKVs loop to
immediately before resolveScannedLock, after the lockResolverRaftReady check.
Ensure the scan cursor is not advanced for the current kvp.Key when Raft is
unavailable, while preserving the existing resolution and outcome-counting flow.

---

Nitpick comments:
In `@kv/compactor_test.go`:
- Around line 163-184: Add a test covering the single-node leader path in FSM
compaction, using StateLeader with NumPeers set to the single-node value and the
existing deadlineCapturingStore setup. Invoke FSMCompactor.SyncOnce and assert
that compactCalled is true, while preserving the existing multi-peer skip test.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 5d36362f-a3b4-42fd-96fc-3776a4e884f9

📥 Commits

Reviewing files that changed from the base of the PR and between 73ed1f3 and a20162e.

📒 Files selected for processing (4)
  • kv/compactor.go
  • kv/compactor_test.go
  • kv/lock_resolver.go
  • kv/lock_resolver_test.go

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant