fix(monitoring): blocks_found decreases when the finding channel disconnects - #802
Open
gimballock wants to merge 3 commits into
Open
fix(monitoring): blocks_found decreases when the finding channel disconnects#802gimballock wants to merge 3 commits into
gimballock wants to merge 3 commits into
Conversation
`sv2_client_blocks_found_total` was recomputed each refresh as a sum over currently connected channels, so the count dropped when the channel that found a block disconnected. Per-channel `blocks_found` lives in the channel's `ShareAccounting` and is discarded with the channel, so no total derived from live channels can be monotonic. Add `Sv2ClientsMonitoring::get_blocks_found_total`, defaulting to the existing live-channel sum so implementors are unaffected, and let implementors that track the count at process scope override it. The method receives the clients `refresh()` has already collected, so it adds no business-logic lock acquisition. The total travels on `MonitoringSnapshot` rather than `Sv2ClientsSummary`, leaving the JSON API schema unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add a `blocks_found_total` accumulator to `ChannelManager`, incremented in each of the two `ShareValidationResult::BlockFound` arms — the only block paths in the pool — and report it via `get_blocks_found_total`. The count now survives both channel-removal paths: `remove_downstream` and `handle_close_channel`. It also captures a channel that connects, finds a block, and disconnects between two scrapes, which the previous derivation missed entirely. The metric stays a `Gauge`. The accumulator is a plain `AtomicU64` on the application's own struct, so no `prometheus` type reaches protocol code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Extend `block_found_detected_in_pool_metrics` to disconnect the client that found the block and re-read the metric, which is the case the previous derivation got wrong. Compare against the count observed before the disconnect rather than a literal: shutdown drains gracefully and the miner can find further low-difficulty regtest blocks meanwhile, so the total may legitimately climb. The invariant is that it never decreases. Make `parse_metric_value` public so the test can read the gauge instead of duplicating the parser. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gimballock
marked this pull request as ready for review
August 24, 2026 22:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
sv2_client_blocks_found_totalis recomputed each refresh as a sum over currently connected channels, so when the channel that found a block goes away, the count drops. A found block should outlive the channel that found it.Evidence
block_found_detected_in_pool_metricsalready mines a regtest block through the full stack. This PR extends it to disconnect the finding client and re-read the metric:Metric 'sv2_client_blocks_found_total' has value 0 but expected: == 1.So the regression reproduces in a running pool, not only in a unit fixture.
Where it comes from
blocks_foundis owned bychannels_sv2::server::share_accountinginside each channel and read viashare_accounting.get_blocks_found(), so the only count that exists is scoped to the channel and dies with it. Two paths end a channel —remove_downstream()andhandle_close_channel()— and a channel that connects, finds a block, and disconnects between two scrapes was never counted at all.The metric arrived in #278 while disconnected clients still lingered in the snapshot (#319, closed 2026-04-13). That leak masked this; fixing #319 exposed it.
Changes
AtomicU64onChannelManager, incremented in each of the twoShareValidationResult::BlockFoundarms — the only block paths in the pool.Sv2ClientsMonitoring::get_blocks_found_total(&clients), defaulting to the current live-channel sum so other implementors are unaffected. It receives the clientsrefresh()has already collected, so it adds no business-logic lock acquisition — the guarantee from fix(monitoring): Eliminate DoS vulnerability via snapshot cache #193 is preserved, andtest_snapshot_cache_eliminates_lock_contentioncovers it.MonitoringSnapshot, notSv2ClientsSummary, so the JSON API schema andopenapi.jsonare unchanged.Neither channel-removal path is touched, and snapshot cadence no longer affects correctness. Per-channel
blocks_foundin the JSON API is unchanged.Instrument: stays a
Gauge. The accumulator is a plainAtomicU64on the application's own struct, so noprometheustype reaches protocol code — the same "expose what the management struct already tracks" pattern, at process scope rather than channel scope. Blocks are rare, so this is not a hot path.Assumption worth a reviewer's eye
That
share_accountingincrementsblocks_foundon the same condition that yieldsShareValidationResult::BlockFound. I checked the app-side arms, not stratum-core's condition. If they can diverge, the process total and the per-channel view would drift.Not included
sv2_server_blocks_found_totalfor JDC and translator; an aggregate on the JSON API summary responses; and coverage for the standard-channel arm, which the current test does not reach because the translator path uses extended channels.Verified
stratum-apps109/109, pool tests, andmonitoring_integration9/9 pass;cargo clippyclean;cargo fmtclean across all three workspaces.🤖 Generated with Claude Code