Skip to content

fix(monitoring): blocks_found decreases when the finding channel disconnects - #802

Open
gimballock wants to merge 3 commits into
stratum-mining:mainfrom
marafoundation:blocks-found-process-accumulator
Open

fix(monitoring): blocks_found decreases when the finding channel disconnects#802
gimballock wants to merge 3 commits into
stratum-mining:mainfrom
marafoundation:blocks-found-process-accumulator

Conversation

@gimballock

Copy link
Copy Markdown
Contributor

sv2_client_blocks_found_total is recomputed each refresh as a sum over currently connected channels, so when the channel that found a block goes away, the count drops. A found block should outlive the channel that found it.

Evidence

block_found_detected_in_pool_metrics already mines a regtest block through the full stack. This PR extends it to disconnect the finding client and re-read the metric:

  • with the fix: passes;
  • with the metric left as a sum over live channels: fails with Metric 'sv2_client_blocks_found_total' has value 0 but expected: == 1.

So the regression reproduces in a running pool, not only in a unit fixture.

Where it comes from

blocks_found is owned by channels_sv2::server::share_accounting inside each channel and read via share_accounting.get_blocks_found(), so the only count that exists is scoped to the channel and dies with it. Two paths end a channel — remove_downstream() and handle_close_channel() — and a channel that connects, finds a block, and disconnects between two scrapes was never counted at all.

The metric arrived in #278 while disconnected clients still lingered in the snapshot (#319, closed 2026-04-13). That leak masked this; fixing #319 exposed it.

Changes

  • AtomicU64 on ChannelManager, incremented in each of the two ShareValidationResult::BlockFound arms — the only block paths in the pool.
  • Sv2ClientsMonitoring::get_blocks_found_total(&clients), defaulting to the current live-channel sum so other implementors are unaffected. It receives the clients refresh() has already collected, so it adds no business-logic lock acquisition — the guarantee from fix(monitoring): Eliminate DoS vulnerability via snapshot cache #193 is preserved, and test_snapshot_cache_eliminates_lock_contention covers it.
  • The total travels on MonitoringSnapshot, not Sv2ClientsSummary, so the JSON API schema and openapi.json are unchanged.

Neither channel-removal path is touched, and snapshot cadence no longer affects correctness. Per-channel blocks_found in the JSON API is unchanged.

Instrument: stays a Gauge. The accumulator is a plain AtomicU64 on the application's own struct, so no prometheus type reaches protocol code — the same "expose what the management struct already tracks" pattern, at process scope rather than channel scope. Blocks are rare, so this is not a hot path.

Assumption worth a reviewer's eye

That share_accounting increments blocks_found on the same condition that yields ShareValidationResult::BlockFound. I checked the app-side arms, not stratum-core's condition. If they can diverge, the process total and the per-channel view would drift.

Not included

sv2_server_blocks_found_total for JDC and translator; an aggregate on the JSON API summary responses; and coverage for the standard-channel arm, which the current test does not reach because the translator path uses extended channels.

Verified

stratum-apps 109/109, pool tests, and monitoring_integration 9/9 pass; cargo clippy clean; cargo fmt clean across all three workspaces.

🤖 Generated with Claude Code

Eric Price and others added 3 commits August 24, 2026 09:25
`sv2_client_blocks_found_total` was recomputed each refresh as a sum over
currently connected channels, so the count dropped when the channel that
found a block disconnected.

Per-channel `blocks_found` lives in the channel's `ShareAccounting` and is
discarded with the channel, so no total derived from live channels can be
monotonic. Add `Sv2ClientsMonitoring::get_blocks_found_total`, defaulting
to the existing live-channel sum so implementors are unaffected, and let
implementors that track the count at process scope override it.

The method receives the clients `refresh()` has already collected, so it
adds no business-logic lock acquisition. The total travels on
`MonitoringSnapshot` rather than `Sv2ClientsSummary`, leaving the JSON API
schema unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add a `blocks_found_total` accumulator to `ChannelManager`, incremented in
each of the two `ShareValidationResult::BlockFound` arms — the only block
paths in the pool — and report it via `get_blocks_found_total`.

The count now survives both channel-removal paths: `remove_downstream` and
`handle_close_channel`. It also captures a channel that connects, finds a
block, and disconnects between two scrapes, which the previous derivation
missed entirely.

The metric stays a `Gauge`. The accumulator is a plain `AtomicU64` on the
application's own struct, so no `prometheus` type reaches protocol code.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Extend `block_found_detected_in_pool_metrics` to disconnect the client that
found the block and re-read the metric, which is the case the previous
derivation got wrong.

Compare against the count observed before the disconnect rather than a
literal: shutdown drains gracefully and the miner can find further
low-difficulty regtest blocks meanwhile, so the total may legitimately
climb. The invariant is that it never decreases.

Make `parse_metric_value` public so the test can read the gauge instead of
duplicating the parser.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant