Skip to content

fix(healthcheck): guard target status fields against data race#412

Open
lafoush wants to merge 1 commit into
fosrl:mainfrom
lafoush:fix/healthcheck-status-data-race
Open

fix(healthcheck): guard target status fields against data race#412
lafoush wants to merge 1 commit into
fosrl:mainfrom
lafoush:fix/healthcheck-status-data-race

Conversation

@lafoush

@lafoush lafoush commented Jul 14, 2026

Copy link
Copy Markdown

Problem

monitorTarget / performHealthCheck mutate a Target's Status, LastError, LastCheck, CheckCount, and consecutive-success/failure counters from the target's monitor goroutine. Meanwhile getAllTargetsUnsafe (called by GetTargets, used for status reporting to the server) copied the whole struct with targetCopy := *target.

The monitor mutex protects the targets map, not the contents of each Target, so these accesses overlap without synchronization. The race detector flags it on any concurrent monitor + GetTargets workload — e.g. the reconnect path, where the status reporter snapshots targets while their monitor goroutines are updating status.

WARNING: DATA RACE
Read at ... by goroutine N:
  healthcheck.(*Monitor).getAllTargetsUnsafe()   // targetCopy := *target
Previous write at ... by goroutine M:
  healthcheck.(*Monitor).monitorTarget()         // target.timer / target.Status

Fix

  • Add a per-Target sync.Mutex guarding the mutable status fields.
  • Replace the whole-struct copy with a snapshot() taken under that lock, copying only the reported fields (Config, Status, LastCheck, LastError, CheckCount). Runtime-only fields (timer/ctx/cancel/client) are intentionally omitted — callers of GetTargets only read the reported fields, and this also avoids copying a sync.Mutex.
  • The lock is not held across the health-check network request, so status snapshots are never blocked for the check timeout.
  • DisableTarget's Status write is guarded too.

Test

Adds healthcheck_test.go with a regression test that exercises concurrent monitoring + snapshotting. It fails under -race without the fix and passes with it:

# without the fix
--- FAIL: TestConcurrentStatusSnapshotNoRace (race detected)
# with the fix
ok  github.com/fosrl/newt/healthcheck

go build, go vet ./healthcheck/, and gofmt are clean.

No functional/behavioral change to health checking — this is purely a concurrency-safety fix.

monitorTarget/performHealthCheck mutate a Target's Status, LastError,
LastCheck, CheckCount and consecutive counters from the target's monitor
goroutine, while getAllTargetsUnsafe copied the whole struct (targetCopy
:= *target) for status reporting. The map lock protects the map, not the
target contents, so these overlap without synchronization. Confirmed by
the race detector on a concurrent monitor + GetTargets workload.

Add a per-Target mutex guarding the mutable status fields and replace the
whole-struct copy with a snapshot() taken under that lock (copying only
the reported fields; runtime-only fields timer/ctx/cancel/client are
omitted). The lock is not held across the health-check network request,
so snapshots are never blocked for the check timeout. DisableTarget's
Status write is guarded too.

Adds healthcheck_test.go with a regression test that fails under -race
without the fix.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant