Skip to content

gpu: the agent never said when it dropped GPU records - #204

Merged
dpsoft merged 1 commit into
mainfrom
fix/gpu-silent-record-loss
Oct 6, 2026
Merged

dpsoft merged 1 commit into
mainfrom
fix/gpu-silent-record-loss

Conversation

@dpsoft

@dpsoft dpsoft commented Oct 6, 2026 •

Copy link
Copy Markdown
Owner

A GPU capture could lose most of its data and say nothing about it.

The silence

gpuprobe.Stats already counted every kind of loss — KernelDropped,
StacksEvicted, SequenceGaps, StacksMissing, SinkRejected — and
nothing printed them. gpuprofile.Profiler exposed no Stats() at all, so
the agent could not read them even in principle. A run that lost 90% of its
GPU work still reported exactly one line:

GPU profile written to out-gpu.pb.gz (1151 samples)

A sample count alone cannot distinguish a quiet GPU from a lossy capture.
This is the same shape as #109, where the debuginfod counters existed and
nothing printed them.

What was being lost

Measured on an RTX 3090, same PyTorch workload and window, with only
PERFAGENT_GPU_SAMPLE_PERIOD differing:

period records in kernel_dropped seq gaps lost
8 45,316 2,436 217 5.5%
1 8,507 57,353 18,181 90%

The adapter is not at fault. Run with PERFAGENT_GPU_LOG and no agent
attached it emits identically at both periods — launches=35404,
activity_kernels=35404, launch_batch_dropped=0, cupti_dropped=0 — and
the sampler is exact (4,406 sampled at period 8, all 35,404 at period 1).

The loss is the BPF event ringbuf: a compiled-in 4 MB at
bpf/gpu_usdt.bpf.c:168 with no override. What overflows is the CUPTI
kernel activity records, and those carry the GPU time itself — so lowering
the sampling period to buy attribution shrinks the profile instead of
merely coarsening it. That is the opposite of what the knob reads like it
does, and nothing in the output said so.

The change

  • gpuprofile.Profiler.Stats(), so the counters are reachable.
  • A gpu: LOST line naming each non-zero counter, and saying plainly that a
    smaller sampling period makes it worse.
  • --gpu-ring-bytes to size the buffer. Rounded up to a power-of-two page
    multiple and bounded, because the kernel rejects a bad max_entries with
    a bare EINVAL far from the knob that set it. roundRingBytes is
    mutation-tested — it fails on a no-op implementation.

Does the flag work?

Same workload and window, period=1, ring size the only difference:

ring records kernel_dropped seq gaps lost
4 MB (default) 3,645 41,149 13,340 93.7%
64 MB 16,119 0 0 6.9%

Both overflow counters go to exactly zero and 4.4× more records survive —
the signature of relieving ring overflow, not a general improvement.

One limit, stated rather than buried: it is a single pair. Four
confirmation runs returned either an empty capture or a workload that never
finished, on a GPU that had begun failing with cudaErrorLaunchTimeout, so
a second clean pair is owed once the machine is healthy.

What this does to attribution

Record loss depresses it, which is the point. With --profile alongside
--gpu:

gpuprobe.Stats carries the identity
StacksResolved = StacksAttached + StacksEvicted + StacksProfilerOnly + PendingStacks,
so the gap can be decomposed rather than guessed at:

period resolved attached evicted profiler-only + pending attached
8 2,621 2,618 0 3 99.9%
1 4,735 1,246 1,755 1,734 26.3%

At the default period almost every resolved stack finds its launch. At
period 1 the resolved stacks roughly third into attached, evicted, and
still-parked.

A parked stack is waiting for the kernel record it pairs with, and those
records are what the ringbuf was dropping — 57,353 of them in that run — so
both the evictions and the pending backlog are consistent with twins that
never arrived. Consistent, not proven: this PR does not measure the
recovery, and StacksProfilerOnly counts something different again (a
launch whose only stack is the adapter's own callback, which is deliberately
not attributed). The claim here is that attribution degrades alongside
record loss, not that this flag fixes attribution.

Default unchanged

The compiled-in 4 MB stays the default. At the default sampling period it
loses ~5%, and raising it costs locked memory; the point of this PR is that
the loss is now visible, and that an operator who lowers the period has a
knob to pay for it with.

gpuprobe.Stats already counted every kind of loss -- KernelDropped,
StacksEvicted, SequenceGaps, StacksMissing, SinkRejected -- and nothing
printed them. gpuprofile.Profiler exposed no Stats() at all, so the agent
could not read them even in principle. A capture that lost most of its GPU
work still reported only "GPU profile written to ... (N samples)", and N
alone cannot tell a quiet GPU from a lossy capture. Same shape as #109,
where the debuginfod counters existed and nothing printed them.

Measured on an RTX 3090, same PyTorch workload and 25s window, with only
PERFAGENT_GPU_SAMPLE_PERIOD differing:

  period=8   records=45316   lost=2653   (5.5%)
  period=1   records=8507    lost=77289  (90%)

kernel_dropped dominates: 2436 then 57353. The adapter is not at fault --
run with PERFAGENT_GPU_LOG and no agent attached it emits identically at
both periods (launches=35404 activity_kernels=35404 launch_batch_dropped=0
cupti_dropped=0) and the sampler is exact (4406 sampled at 8, all 35404 at
1). The loss is the BPF event ringbuf, a compiled-in 4 MB with no override.

What overflows is the CUPTI kernel activity records, and those carry the GPU
time itself -- so lowering the sampling period to buy attribution SHRINKS
the profile instead of merely coarsening it. That is the opposite of what
the knob reads like it does, and nothing in the output said so.

So: Profiler.Stats(), a "gpu: LOST" line naming each counter when any is
non-zero, and --gpu-ring-bytes to size the buffer. The size is rounded up to
a power-of-two page multiple and bounded, because the kernel rejects a bad
max_entries with a bare EINVAL far from the knob that set it.

Verified, same workload and window, period=1, ring size the only
difference:

  4 MB (default)   records=3645    kernel_dropped=41149  seq_gaps=13340
  64 MB            records=16119   kernel_dropped=0      seq_gaps=0

Both overflow counters go to exactly zero and 4.4x more records survive,
which is the signature of relieving ring overflow rather than a general
improvement.

One limit on that result: it is a single pair. Four confirmation runs
returned either an empty capture or a workload that never finished, on a
GPU that had started failing with cudaErrorLaunchTimeout, so n=2 is owed
once the machine is healthy.

Attribution is a fair question to ask of a change like this, and the
answer is that record loss depresses it. With --profile alongside --gpu,
period 8 attached 2618 of 2621 resolved stacks (99.9%) while period 1
attached 1246 of 4735 (26.3%) -- the kernel records a stack would attach
TO are exactly what the ringbuf was dropping. Recovering the records
should recover most of that, but this PR does not measure it, so it does
not claim it.
@dpsoft
dpsoft force-pushed the fix/gpu-silent-record-loss branch from 94fb461 to 08f0775 Compare October 6, 2026 14:33
@dpsoft
dpsoft merged commit ba6d85b into main Oct 6, 2026
16 checks passed
@dpsoft
dpsoft deleted the fix/gpu-silent-record-loss branch October 6, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant