Repository navigation
gpu: the agent never said when it dropped GPU records - #204
Merged
Merged
Conversation
gpuprobe.Stats already counted every kind of loss -- KernelDropped, StacksEvicted, SequenceGaps, StacksMissing, SinkRejected -- and nothing printed them. gpuprofile.Profiler exposed no Stats() at all, so the agent could not read them even in principle. A capture that lost most of its GPU work still reported only "GPU profile written to ... (N samples)", and N alone cannot tell a quiet GPU from a lossy capture. Same shape as #109, where the debuginfod counters existed and nothing printed them. Measured on an RTX 3090, same PyTorch workload and 25s window, with only PERFAGENT_GPU_SAMPLE_PERIOD differing: period=8 records=45316 lost=2653 (5.5%) period=1 records=8507 lost=77289 (90%) kernel_dropped dominates: 2436 then 57353. The adapter is not at fault -- run with PERFAGENT_GPU_LOG and no agent attached it emits identically at both periods (launches=35404 activity_kernels=35404 launch_batch_dropped=0 cupti_dropped=0) and the sampler is exact (4406 sampled at 8, all 35404 at 1). The loss is the BPF event ringbuf, a compiled-in 4 MB with no override. What overflows is the CUPTI kernel activity records, and those carry the GPU time itself -- so lowering the sampling period to buy attribution SHRINKS the profile instead of merely coarsening it. That is the opposite of what the knob reads like it does, and nothing in the output said so. So: Profiler.Stats(), a "gpu: LOST" line naming each counter when any is non-zero, and --gpu-ring-bytes to size the buffer. The size is rounded up to a power-of-two page multiple and bounded, because the kernel rejects a bad max_entries with a bare EINVAL far from the knob that set it. Verified, same workload and window, period=1, ring size the only difference: 4 MB (default) records=3645 kernel_dropped=41149 seq_gaps=13340 64 MB records=16119 kernel_dropped=0 seq_gaps=0 Both overflow counters go to exactly zero and 4.4x more records survive, which is the signature of relieving ring overflow rather than a general improvement. One limit on that result: it is a single pair. Four confirmation runs returned either an empty capture or a workload that never finished, on a GPU that had started failing with cudaErrorLaunchTimeout, so n=2 is owed once the machine is healthy. Attribution is a fair question to ask of a change like this, and the answer is that record loss depresses it. With --profile alongside --gpu, period 8 attached 2618 of 2621 resolved stacks (99.9%) while period 1 attached 1246 of 4735 (26.3%) -- the kernel records a stack would attach TO are exactly what the ringbuf was dropping. Recovering the records should recover most of that, but this PR does not measure it, so it does not claim it.
dpsoft
force-pushed
the
fix/gpu-silent-record-loss
branch
from
October 6, 2026 14:33
94fb461 to
08f0775
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A GPU capture could lose most of its data and say nothing about it.
The silence
gpuprobe.Statsalready counted every kind of loss —KernelDropped,StacksEvicted,SequenceGaps,StacksMissing,SinkRejected— andnothing printed them.
gpuprofile.Profilerexposed noStats()at all, sothe agent could not read them even in principle. A run that lost 90% of its
GPU work still reported exactly one line:
A sample count alone cannot distinguish a quiet GPU from a lossy capture.
This is the same shape as #109, where the debuginfod counters existed and
nothing printed them.
What was being lost
Measured on an RTX 3090, same PyTorch workload and window, with only
PERFAGENT_GPU_SAMPLE_PERIODdiffering:The adapter is not at fault. Run with
PERFAGENT_GPU_LOGand no agentattached it emits identically at both periods —
launches=35404,activity_kernels=35404,launch_batch_dropped=0,cupti_dropped=0— andthe sampler is exact (4,406 sampled at period 8, all 35,404 at period 1).
The loss is the BPF event ringbuf: a compiled-in 4 MB at
bpf/gpu_usdt.bpf.c:168with no override. What overflows is the CUPTIkernel activity records, and those carry the GPU time itself — so lowering
the sampling period to buy attribution shrinks the profile instead of
merely coarsening it. That is the opposite of what the knob reads like it
does, and nothing in the output said so.
The change
gpuprofile.Profiler.Stats(), so the counters are reachable.gpu: LOSTline naming each non-zero counter, and saying plainly that asmaller sampling period makes it worse.
--gpu-ring-bytesto size the buffer. Rounded up to a power-of-two pagemultiple and bounded, because the kernel rejects a bad
max_entrieswitha bare
EINVALfar from the knob that set it.roundRingBytesismutation-tested — it fails on a no-op implementation.
Does the flag work?
Same workload and window, period=1, ring size the only difference:
Both overflow counters go to exactly zero and 4.4× more records survive —
the signature of relieving ring overflow, not a general improvement.
One limit, stated rather than buried: it is a single pair. Four
confirmation runs returned either an empty capture or a workload that never
finished, on a GPU that had begun failing with
cudaErrorLaunchTimeout, soa second clean pair is owed once the machine is healthy.
What this does to attribution
Record loss depresses it, which is the point. With
--profilealongside--gpu:gpuprobe.Statscarries the identityStacksResolved = StacksAttached + StacksEvicted + StacksProfilerOnly + PendingStacks,so the gap can be decomposed rather than guessed at:
At the default period almost every resolved stack finds its launch. At
period 1 the resolved stacks roughly third into attached, evicted, and
still-parked.
A parked stack is waiting for the kernel record it pairs with, and those
records are what the ringbuf was dropping — 57,353 of them in that run — so
both the evictions and the pending backlog are consistent with twins that
never arrived. Consistent, not proven: this PR does not measure the
recovery, and
StacksProfilerOnlycounts something different again (alaunch whose only stack is the adapter's own callback, which is deliberately
not attributed). The claim here is that attribution degrades alongside
record loss, not that this flag fixes attribution.
Default unchanged
The compiled-in 4 MB stays the default. At the default sampling period it
loses ~5%, and raising it costs locked memory; the point of this PR is that
the loss is now visible, and that an operator who lowers the period has a
knob to pay for it with.