Skip to content

docs: lead the README with GPU, and correct three stale claims - #202

Merged
dpsoft merged 7 commits into
mainfrom
docs/readme-gpu-first
Oct 6, 2026
Merged

dpsoft merged 7 commits into
mainfrom
docs/readme-gpu-first

Conversation

@dpsoft

@dpsoft dpsoft commented Oct 6, 2026

Copy link
Copy Markdown
Owner

The README described a profiler that no longer exists in two places, and
omitted a capability it has had since the CUPTI adapter landed.

What was wrong

  • Python frames. The Cross-language section said the BPF frame-chain
    walker was "not built yet" behind python: walk the interpreter frame chain from BPF instead of injecting perf trampolines #83 and that perf-agent had "no
    Python-level frames of its own". It landed: CPython 3.12-3.14 walk, Fedora's
    PGO-split libpython3.14 included. The Usage section repeated the stale
    claim a second time, pointing readers at python -X perf as the only path.

  • GPU. Nothing above the Architecture section mentioned GPU profiling at
    all. It is the most distinctive thing the tool does and it was invisible to
    anyone reading the front page.

  • The tagline listed CPU, off-CPU and PMU only.

What this changes

GPU is now the first entry under What you can do, with a real capture above
it: a live PyTorch step from pa_train_step (torch_workload.py:76) under
Thread.run, through torch::autograd and cublasSgemm_v2, across the
[gpu:launch] boundary into the CUDA kernel — Python, C++ and GPU in one
tree. The interactive HTML is checked in beside it.

The GPU example documents both steps rather than implying one command:
--gpu writes its own profile alongside --profile/--offcpu, and
cmd/flamegraph -fuse joins them. Both constraints that otherwise capture
nothing silently are stated — injection must precede cuInit, and
--gpu-shim must be the inode the target actually mapped.

Python's entry now says what is true, including the per-PID enrolment limit
(#194): a --pid capture walks Python, a system-wide -a one does not.

Notes

  • No version bump. The release notes live separately.
  • Everything load-bearing was checked against source, not memory: --gpu and
    --gpu-shim at main.go:37-45, the -fuse/-in syntax from
    cmd/flamegraph/main.go:13, the adapter filename from shim/Makefile:7,
    the 3.12-3.14 range from pyunwind/attach_live_test.go:169. All six
    frame names in the caption were grepped out of the committed HTML, and all
    10 local links resolve.
  • The hero PNG is the default sampling period (8), so most GPU time sits
    under [gpu:launch unsampled] with no CPU caller. That is what period 8
    predicts, and the interactive page states it in its own banner. A denser
    picture would mean shipping a non-default configuration as the hero.
  • docs/flamegraph-pytorch-gpu.html is 600 KB — self-contained by design.
    Happy to host it externally instead if you would rather not carry it.

dpsoft added 7 commits October 5, 2026 21:49
The README described a profiler that no longer exists in two places and a
capability it now has in a third.

- Python frames: the Cross-language section said the BPF frame-chain walker
  was "not built yet" behind #83, and Usage repeated it. It landed; CPython
  3.12-3.14 walk, Fedora's PGO-split libpython included. Replaced with what
  is true today, including the per-PID enrolment limit (#194).

- GPU: nothing above the Architecture section said perf-agent profiles GPUs
  at all. It is now the first thing under "What you can do", with a real
  capture of a PyTorch step -- Python frames through torch::autograd and
  cublasSgemm_v2, across [gpu:launch] into the kernel -- and the two
  commands that produce it.

- The tagline listed CPU, off-CPU and PMU only.

The GPU example documents both steps rather than implying one: --gpu writes
its own profile alongside --profile/--offcpu, and cmd/flamegraph -fuse joins
them. Both constraints that silently capture nothing are stated -- injection
must precede cuInit, and --gpu-shim must be the inode the target mapped.
Two things the front page got wrong, both reported from reading it.

The flame graph sat directly under Quickstart, whose snippet captures a CPU
profile and inspects it with `go tool pprof`. That command cannot produce
that picture, so the first thing a reader saw was a command and an unrelated
result. The image now follows the two GPU commands that actually produce it,
and its caption says so.

The architecture diagram predated the GPU pipeline and the Python walker. It
showed neither, and its OUTPUT box listed only on-CPU, off-CPU and PMU --
omitting GPU profiles and the HTML flame graph entirely. Redrawn with the
target process (where the CUPTI adapter is loaded at cuInit), gpu_usdt.bpf.c
and pywalk_probe.bpf.c on the kernel side, and gpuprobe.Consumer,
gpu.Timeline and gpu.ModuleStore on the user-space side, which is where the
kernel-to-launching-stack join on CUPTI's correlation id happens.
A flame built from profile bars: row widths taper and their centres drift
right as they climb, so the silhouette leans like a flame while every bar
stays a legible stack frame. Colour climbs application -> CPU/GPU boundary
-> GPU kernel using the flame graph's own hues (335 / 28 / 184 / 128), so
the mark and the product it names are coloured by the same rule.

Four files, because one drawing does not survive every size:

  perf-agent-mark.svg      the mark, good down to ~32px
  perf-agent-favicon.svg   redrawn at 16px, three rows -- NOT a scaled copy;
                           the full mark loses its top rows below ~24px
  perf-agent-lockup.svg    mark + wordmark, with a dark-scheme text colour
  perf-agent-tile.svg      the mark inset on a dark ground, for avatars

The mark itself carries no text, so it needs no light/dark variant; only the
lockup's wordmark switches, via prefers-color-scheme.
The logo was sitting as a bare mark stacked above an `# perf-agent`
heading, which is both weaker than the lockup and says the name twice. The
lockup is the artwork drawn for exactly this position, so it replaces both:
mark and wordmark set together, at the proportions they were spaced for.

The H1 goes with it rather than repeating the wordmark underneath. GitHub
already titles the page with the repository name, and the lockup carries an
alt of "perf-agent" so the name is still read aloud and still indexed.
The ASCII diagram had grown past what box-drawing characters can carry: a
hundred-odd lines, arrows that had to be read twice to see where they
started, and no room to show the maps without crowding the programs. It also
rendered at the reader's terminal font, so the alignment it depended on was
not guaranteed.

Same content, drawn properly, with three things the ASCII version got wrong
or omitted:

  - cpu.bpf.c and perf.bpf.c no longer appear to tail-call into the
    interpreter walker. They cannot: one reads counters and the other
    aggregates FP stacks in the kernel. Only the programs that walk a user
    stack hand off.

  - the GPU path has its own events ringbuf, and it is now drawn. It was
    missing entirely, which mattered: that buffer is where GPU records are
    lost when the sampling period is lowered.

  - gpuprobe.Consumer reads that ringbuf, not the FP aggregated maps. The
    old arrow implied otherwise.

The SVG carries its own light and dark palettes via prefers-color-scheme, so
it reads on both GitHub themes, and an alt description for anyone who cannot
see it.
456 lines, 20 KB. Most of it was not unique content — it was a second,
shorter copy of files that already exist and are far more thorough.

  README "Building"  25 lines   BUILDING.md is 235
  README "Testing"   22 lines   TESTING.md  is 342

Two copies of the same instructions means one of them is wrong eventually,
and the README copy is the one strangers read first. Those two sections are
deleted rather than moved; the real files are linked instead.

The rest moves to docs/, one file per topic, next to the notes that were
already there (gpu-injection, cpython-frame-walking, debuginfod-symbolization,
perf-data-output):

  docs/usage.md          Usage + Flags, with a real GPU section
  docs/requirements.md   kernel, capabilities, distro notes
  docs/output.md         file naming, pprof fidelity, PMU format
  docs/use-cases.md      the seven capability sections in full
  docs/library-usage.md  embedding the agent in Go
  docs/architecture.md   the diagram and what it means

What is left is 87 lines: the mark, one line of what this is, the capture
that proves it, six one-line capabilities each pointing at its own document,
a quickstart that fits on a screen, and a table of where everything went.

The table of contents went too. It listed fourteen sections of a page you
now scroll in a few seconds, and GitHub renders an outline button anyway.

Usage gained a GPU section it did not have: the two commands, both failure
modes that otherwise capture nothing silently (injection must precede
cuInit; --gpu-shim must be the inode the target mapped), and a pointer to
what lowering PERFAGENT_GPU_SAMPLE_PERIOD costs. Every link in the README
and in all six new documents was checked to resolve.
The hero image was screenshotted at a 1060px viewport against a page whose
graph is 1285px tall, so the bottom four rows were sliced off: start_thread,
the wide [gpu:launch unsampled] bar, and the CPU and GPU root bars.

Those rows are what make the rest legible. Without the roots there is
nothing anchoring the two halves, and without the unsampled bar the 88% of
GPU time it represents is simply absent -- so the middle of the picture read
as an unexplained empty hole rather than as the shape of the data.

Recaptured from the same committed HTML at full height. No new profiling
run: same capture, same numbers (42.746 s, 51,340 samples, 204 stacks), just
the whole of it.
@dpsoft
dpsoft merged commit 65405d9 into main Oct 6, 2026
16 checks passed
@dpsoft
dpsoft deleted the docs/readme-gpu-first branch October 6, 2026 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant