Skip to content

kprobe: don't exit when perf record gt 4096 bytes - #385

Open
nicholasberlin wants to merge 2 commits into
mainfrom
fix/perf-oversized-event-fatal-exit
Open

kprobe: don't exit when perf record gt 4096 bytes#385
nicholasberlin wants to merge 2 commits into
mainfrom
fix/perf-oversized-event-fatal-exit

Conversation

@nicholasberlin

Copy link
Copy Markdown
Contributor

Problem

perf_mmap_read() linearized a record that wraps the ring into a fixed wrapped_event_buf[4096] and errx(1)'d the whole process whenever header.size exceeded it. A single perf record can legally be bigger: header.size is a u16, and the sched_process_exec tracepoint records the full exec path (up to PATH_MAX) via PERF_SAMPLE_RAW. Execing a binary through a path longer than ~4KB therefore killed any libquark consumer on the kprobe backend:

quark-test: getting an event larger than wrapped buf

Fix

Replace the fixed per-cpu buffer with one heap buffer per queue of UINT16_MAX + 1 bytes: no record can exceed it by type, and the scratch is never live across two rings since each record is fully consumed before the next ring is read.

While here, harden the read path:

  • Snapshot header.size once instead of re-reading it from the shared writable mapping.
  • Reject corrupt records smaller than a header — consuming them would spin on the same offset forever.
  • static_assert that the ring can hold the largest possible record, so a corrupt data_head can't walk the wrap copy out of the mapping.
  • Drop the now-unused err.h.

Net effect is stronger than the old guard: even a fully corrupt data_head can no longer cause an out-of-bounds read or write — worst case is a garbage event, not heap corruption or a dead process.

Testing

New t_exec_long_path (kprobe) execs a copy of true through a PATH_MAX - 1 path and asserts the recorded exe matches byte-for-byte, proving the oversized record survives and is linearized intact.

  • Against the previous code, the test reproduces the fatal exit (getting an event larger than wrapped buf, exit 1).
  • With the fix, the full quark-test suite passes on both kprobe and ebpf backends (qemu, 6.8.0-65-generic).

@nicholasberlin
nicholasberlin requested a review from a team as a code owner August 12, 2026 20:32
@nicholasberlin

Copy link
Copy Markdown
Contributor Author

CI failures root-caused and fixed in e15a86b: kernels older than 5.16 cap perf tracepoint records at PERF_MAX_TRACE_SIZE (2048) and silently drop anything bigger, so the PATH_MAX exec record the test relies on never arrives (exe stays inherited from fork and the strcmp assert fired). Backports make version checks unreliable — Ubuntu's 5.4 drops the record while RHEL's 4.18.0-553 delivers it — so the test now probes the behavior: a mid-sized path below the cap is asserted strictly on every kernel, and the full PATH_MAX check applies only when the oversized record is actually delivered, skipping otherwise (a partial copy of the long path still fails, since that would be quark corrupting the record rather than the kernel dropping it).

Note this also means the fatal errx being removed here was unreachable on those older kernels — no record over 2048 bytes could be delivered — so skipping the strict check there matches reality.

Verified locally: 5.4.0-150-generic (skips), 4.18.0-553.el8_10 (strict), 6.8.0-65-generic (strict); full suite green on all three.

perf_mmap_read() linearized a record that wraps the ring into a fixed
wrapped_event_buf[4096] and errx(1)'d the whole process whenever
header.size exceeded it. A single perf record can legally be bigger:
header.size is a u16 and the sched_process_exec tracepoint records the
full exec path (up to PATH_MAX) via PERF_SAMPLE_RAW, so execing a binary
through a long path crashed any libquark consumer on the kprobe backend.

Replace the fixed per-cpu buffer with one heap buffer per queue of
UINT16_MAX + 1 bytes: no record can exceed it by type, and the scratch
is never live across two rings since each record is fully consumed
before the next ring is read. While here, harden the read path: snapshot
header.size instead of re-reading it from the shared ring, reject
corrupt records smaller than a header (consuming them would spin on the
same offset forever), and static_assert that the ring itself can hold
the largest possible record so a corrupt data_head can't walk the wrap
copy out of the mapping. Drop the now-unused err.h.

Add t_exec_long_path, which execs through a PATH_MAX - 1 path and checks
the recorded exe: it reproduces the old fatal exit ("getting an event
larger than wrapped buf") on the previous code and verifies the record
is linearized intact on the new one. Full quark-test suite passes on
both backends.
Kernels older than 5.16 cap a perf tracepoint record at
PERF_MAX_TRACE_SIZE(2048) and silently drop anything bigger, so the
PATH_MAX exec record t_exec_long_path relies on never arrives: exe stays
inherited from the fork and the strcmp assert fails. Ubuntu's 5.4 drops
it while RHEL's 4.18 delivers it, so probe the behavior instead of the
kernel version: exec through a mid-sized path below the cap first, which
must work strictly everywhere, then only require the full PATH_MAX exe
when the record actually arrived, skipping otherwise. A partial copy of
the long path still fails, that would be quark corrupting the record
rather than the kernel dropping it.

Verified on 5.4.0-150-generic (skips), 4.18.0-553.el8_10 (strict) and
6.8.0-65-generic (strict), full suite green on all three.
@nicholasberlin
nicholasberlin force-pushed the fix/perf-oversized-event-fatal-exit branch from e15a86b to 1429daa Compare August 13, 2026 13:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant