Add generic stage-aware perf hook infrastructure (#683)#683
Open
charles-typ wants to merge 1 commit into
Open
Conversation
meta-cla
Bot
added
the
CLA Signed
|
@charles-typ has exported this pull request. If you are a Meta employee, you can view the originating Diff in D108110315. |
meta-codesync
Bot
changed the title
charles-typ
added a commit
to charles-typ/DCPerf
that referenced
this pull request
Jul 8, 2026
Summary:
The WDL bench `prod_set` job runs ~25 individual sub-benchmarks
(memcpy_benchmark, openssl, lzbench, ProtocolBench, ...) back-to-back in a
single benchpress run. Today the perf hook spans the whole run, so PMU +
sysstat data ends up smeared across all sub-benchmarks in one set of CSVs.
Distinguishing IPC, topdown breakdown, mpstat etc. per sub-benchmark
requires teasing apart timestamps after the fact, which is brittle.
This change adds an opt-in stage-aware mode to the perf hook so each
sub-benchmark gets its own folder of perf data:
benchmark_metrics_<uuid>/
memcpy_benchmark/
mpstat.csv
mem-stat.csv
perf-stat.csv
topdown-... .csv
(etc -- one set per enabled perf monitor)
hash_hash_benchmark/
...
...
The mechanism:
1. The perf hook accepts a new option `stage_aware: true`. When set,
`before_job` does NOT start any monitors. Instead it creates a FIFO
under benchmark_metrics_<uuid>/perf_stage.fifo, advertises its path
via the env var BENCHPRESS_PERF_STAGE_FIFO, and spawns a
coordinator thread.
2. The coordinator reads commands from the FIFO. Each
"START <stage_name>" allocates a fresh set of perf monitors with
that stage name as a sub-folder; each "STOP" terminates the
monitors and writes their CSVs. Multiple START/STOP cycles are
supported. `after_job` writes a final __EXIT__ to drain the
coordinator.
3. Every existing perf monitor (mpstat, memstat, netstat, perfstat,
vmstat, cpufreq*, power, topdown -- including IntelPerfSpect/3,
BasePerfUtil, AMDPerfUtil, ARMPerfUtil, NVPerfUtil,
NeoVerseV3PerfUtil) gains a `subdir` constructor arg that the base
`Monitor.gen_path` joins under benchmark_metrics_<uuid>/. Existing
callers that don't pass `subdir` keep their flat layout, so default
mode is byte-for-byte unchanged.
4. `packages/wdl_bench/run_prod.sh` emits "START <bench>" before each
sub-benchmark and "STOP" after. The emit is gated on the
BENCHPRESS_PERF_STAGE_FIFO env var, so running prod_set without
stage-aware mode (or without the perf hook at all) is unchanged.
5. A new job entry `prod_set_per_stage_perf` in jobs_wdl.yml
documents the wiring; it has the same shape as `prod_set`. To
activate, run:
./benchpress run prod_set_per_stage_perf \\
-k perf -a '{"perf": {"stage_aware": true}}'
Differential Revision: D108110315
charles-typ
force-pushed
the
export-D108110315-to-v2-beta
branch
from
c0c3410 to
0f91b6c
Compare
charles-typ
added a commit
to charles-typ/DCPerf
that referenced
this pull request
Jul 8, 2026
Summary:
The WDL bench `prod_set` job runs ~25 individual sub-benchmarks
(memcpy_benchmark, openssl, lzbench, ProtocolBench, ...) back-to-back in a
single benchpress run. Today the perf hook spans the whole run, so PMU +
sysstat data ends up smeared across all sub-benchmarks in one set of CSVs.
Distinguishing IPC, topdown breakdown, mpstat etc. per sub-benchmark
requires teasing apart timestamps after the fact, which is brittle.
This change adds an opt-in stage-aware mode to the perf hook so each
sub-benchmark gets its own folder of perf data:
benchmark_metrics_<uuid>/
memcpy_benchmark/
mpstat.csv
mem-stat.csv
perf-stat.csv
topdown-... .csv
(etc -- one set per enabled perf monitor)
hash_hash_benchmark/
...
...
The mechanism:
1. The perf hook accepts a new option `stage_aware: true`. When set,
`before_job` does NOT start any monitors. Instead it creates a FIFO
under benchmark_metrics_<uuid>/perf_stage.fifo, advertises its path
via the env var BENCHPRESS_PERF_STAGE_FIFO, and spawns a
coordinator thread.
2. The coordinator reads commands from the FIFO. Each
"START <stage_name>" allocates a fresh set of perf monitors with
that stage name as a sub-folder; each "STOP" terminates the
monitors and writes their CSVs. Multiple START/STOP cycles are
supported. `after_job` writes a final __EXIT__ to drain the
coordinator.
3. Every existing perf monitor (mpstat, memstat, netstat, perfstat,
vmstat, cpufreq*, power, topdown -- including IntelPerfSpect/3,
BasePerfUtil, AMDPerfUtil, ARMPerfUtil, NVPerfUtil,
NeoVerseV3PerfUtil) gains a `subdir` constructor arg that the base
`Monitor.gen_path` joins under benchmark_metrics_<uuid>/. Existing
callers that don't pass `subdir` keep their flat layout, so default
mode is byte-for-byte unchanged.
4. `packages/wdl_bench/run_prod.sh` emits "START <bench>" before each
sub-benchmark and "STOP" after. The emit is gated on the
BENCHPRESS_PERF_STAGE_FIFO env var, so running prod_set without
stage-aware mode (or without the perf hook at all) is unchanged.
5. A new job entry `prod_set_per_stage_perf` in jobs_wdl.yml
documents the wiring; it has the same shape as `prod_set`. To
activate, run:
./benchpress run prod_set_per_stage_perf \\
-k perf -a '{"perf": {"stage_aware": true}}'
Differential Revision: D108110315
charles-typ
added a commit
to charles-typ/DCPerf
that referenced
this pull request
Jul 8, 2026
Summary:
The WDL bench `prod_set` job runs ~25 individual sub-benchmarks
(memcpy_benchmark, openssl, lzbench, ProtocolBench, ...) back-to-back in a
single benchpress run. Today the perf hook spans the whole run, so PMU +
sysstat data ends up smeared across all sub-benchmarks in one set of CSVs.
Distinguishing IPC, topdown breakdown, mpstat etc. per sub-benchmark
requires teasing apart timestamps after the fact, which is brittle.
This change adds an opt-in stage-aware mode to the perf hook so each
sub-benchmark gets its own folder of perf data:
benchmark_metrics_<uuid>/
memcpy_benchmark/
mpstat.csv
mem-stat.csv
perf-stat.csv
topdown-... .csv
(etc -- one set per enabled perf monitor)
hash_hash_benchmark/
...
...
The mechanism:
1. The perf hook accepts a new option `stage_aware: true`. When set,
`before_job` does NOT start any monitors. Instead it creates a FIFO
under benchmark_metrics_<uuid>/perf_stage.fifo, advertises its path
via the env var BENCHPRESS_PERF_STAGE_FIFO, and spawns a
coordinator thread.
2. The coordinator reads commands from the FIFO. Each
"START <stage_name>" allocates a fresh set of perf monitors with
that stage name as a sub-folder; each "STOP" terminates the
monitors and writes their CSVs. Multiple START/STOP cycles are
supported. `after_job` writes a final __EXIT__ to drain the
coordinator.
3. Every existing perf monitor (mpstat, memstat, netstat, perfstat,
vmstat, cpufreq*, power, topdown -- including IntelPerfSpect/3,
BasePerfUtil, AMDPerfUtil, ARMPerfUtil, NVPerfUtil,
NeoVerseV3PerfUtil) gains a `subdir` constructor arg that the base
`Monitor.gen_path` joins under benchmark_metrics_<uuid>/. Existing
callers that don't pass `subdir` keep their flat layout, so default
mode is byte-for-byte unchanged.
4. `packages/wdl_bench/run_prod.sh` emits "START <bench>" before each
sub-benchmark and "STOP" after. The emit is gated on the
BENCHPRESS_PERF_STAGE_FIFO env var, so running prod_set without
stage-aware mode (or without the perf hook at all) is unchanged.
5. A new job entry `prod_set_per_stage_perf` in jobs_wdl.yml
documents the wiring; it has the same shape as `prod_set`. To
activate, run:
./benchpress run prod_set_per_stage_perf \\
-k perf -a '{"perf": {"stage_aware": true}}'
Differential Revision: D108110315
meta-codesync
Bot
changed the title
charles-typ
force-pushed
the
export-D108110315-to-v2-beta
branch
from
0f91b6c to
b4a8596
Compare
charles-typ
added a commit
to charles-typ/DCPerf
that referenced
this pull request
Jul 8, 2026
Summary:
Benchpress's perf hook spans an entire benchmark run today, so PMU + sysstat
data ends up smeared across every sub-benchmark in one set of CSVs. For
benchmarks that run many workloads back-to-back (WDL prod_set, SPEC2017
intrate, ...), distinguishing IPC, topdown breakdown, mpstat etc. per
sub-benchmark requires teasing apart timestamps after the fact, which is
brittle.
This diff adds a generic, opt-in stage-aware mode to the perf hook. It is not
tied to any specific benchmark -- consumers (WDL, SPEC, ...) land on top of
this and just emit stage markers.
The mechanism:
1. The perf hook accepts a new option `stage_aware: true`. When set,
`before_job` does NOT start any monitors. Instead it creates a FIFO
under benchmark_metrics_<uuid>/perf_stage.fifo, advertises its path
via the env var BENCHPRESS_PERF_STAGE_FIFO, and spawns a
coordinator thread.
2. The coordinator reads commands from the FIFO. Each
"START <stage_name>" allocates a fresh set of perf monitors with
that stage name as a sub-folder; each "STOP" terminates the
monitors and writes their CSVs. Multiple START/STOP cycles are
supported. `after_job` writes a final __EXIT__ to drain the
coordinator.
3. Every existing perf monitor (mpstat, memstat, netstat, perfstat,
vmstat, cpufreq*, power, topdown -- including IntelPerfSpect/3,
BasePerfUtil, AMDPerfUtil, ARMPerfUtil, NVPerfUtil,
NeoVerseV3PerfUtil) gains a `subdir` constructor arg that the base
`Monitor.gen_path` joins under benchmark_metrics_<uuid>/. Existing
callers that don't pass `subdir` keep their flat layout, so default
mode is byte-for-byte unchanged.
The resulting per-stage layout, e.g.:
benchmark_metrics_<uuid>/
<stage_name>/
mpstat.csv
mem-stat.csv
perf-stat.csv
topdown-... .csv
Benchmark-specific wiring (WDL prod_set, SPEC2017 intrate) lands in later
diffs in this stack.
Differential Revision: D108110315
meta-codesync
Bot
changed the title
charles-typ
added a commit
to charles-typ/DCPerf
that referenced
this pull request
Jul 9, 2026
Summary:
Benchpress's perf hook spans an entire benchmark run today, so PMU + sysstat
data ends up smeared across every sub-benchmark in one set of CSVs. For
benchmarks that run many workloads back-to-back (WDL prod_set, SPEC2017
intrate, ...), distinguishing IPC, topdown breakdown, mpstat etc. per
sub-benchmark requires teasing apart timestamps after the fact, which is
brittle.
This diff adds a generic, opt-in stage-aware mode to the perf hook. It is not
tied to any specific benchmark -- consumers (WDL, SPEC, ...) land on top of
this and just emit stage markers.
The mechanism:
1. The perf hook accepts a new option `stage_aware: true`. When set,
`before_job` does NOT start any monitors. Instead it creates a FIFO
under benchmark_metrics_<uuid>/perf_stage.fifo, advertises its path
via the env var BENCHPRESS_PERF_STAGE_FIFO, and spawns a
coordinator thread.
2. The coordinator reads commands from the FIFO. Each
"START <stage_name>" allocates a fresh set of perf monitors with
that stage name as a sub-folder; each "STOP" terminates the
monitors and writes their CSVs. Multiple START/STOP cycles are
supported. `after_job` writes a final __EXIT__ to drain the
coordinator.
3. Every existing perf monitor (mpstat, memstat, netstat, perfstat,
vmstat, cpufreq*, power, topdown -- including IntelPerfSpect/3,
BasePerfUtil, AMDPerfUtil, ARMPerfUtil, NVPerfUtil,
NeoVerseV3PerfUtil) gains a `subdir` constructor arg that the base
`Monitor.gen_path` joins under benchmark_metrics_<uuid>/. Existing
callers that don't pass `subdir` keep their flat layout, so default
mode is byte-for-byte unchanged.
The resulting per-stage layout, e.g.:
benchmark_metrics_<uuid>/
<stage_name>/
mpstat.csv
mem-stat.csv
perf-stat.csv
topdown-... .csv
Benchmark-specific wiring (WDL prod_set, SPEC2017 intrate) lands in later
diffs in this stack.
Reviewed By: YifanYuan3
Differential Revision: D108110315
charles-typ
force-pushed
the
export-D108110315-to-v2-beta
branch
from
b4a8596 to
66e91a1
Compare
charles-typ
added a commit
to charles-typ/DCPerf
that referenced
this pull request
Jul 9, 2026
Summary:
Benchpress's perf hook spans an entire benchmark run today, so PMU + sysstat
data ends up smeared across every sub-benchmark in one set of CSVs. For
benchmarks that run many workloads back-to-back (WDL prod_set, SPEC2017
intrate, ...), distinguishing IPC, topdown breakdown, mpstat etc. per
sub-benchmark requires teasing apart timestamps after the fact, which is
brittle.
This diff adds a generic, opt-in stage-aware mode to the perf hook. It is not
tied to any specific benchmark -- consumers (WDL, SPEC, ...) land on top of
this and just emit stage markers.
The mechanism:
1. The perf hook accepts a new option `stage_aware: true`. When set,
`before_job` does NOT start any monitors. Instead it creates a FIFO
under benchmark_metrics_<uuid>/perf_stage.fifo, advertises its path
via the env var BENCHPRESS_PERF_STAGE_FIFO, and spawns a
coordinator thread.
2. The coordinator reads commands from the FIFO. Each
"START <stage_name>" allocates a fresh set of perf monitors with
that stage name as a sub-folder; each "STOP" terminates the
monitors and writes their CSVs. Multiple START/STOP cycles are
supported. `after_job` writes a final __EXIT__ to drain the
coordinator.
3. Every existing perf monitor (mpstat, memstat, netstat, perfstat,
vmstat, cpufreq*, power, topdown -- including IntelPerfSpect/3,
BasePerfUtil, AMDPerfUtil, ARMPerfUtil, NVPerfUtil,
NeoVerseV3PerfUtil) gains a `subdir` constructor arg that the base
`Monitor.gen_path` joins under benchmark_metrics_<uuid>/. Existing
callers that don't pass `subdir` keep their flat layout, so default
mode is byte-for-byte unchanged.
The resulting per-stage layout, e.g.:
benchmark_metrics_<uuid>/
<stage_name>/
mpstat.csv
mem-stat.csv
perf-stat.csv
topdown-... .csv
Benchmark-specific wiring (WDL prod_set, SPEC2017 intrate) lands in later
diffs in this stack.
Reviewed By: YifanYuan3
Differential Revision: D108110315
charles-typ
force-pushed
the
export-D108110315-to-v2-beta
branch
from
66e91a1 to
45e225f
Compare
charles-typ
added a commit
to charles-typ/DCPerf
that referenced
this pull request
Jul 9, 2026
Summary:
Benchpress's perf hook spans an entire benchmark run today, so PMU + sysstat
data ends up smeared across every sub-benchmark in one set of CSVs. For
benchmarks that run many workloads back-to-back (WDL prod_set, SPEC2017
intrate, ...), distinguishing IPC, topdown breakdown, mpstat etc. per
sub-benchmark requires teasing apart timestamps after the fact, which is
brittle.
This diff adds a generic, opt-in stage-aware mode to the perf hook. It is not
tied to any specific benchmark -- consumers (WDL, SPEC, ...) land on top of
this and just emit stage markers.
The mechanism:
1. The perf hook accepts a new option `stage_aware: true`. When set,
`before_job` does NOT start any monitors. Instead it creates a FIFO
under benchmark_metrics_<uuid>/perf_stage.fifo, advertises its path
via the env var BENCHPRESS_PERF_STAGE_FIFO, and spawns a
coordinator thread.
2. The coordinator reads commands from the FIFO. Each
"START <stage_name>" allocates a fresh set of perf monitors with
that stage name as a sub-folder; each "STOP" terminates the
monitors and writes their CSVs. Multiple START/STOP cycles are
supported. `after_job` writes a final __EXIT__ to drain the
coordinator.
3. Every existing perf monitor (mpstat, memstat, netstat, perfstat,
vmstat, cpufreq*, power, topdown -- including IntelPerfSpect/3,
BasePerfUtil, AMDPerfUtil, ARMPerfUtil, NVPerfUtil,
NeoVerseV3PerfUtil) gains a `subdir` constructor arg that the base
`Monitor.gen_path` joins under benchmark_metrics_<uuid>/. Existing
callers that don't pass `subdir` keep their flat layout, so default
mode is byte-for-byte unchanged.
The resulting per-stage layout, e.g.:
benchmark_metrics_<uuid>/
<stage_name>/
mpstat.csv
mem-stat.csv
perf-stat.csv
topdown-... .csv
Benchmark-specific wiring (WDL prod_set, SPEC2017 intrate) lands in later
diffs in this stack.
Reviewed By: YifanYuan3
Differential Revision: D108110315
Summary:
Benchpress's perf hook spans an entire benchmark run today, so PMU + sysstat
data ends up smeared across every sub-benchmark in one set of CSVs. For
benchmarks that run many workloads back-to-back (WDL prod_set, SPEC2017
intrate, ...), distinguishing IPC, topdown breakdown, mpstat etc. per
sub-benchmark requires teasing apart timestamps after the fact, which is
brittle.
This diff adds a generic, opt-in stage-aware mode to the perf hook. It is not
tied to any specific benchmark -- consumers (WDL, SPEC, ...) land on top of
this and just emit stage markers.
The mechanism:
1. The perf hook accepts a new option `stage_aware: true`. When set,
`before_job` does NOT start any monitors. Instead it creates a FIFO
under benchmark_metrics_<uuid>/perf_stage.fifo, advertises its path
via the env var BENCHPRESS_PERF_STAGE_FIFO, and spawns a
coordinator thread.
2. The coordinator reads commands from the FIFO. Each
"START <stage_name>" allocates a fresh set of perf monitors with
that stage name as a sub-folder; each "STOP" terminates the
monitors and writes their CSVs. Multiple START/STOP cycles are
supported. `after_job` writes a final __EXIT__ to drain the
coordinator.
3. Every existing perf monitor (mpstat, memstat, netstat, perfstat,
vmstat, cpufreq*, power, topdown -- including IntelPerfSpect/3,
BasePerfUtil, AMDPerfUtil, ARMPerfUtil, NVPerfUtil,
NeoVerseV3PerfUtil) gains a `subdir` constructor arg that the base
`Monitor.gen_path` joins under benchmark_metrics_<uuid>/. Existing
callers that don't pass `subdir` keep their flat layout, so default
mode is byte-for-byte unchanged.
The resulting per-stage layout, e.g.:
benchmark_metrics_<uuid>/
<stage_name>/
mpstat.csv
mem-stat.csv
perf-stat.csv
topdown-... .csv
Benchmark-specific wiring (WDL prod_set, SPEC2017 intrate) lands in later
diffs in this stack.
Reviewed By: YifanYuan3
Differential Revision: D108110315
charles-typ
force-pushed
the
export-D108110315-to-v2-beta
branch
from
45e225f to
52d755d
Compare
meta-codesync Bot
pushed a commit
that referenced
this pull request
Jul 9, 2026
Summary: Pull Request resolved: #683 Benchpress's perf hook spans an entire benchmark run today, so PMU + sysstat data ends up smeared across every sub-benchmark in one set of CSVs. For benchmarks that run many workloads back-to-back (WDL prod_set, SPEC2017 intrate, ...), distinguishing IPC, topdown breakdown, mpstat etc. per sub-benchmark requires teasing apart timestamps after the fact, which is brittle. This diff adds a generic, opt-in stage-aware mode to the perf hook. It is not tied to any specific benchmark -- consumers (WDL, SPEC, ...) land on top of this and just emit stage markers. The mechanism: 1. The perf hook accepts a new option `stage_aware: true`. When set, `before_job` does NOT start any monitors. Instead it creates a FIFO under benchmark_metrics_<uuid>/perf_stage.fifo, advertises its path via the env var BENCHPRESS_PERF_STAGE_FIFO, and spawns a coordinator thread. 2. The coordinator reads commands from the FIFO. Each "START <stage_name>" allocates a fresh set of perf monitors with that stage name as a sub-folder; each "STOP" terminates the monitors and writes their CSVs. Multiple START/STOP cycles are supported. `after_job` writes a final __EXIT__ to drain the coordinator. 3. Every existing perf monitor (mpstat, memstat, netstat, perfstat, vmstat, cpufreq*, power, topdown -- including IntelPerfSpect/3, BasePerfUtil, AMDPerfUtil, ARMPerfUtil, NVPerfUtil, NeoVerseV3PerfUtil) gains a `subdir` constructor arg that the base `Monitor.gen_path` joins under benchmark_metrics_<uuid>/. Existing callers that don't pass `subdir` keep their flat layout, so default mode is byte-for-byte unchanged. The resulting per-stage layout, e.g.: benchmark_metrics_<uuid>/ <stage_name>/ mpstat.csv mem-stat.csv perf-stat.csv topdown-... .csv Benchmark-specific wiring (WDL prod_set, SPEC2017 intrate) lands in later diffs in this stack. Reviewed By: YifanYuan3 Differential Revision: D108110315 fbshipit-source-id: 0885ec912cc8b91359900b187ef3df83030bf9a1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Benchpress's perf hook spans an entire benchmark run today, so PMU + sysstat
data ends up smeared across every sub-benchmark in one set of CSVs. For
benchmarks that run many workloads back-to-back (WDL prod_set, SPEC2017
intrate, ...), distinguishing IPC, topdown breakdown, mpstat etc. per
sub-benchmark requires teasing apart timestamps after the fact, which is
brittle.
This diff adds a generic, opt-in stage-aware mode to the perf hook. It is not
tied to any specific benchmark -- consumers (WDL, SPEC, ...) land on top of
this and just emit stage markers.
The mechanism:
The perf hook accepts a new option
stage_aware: true. When set,before_jobdoes NOT start any monitors. Instead it creates a FIFOunder benchmark_metrics_/perf_stage.fifo, advertises its path
via the env var BENCHPRESS_PERF_STAGE_FIFO, and spawns a
coordinator thread.
The coordinator reads commands from the FIFO. Each
"START <stage_name>" allocates a fresh set of perf monitors with
that stage name as a sub-folder; each "STOP" terminates the
monitors and writes their CSVs. Multiple START/STOP cycles are
supported.
after_jobwrites a final EXIT to drain thecoordinator.
Every existing perf monitor (mpstat, memstat, netstat, perfstat,
vmstat, cpufreq*, power, topdown -- including IntelPerfSpect/3,
BasePerfUtil, AMDPerfUtil, ARMPerfUtil, NVPerfUtil,
NeoVerseV3PerfUtil) gains a
subdirconstructor arg that the baseMonitor.gen_pathjoins under benchmark_metrics_/. Existingcallers that don't pass
subdirkeep their flat layout, so defaultmode is byte-for-byte unchanged.
The resulting per-stage layout, e.g.:
benchmark_metrics_/
<stage_name>/
mpstat.csv
mem-stat.csv
perf-stat.csv
topdown-... .csv
Benchmark-specific wiring (WDL prod_set, SPEC2017 intrate) lands in later
diffs in this stack.
Reviewed By: YifanYuan3
Differential Revision: D108110315