Emit per-stage overall metrics for Manifold (#684)#684
Open
charles-typ wants to merge 2 commits into
Open
Conversation
meta-cla
Bot
added
the
CLA Signed
|
@charles-typ has exported this pull request. If you are a Meta employee, you can view the originating Diff in D108195608. |
meta-codesync
Bot
changed the title
charles-typ
added a commit
to charles-typ/DCPerf
that referenced
this pull request
Jul 8, 2026
Summary:
D108110315 teaches the benchpress perf hook to collect PMU + sysstat data in
per-WDL-sub-benchmark stage directories for WDL prod_set. The raw staged CSVs
are preserved by perfpub's recursive Manifold upload, but users still need a
single processed summary per sub-benchmark, analogous to the normal
`overall-metrics.csv` that perfpub emits for a single benchmark.
This change adds stage-aware summary generation to perfpub:
benchmark_metrics_<run_id>/
memcpy_benchmark/
overall-metrics.csv # NEW
perf-stat.csv
nv-perf-collector-summary.csv
...
hash_hash_benchmark/
overall-metrics.csv # NEW
...
wdl_stage_overall_metrics.csv # NEW aggregate index
wdl_stage_overall_metrics.json # NEW aggregate index
wdl_stage_perf_summary.csv # generic numeric summary from prior patch
wdl_stage_perf_summary.json
Implementation details:
- For each immediate stage subdir that contains CSVs, perfpub temporarily
chdirs into that subdir and reuses the same reader functions as the top-level
path (`read_mpstat`, `read_memstat`, `read_cpufreq_*`, `read_perfstat`,
`read_nv_perf_collector`, `read_arm_perf_collector`, etc.). This gives each
stage the same processed metric lines as a normal single benchmark's
`overall-metrics.csv`.
- Adds the WDL sub-benchmark score at the top when the score exists in the
parent benchmark metrics JSON.
- Writes per-stage `overall-metrics.csv`, plus top-level CSV/JSON indices for
easy discovery in Manifold.
- No XDB/dashboard changes. The goal is Manifold artifact usability.
- `sample_avg_from_csv()` now tolerates partial CSV schemas by selecting only
requested metric columns that exist and warning about missing ones. This is
useful for stage dirs where a monitor didn't emit the full standard set of
columns.
Differential Revision: D108195608
charles-typ
force-pushed
the
export-D108195608-to-v2-beta
branch
from
05ef811 to
e16ead6
Compare
charles-typ
added a commit
to charles-typ/DCPerf
that referenced
this pull request
Jul 8, 2026
Summary:
D108110315 teaches the benchpress perf hook to collect PMU + sysstat data in
per-WDL-sub-benchmark stage directories for WDL prod_set. The raw staged CSVs
are preserved by perfpub's recursive Manifold upload, but users still need a
single processed summary per sub-benchmark, analogous to the normal
`overall-metrics.csv` that perfpub emits for a single benchmark.
This change adds stage-aware summary generation to perfpub:
benchmark_metrics_<run_id>/
memcpy_benchmark/
overall-metrics.csv # NEW
perf-stat.csv
nv-perf-collector-summary.csv
...
hash_hash_benchmark/
overall-metrics.csv # NEW
...
wdl_stage_overall_metrics.csv # NEW aggregate index
wdl_stage_overall_metrics.json # NEW aggregate index
wdl_stage_perf_summary.csv # generic numeric summary from prior patch
wdl_stage_perf_summary.json
Implementation details:
- For each immediate stage subdir that contains CSVs, perfpub temporarily
chdirs into that subdir and reuses the same reader functions as the top-level
path (`read_mpstat`, `read_memstat`, `read_cpufreq_*`, `read_perfstat`,
`read_nv_perf_collector`, `read_arm_perf_collector`, etc.). This gives each
stage the same processed metric lines as a normal single benchmark's
`overall-metrics.csv`.
- Adds the WDL sub-benchmark score at the top when the score exists in the
parent benchmark metrics JSON.
- Writes per-stage `overall-metrics.csv`, plus top-level CSV/JSON indices for
easy discovery in Manifold.
- No XDB/dashboard changes. The goal is Manifold artifact usability.
- `sample_avg_from_csv()` now tolerates partial CSV schemas by selecting only
requested metric columns that exist and warning about missing ones. This is
useful for stage dirs where a monitor didn't emit the full standard set of
columns.
Differential Revision: D108195608
charles-typ
force-pushed
the
export-D108195608-to-v2-beta
branch
from
e16ead6 to
a3bcf94
Compare
meta-codesync
Bot
changed the title
charles-typ
force-pushed
the
export-D108195608-to-v2-beta
branch
from
a3bcf94 to
2a52a5b
Compare
Summary:
Benchpress's perf hook spans an entire benchmark run today, so PMU + sysstat
data ends up smeared across every sub-benchmark in one set of CSVs. For
benchmarks that run many workloads back-to-back (WDL prod_set, SPEC2017
intrate, ...), distinguishing IPC, topdown breakdown, mpstat etc. per
sub-benchmark requires teasing apart timestamps after the fact, which is
brittle.
This diff adds a generic, opt-in stage-aware mode to the perf hook. It is not
tied to any specific benchmark -- consumers (WDL, SPEC, ...) land on top of
this and just emit stage markers.
The mechanism:
1. The perf hook accepts a new option `stage_aware: true`. When set,
`before_job` does NOT start any monitors. Instead it creates a FIFO
under benchmark_metrics_<uuid>/perf_stage.fifo, advertises its path
via the env var BENCHPRESS_PERF_STAGE_FIFO, and spawns a
coordinator thread.
2. The coordinator reads commands from the FIFO. Each
"START <stage_name>" allocates a fresh set of perf monitors with
that stage name as a sub-folder; each "STOP" terminates the
monitors and writes their CSVs. Multiple START/STOP cycles are
supported. `after_job` writes a final __EXIT__ to drain the
coordinator.
3. Every existing perf monitor (mpstat, memstat, netstat, perfstat,
vmstat, cpufreq*, power, topdown -- including IntelPerfSpect/3,
BasePerfUtil, AMDPerfUtil, ARMPerfUtil, NVPerfUtil,
NeoVerseV3PerfUtil) gains a `subdir` constructor arg that the base
`Monitor.gen_path` joins under benchmark_metrics_<uuid>/. Existing
callers that don't pass `subdir` keep their flat layout, so default
mode is byte-for-byte unchanged.
The resulting per-stage layout, e.g.:
benchmark_metrics_<uuid>/
<stage_name>/
mpstat.csv
mem-stat.csv
perf-stat.csv
topdown-... .csv
Benchmark-specific wiring (WDL prod_set, SPEC2017 intrate) lands in later
diffs in this stack.
Reviewed By: YifanYuan3
Differential Revision: D108110315
Summary:
The generic stage-aware perf hook (D108110315) collects PMU + sysstat data in
per-stage directories under benchmark_metrics_<run_id>/<stage>/. The raw
staged CSVs are preserved by perfpub's recursive Manifold upload, but users
still need a single processed summary per stage, analogous to the normal
`overall-metrics.csv` that perfpub emits for a single benchmark.
This change adds stage-aware summary generation to perfpub. It is
benchmark-agnostic: any benchmark that drives the stage-aware hook (WDL
prod_set, SPEC2017 intrate, ...) produces stage subdirs, and perfpub
summarizes each one the same way.
benchmark_metrics_<run_id>/
<stage>/ # e.g. memcpy_benchmark, 500.perlbench_r
overall-metrics.csv # NEW
perf-stat.csv
nv-perf-collector-summary.csv
...
stage_overall_metrics.csv # NEW aggregate index
stage_overall_metrics.json # NEW aggregate index
stage_perf_summary.csv # generic numeric summary from prior patch
stage_perf_summary.json
Implementation details:
- For each immediate stage subdir that contains CSVs, perfpub temporarily
chdirs into that subdir and reuses the same reader functions as the top-level
path (`read_mpstat`, `read_memstat`, `read_cpufreq_*`, `read_perfstat`,
`read_nv_perf_collector`, `read_arm_perf_collector`, etc.). This gives each
stage the same processed metric lines as a normal single benchmark's
`overall-metrics.csv`.
- Adds the stage's score at the top when the score exists in the parent
benchmark metrics JSON.
- Writes per-stage `overall-metrics.csv`, plus top-level CSV/JSON indices for
easy discovery in Manifold.
- No XDB/dashboard changes. The goal is Manifold artifact usability.
- `sample_avg_from_csv()` now tolerates partial CSV schemas by selecting only
requested metric columns that exist and warning about missing ones. This is
useful for stage dirs where a monitor didn't emit the full standard set of
columns.
Reviewed By: YifanYuan3
Differential Revision: D108195608
meta-codesync
Bot
changed the title
charles-typ
force-pushed
the
export-D108195608-to-v2-beta
branch
from
2a52a5b to
b63ba6b
Compare
meta-codesync Bot
pushed a commit
that referenced
this pull request
Jul 9, 2026
Summary: Pull Request resolved: #684 The generic stage-aware perf hook (D108110315) collects PMU + sysstat data in per-stage directories under benchmark_metrics_<run_id>/<stage>/. The raw staged CSVs are preserved by perfpub's recursive Manifold upload, but users still need a single processed summary per stage, analogous to the normal `overall-metrics.csv` that perfpub emits for a single benchmark. This change adds stage-aware summary generation to perfpub. It is benchmark-agnostic: any benchmark that drives the stage-aware hook (WDL prod_set, SPEC2017 intrate, ...) produces stage subdirs, and perfpub summarizes each one the same way. benchmark_metrics_<run_id>/ <stage>/ # e.g. memcpy_benchmark, 500.perlbench_r overall-metrics.csv # NEW perf-stat.csv nv-perf-collector-summary.csv ... stage_overall_metrics.csv # NEW aggregate index stage_overall_metrics.json # NEW aggregate index stage_perf_summary.csv # generic numeric summary from prior patch stage_perf_summary.json Implementation details: - For each immediate stage subdir that contains CSVs, perfpub temporarily chdirs into that subdir and reuses the same reader functions as the top-level path (`read_mpstat`, `read_memstat`, `read_cpufreq_*`, `read_perfstat`, `read_nv_perf_collector`, `read_arm_perf_collector`, etc.). This gives each stage the same processed metric lines as a normal single benchmark's `overall-metrics.csv`. - Adds the stage's score at the top when the score exists in the parent benchmark metrics JSON. - Writes per-stage `overall-metrics.csv`, plus top-level CSV/JSON indices for easy discovery in Manifold. - No XDB/dashboard changes. The goal is Manifold artifact usability. - `sample_avg_from_csv()` now tolerates partial CSV schemas by selecting only requested metric columns that exist and warning about missing ones. This is useful for stage dirs where a monitor didn't emit the full standard set of columns. Reviewed By: YifanYuan3 Differential Revision: D108195608 fbshipit-source-id: 0474d197315647e464c0c06aa67dbc542077e3d8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
The generic stage-aware perf hook (D108110315) collects PMU + sysstat data in
per-stage directories under benchmark_metrics_<run_id>//. The raw
staged CSVs are preserved by perfpub's recursive Manifold upload, but users
still need a single processed summary per stage, analogous to the normal
overall-metrics.csvthat perfpub emits for a single benchmark.This change adds stage-aware summary generation to perfpub. It is
benchmark-agnostic: any benchmark that drives the stage-aware hook (WDL
prod_set, SPEC2017 intrate, ...) produces stage subdirs, and perfpub
summarizes each one the same way.
benchmark_metrics_<run_id>/
/ # e.g. memcpy_benchmark, 500.perlbench_r
overall-metrics.csv # NEW
perf-stat.csv
nv-perf-collector-summary.csv
...
stage_overall_metrics.csv # NEW aggregate index
stage_overall_metrics.json # NEW aggregate index
stage_perf_summary.csv # generic numeric summary from prior patch
stage_perf_summary.json
Implementation details:
chdirs into that subdir and reuses the same reader functions as the top-level
path (
read_mpstat,read_memstat,read_cpufreq_*,read_perfstat,read_nv_perf_collector,read_arm_perf_collector, etc.). This gives eachstage the same processed metric lines as a normal single benchmark's
overall-metrics.csv.benchmark metrics JSON.
overall-metrics.csv, plus top-level CSV/JSON indices foreasy discovery in Manifold.
sample_avg_from_csv()now tolerates partial CSV schemas by selecting onlyrequested metric columns that exist and warning about missing ones. This is
useful for stage dirs where a monitor didn't emit the full standard set of
columns.
Reviewed By: YifanYuan3
Differential Revision: D108195608