PR5/8: Adding scaling distribution to hwgraph#124
Open
ErwanAliasr1 wants to merge 16 commits into
Open
Conversation
The turbostat "starting/stopping background monitoring" messages were printed for every single job, making the output very verbose and hard to read on runs with many jobs. This information is still useful for debugging, so rather than removing it, it is now gated behind a new --verbose (-v) flag (off by default). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Erwan Velu <e.velu@criteo.com>
Pass --yaml to every stress-ng stressor so it writes its metrics next to the stdout file (as <name>.yaml). We don't parse it yet, but capturing the structured YAML output gives us a machine-readable source to build richer parsing features on later, rather than scraping the human-oriented stdout. Add a parsing fixture for stress-ng 0.21.03 under hwbench/tests/parsing/stressng/v02103b/, including the real .yaml artifact alongside stdout, so the auto-discovered parsing tests now cover this version. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Erwan Velu <e.velu@criteo.com>
ErwanAliasr1
changed the title
Benchmark output files -- stdout, stderr, version-stdout, version-stderr and, for stress-ng, the --yaml metrics file -- were named after the engine's `name` (e.g. "stressngint128"). That name is shared by every run a single job expands into: the scaling matrix (hosting_cpu_cores_scaling / stressor_range) turns one job into many runs that all reuse it. As they share one output directory, each iteration overwrote the previous one's files, leaving only the last iteration on disk. Introduce ExternalBench.output_basename, which prefixes the engine `name` with get_name_with_position() -- the per-benchmark id already used as the results.json key (and the expanded-job section header). _write_output and the stress-ng --yaml file now use it, so files look like "cpu10_5_stressngint128-stdout" / "....yaml" and map one-to-one to their results.json entry. `name` keeps its original meaning; External falls back to it for non-benchmark commands (External_Simple). While here, fix a latent bug this path exposes: get_name_with_position() guards on `if not self.benchmark`, but BenchmarkParameters never initialised that attribute (only set_benchmark() did), so reading it before the owning Benchmark is attached raised AttributeError instead of falling back to the bare job name. Initialise self.benchmark = None. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Erwan Velu <e.velu@criteo.com>
Add a per-NUMA-domain view of the per-core CPU metrics (frequency, IPC and
core power): instead of hundreds of unreadable per-core lines, plot one
averaged line per NUMA domain. Following the per-core convention, each metric
is rendered as "all_numa" (every core of each domain) and "pinned_numa" (only
the cores pinned during the job, grouped by their domain, dropping domains
with no pinned core; annotated with the pinned range).
This requires the NUMA topology in the trace, which hwbench collected but did
not serialize. cpu.dump() now writes:
- numa_nodes: {domain: [logical cores]}
- numa_distances: the inter-node distance matrix (kept for future
topology-aware features)
Both are parsed from numactl -H; NUMA.get_distances() exposes the matrix that
was previously only used to derive quadrants. Traces predating this change
lack the topology and simply skip the NUMA graphs.
Tests reuse the existing NUMA fixtures: distance-matrix assertions in the
parse tests, a cpu.dump() topology test on the mocked hardware.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Erwan Velu <e.velu@criteo.com>
Complement the per-NUMA-domain line graphs with two heatmaps: - A per-host NUMA distance heatmap (domain x domain, color = inter-domain distance), rendered once per trace in environment/by_host/<host>/, so the topology (which domains are close/far) is visible independently of any benchmark. - A per-metric NUMA domain x time heatmap for frequency and IPC, next to the corresponding line graph in all_numa/pinned_numa: Y = NUMA domains, X = time (as in the line graphs), color = the domain's value at each step. A legend box on the left lists each domain's cores condensed via cpu_list_to_range() (individually bracketed, right-aligned, centered dash), like the component legend of the other graphs. Graph.render() gains an extra_legend parameter so a manually placed legend is included in the tight bounding box. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Erwan Velu <e.velu@criteo.com>
IPC is usually interesting to see how cores are performing. The frequency is not always enough to get a good estimation of the core's performance, so let's report the IPC too. Add an "ipc" SMP scaling graph, mirroring the cpu_clock one: add_perf() gains ipc/ipc_err/ipc_cores parameters that aggregate the min/mean/max IPC across cores (optionally restricted to the pinned cores), and smp_scaling renders an ipc graph type with the same all_cores/pinned_cores variants and errorbars. IPC is not always collected, so it is only aggregated and rendered when the trace actually has an IPC context; IPC-less traces are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Erwan Velu <e.velu@criteo.com>
Compare traces per NUMA domain across the scaling sweep. The first trace is the reference; one heatmap is produced for each other trace, with the compared trace name in the filename (scaling_<type>_numa_delta_<ref>_vs_<other>_<engine>). Y = NUMA domains, X = scaling step (worker count), color = signed delta value(reference) - value(other): red when the compared trace is lower than the reference, green when higher, white when equal (diverging map centered at 0). Rendered for frequency (cpu_clock) and IPC (cpu_ipc, when present), each in the all_cores and (when the sweep pins cores) pinned_cores directories next to the line graphs. On the pinned view, domains not pinned at a given step have no data and are drawn black. This will help understanding what NUMA mode was more or less active during a step of the scaling. A left box lists each domain's cores (condensed, via numa_core_blocks) and the colour legend is a caption under the graph. Only produced when comparing at least two traces exposing the NUMA topology. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Erwan Velu <e.velu@criteo.com>
The lack of column and alignment in the header made it harder to read. Before: System: server_x_y_z Bios 3.2 Kernel 8.10.1 After: System:server_x_y_z Bios:3.2 Kernel:8.10.1 Signed-off-by: Erwan Velu <e.velu@criteo.com>
With large data like cpu frequencies, the legend box tends to collide with the Y-AXIS making it unreadable. Let's move this box a bit away to the left to avid this. Signed-off-by: Erwan Velu <e.velu@criteo.com>
The "No samples found in <metric>, ignoring metric." messages polluted the output when a metric is missing. A typical output was showing a missing PDU metric which can be a normal thing. So let's only print them if "--verbose" is requested. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Erwan Velu <e.velu@criteo.com>
The stats output printed one max line per power metric, including metrics that had no data at all, e.g. "PDU max : 0.00 Watts in " with an empty bench name. That 0.00 line looked like a buggy processing from hwgraph rather than what it actually was: a simply missing metric. Metrics are now only displayed if valid data is found. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Erwan Velu <e.velu@criteo.com>
Reading a value off the smp_scaling line graphs was imprecise with only the major Y ticks. Add a single intermediate horizontal gridline between two Y ticks (AutoMinorLocator(2)), a bit bolder than the default minor grid so it is actually readable, but still lighter than the solid major gridlines. Applies to the scaling line graphs (perf, perf_watt, watts, cpu_clock, cpu_ipc); the delta heatmaps are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Erwan Velu <e.velu@criteo.com>
The per-trace legend of the smp_scaling graphs shows, for each trace, "<name> [min; mean; stddev; max]". Each label was padded to its own length, so traces with different name lengths produced a misaligned table where the [min; mean; stddev; max] columns did not line up, making the values harder to read and compare between traces. Pad the trace name and the values to a width common to all the traces of the graph (like the environment graphs already do), so the columns stay aligned whatever the trace names. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
For every per-core CPU metric (Core frequency, Core IPC and CPU Core power consumption), render an additional graph showing how that metric is distributed across the cores at steady state. Each core's mean value over the run becomes one data point; a violin shows the density and an overlaid box shows the median (red), mean (green dashed), quartiles and outliers. Where the line graphs answer "how did the metric evolve over time?", these answer "how uniform were the cores?" -- immediately exposing stragglers, bimodal behaviour (e.g. boosted vs throttled cores) and the spread that a single averaged curve hides. Like the other per-core graphs they are rendered twice, once over all the cores (all_cores) and once restricted to the cores pinned during the job (pinned_cores), and land in the same per-metric directories next to their line graphs. The Y axis is autoscaled rather than zero-based: a distribution is unreadable squished against a zero baseline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Erwan Velu <e.velu@criteo.com>
Mirror the per-core steady-state distribution graphs on the SMP scaling
views. For each trace and each per-core metric (Core frequency, Core IPC
and CPU Core power consumption), one graph plots a violin + box per
scaling step (X = worker count, evenly spaced), showing how the
core-to-core distribution of that metric evolves as the sweep grows.
The scaling line graphs plot a single averaged value per step; these
expose the spread that average hides -- cores that start to diverge or
throttle only past a given worker count, bimodal frequency behaviour, or
a widening per-core power spread. The box reports median (red), mean
(green dashed), quartiles and outliers.
Like the other per-core scaling graphs they are rendered for all_cores
and, when the sweep pins cores, pinned_cores, landing in the cpu_clock,
cpu_ipc and cpu_core_power directories next to the matching line graphs.
The Y axis is autoscaled rather than zero-based: a distribution is
unreadable squished against a zero baseline.
Why violin graphs at both levels: the per-benchmark (steady-state) and
the per-scaling-step distributions answer two complementary questions
that averages and time-series curves both flatten.
- The individual, steady-state violin answers "for this one operating
point, how uniform is the hardware?". A tight body means every core
behaves alike; a long tail or a second lobe exposes a straggler, an
asymmetric NUMA/boost domain or a mis-pinned core -- something a
single mean value, and even the time-series line graph, hides because
they collapse every core into one number per instant.
- The scaling violins put those same distributions side by side along
the load axis, so the shape itself becomes the signal: you watch the
spread widen or split exactly at the worker count where cores begin
to contend for power/thermal budget, and you can tell a uniform
slow-down (body drops as a block) from a divergence (body stretches
or splits) -- a distinction a scaling line of per-step averages
cannot make.
Together they let a reader first confirm a single point is healthy, then
follow how that health degrades under increasing load, using the same
visual vocabulary at both zoom levels.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Erwan Velu <e.velu@criteo.com>
Add a distribution view for per-core CPU metrics (frequency, IPC, core power) aggregated by NUMA domain, complementing the existing per-domain line graphs and heatmaps: - numa_distribution_graph: one violin + box per NUMA domain for a single benchmark job (steady-state), landing next to the matching line graph in the same all_numa/pinned_numa directories. - render_numa_scaling_ridgelines: the SMP-scaling counterpart. A first grouped-violin attempt (one violin per domain per step, all in one axes) became unreadable past a handful of domains and steps. After reviewing sample renderings of several alternatives (median+IQR line, small multiples, spread heatmap, ridgeline), the ridgeline design was picked: one panel per scaling step -- every step, laid out as a grid so the figure grows in rows rather than becoming unreadable -- each panel a stacked density (ridgeline) per NUMA domain, preserving the full distribution shape (skew, bimodality) that a single averaged value would flatten. The grid's header/footer are reserved as a constant number of inches (not a fraction of the figure) so they keep the same size regardless of how many rows a long sweep needs. Both are rendered for all_numa (every core of each domain) and, when the sweep pins cores, pinned_numa. Also mark which CPU package each NUMA domain belongs to: hwbench does not record this directly, but same-package domains are always much closer to each other in the NUMA distance matrix than cross-package ones, so _numa_domains_by_package groups domains whose mutual distance is below a fixed threshold (union-find), with no need to know the socket count up front. Each scaling ridgeline panel gets a pale background wash per package behind its ridges, with a "CPU package <n>" legend, colour-matched to the washes, centered below the whole grid. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Erwan Velu <e.velu@criteo.com>
ErwanAliasr1
force-pushed
the
hwgraph-scaling-distribution
branch
from
July 16, 2026 10:32
73fc3b1 to
62bd4d9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds some per-core and per numa distributions graphs