Skip to content

Raspberry Pi support for AXI bus events in Linux perf tool - #7671

Open
captain5050 wants to merge 3 commits into
raspberrypi:rpi-6.18.yfrom
captain5050:rpi-6.18.y
Open

captain5050 wants to merge 3 commits into
raspberrypi:rpi-6.18.yfrom
captain5050:rpi-6.18.y

Conversation

@captain5050

Copy link
Copy Markdown
Contributor

I've addressed review feedback from:
#7571
but in doing so accidentally closed the PR. Apologies, hopefully things are in a better state this time.

Add device tree bindings for the Broadcom Raspberry Pi AXI PMU block
found on BCM2835, BCM2711, and BCM2712 SoCs.

Depending on the SoC and security configuration, these PMUs support
either direct MMIO access for the System monitor or routing via the
Raspberry Pi firmware mailbox IPC for the VideoCore VPU monitor.

Signed-off-by: Ian Rogers <irogers@google.com>
Add a PMU driver for the Raspberry Pi AXI performance monitors. It
provides access to the System and VideoCore performance counters on
Broadcom BCM2835 and BCM2711 platforms natively through the perf
subsystem.

The driver implements bus-watcher refcounting, group validation, CPU
hotplug migration, and sysfs event aliases mapping hardware-specific
events to AXI pipelines (such as DMA, V3D, HVS, and ISP). Memory-mapped
endpoints for the primary System bus are read synchronously, while the
VideoCore VPU monitor is polled asynchronously in process context via
the Raspberry Pi firmware mailbox interface.

Signed-off-by: Ian Rogers <irogers@google.com>
Extend the Raspberry Pi AXI PMU driver to support the Broadcom BCM2712
architecture used in the Raspberry Pi 5.

The BCM2712 SoC drops the legacy VideoCore VPU monitor and exposes a
unified System monitor via MMIO mapping distinctly different hardware
pipelines (such as DISPLAY_TOP, ARGON_TOP, BSTM_TOP, PCIE_01, and XPT)
and expanded 6-bit AXI master ID filter tables.

Signed-off-by: Ian Rogers <irogers@google.com>
@popcornmix

Copy link
Copy Markdown
Collaborator

Got a quite long Claude review.

I've been through the specs and tested poking registers on 2711, 2712 and 2712D0 so the points in the review should be accurate.

Note, there were some bugs in downstream driver (and some new ones in the new code).
Note, the mailbox interface for reading VPU monitor on Pi5 is not working for you ("2026/10/05" firmware was a custom one with that fixes - you can assume that will be default behaviour in near future).

If you'd like I can get claude to suggest patches to your commits (but it is also fine if you'd like to do this).

@popcornmix

popcornmix commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Thanks for the update — this is a big improvement on #7571. I checked it against drivers/perf/raspberrypi_axi_monitor.c again, built it for arm64 (W=1, with lockdep) at both the 2835 commit and the tip, and ran checkpatch. Both commits build without warnings. checkpatch reports only line lengths and 3 minor CHECKs. I haven't run the driver itself on hardware, but I have tested the registers directly on 2711 B0, 2712 C0 and 2712 D0 (results below).

What's fixed since #7571

  • The BCM2835, BCM2711 and BCM2712 system-bus, VPU-bus and filter tables now match the downstream driver exactly. I checked every one of the 781 event aliases by script: each name matches its monitor/bus/counter.
  • CHIP_BCM2711 is now separate.
  • The mailbox address is now read from the raw reg cell, not resource->start.
  • The VPU monitor always uses the mailbox on 2835/2711.
  • scale/unit are gone, counters use the full 32 bits, and all 11 counters are exposed.
  • Kconfig now uses ARCH_BCM2835 and excludes RPI_AXIPERF.

Blocking

1. The driver can't be enabled in any Raspberry Pi defconfig

depends on !RPI_AXIPERF, but every RPi defconfig sets CONFIG_RPI_AXIPERF=m (bcmrpi, bcm2709, bcm2711, bcm2711_rt, bcm2712). As it stands, nobody gets this driver. We need to decide whether it replaces the debugfs driver, and if so:

  • update the defconfigs
  • update arch/arm/boot/dts/overlays/README (the axiperf param still points at /sys/kernel/debug/raspberrypi_axi_monitor)

2. The BCM2712 VPU monitor has been dropped

config_is_valid() rejects monitor=1 on 2712, and probe skips reg[1]. The commit message says "BCM2712 drops the legacy VideoCore VPU monitor". It doesn't: 2712 has a VPU monitor, and I've had it counting through the firmware mailbox on a Pi 5 (see the hardware tests below).

2712 should take the same mailbox path as 2711, using the 2711 VPU table, which is identical to downstream's 2712 one. The VPU address in the DT is wrong, though: see the VPU monitor section below.

3. The binding doesn't match the in-tree DT

  • The in-tree nodes have firmware = <&firmware>;, but the binding doesn't list firmware and has additionalProperties: false. dtbs_check will fail on every Pi.
  • The 2712 branch has maxItems: 1, but bcm2712-ds.dtsi has two reg entries.
  • reg should be minItems: 2 for 2835/2711. Probe fails without reg[1].
  • The description says the VPU region is "memory-mapped if supplied". It is never memory-mapped: it is a VideoCore address passed to the firmware.

Related: the driver finds the firmware with of_find_compatible_node(NULL, NULL, "raspberrypi,bcm2835-firmware") and ignores the firmware phandle. Please use the phandle, as downstream does.

4. Register layout doesn't match the hardware docs (2711 B0, 2712 C0, 2712 D0)

I checked the system monitor (APERF0) register docs for 2711 B0, 2712 C0 and 2712 D0. The bus-watcher layout changes between chip revisions, and some of it is wrong in both this driver and the downstream one.

BWx_CTRL ID filter field

2711 B0 2712 C0 2712 D0
ID [12:8], 5 bits [13:8], 6 bits: master ID, i.e. AXI ID bits [8:3] [17:8], 10 bits: full AXI ID
ID_MASK — — [27:18], reset value 0x1f8 (masks off the bottom 3 bits)
PROT_EN / PROT — — [7] / [6]
BUS [4:0] [4:0] [4:0], bit 5 spare
  • D0: the filter is broken. The driver builds the whole control word from scratch, so it writes ID_MASK = 0. The match rule is (axi_id & ID_MASK) == (ID & ID_MASK), so a mask of 0 matches every transaction: on D0, every filter event counts all traffic. On D0 the master ID must also be written as master << 3 into ID, with ID_MASK = 0x1f8. The docs also say that on 2712 the V3D and ARM buses don't use the {master, 3 bits} ID scheme.

  • D0 needs its own compatible. Pinctrl already uses separate brcm,bcm2712c0-* and brcm,bcm2712d0-* compatibles, and bcm2712-d-rpi-5-b.dts switches between them. axiperf needs the same, for example brcm,bcm2712d0-axiperf, added to the binding as well.

  • The 2712 filter numbers are wrong (here and downstream). The hardware master IDs leave gaps; the tables number them with no gaps. They agree only up to VPU_DC1 = 6:

    Master Driver Hardware
    VPU_L2 7 8
    DMA2 8 9
    ARM 10 11
    DMA0 11 12
    PCIE0 15 18
    HVS 22 32
    USB 31 41
    PISP 34 44
    MPUCACHE1 50 63

    The full list: VPU_UC0=1 … VPU_DC1=6, VPU_L2=8, DMA2=9, VPU_DEBUG=10, ARM=11, DMA0=12, DMA1=13, RAAGA=14, BBSI=16, PCIE0..2=18..20, UMR=27, SAGE=28, HVDP=29, BSP=30, HVS..USB=32..41, ARGON..EMMC2=42..48, TRC=52, BSTM0..MPUCACHE1=53..63.

    So the 2712 enum needs re-basing, and BCM2712_FLT_MAX must be 64. The 6-bit config:10-15 field is the right size for C0. For D0, decide whether to expose the raw ID and mask (10 + 10 bits) or keep the master-ID field and encode it per revision.

  • BW_CTRL_BUS_FILTER_MASK and BW_CTRL_BUS_WATCH_MASK should be chosen per chip, not shared: ID is [12:8] on 2711, and BUS is [4:0] everywhere, not [5:0].

Counter registers

Counter Width / meaning
ATRANS, WTRANS, RTRANS 32 bits. W/R count beats; ATRANS counts address transactions.
ATWAIT, WTWAIT, RTWAIT 28 bits ([27:0])
AMAX, WMAX, RMAX 28 bits; peak latency, not a running total
RPEND 8 bits; the number of reads currently pending, not a running total
READ_ATRANS (0x2c) 2712 only. It does not exist on 2711, where 0x2c is unused.
BWx_DEBUG (0x3c) 2712 D0 only. READ_BURST makes RTRANS count bursts instead of beats; the driver should write 0 to it.
  • The wait counters wrap at 2^28. The *wait deltas use u32 subtraction, so when a 28-bit counter wraps, the delta comes out about 4×10⁹ too large. Mask the delta to the counter's width, e.g. (new - prev) & GENMASK(27, 0). At a bus clock of about 500 MHz, a wait counter on a busy bus wraps after about 0.5 s. The 100 ms poll is fast enough to catch that, but only if the delta is masked.
  • amax/wmax/rmax/rpend aren't running totals. They go through the same delta-summing as the counters, which gives meaningless results; for example, a second event that shares a watcher reports max - baseline. Report them as raw values, or drop them.
  • Remove ratrans on 2835/2711. It reads an address with no register behind it.
  • GEN_CTRL differs between chips.
    • 2711 has only ENABLE [0] and RESET [1]; there is no WATCH bit.
    • On 2712, RESET is self-clearing and only works while EN is set. The driver writes GEN_CTL_RESET_BIT on its own, so on 2712 that write does nothing. Write EN | RESET.

VPU monitor (APERF1)

  • Same layout as the system monitor. On each of 2711 B0, 2712 C0 and 2712 D0, the VPU monitor's registers match that chip's system monitor field for field. So every problem above applies to the VPU monitor too:

    • 28-bit wait/max counters and the 8-bit RPEND
    • no READ_ATRANS on 2711
    • D0 ID_MASK, plus BWx_DEBUG on D0
    • the 2712 GEN_CTRL reset behaviour
  • 2712 has a VPU monitor. This confirms blocking item 2: it should be supported on 2712, not dropped.

  • The DT has the wrong VPU address. The docs give 0x7ee00000 (VideoCore address) for all three chips; the DT uses 0x7ee08000 on 2835/2711 and 0x7e000000 on 2712.

    • On 2712, only 0x7ee00000 works (tested below).
    • On 2711, 0x7ee08000 is accepted by the firmware but nothing reaches the hardware.

    So reg[1] in bcm270x.dtsi and bcm2712-ds.dtsi needs fixing, and the binding example with it. It also means the VPU figures from the existing debugfs driver have never been valid.

  • Bus 13 exists. The docs list VPU buses only up to 11 (2711) or 12 (2712), but on 2712 C0 bus 13 (L2_IN) counts traffic, so the driver tables are right to include it.

Hardware tests

I tested directly on three boards, using busybox devmem for the system monitor and vcmailbox GET/SET_PERIPH_REG for the VPU monitor:

  • Pi 4B (2711 B0), firmware Aug 10 2026
  • Pi 5 (2712 C0), firmware 2026/10/05
  • Pi 500 (2712 D0), firmware 2026/10/01
Test 2711 B0 2712 C0 2712 D0
BW0_CTRL reset value 0 0 0x07e00000 (ID_MASK = 0x1f8), as documented
BW0_CTRL writable bits (write 0xffffffff) 0xf0001fff: ID is 5 bits 0xf0003fff: ID is 6 bits —
Filter with a non-existent master ID, ID_MASK = 0 (what the driver writes) — — counts about the same as no filter
Filter with a non-existent master ID, ID_MASK = 0x1f8 0 (ID 31) 0 (ID 7) 0
Which master IDs show traffic, scanning all 64 (C0) — bus 5 during network receive: only 20 (PCIE2, where RP1 sits). Bus 1: only 32 (HVS). Bus 9 during SD reads: only 48 (EMMC2). Bus 11: 1 (VPU_UC0) and 11 (ARM). All as numbered in the docs. —
PR's numbers for those masters (PCIE2 = 17, HVS = 22, EMMC2 = 38, ARM = 10) — none of them match —
ARM filter (30) on bus 10 counts, as downstream numbers it — —
Register at 0x2c (READ_ATRANS) reads 0x41584950 ("AXIP" filler value): no register there counts —
RPEND under load 255, then 1: an instantaneous value 255 —
GEN_CTRL RESET bit sticks — self-clearing (0x7 reads back as 0x5)
GEN_CTRL = EN only (no WATCH_EN) — — counters don't count
VPU mailbox at 0x7ee08000 firmware accepts it, but writes don't stick and everything reads 0 reads 0 —
VPU mailbox at 0x7ee00000 firmware rejects the address works: control register holds its value, and counters count on buses 1, 3, 7, 9, 10, 11, 13 —
VPU mailbox at 0x7e000000 — reads 0 —
VPU mailbox, any address — — firmware returns error status 0x80000001 for the whole call

So, whether the VPU monitor works depends on the firmware. The Pi 4 firmware I have checks addresses against 0x7ee08000, which has nothing behind it. The 2026/10/01 firmware on the Pi 500 doesn't support the call on 2712 at all. The driver should cope with the firmware refusing the request, for example by not registering the VPU events.

On bus 3, IDs 16 to 43 all count, with steadily falling totals. Those are the ARM's own AXI IDs, which the docs say don't follow the master-ID scheme, so a master filter means nothing on the ARM bus (and, per the docs, on V3D).

The bus names may also be misleading. Network DMA from RP1 (PCIe2) shows up on bus 5 (BSTM_TOP), not bus 6 (PCIE_01), and SD card DMA on bus 9 (SRC). These could be fabric port names rather than the devices they serve, but users picking pcie_01_* to measure RP1 will see nothing. It's worth checking the bus names with someone who knows the fabric.

Should fix

  • One mailbox call per counter. The VPU work does one GET_PERIPH_REG per event, every 100 ms. With many VPU events open, that is dozens of firmware calls per tick. A single call could read all of a watcher's counters at once, as downstream does.
  • 2711: AIO can't be selected. Filter index 0 is AIO, but filter=0 means "no filter". This is the same as downstream.
  • The Kconfig help is misleading. "Expect 100ms ping latencies to the VideoCore firmware mailbox" isn't accurate. The real behaviour is that VPU counts are polled every 100 ms, so perf stat misses up to 100 ms at the end of a run. It also doesn't apply to 2712.
  • Firmware disabled. With RASPBERRYPI_FIRMWARE=n, devm_rpi_firmware_get() returns NULL, so probe defers forever on 2835/2711.
  • No Documentation/admin-guide/perf/ entry (raised last time).

Minor / style

  • About 2,300 of the 3,400 lines are PMU_EVENT_ATTR_STRING boilerplate: 11 counters × each bus × 3 SoCs. A per-bus macro that expands the 11 counters, or building the attributes at probe from the name tables, would make this far smaller and easier to check.
  • The comments are very long and often just restate the code. Examples:
    • "Use READ_ONCE to prevent KCSAN…"
    • "~10-20 nanoseconds"
    • the rpi_axi_pmu__exit() comment: the claim that the work queue is "permanently sealed" is still there
    • the timer kernel-doc says "hard IRQ", but the timer is HRTIMER_MODE_REL_SOFT
  • rpi_axi_pmu_stop() sets PERF_HES_UPTODATE for VPU events, which haven't actually been updated.
  • U32_MAX is used to mean "read failed" on the MMIO path, where it is also a valid counter value. That reading gets skipped. It's harmless, because the next read recovers, but the MMIO path can't fail, so the check isn't needed there.
  • checkpatch --strict: about 165 lines over 100 columns, 1 mutex without a comment, 2 misaligned lines.

@captain5050

Copy link
Copy Markdown
Contributor Author

Thanks again for the help! Wrt Claude helping with my patches or me acting on the reviews, I'm easy. There's a bit of a delay with me getting the work done, but also the patches aren't urgent. If you would like to have a crack at it I don't mind. The next version should address the feedback above and also some device tree nits that were posted on LKML: https://lore.kernel.org/linux-perf-users/20261002175738.3242646-1-irogers@google.com/

@popcornmix

Copy link
Copy Markdown
Collaborator

I've pushed a tree with some claude fixes to https://github.com/popcornmix/linux/tree/captain5050
It does need updated bootloader for working VPU stats (I'll push that out soon).

@popcornmix

Copy link
Copy Markdown
Collaborator

firmware (for pi0-4) : here
bootloader (for pi5) : here

I've updated my branch to include any fixes I've found: https://github.com/popcornmix/linux/tree/captain5050
Feel free to squash as you see fit.

@popcornmix

Copy link
Copy Markdown
Collaborator

Raspberry Pi AXI PMU: hardware validation

This describes how the bus tables, master IDs and register handling in drivers/perf/rpi_axi_pmu.c were checked on real hardware. It covers every chip the driver supports, and for each value it records where it came from:

  • Doc: Broadcom's register documentation for the AXI performance monitors.
  • Downstream: the tables in the existing debugfs driver, raspberrypi_axi_monitor.c.
  • Measured: confirmed directly on hardware, as described below.

The Broadcom documentation for BCM2712 describes the set-top box variant of the chip. Several of its port names refer to blocks that aren't used on a Raspberry Pi, and on BCM2712 D0 the port numbering differs from the documentation. Where the documentation and the measurements disagree, the driver follows the measurements.

Hardware

Chip Board used
BCM2835 Raspberry Pi 1 B+
BCM2837 Raspberry Pi 3 B+
BCM2711 B0 Raspberry Pi 4 B
BCM2712 C0 Raspberry Pi 5 (rev 1.0)
BCM2712 D0 Raspberry Pi 5 (rev 1.2) and Raspberry Pi 500

All boards ran a 6.18 Raspberry Pi kernel with current firmware.

Method

1. Direct register probing (driver unloaded)

A small user-space tool programmed the bus watchers directly:

  • the System monitor through /dev/mem
  • the VPU monitor through the firmware's GET/SET_PERIPH_REG mailbox calls

Working outside the driver allowed:

  • any bus index (0–31) and any ID filter, including values outside the driver's tables
  • reading all eleven counters of a watcher at once

For each workload, the tool:

  1. Scanned every bus index on both monitors and compared it with an idle run.
  2. On each bus that rose clearly above idle, scanned every master ID filter to find which masters produced the traffic.

2. Workloads used to exercise known masters

Workload Exercises
CPU memory read, write and copy over a 64 MiB buffer Arm
Network receive and transmit at line rate Ethernet (GENET or USB on older boards; RP1 over PCIe on BCM2712)
SD card reads (O_DIRECT) SD/eMMC controller
NVMe reads External PCIe (BCM2712)
USB mass-storage reads VL805 over PCIe (BCM2711)
dmatest on individual DMA controllers and channels DMA engines
HDMI audio playback DMA1 (BCM2712)
Display on vs blanked through fb0 HVS scanout
DRM writeback on each writeback connector HVS writeback outputs (BCM2712)
Headless GLES render loop V3D
HEVC decode (ffmpeg -hwaccel drm) HEVC decoder (BCM2711, BCM2712)
H.264 decode and encode (h264_v4l2m2m) VideoCore video blocks (BCM2835–BCM2711)
ISP conversion through bcm2835-codec ISP (BCM2835–BCM2711)
Camera capture: raw unicam, and through libcamera UNICAM, ISP, PiSP back end
WiFi scans WiFi SDIO controller
vcgencmd firmware calls, and vcgencmd memtest with cached and uncached flags VPU

3. Checks through the driver

The driver was then loaded and checked with perf stat -a on every chip:

  • event lists
  • filtered events against unfiltered ones
  • multiplexing
  • interval mode (-I)
  • VPU events, with and without firmware support

Results: System monitor (APERF0)

"Unconfirmed" means no available workload produced traffic on that bus. The name is kept from the documentation (or from the downstream driver, where the documentation gives none).

BCM2835 (event names as in the driver)

Bus Event Name from Measured
0 dma_l2 Downstream DMA0 (master 16) writes during SD reads
1 trans Downstream Unconfirmed
2 jpeg Downstream Unconfirmed
3 system_uc Downstream ISP (11) writes
4 dma_uc Downstream Unconfirmed on BCM2835 (measured on BCM2837)
5 system_l2 Downstream Every L2-cached master: Arm 30, HVS 10, DMA0 16, ISP 11, video 13/20, USB 24, HOST_PORT 8
6 ccp2tx Downstream Unconfirmed
7 mphi_rx Downstream Unconfirmed
8 mphi_tx Downstream HOST_PORT (8), small
9 hvs Downstream HVS (10); falls to 0 when the display is blanked
10 h264 Downstream Unconfirmed on BCM2835 (measured on BCM2837)
11 isp Downstream ISP (11)
12 v3d Downstream Unconfirmed
13 peripheral Downstream DMA0 (16) reading the SD FIFO; Arm; CORE0_V (1)
14 cpu_uc Downstream Unconfirmed on BCM2835 (measured on BCM2837)
15 cpu_l2 Downstream Arm (30)

BCM2837 (same table as BCM2835)

Bus Event Measured
3 system_uc Every master: Arm 30, DMA 16/17, V3D 25, HVS 10, ISP 11, video 13/22, CORE0_V 1
4 dma_uc DMA0 (16) during SD reads; DMA1 (17) under dmatest
8 mphi_tx HOST_PORT (8), small
9 hvs HVS (10); falls to 0 when the display is blanked
10 h264 VIDEO_FME (22), VIDEO_SD2AXI (13), VIDEO_CME (20) during H.264 encode
11 isp ISP (11)
13 peripheral CORE0_V (1), Arm (30), DMA0 (16)
14 cpu_uc Arm (30)
12 v3d No traffic under GPU load. V3D (25) appears on system_uc instead. Possibly only V3D's L2-cached path; unconfirmed
0–2, 5–7, 15 Unconfirmed

BCM2711 B0

Bus Event Name from Measured
0 dma_l2 Downstream Unconfirmed
1 trans Downstream Unconfirmed
2 jpeg Downstream Unconfirmed
3 vpu_uc Downstream CORE0_V (1), L2_MAIN (7); uncached VPU accesses from vcgencmd memtest land here
4 dma_uc Downstream DMA0 (16) and DMA1 (17)
5 system_l2 Downstream VIDEO_SD2AXI (13) during H.264 decode
6 hvs Downstream HVS; falls to 0 when the display is blanked (see the note on master IDs below)
7 argon Downstream HEVC decoder (ARGON, 8), plus ISP (11) and EMMC2 (31): a shared port
8 h264 Downstream VIDEO_FME (22), VIDEO_SD2AXI (13), VIDEO_CME (20)
9 peripheral Downstream CORE0_V (1), Arm (30)
10 arm_uc Downstream Arm (30): all CPU memory traffic
11 arm_l2 Downstream Unconfirmed

On BCM2711, none of these appeared on any bus of either monitor:

  • V3D, under GPU load
  • GENET, during network traffic
  • the VL805 USB controller (PCIe), during USB disk reads
  • UNICAM, during direct capture at about 78 MB/s

These blocks seem to sit outside the part of the fabric that the monitors cover.

BCM2712 C0

Bus Event Doc name Measured
0 vpu_uc VPU_UC VPU_UC0 (1) under VPU load
1 display DISPLAY_TOP HVS (32); 0 when the display is blanked. Also the writeback outputs MOP0 (34) and MOP1 (35)
2 v3d V3D V3D, under GPU load (V3D assigns its own AXI IDs)
3 arm ARM Arm, under CPU memory load (the Arm assigns its own AXI IDs)
4 xpt XPT Unconfirmed (XPT is not used on a Pi)
5 pcie_02 BSTM_TOP PCIe2 (20): RP1 on a Pi 5, i.e. network, RP1 DMA and camera frames
6 pcie_01 PCIE_01 PCIE1 (19): NVMe on the external PCIe connector
7 argon ARGON_TOP HEVC decoder (42) and PiSP back end (44)
8 emmc2 ARB3 The EMMC2 controller (WiFi on a Pi 5), during WiFi scans; master ID 39
9 sd_dma SRC SD card (48), DMA0 (12) and DMA1 (13), including HDMI audio through DMA1
10 hvdp HVDP Unconfirmed (HVDP is not used on a Pi)
11 per PER VPU_UC0 (1) and Arm (11) peripheral accesses
12 system_l2 SYSTEM_L2 Unconfirmed

Bold names differ from the documentation, because the documented name describes a block the Pi doesn't use.

BCM2712 D0

On D0, three ports are removed: XPT, PCIE_01 and HVDP. The rest are renumbered to fill the gaps, and the external PCIe moves onto the argon port.

The driver detects D0 at probe time: only D0 lets the watcher's ID mask field be written.

D0 bus Event Same port as C0 bus Measured on D0
0 vpu_uc 0 VPU_UC0 (1)
1 display 1 HVS (32); 0 when the display is blanked
2 v3d 2 V3D
3 arm 3 Arm
4 pcie_02 5 PCIe2 (20): RP1 network, DMA and camera
5 argon 7 HEVC decoder (42), PiSP back end (44), external PCIe NVMe (19)
6 emmc2 8 WiFi scans, master 39
7 sd_dma 9 SD card (48), DMA0 (12)
8 per 11 VPU_UC0 (1), Arm (11)
9 system_l2 12 Unconfirmed

These measurements were spread across two D0 boards (a Pi 5 rev 1.2 and a Pi 500), and every bus the two have in common agreed.

Indices beyond the table

On every chip, bus indices past the end of the table count nothing at idle. Under heavy load, some of them count once per clock cycle. They are not real ports, so the driver rejects any bus number beyond each chip's table.

Results: VPU monitor (APERF1)

The downstream BCM2835 VPU table is wrong from index 5 onward, on both BCM2835 and BCM2837. All three older chips actually follow the BCM2711 layout, which has no L2_FLUSH entry, and BCM2835/BCM2837 add a working index 14.

The driver now uses this layout for all of them. BCM2712 uses the BCM2711 list, and its bus 13 was measured.

Bus Event (vpu_…) Name from Measured
0 vpu1_d_l2 Downstream Unconfirmed (VPU core 1 not seen)
1 vpu0_d_l2 Downstream DCACHE0 (3); cached VPU accesses (memtest flags 0x20)
2 vpu1_i_l2 Downstream Unconfirmed
3 vpu0_i_l2 Downstream ICACHE0 (2), on every chip
4 system_l2 Downstream Every L2-cached master (BCM2835); video (BCM2711, BCM2837)
5 dma_l2 Downstream (BCM2711 list) Unconfirmed
6 vpu1_d_uc Downstream (BCM2711 list) Unconfirmed
7 vpu0_d_uc Downstream (BCM2711 list) CORE0_V (1) on every chip; uncached VPU accesses (memtest flags 0x24). The downstream BCM2835 table calls this VPU1_D_UC
8 vpu1_i_uc Downstream (BCM2711 list) Unconfirmed
9 vpu0_i_uc Downstream (BCM2711 list) Unconfirmed
10 vpu_uc Downstream (BCM2711 list) Uncached VPU traffic, and ISP writes (BCM2835). The downstream BCM2835 table calls this VPU0_I_UC, an instruction-fetch bus, which can't carry writes
11 l2_out Downstream (BCM2711 list) L2_MAIN (7), on every chip. The downstream BCM2835 table calls this SYSTEM_UC
12 dma_uc Downstream (BCM2711 list) Unconfirmed
13 l2_in Downstream (BCM2711 list) L2_MAIN (7) and the VPU caches. Present on BCM2712 too, although the documentation's list stops at 12
14 sdram Downstream (BCM2835 list) BCM2835/BCM2837 only: all masters plus the VPU caches. Not present on BCM2711

vcgencmd memtest 0x10000 10000 <flags> on BCM2711 separates the cached and uncached paths cleanly (transactions per 2 s):

Flags vpu0_d_l2 l2_in vpu0_d_uc vpu_uc System vpu_uc
0x20 (cached) 82.0 M 82.4 M 10 k 69 k 58 k
0x24 (uncached) 0.1 M 0.6 M 163.9 M 163.9 M 84.6 M

Results: master IDs (filter values)

BCM2835–BCM2711 (5-bit filter)

Measured, matching the downstream filter list:

  • CORE0_V 1, ICACHE0 2, DCACHE0 3, L2_MAIN 7, HOST_PORT 8
  • ISP 11, VIDEO_SD2AXI 13, DMA0 16, DMA1 17, VIDEO_CME 20, VIDEO_FME 22
  • USB 24 (BCM2835), V3D0 25 (BCM2837)
  • Arm/CPU 30
  • On BCM2711 also: ARGON 8 and EMMCSTB 31

Exceptions:

  • BCM2711 HVS uses master 0. The downstream list calls master 0 "AIO" and puts HVS at 10. Filter value 0 means "no filter" in the driver's event format, so on BCM2711 the HVS traffic can't be singled out with a filter.
  • HVS is master 10 on BCM2835 and BCM2837, as listed.

BCM2712 (6-bit filter, the master number at AXI ID bits [8:3])

The documentation's numbering is sparse, with gaps. The previous driver table numbered the masters consecutively instead, so from VPU_L2 onward every filter selected the wrong master. The driver now uses the documented numbers.

Measured:

Master ID
VPU_UC0 1
VPU_IC0 2
VPU_DC0 3
VPU_L2 8 (small)
Arm (peripheral accesses) 11
DMA0 12
DMA1 13
PCIE1 (external connector) 19
PCIE2 (RP1) 20
HVS 32
MOP0, MOP1 (HVS writeback #0, #1) 34, 35
EMMC0 (WiFi controller) 39
ARGON (HEVC) 42
PISP (back end) 44
EMMC2 (SD card) 48

Notes:

  • V3D and the Arm use their own AXI IDs, not the master numbering. The documentation says the same.
  • The documentation's EMMC master names don't match the controller numbering. The controller that bus 8 serves is EMMC2, but its AXI master ID is 39, which the documentation calls "EMMC0". The SD card controller's master ID is 48, which the documentation calls "EMMC2". The filter numbers are the measured ones, so filtering works; only the names in the master enum may mislead.

Never seen on any bus, under every workload available:

  • VPU core 1 (4, 5, 6), VPU_DEBUG 10, DMA2 9, SYS_DMA 59
  • TRC 52 (probably debug trace)
  • HVS_WMK 33
  • MMU/MPU caches 60–63: IOMMU page walks are probably counted under the client's ID
  • Blocks unused on a Pi: RAAGA, BBSI, UMR, SAGE, HVDP, BSP, MBVN, XPT, BSTM, AIO, MAP, GENET, USB, UNICAM, PISPFE, JPEG, DSI
  • Not testable on the boards used: PCIE0 18, EMMC1 47

Results: register layout and counters

Watcher control fields

The writable bits were found by writing all ones to BWx_CTRL and reading the value back.

Bus field ID field ID mask Source
BCM2835/BCM2837 [4:0] [12:8] — Doc (BCM2711) and downstream. Filters 1–31 behave as expected
BCM2711 [4:0] [12:8] — Measured: writable bits 0xf0001fff
BCM2712 C0, both monitors [4:0] [13:8], master number — Measured: writable bits 0xf0003fff
BCM2712 D0, System monitor [4:0] [17:8], full AXI ID; master in ID bits [8:3] [27:18]; reset value 0x1f8 Doc, and measured: writable bits 0xf7fdffff
BCM2712 D0, VPU monitor [4:0] [13:8], master number [23:18], 6 bits Measured: writable bits 0xf0fc3fff. Differs from the D0 System monitor

On D0 the driver writes the ID mask explicitly. With a mask of 0, every transaction matches, so a filter would count all traffic. That was observed on hardware.

Counters

Counter Behaviour Source
atrans Address transactions, reads plus writes Doc; measured
ratrans Read address transactions only; absent on BCM2835–BCM2711 (the register reads the filler value 0x41584950, "AXIP") Doc; measured
rtrans, wtrans Data beats. Arm bursts are 2 beats (BCM2712). On the BCM2712 sd_dma bus a beat is 16 bytes, measured from HDMI audio's known data rate Measured
*wait, *max 28-bit; *max is a peak value, not a running total Doc. The 28-bit wrap wasn't reached in testing
rpend 8-bit; an instantaneous count of pending reads (0–255 seen) Doc; measured

The driver masks counter deltas to these widths, and reports *max and rpend as their current values rather than summing them.

Global control

Behaviour Source
BCM2712: the RESET bit clears itself Measured on D0
BCM2712: counters only count while WATCH_EN is set as well as EN Measured on D0
BCM2711: only ENABLE and RESET are documented; writing the BCM2712 WATCH_EN bit is harmless Doc; measured

Results: firmware access to the VPU monitor

The VPU monitor isn't accessible from the ARM. The driver reaches it through the firmware's GET/SET_PERIPH_REG calls, using the address in the second reg entry of the device tree.

Chip VPU monitor address Source
BCM2835/BCM2837 0x7ee08000 Measured: the only address the firmware accepts, and it reaches the hardware
BCM2711 0x7ee00000 Doc; measured. The previous DT value 0x7ee08000 is accepted by older firmware but reads zero
BCM2712 0x7ee00000 Doc; measured. The previous DT value 0x7e000000 reads zero

The device-tree changes in this series set these addresses.

Older firmware either rejects these calls or returns zeros. At probe, the driver writes a GEN_CTRL bit through the firmware and reads it back. If that fails, it logs one warning and hides the VPU events, so it never reports counts of zero by mistake.

A single 44-word GET_PERIPH_REG call reads all three VPU watchers on every chip tested.

Results: checks through the driver

All of the following were run with perf stat -a on every chip unless noted.

  • Event lists: match the tables above. That includes the separate D0 table, picked at probe.
  • Filters: filtered counts equal unfiltered counts for the expected master, and are 0 for others. Examples:
    • BCM2712: argon with filter 42, pcie_02 with 20, emmc2 with 39, sd_dma with 13 during DMA1 transfers.
    • D0 VPU filter 2 against 3.
  • Consistency: on the VPU monitor, vpu0_d_l2 atrans = rtrans + wtrans exactly, because all counters of a watcher are sampled together.
  • Multiplexing: with more VPU events open than watchers, the time-scaled counts land within about 4–12% of each event counted alone. The shortfall is the last ≤100 ms of each slice, which can't be read when an event is stopped. The driver defaults to a 1 s multiplexing interval when the VPU monitor is in use. With perf's default 4 ms interval, VPU events counted little or nothing.
  • Interval mode (-I 500): works for System and VPU events.
  • Per-task events: refused, as this is an uncore PMU.

Known hardware quirks found during testing

  • BCM2837: a watcher set to VPU bus 15 counts only while another watcher is set to bus 14, and then reads almost the same. This was reproduced with the driver unloaded, so it's hardware behaviour. The current VPU table stops at 14 for these chips.
  • BCM2712: some HEVC decoder transactions set AXI ID bit 9, although the documentation describes 9-bit IDs. Filtering on master 42 still works, because the D0 mask covers only ID bits [8:3].

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants