Skip to content

Pi 5 RP1: pio transer stuck when using channel dma2chan2 #7642

Description

@glaspiet

Describe the bug

Since 6.18.50, i am experiencing a problem when i run a slightly modified version of piotest on a Raspberry Pi 5: every 2nd run, the program hangs indefinitely inside rp1_pio_sm_xfer_data32_user() .

If i add print statements to piotest, the output shows that the for loop hangs on the second call to pio_sm_xfer_data from piolib:

timeout --signal=TERM --kill-after=5s 10s     ./pio-reproducer/pio-build-piotest/examples/piotest 2
Loaded program at -1, using sm 1, gpio 2
transfer 0 BEGIN
transfer 0 END rc=0
transfer 1 BEGIN

If i configure multiple buffers, for example 3 instead of 1: pio_sm_config_xfer(pio, sm, PIO_DIR_TO_SM, 2 * 1024 * 1024, 3);,
the program will get stuck at the 4th call to pio_sm_xfer_data.

Stacktrace shows that the program is stuck inside rp1_pio_sm_xfer_data32_user:

[<0>] rp1_pio_sm_xfer_data32_user+0x2c8/0x6a0 [rp1_pio]
[<0>] rp1_pio_sm_xfer_data_user+0x48/0x80 [rp1_pio]
[<0>] rp1_pio_ioctl+0x1a4/0x248 [rp1_pio]
[<0>] __arm64_sys_ioctl+0xb4/0x120
[<0>] invoke_syscall+0x4c/0x100
[<0>] el0_svc_common.constprop.0+0x48/0xf0
[<0>] do_el0_svc+0x24/0x38
[<0>] el0_svc+0x38/0x120
[<0>] el0t_64_sync_handler+0xa0/0xe8
[<0>] el0t_64_sync+0x198/0x1a0
=== /proc/2468/task/2469 ===
[<0>] rp1_pio_irq_wait+0x40/0xa8 [rp1_pio]
[<0>] rp1_pio_ioctl+0x1a4/0x248 [rp1_pio]
[<0>] __arm64_sys_ioctl+0xb4/0x120
[<0>] invoke_syscall+0x4c/0x100
[<0>] el0_svc_common.constprop.0+0x48/0xf0
[<0>] do_el0_svc+0x24/0x38
[<0>] el0_svc+0x38/0x120
[<0>] el0t_64_sync_handler+0xa0/0xe8
[<0>] el0t_64_sync+0x198/0x1a0

Whenever the problem occurs, running cat /sys/kernel/debug/dmaengine/summary always shows that channel dma2chan2 is being used:

dma2 (1f00188000.dma): number of channels: 8
 dma2chan2    | 1f00178000.pio:tx0

Conversely, if the program runs without problems, it always uses dma2chan0.

After the successful run, the following logs appear (journalctl -kf -o short-monotonic):

[ 3346.002998] cpdis-2505-117 kernel: dma dma2chan0: dma2chan0 failed to stop
[ 3346.003394] cpdis-2505-117 kernel: dma dma2chan0: dma2chan0 is non-idle!

Then during the second run, the program hangs and the logs show:

[ 3357.420030] cpdis-2505-117 kernel: dma dma2chan0: dma2chan0 is non-idle!
[ 3362.420028] cpdis-2505-117 kernel: rp1-pio 1f00178000.pio: DMA bounce interrupted

It seems that dma2chan0 cannot be stopped during the 1st run, after which dma2chan2 is selected during the 2nd run, but this channel never finishes. However dma2chan0 is stopped somehow during the 2nd run because on the 3rd run, the programs runs OK again.

I ran into this because i am controlling a led panel, using the hzeller lib. But the most basic way to reproduce the issue is to use a modified piotest without anything attached to the raspberrpy pi.

Steps to reproduce the behaviour

sudo apt update
sudo apt install git build-essential cmake
mkdir /tmp/pio-reproducer && cd pio-reproducer
git clone https://github.com/raspberrypi/utils.git rpi-utils

In rpi-utils/piolib/examples/piotest.c, replace:

      pio_sm_xfer_data(pio, sm, PIO_DIR_TO_SM, sizeof(databuf), databuf);

with:

      fprintf(stderr, "transfer %d BEGIN\n", pass + 1);
      fflush(stderr);

      pio_sm_xfer_data(pio, sm, PIO_DIR_TO_SM,
                       sizeof(databuf), databuf);

      fprintf(stderr, "transfer %d END\n", pass + 1);
      fflush(stderr);

and replace:

      const uint pixels = 60;
      .
      .
      pio_sm_config_xfer(pio, sm, PIO_DIR_TO_SM, 256, 1);

with:

      const uint pixels = 6144;
      .
      .
      pio_sm_config_xfer(pio, sm, PIO_DIR_TO_SM,
                 2 * 1024 * 1024, 1);

Build and run:

cmake -S rpi-utils/piolib -B pio-build-piotest \
      -DCMAKE_BUILD_TYPE=Debug \
      -DBUILD_SHARED_LIBS=OFF

cmake --build pio-build-piotest --target piotest -j2

sudo timeout --signal=TERM --kill-after=5s 10s \
      ./pio-build-piotest/examples/piotest 2

With this program i can reproduce the issue every second time i run it. Even taking longer pauses between runs (30 seconds) does not resolve the issue.

After the successful run, the following logs appear (journalctl -kf -o short-monotonic):

[ 3346.002998] cpdis-2505-117 kernel: dma dma2chan0: dma2chan0 failed to stop
[ 3346.003394] cpdis-2505-117 kernel: dma dma2chan0: dma2chan0 is non-idle!

Then during the second run, the program hangs and the logs show:

[ 3357.420030] cpdis-2505-117 kernel: dma dma2chan0: dma2chan0 is non-idle!
[ 3362.420028] cpdis-2505-117 kernel: rp1-pio 1f00178000.pio: DMA bounce interrupted

The third run will be ok, with logs as in the first run. During the fourth run the problem occurs again. And so on and so forth.

The issue can be reproduced with the original piotest, but it occurs far less often, usually every 50th time or so. I tried two different raspberry's, same issue.

Device (s)

Raspberry Pi 5

System

Raspberry Pi 5 Model B Rev 1.1
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
NAME="Debian GNU/Linux"
VERSION_ID="13"
VERSION="13 (trixie)"

Raspberry Pi reference 2026-04-21
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 4ad56cc850fa60adcc7f07dc15879bc95cc1d281, stage2

Linux cpdis-2505-122 6.18.50+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.18.50-1+rpt1 (2026-09-11) aarch64 GNU/Linux
Revision : d04171
Serial : 935509188ec7e445
Model : Raspberry Pi 5 Model B Rev 1.1
Throttled flag : throttled=0x0
.
.
[ 0.000000] Linux version 6.18.50+rpt-rpi-2712 (serge@raspberrypi.com) (aarch64-linux-gnu-gcc-14 (Debian 14.2.0-19) 14.2.0, GNU ld (GNU Binutils for Debian) 2.44) #1 SMP PREEMPT Debian 1:6.18.50-1+rpt1 (2026-09-11)

Logs

No response

Additional context

No response

Activity

  1. changed the title [-]Pi 5 RP1 PIO: channel problem[/-] [+]Pi 5 RP1: pio transer stuck when using channel dma2chan2[/+] on Sep 25, 2026
  2. pelwell commented on Sep 25, 2026

    @pelwell
    Contributor

    There seems to be a problem associated with killing the application - even before running for a second time you see:

    [  676.469606] rp1-pio 1f00178000.pio: DMA bounce interrupted
    [  676.520073] dma dma2chan0: dma2chan0 failed to stop
    [  676.520141] dma dma2chan0: dma2chan0 is non-idle!
    

    Actually, I think it's a general problem stopping mid-DMA transfer, one which is exposed by the larger volumes of data and slow clock (800kHz). Either way, I'll get to the bottom of it.

  3. pelwell commented on Sep 28, 2026

    @pelwell
    Contributor

    #7654 should fix the problem by adding support for the ABORT feature of the DMA controller. It also reduces the number of alarming error messages in the kernel log for expected situations.

    In about 15 minutes you should be able to install a trial build by running sudo rpi-update pulls/7654, but read the caveats first.

    By the way, the parameters to your timeout command don't make a lot of sense - the kill signal is only used as a last resort if the other signal (default SIGTERM) sent after the timeout doesn't result in the process exiting, so it only has any effect if the interval is larger than the normal timeout.

  4. glaspiet commented on Sep 28, 2026

    @glaspiet
    Author

    That's quick, thanks a lot! Just installed the trial build, i'll let my program run for a while.

    By the way, the parameters to your timeout command don't make a lot of sense - the kill signal is only used as a last resort if the other signal (default SIGTERM) sent after the timeout doesn't result in the process exiting, so it only has any effect if the interval is larger than the normal timeout.

    According to the docs, duration specified in --kill-after=duration starts from the point where timeout sends the initial signal, so in this case would be send after 10+5 seconds. But signal=TERM does not achieve anything indeed, thanks for pointing that out.

  5. pelwell commented on Sep 29, 2026

    @pelwell
    Contributor

    Thanks for the correction re the kill-after interval; although less obviously worded, the version of the man page we install does mention that if one looks hard enough.

    Having satisfied myself that they are correct, the changes to the DMA and PIO drivers have now been merged and will be in future releases.

  6. glaspiet commented on Sep 30, 2026

    @glaspiet
    Author

    My program has been running without any failures since i installed the trial build. Thanks again for the quick fix!

  7. pelwell commented on Sep 30, 2026

    @pelwell
    Contributor

    Thanks for the feedback. The PR has been merged, so future kernel releases will include the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions