From 9710349a141eaa5489c7818f27500be5264683e2 Mon Sep 17 00:00:00 2001 From: prbasyal_amdeng Date: Wed, 7 Oct 2026 11:28:37 -0400 Subject: [PATCH 1/3] Known issue updated --- docs/about/release-notes.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/about/release-notes.md b/docs/about/release-notes.md index aecdb504..8e9f90a6 100644 --- a/docs/about/release-notes.md +++ b/docs/about/release-notes.md @@ -557,6 +557,14 @@ On AMD Instinct MI350P GPUs, the RAS error injection and query interfaces (used See [GitHub issue #8794](https://github.com/ROCm/TheRock/issues/8794). +### ComfyUI workloads might show lower performance on Radeon Pro W7900 GPUs + +ComfyUI-based AI workloads running on Radeon Pro W7900 GPUs might experience lower performance compared to previous ROCm releases. Impacted workloads include image and video generation models such as Stable Diffusion, Wan, and LTX. + +### Device-to-device hipMemcpyAsync might fail on AMD Instinct MI200 Series GPUs + +On AMD Instinct MI200 Series GPUs, device-to-device `hipMemcpyAsync` copies that use the DMA engine path (for example, with the `hipMemcpyDeviceToDeviceNoCU` flag) might fail without changing the destination buffer and reporting an error. This happens because the copy path selects an SDMA engine without properly validating which engines are actually usable for that GPU pair, so the transfer can target an engine that never completes the write. Although this issue was validated on PCIe-connected systems, the root cause is topology-agnostic and might also affect xGMI-linked systems. As a workaround, use hipMemcpyDeviceToDevice (without NoCU), which copies through compute units instead of DMA and is not affected. + ## ROCm resolved issues The following notable issues have been fixed in ROCm 10.1.0. From 297318b5c987ed55d84bea74fd75abdbb1469547 Mon Sep 17 00:00:00 2001 From: prbasyal_amdeng Date: Wed, 7 Oct 2026 14:13:35 -0400 Subject: [PATCH 2/3] GPU page fault issue added --- docs/about/release-notes.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/about/release-notes.md b/docs/about/release-notes.md index 8e9f90a6..327b6a80 100644 --- a/docs/about/release-notes.md +++ b/docs/about/release-notes.md @@ -563,7 +563,11 @@ ComfyUI-based AI workloads running on Radeon Pro W7900 GPUs might experience low ### Device-to-device hipMemcpyAsync might fail on AMD Instinct MI200 Series GPUs -On AMD Instinct MI200 Series GPUs, device-to-device `hipMemcpyAsync` copies that use the DMA engine path (for example, with the `hipMemcpyDeviceToDeviceNoCU` flag) might fail without changing the destination buffer and reporting an error. This happens because the copy path selects an SDMA engine without properly validating which engines are actually usable for that GPU pair, so the transfer can target an engine that never completes the write. Although this issue was validated on PCIe-connected systems, the root cause is topology-agnostic and might also affect xGMI-linked systems. As a workaround, use hipMemcpyDeviceToDevice (without NoCU), which copies through compute units instead of DMA and is not affected. +On AMD Instinct MI200 Series GPUs, device-to-device `hipMemcpyAsync` copies that use the DMA engine path (for example, with the `hipMemcpyDeviceToDeviceNoCU` flag) might fail without changing the destination buffer and reporting an error. This happens because the copy path selects an SDMA engine without properly validating which engines are actually usable for that GPU pair, so the transfer can target an engine that never completes the write. Although this issue was validated on PCIe-connected systems, the root cause is topology-agnostic and might also affect xGMI-linked systems. As a workaround, use `hipMemcpyDeviceToDevice` (without `NoCU`), which copies through compute units instead of DMA and is not affected. + +### Multi-application workloads might experience an intermittent GPU page fault on some Instinct GPUs + +Running multiple concurrent applications with high GPU memory-transfer activity might intermittently trigger a GPU page fault after several hours of runtime on some Instinct GPUs. When this happens, the affected job stops responding and must be restarted. The issue is more likely to occur when GPU memory is already under pressure from a recently run workload. ## ROCm resolved issues From 4fbb51cc65fdabb300d107b190478d980e11614b Mon Sep 17 00:00:00 2001 From: prbasyal_amdeng Date: Wed, 7 Oct 2026 18:29:39 -0400 Subject: [PATCH 3/3] GPU page fault issue added --- docs/about/release-notes.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/docs/about/release-notes.md b/docs/about/release-notes.md index 327b6a80..10618f61 100644 --- a/docs/about/release-notes.md +++ b/docs/about/release-notes.md @@ -553,21 +553,19 @@ Running workloads under Valgrind on WSL2 might stop responding during ROCm runti ### RAS error injection and query are unavailable on AMD Instinct MI350P GPUs -On AMD Instinct MI350P GPUs, the RAS error injection and query interfaces (used to validate error-handling paths, for example with the amdgpuras tool) are not supported in ROCm 10.1.0. Injection commands fail, and query commands report that the block doesn't support querying. This affects only manual RAS error injection and query tooling. Normal GPU reliability features such as ECC detection and error recovery during regular operation are not affected. - -See [GitHub issue #8794](https://github.com/ROCm/TheRock/issues/8794). +On AMD Instinct MI350P GPUs, the RAS error injection and query interfaces (used to validate error-handling paths, for example with the amdgpuras tool) are not supported in ROCm 10.1.0. Injection commands fail, and query commands report that the block doesn't support querying. This affects only manual RAS error injection and query tooling. Normal GPU reliability features such as ECC detection and error recovery during regular operation are not affected. See [GitHub issue #8794](https://github.com/ROCm/TheRock/issues/8794). ### ComfyUI workloads might show lower performance on Radeon Pro W7900 GPUs -ComfyUI-based AI workloads running on Radeon Pro W7900 GPUs might experience lower performance compared to previous ROCm releases. Impacted workloads include image and video generation models such as Stable Diffusion, Wan, and LTX. +ComfyUI-based AI workloads running on Radeon Pro W7900 GPUs might experience lower performance compared to previous ROCm releases. Impacted workloads include image and video generation models such as Stable Diffusion, Wan, and LTX. See [GitHub issue #8842](https://github.com/ROCm/TheRock/issues/8842). ### Device-to-device hipMemcpyAsync might fail on AMD Instinct MI200 Series GPUs -On AMD Instinct MI200 Series GPUs, device-to-device `hipMemcpyAsync` copies that use the DMA engine path (for example, with the `hipMemcpyDeviceToDeviceNoCU` flag) might fail without changing the destination buffer and reporting an error. This happens because the copy path selects an SDMA engine without properly validating which engines are actually usable for that GPU pair, so the transfer can target an engine that never completes the write. Although this issue was validated on PCIe-connected systems, the root cause is topology-agnostic and might also affect xGMI-linked systems. As a workaround, use `hipMemcpyDeviceToDevice` (without `NoCU`), which copies through compute units instead of DMA and is not affected. +On AMD Instinct MI200 Series GPUs, device-to-device `hipMemcpyAsync` copies that use the DMA engine path (for example, with the `hipMemcpyDeviceToDeviceNoCU` flag) might fail without changing the destination buffer and reporting an error. This happens because the copy path selects an SDMA engine without properly validating which engines are actually usable for that GPU pair, so the transfer can target an engine that never completes the write. Although this issue was validated on PCIe-connected systems, the root cause is topology-agnostic and might also affect xGMI-linked systems. As a workaround, use `hipMemcpyDeviceToDevice` (without `NoCU`), which copies through compute units instead of DMA and is not affected. See [GitHub issue #8843](https://github.com/ROCm/TheRock/issues/8843). ### Multi-application workloads might experience an intermittent GPU page fault on some Instinct GPUs -Running multiple concurrent applications with high GPU memory-transfer activity might intermittently trigger a GPU page fault after several hours of runtime on some Instinct GPUs. When this happens, the affected job stops responding and must be restarted. The issue is more likely to occur when GPU memory is already under pressure from a recently run workload. +Running multiple concurrent applications with high GPU memory-transfer activity, including SDMA-driven copies and SDMA engine stalls, might intermittently trigger a GPU page fault on some Instinct GPUs. When this happens, the affected job stops responding and must be restarted. In some cases, the GPU queue does not recover until a timeout of about 15 minutes expires, and the kernel log reports a retry page fault. The issue is more likely to occur when GPU memory is already under pressure from a recently run workload. See [GitHub issue #8844](https://github.com/ROCm/TheRock/issues/8844). ## ROCm resolved issues