gpu.error.FspRpcError occurs when setting CC mode #13
Description
I verified that the current CC mode is set to off. However, even though the GPU is bound to the NVIDIA driver, running nvidia-smi takes a long time and eventually returns “no devices”.
juntaek@h100:~$ sudo python3 /shared/gpu-admin-tools/nvidia_gpu_tools.py --gpu-bdf=3a:00.0 --query-cc-mode
NVIDIA GPU Tools version v2025.07.21o
Command line arguments: ['/shared/gpu-admin-tools/nvidia_gpu_tools.py', '--gpu-bdf=3a:00.0', '--query-cc-mode']
2025-09-16,12:34:15.923 WARNING GPU 0000:3a:00.0 ? 0x2321 BAR0 0x0 was in D0/control:auto, forced power control to on. New state D0
Topo:
Intel root port 0000:37:01.0
PciBridge 0000:38:00.0 c030:1000
PciBridge 0000:39:00.0 c030:1000
GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000
2025-09-16,12:34:15.925 INFO Selected GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000
2025-09-16,12:34:15.925 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 CC mode is off
2025-09-16,12:34:15.926 WARNING GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 restoring power control to autoWhen I try to set the CC mode to off again using the --set-cc-mode option, I encounter the following error. How can I resolve this issue?
juntaek@h100:~$ sudo python3 /shared/gpu-admin-tools/nvidia_gpu_tools.py --set-cc-mode=off --reset-after-cc-mode-switch --gpu-bdf=3a:00.0
NVIDIA GPU Tools version v2025.07.21o
Command line arguments: ['/shared/gpu-admin-tools/nvidia_gpu_tools.py', '--set-cc-mode=off', '--reset-after-cc-mode-switch', '--gpu-bdf=3a:00.0']
2025-09-16,12:25:49.804 WARNING GPU 0000:3a:00.0 ? 0x2321 BAR0 0x0 was in D0/control:auto, forced power control to on. New state D0
Topo:
Intel root port 0000:37:01.0
PciBridge 0000:38:00.0 c030:1000
PciBridge 0000:39:00.0 c030:1000
GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000
2025-09-16,12:25:49.806 INFO Selected GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000
File "/shared/gpu-admin-tools/cli/per_gpu.py", line 198, in main_per_gpu
gpu.set_cc_mode(opts.set_cc_mode)
File "/shared/gpu-admin-tools/nvidia_gpu_tools.py", line 3722, in set_cc_mode
self.fsp_rpc.prc_knob_check_and_write(PrcKnob.PRC_KNOB_ID_BAR0_DECOUPLER.value, bar0_decoupler_val)
File "/shared/gpu-admin-tools/nvidia_gpu_tools.py", line 2881, in prc_knob_check_and_write
old_value = self.prc_knob_read(knob_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/shared/gpu-admin-tools/nvidia_gpu_tools.py", line 2851, in prc_knob_read
data = self.prc_cmd([prc])
^^^^^^^^^^^^^^^^^^^
File "/shared/gpu-admin-tools/nvidia_gpu_tools.py", line 2777, in prc_cmd
return self.send_cmd(0x13, data, sync=sync)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/shared/gpu-admin-tools/nvidia_gpu_tools.py", line 2772, in send_cmd
raise FspRpcError(self, mdata[4], mdata)
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 boot_status 0x200bc = 0xff
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 boot_flags 0x20120 = 0x0
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 fsp_status_0 0x8f0320 = 0x40
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 fsp_status_1 0x8f0324 = 0x110186
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 fsp_status_2 0x8f0328 = 0x1fa824
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 fsp_status_3 0x8f032c = 0xbadf57eb
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 last_reset 0x9128c = 0x0
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 prc_0 0x92de0 = 0x0
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 prc_1 0x92de4 = 0x0
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 prc_2 0x92de8 = 0x0
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 prc_cold 0x92dc0 = 0x0
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_00 0x1400 = 0x0
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_01 0x1404 = 0x0
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_02 0x1408 = 0x0
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_03 0x140c = 0x0
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_04 0x1410 = 0x0
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_05 0x1414 = 0x0
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_06 0x1418 = 0x0
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_07 0x141c = 0x0
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_08 0x1420 = 0x0
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_09 0x1424 = 0x0
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_10 0x1428 = 0x0
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_11 0x142c = 0x0
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_12 0x1430 = 0x0
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_13 0x1434 = 0x0
2025-09-16,12:25:49.828 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_14 0x1438 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_15 0x143c = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_16 0x1440 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_17 0x1444 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_18 0x1448 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_19 0x144c = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_20 0x1450 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_21 0x1454 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_22 0x1458 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_23 0x145c = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_24 0x1460 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_25 0x1464 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_26 0x1468 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_27 0x146c = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_28 0x1470 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_29 0x1474 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_30 0x1478 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_31 0x147c = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_32 0x1480 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_33 0x1484 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_34 0x1488 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_35 0x148c = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_36 0x1490 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_37 0x1494 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_38 0x1498 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_39 0x149c = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_40 0x14a0 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_41 0x14a4 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_42 0x14a8 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_43 0x14ac = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_44 0x14b0 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_45 0x14b4 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_46 0x14b8 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_47 0x14bc = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_48 0x14c0 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_49 0x14c4 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_50 0x14c8 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_51 0x14cc = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_52 0x14d0 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_53 0x14d4 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_54 0x14d8 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_55 0x14dc = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_56 0x14e0 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_57 0x14e4 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_58 0x14e8 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_59 0x14ec = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_60 0x14f0 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_61 0x14f4 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_62 0x14f8 = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_63 0x14fc = 0x0
2025-09-16,12:25:49.829 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 nvlink debug print
Traceback (most recent call last):
File "/shared/gpu-admin-tools/cli/per_gpu.py", line 198, in main_per_gpu
gpu.set_cc_mode(opts.set_cc_mode)
File "/shared/gpu-admin-tools/nvidia_gpu_tools.py", line 3722, in set_cc_mode
self.fsp_rpc.prc_knob_check_and_write(PrcKnob.PRC_KNOB_ID_BAR0_DECOUPLER.value, bar0_decoupler_val)
File "/shared/gpu-admin-tools/nvidia_gpu_tools.py", line 2881, in prc_knob_check_and_write
old_value = self.prc_knob_read(knob_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/shared/gpu-admin-tools/nvidia_gpu_tools.py", line 2851, in prc_knob_read
data = self.prc_cmd([prc])
^^^^^^^^^^^^^^^^^^^
File "/shared/gpu-admin-tools/nvidia_gpu_tools.py", line 2777, in prc_cmd
return self.send_cmd(0x13, data, sync=sync)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/shared/gpu-admin-tools/nvidia_gpu_tools.py", line 2772, in send_cmd
raise FspRpcError(self, mdata[4], mdata)
gpu.error.FspRpcError: GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 FSP-RPC failed with error 0x202. Data ['0xc0000000', '0x1510de7e', '0x3', '0x13', '0x202']
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/shared/gpu-admin-tools/nvidia_gpu_tools.py", line 4429, in <module>
cli.main.main()
File "/shared/gpu-admin-tools/cli/main.py", line 359, in main
if not main_per_gpu(d, opts):
^^^^^^^^^^^^^^^^^^^^^
File "/shared/gpu-admin-tools/cli/per_gpu.py", line 203, in main_per_gpu
prc_knobs = gpu.query_prc_knobs()
^^^^^^^^^^^^^^^^^^^^^
File "/shared/gpu-admin-tools/nvidia_gpu_tools.py", line 1815, in query_prc_knobs
knob_value = self.fsp_rpc.prc_knob_read(knob.value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/shared/gpu-admin-tools/nvidia_gpu_tools.py", line 2851, in prc_knob_read
data = self.prc_cmd([prc])
^^^^^^^^^^^^^^^^^^^
File "/shared/gpu-admin-tools/nvidia_gpu_tools.py", line 2777, in prc_cmd
return self.send_cmd(0x13, data, sync=sync)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/shared/gpu-admin-tools/nvidia_gpu_tools.py", line 2772, in send_cmd
raise FspRpcError(self, mdata[4], mdata)
gpu.error.FspRpcError: GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 FSP-RPC failed with error 0x202. Data ['0xc0000000', '0x1510de7e', '0x3', '0x13', '0x202']
2025-09-16,12:25:49.891 WARNING GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 restoring power control to autoHere is the debug log:
juntaek@h100:~$ sudo python3 /shared/gpu-admin-tools/nvidia_gpu_tools.py --debug-dump --log debug --gpu-bdf=3a:00.0
NVIDIA GPU Tools version v2025.07.21o
Command line arguments: ['/shared/gpu-admin-tools/nvidia_gpu_tools.py', '--debug-dump', '--log', 'debug', '--gpu-bdf=3a:00.0']
2025-09-16,12:27:16.985 WARNING GPU 0000:3a:00.0 ? 0x2321 BAR0 0x0 was in D0/control:auto, forced power control to on. New state D0
Topo:
Intel root port 0000:37:01.0
PciBridge 0000:38:00.0 c030:1000
PciBridge 0000:39:00.0 c030:1000
GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000
2025-09-16,12:27:16.987 INFO Selected GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 debug dump:
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 boot_status 0x200bc = 0xff
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 boot_flags 0x20120 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 fsp_status_0 0x8f0320 = 0x40
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 fsp_status_1 0x8f0324 = 0x110186
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 fsp_status_2 0x8f0328 = 0x1fa824
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 fsp_status_3 0x8f032c = 0xbadf57eb
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 last_reset 0x9128c = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 prc_0 0x92de0 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 prc_1 0x92de4 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 prc_2 0x92de8 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 prc_cold 0x92dc0 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_00 0x1400 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_01 0x1404 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_02 0x1408 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_03 0x140c = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_04 0x1410 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_05 0x1414 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_06 0x1418 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_07 0x141c = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_08 0x1420 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_09 0x1424 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_10 0x1428 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_11 0x142c = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_12 0x1430 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_13 0x1434 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_14 0x1438 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_15 0x143c = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_16 0x1440 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_17 0x1444 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_18 0x1448 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_19 0x144c = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_20 0x1450 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_21 0x1454 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_22 0x1458 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_23 0x145c = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_24 0x1460 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_25 0x1464 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_26 0x1468 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_27 0x146c = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_28 0x1470 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_29 0x1474 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_30 0x1478 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_31 0x147c = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_32 0x1480 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_33 0x1484 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_34 0x1488 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_35 0x148c = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_36 0x1490 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_37 0x1494 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_38 0x1498 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_39 0x149c = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_40 0x14a0 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_41 0x14a4 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_42 0x14a8 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_43 0x14ac = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_44 0x14b0 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_45 0x14b4 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_46 0x14b8 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_47 0x14bc = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_48 0x14c0 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_49 0x14c4 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_50 0x14c8 = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_51 0x14cc = 0x0
2025-09-16,12:27:16.988 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_52 0x14d0 = 0x0
2025-09-16,12:27:16.989 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_53 0x14d4 = 0x0
2025-09-16,12:27:16.989 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_54 0x14d8 = 0x0
2025-09-16,12:27:16.989 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_55 0x14dc = 0x0
2025-09-16,12:27:16.989 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_56 0x14e0 = 0x0
2025-09-16,12:27:16.989 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_57 0x14e4 = 0x0
2025-09-16,12:27:16.989 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_58 0x14e8 = 0x0
2025-09-16,12:27:16.989 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_59 0x14ec = 0x0
2025-09-16,12:27:16.989 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_60 0x14f0 = 0x0
2025-09-16,12:27:16.989 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_61 0x14f4 = 0x0
2025-09-16,12:27:16.989 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_62 0x14f8 = 0x0
2025-09-16,12:27:16.989 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 vbios_scratch_63 0x14fc = 0x0
2025-09-16,12:27:16.989 INFO GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 nvlink debug print
2025-09-16,12:27:16.989 WARNING GPU 0000:3a:00.0 H100 0x2321 BAR0 0x7e042000000 restoring power control to auto