cuda-checkpoint default restore preserves stale cuDeviceGetPCIBusId cache across cross-node gpu migration; NVML lookups fail as NVML look ups include cudaDeviceGetPciBusId() which returns old id and causes failure post restore #48

Summary

After a restore of a CUDA process onto a different node with different gpu uuids, but same pcie layout, cuDeviceGetPCIBusId(ord) returns the dump-side (source node) bus IDs, while cuDeviceGetUuid_v2(ord) and nvidia-smi both correctly return the restore-side (destination node) physical identities.

This mixed/inconsistent libcuda state breaks downstream consumers that resolve identity by bus ID. Specifically, NCCL's nvmlDeviceGetHandleByPciBusId() fails with Not Found during communicator initialization post restore, because NVML on the restore node has the destination's physical bus IDs, not the dumped ones.

This appears upstream of the issues tracked in #35 and #47 (--device-map) — the failure occurs on the default cuda-checkpoint restore path, with no gpuPairs involved.

Environment:

• Hardware: 8× NVIDIA B200 (180 GB HBM3e), same-SKU on both source and destination nodes (identical PCI layouts 03/13/63/73/83/93/E3/F3:00.0)
• NVIDIA driver: 580.105.08 on both nodes
• cuda-checkpoint tool: 570.86.10 (5976-byte stub binary forwarding via cuGetExportTable)
• CUDA toolkit in container: 12.8
• Framework: PyTorch 2.11, torchrun --nproc_per_node=4, NCCL 2.27.5
• Kubernetes: container partial-GPU allocation (4 of 8 GPUs per pod), device-plugin–driven

Expected

Either:

• (a) cuDeviceGetPCIBusId(ord) returns the restore-side physical bus ID (63/83/E3/F3), consistent with cuDeviceGetUuid_v2 and nvidia-smi; or
• (b) Documentation clearly states that bus IDs are preserved from dump-side (for ordinal-identity semantics), so NCCL / downstream consumers can reconcile.

Per the maintainer comment in #46 ("by default, it should restore GPU 0 to GPU 0"), behavior (a) appears intended. Current behavior is neither (a) nor documented.

Actual

Mixed/inconsistent libcuda state observed. Exact probe output from the restored process:

UUID is correctly re-resolved against the kernel; bus ID is not.

NCCL init failure (immediately downstream): FAIL: NCCL error in: /pytorch/torch/csrc/distributed/c10d/NCCLUtils.cpp:94,
unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.27.5
ncclSystemError: System call (e.g. socket, malloc) or external library call failed
or device error.
Last error:
nvmlDeviceGetHandleByPciBusId() failed: Not Found

Mechanism: NCCL reads the ordinal→bus_id mapping from libcuda (gets 03/13/63/73), then calls nvmlDeviceGetHandleByPciBusId("0000:03:00.0") against the restore-side NVML (which sees 63/83/E3/F3) → Not Found → NCCL abort.

Questions

1. Is this cuDeviceGetPCIBusId behavior intentional (ordinal-preserving restore, surface as OLD bus IDs by design)?
2. If intentional, how should a downstream consumer like NCCL know the libcuda bus-ID view is no longer authoritative?
3. If unintentional: what path refreshes libcuda's bus-ID table post-restore? Is there a public API, or is this a cuda-checkpoint restore-path gap?
4. Does the --device-map fix planned in Cannot get GPU migration to work with CUDA 13 and driver 580 #35/Could not restore on process ID 3406143: "invalid argument" #47 also address this default-path cache staleness, or are they independent issues?

What this means

libcuda has a stale bus-ID cache that dump preserved. For each ordinal, it returns Node A's dump-time bus IDs (03/13/63/73) while simultaneously returning Node B's fresh UUIDs.

This is a mixed/inconsistent libcuda state:

• cuDeviceGetPCIBusId(ord) → process-memory cache from Node A (OLD)
• cuDeviceGetUuid_v2(ord) → kernel query (NEW)
• nvidia-smi (independent process) → kernel query (NEW)