使用simai 模拟物理发包，但是没有办法running起来，一直显示打开设备失败

[Tupperis]

mpirun --prefix /usr/local/openmpi-4.1.6 -np 2
-hostfile ./hostfile
--allow-run-as-root
--mca pml ob1
--mca btl self,tcp,vader
--mca btl_openib_warn_no_device_params_found 0
-x AS_LOG_LEVEL=0
./bin/SimAI_phynet ./hostlist -g 2
-w /home/SimAI/aicb/results/workload/None-gpt_13B-world_size2-tp2-pp1-ep1-gbs32-mbs1-seq4096-MOE-False-GEMM-False-flash_attn-True.txt
-i 3
SimAI_phynet: /home/SimAI/astra-sim-alibabacloud/astra-sim/system/SimAiFlowModelRdma.cc:706: int FlowPhyRdma::ibv_init(): Assertion g_ibv_ctx != NULL' failed. [test:3070717] *** Process received signal *** [test:3070717] Signal: Aborted (6) [test:3070717] Signal code: (-6) [test:3070717] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x78a877a45330] [test:3070717] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x78a877a9eb2c] [test:3070717] [ 2] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x78a877a4527e] [test:3070717] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0xdf)[0x78a877a288ff] [test:3070717] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x2881b)[0x78a877a2881b] [test:3070717] [ 5] /lib/x86_64-linux-gnu/libc.so.6(+0x3b517)[0x78a877a3b517] [test:3070717] [ 6] ./bin/SimAI_phynet(+0x314c7)[0x5885632614c7] [test:3070717] [ 7] ./bin/SimAI_phynet(+0x20c68)[0x588563250c68] [test:3070717] [ 8] /lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x78a877a2a1ca] [test:3070717] [ 9] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x78a877a2a28b] [test:3070717] [10] ./bin/SimAI_phynet(+0x1bf35)[0x58856324bf35] [test:3070717] *** End of error message *** SimAI_phynet: /home/SimAI/astra-sim-alibabacloud/astra-sim/system/SimAiFlowModelRdma.cc:706: int FlowPhyRdma::ibv_init(): Assertion g_ibv_ctx != NULL' failed.
[test-Super-Server:40524] *** Process received signal ***
[test-Super-Server:40524] Signal: Aborted (6)
[test-Super-Server:40524] Signal code: (-6)
[test-Super-Server:40524] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7e33a3442520]
[test-Super-Server:40524] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7e33a34969fc]
[test-Super-Server:40524] [ 2] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7e33a3442476]
[test-Super-Server:40524] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7e33a34287f3]
[test-Super-Server:40524] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x2871b)[0x7e33a342871b]
[test-Super-Server:40524] [ 5] /lib/x86_64-linux-gnu/libc.so.6(+0x39e96)[0x7e33a3439e96]
[test-Super-Server:40524] [ 6] ./bin/SimAI_phynet(+0x30961)[0x59ba865f3961]
[test-Super-Server:40524] [ 7] ./bin/SimAI_phynet(+0x1fc95)[0x59ba865e2c95]
[test-Super-Server:40524] [ 8] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7e33a3429d90]
[test-Super-Server:40524] [ 9] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7e33a3429e40]
[test-Super-Server:40524] [10] ./bin/SimAI_phynet(+0x1b015)[0x59ba865de015]
[test-Super-Server:40524] *** End of error message ***

Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.

---

mpirun noticed that process rank 1 with PID 40524 on node node02 exited on signal 6 (Aborted).

ibdev2netdev(const char indev[64], char netdev[64]) {
  if (strncmp(indev, "mlx5_bond_", 10) == 0) {
    strcpy(netdev, "bond");
    strcat(netdev, indev + 10);
  } else {
    strcpy(netdev, indev);
  }
}
这串代码逻辑看起来用的奇怪，感觉找到了内核的网口名不是ib设备的

[tianhao909]

【人工回复】
hi， 修改 ibdev2netdev() 函数，利用 sysfs 自动查询真实网口名，同时修改 ibv_init() 中的调用 试试？
有什么问题也可以追问看看， 也可以用coding agent 一起调一调bug； 比如 qoder 、cc、 codex 等等

你可以用你们的数据校准校准，和真机测试做一下校准，看看这块仿真器功能对于你们场景的精度怎么样

[tianhao909]

【Agent A回复】【仅供参考】

CN

Issue #280 回复：SimAI 物理发包 RDMA 设备打开失败

问题确认

感谢您的反馈！您的分析方向是正确的。ibdev2netdev() 函数的逻辑确实存在问题，导致在非阿里云 bond 网卡环境下无法正确完成 RDMA 设备初始化。

根因分析

问题出在 astra-sim/system/SimAiFlowModelRdma.cc 中的 ibdev2netdev() 函数（L664-671）：

static void 
ibdev2netdev(const char indev[64], char netdev[64]) {
  if (strncmp(indev, "mlx5_bond_", 10) == 0) {
    strcpy(netdev, "bond");
    strcat(netdev, indev + 10);
  } else {
    strcpy(netdev, indev);  // 问题所在
  }
}

具体原因：

1. IB 设备名与 Linux 网络接口名不一致：标准 Mellanox/NVIDIA 网卡的 IB 设备名为 mlx5_0、mlx5_1 等，但对应的 Linux 网络接口名通常为 eth0、ens1f0、enp1s0f0 等。else 分支直接将 IB 设备名（如 mlx5_0）当作网络接口名传入 ioctl(SIOCGIFADDR)，导致 IP 查询失败。

2. 仅适配了阿里云 bond 场景：mlx5_bond_X → bondX 的映射规则只在阿里云特定 bond 网卡配置下有效，不具备通用性。

3. 设备匹配流程：ibv_init() 中遍历所有 IB 设备，通过 ibdev2netdev() 获取网口名，再通过 ioctl 查询 IP，与 hostlist 中的 IP 比对。由于网口名错误，所有设备都无法匹配，最终 ibv_open_device() 操作的是未匹配/未初始化的指针，触发断言失败。

临时绕过方案

如果您的环境急需运行，可以尝试以下临时方案：

方案1：手动创建符号链接（不推荐，仅供验证）

确认您的 IB 设备名和对应网口名：

# 查看 IB 设备
ibv_devices

# 查看 IB 设备对应的网络接口
cat /sys/class/infiniband/mlx5_0/ports/1/gid_attrs/ndevs/0
# 例如输出：ens1f0

方案2：本地修改代码后重新编译

修改 ibdev2netdev() 函数，利用 sysfs 自动查询真实网口名：

static void 
ibdev2netdev(const char indev[64], char netdev[64], int port, int gid_idx) {
  char path[256];
  // 通过 sysfs 查询 IB 设备对应的网络接口名
  snprintf(path, sizeof(path), 
    "/sys/class/infiniband/%s/ports/%d/gid_attrs/ndevs/%d", indev, port, gid_idx);
  
  FILE* fp = fopen(path, "r");
  if (fp) {
    if (fscanf(fp, "%63s", netdev) != 1) {
      strcpy(netdev, indev); // fallback
    }
    fclose(fp);
  } else if (strncmp(indev, "mlx5_bond_", 10) == 0) {
    // 兼容旧的 bond 映射逻辑
    strcpy(netdev, "bond");
    strcat(netdev, indev + 10);
  } else {
    strcpy(netdev, indev); // fallback
  }
}

同时修改 ibv_init() 中的调用：

ibdev2netdev(dev_lst[i]->name, netdev, IB_PORT, gid_index);

并修复以下附加问题：

• 将 struct ibv_device* ibv_dev; 改为 struct ibv_device* ibv_dev = NULL;（防止未初始化指针的未定义行为）
• 将 assert(g_ibv_ctx != NULL) 替换为更友好的错误日志并返回错误码

环境检查步骤

在运行 SimAI_phynet 之前，请确认以下检查项全部通过：

# 1. 确认 IB 设备存在
ibv_devices
# 期望输出：列出 mlx5_0 等设备

# 2. 确认设备状态正常
ibv_devinfo
# 期望看到 state: PORT_ACTIVE

# 3. 确认 sysfs 中能查到对应网口名
cat /sys/class/infiniband/mlx5_0/ports/1/gid_attrs/ndevs/0
# 期望输出：类似 ens1f0、eth0 等

# 4. 确认该网口有 IP 地址且与 hostlist 中的 IP 一致
ip addr show <网口名>

# 5. 确认 RDMA 基本通信正常
# 服务端：ib_write_bw -d mlx5_0
# 客户端：ib_write_bw -d mlx5_0 <服务端IP>

# 6. 确认 GID index 正确（-i 参数）
cat /sys/class/infiniband/mlx5_0/ports/1/gids/3
# 使用 RoCEv2 时通常选择非零且对应正确网口的 GID index

补充说明

• -i 参数（gid_index）需要与您实际的 RoCE GID index 对应，不同环境下可能不同（常见为 0 或 3）
• hostlist 文件中的 IP 必须是 RDMA 网卡对应网口的 IP，而非管理口 IP
• 当前代码是为阿里云内部特定 bond 网卡环境编写的，通用场景下需要上述修改

如果以上信息不足以解决您的问题，请提供以下信息以便进一步定位：

1. ibv_devices 的输出
2. ibv_devinfo -v 的输出
3. ip addr show 的输出（关注 RDMA 网卡对应网口）
4. hostlist 文件内容
5. 您使用的 -i 参数值

EN

Issue #280 Reply: SimAI Physical Packet Injection - RDMA Device Open Failure

Problem Confirmation

Thank you for your report! Your analysis is on the right track. The ibdev2netdev() function indeed has a logic issue that prevents proper RDMA device initialization in non-Alibaba Cloud (non-bond NIC) environments.

Root Cause Analysis

The issue is in astra-sim/system/SimAiFlowModelRdma.cc, the ibdev2netdev() function (L664-671):

static void 
ibdev2netdev(const char indev[64], char netdev[64]) {
  if (strncmp(indev, "mlx5_bond_", 10) == 0) {
    strcpy(netdev, "bond");
    strcat(netdev, indev + 10);
  } else {
    strcpy(netdev, indev);  // BUG: uses IB device name as network interface name
  }
}

Specific Causes:

1. IB device name != Linux network interface name: Standard Mellanox/NVIDIA NICs have IB device names like mlx5_0, mlx5_1, while their corresponding Linux network interface names are typically eth0, ens1f0, enp1s0f0, etc. The else branch directly passes the IB device name (e.g., mlx5_0) as the network interface name to ioctl(SIOCGIFADDR), causing the IP lookup to fail.

2. Only adapted for Alibaba Cloud bond scenario: The mlx5_bond_X → bondX mapping only works in Alibaba Cloud's specific bond NIC configuration and lacks generality.

3. Device matching flow: ibv_init() iterates through all IB devices, calls ibdev2netdev() to get the network interface name, then queries the IP via ioctl and compares against the IP in hostlist. Due to incorrect interface names, no device can match. The code then operates on an uninitialized/unmatched pointer, triggering the assertion failure.

Temporary Workaround

If you need to run SimAI_phynet urgently, here are some temporary approaches:

Option 1: Verify your IB device mapping

First, check your IB device name and its corresponding network interface:

# List IB devices
ibv_devices

# Check the network interface mapped to the IB device
cat /sys/class/infiniband/mlx5_0/ports/1/gid_attrs/ndevs/0
# Expected output: e.g., ens1f0

Option 2: Local code modification and recompile

Modify the ibdev2netdev() function to automatically query the real network interface name via sysfs:

static void 
ibdev2netdev(const char indev[64], char netdev[64], int port, int gid_idx) {
  char path[256];
  // Query via sysfs: /sys/class/infiniband/<dev>/ports/<port>/gid_attrs/ndevs/<gid>
  snprintf(path, sizeof(path), 
    "/sys/class/infiniband/%s/ports/%d/gid_attrs/ndevs/%d", indev, port, gid_idx);
  
  FILE* fp = fopen(path, "r");
  if (fp) {
    if (fscanf(fp, "%63s", netdev) != 1) {
      strcpy(netdev, indev); // fallback
    }
    fclose(fp);
  } else if (strncmp(indev, "mlx5_bond_", 10) == 0) {
    // Backward compatible: old bond mapping logic
    strcpy(netdev, "bond");
    strcat(netdev, indev + 10);
  } else {
    strcpy(netdev, indev); // fallback
  }
}

Also modify the call site in ibv_init():

ibdev2netdev(dev_lst[i]->name, netdev, IB_PORT, gid_index);

Additional fixes:

• Change struct ibv_device* ibv_dev; to struct ibv_device* ibv_dev = NULL; (prevent undefined behavior from uninitialized pointer)
• Replace assert(g_ibv_ctx != NULL) with a user-friendly error message and return an error code

Environment Verification Steps

Before running SimAI_phynet, please verify the following checks pass:

# 1. Verify IB devices exist
ibv_devices
# Expected: lists devices like mlx5_0

# 2. Verify device status is active
ibv_devinfo
# Expected: state: PORT_ACTIVE

# 3. Verify sysfs can resolve the network interface name
cat /sys/class/infiniband/mlx5_0/ports/1/gid_attrs/ndevs/0
# Expected: something like ens1f0, eth0, etc.

# 4. Verify the interface has an IP matching your hostlist
ip addr show <interface_name>

# 5. Verify basic RDMA communication works
# Server: ib_write_bw -d mlx5_0
# Client: ib_write_bw -d mlx5_0 <server_ip>

# 6. Verify correct GID index (-i parameter)
cat /sys/class/infiniband/mlx5_0/ports/1/gids/3
# For RoCEv2, typically choose a non-zero GID index that corresponds to the correct interface

Additional Notes

• The -i parameter (gid_index) must correspond to your actual RoCE GID index, which varies across environments (commonly 0 or 3)
• The IPs in the hostlist file must be the IPs of the RDMA NIC's network interfaces, NOT the management interface IPs
• The current code was written for Alibaba Cloud's specific bond NIC environment; generic environments require the modifications described above

If the above information is insufficient to resolve your issue, please provide:

1. Output of ibv_devices
2. Output of ibv_devinfo -v
3. Output of ip addr show (focusing on RDMA NIC interfaces)
4. Contents of your hostlist file
5. The -i parameter value you are using

[tianhao909]

【Agent B回复】【仅供参考】

CN

中文版 issue 回复

1. 需求背景总结

这个 issue 是在使用 SimAI_phynet 做 physical network / RoCEv2 物理发包 时，程序在初始化 RDMA verbs 设备阶段失败：

FlowPhyRdma::ibv_init(): Assertion `g_ibv_ctx != NULL' failed.

用户运行的是 SimAI_phynet，输入包含：

• MPI hostfile
• SimAI hostlist
• AICB 生成的 workload
• -g 2
• -i 3 指定 RDMA gid_index

根据现有代码，SimAI_phynet 会在启动时调用 BootStrapNet() 读取 hostlist，将 local_rank 映射到一个 IP；随后 FlowPhyRdma::ibv_init() 遍历本机 RDMA 设备列表，尝试通过“RDMA 设备名 -> Linux 网卡名 -> 网卡 IP”的方式找到与 hostlist[local_rank] 对应的 RDMA 设备，然后调用 ibv_open_device()。

用户怀疑的点是对的：当前 ibdev2netdev() 的逻辑比较脆弱，它把普通 mlx5_0 这类 IB/RDMA device name 当成 Linux netdev name 使用；除 mlx5_bond_* 这种特殊命名外，通常无法正确映射到 ethX / ensX / bondX 等内核网卡名。

---

2. 涉及文件/模块

主要相关代码

1. RDMA 初始化与设备选择

• astra-sim-alibabacloud/astra-sim/system/SimAiFlowModelRdma.cc
• 重点位置：
  • ibdev2netdev()：
    

    SimAI/astra-sim-alibabacloud/astra-sim/system/SimAiFlowModelRdma.cc

    Lines 663 to 671 in f5efb5a

    	static void
    	ibdev2netdev(const char indev[64], char netdev[64]) {
    	if (strncmp(indev, "mlx5_bond_", 10) == 0) {
    	strcpy(netdev, "bond");
    	strcat(netdev, indev + 10);
    	} else {
    	strcpy(netdev, indev);
    	}
    	}

  • FlowPhyRdma::ibv_init()：
    

    SimAI/astra-sim-alibabacloud/astra-sim/system/SimAiFlowModelRdma.cc

    Lines 673 to 708 in f5efb5a

    	int
    	FlowPhyRdma::ibv_init(void) {
    	int nb_dev = 0;
    	struct ibv_device **dev_lst = NULL;
    	struct ibv_device* ibv_dev;
    	in_addr_t src_addr = inet_addr(rank2addr[local_rank].c_str());
    	MockNcclLog* NcclLog = MockNcclLog::getInstance();
    	dev_lst = ibv_get_device_list(&nb_dev);
    	assert(nb_dev > 0);
    	assert(dev_lst != NULL);
    	for (int i = 0; i < nb_dev; ++i) {
    	in_addr_t ia = 0;
    	char netdev[64];
    	ibdev2netdev(dev_lst[i]->name,netdev);
    	int ret = zeta_util_ifname_to_inet_addr(netdev, &ia);
    	NcclLog->writeLog(
    	NcclLogLevel::DEBUG, "netdev %s ibdev %s",netdev,dev_lst[i]->name);
    	if (ret == 0 && ia == src_addr) {
    	ibv_dev = dev_lst[i];
    	break;
    	}
    	}
    	if (ibv_dev == NULL) {
    	NcclLog->writeLog(NcclLogLevel::ERROR,"ibv Device not found");
    	return -1;
    	} else{
    	NcclLog->writeLog(
    	NcclLogLevel::DEBUG, "ibv device %s init success ", ibv_dev->dev_name);
    	}
    	g_ibv_ctx = ibv_open_device(ibv_dev);
    	assert(g_ibv_ctx != NULL);
    	return 0;

2. hostlist 读取与 rank 到 IP 的映射

• astra-sim-alibabacloud/astra-sim/system/BootStrapnet.cc
• 重点位置：
  

  SimAI/astra-sim-alibabacloud/astra-sim/system/BootStrapnet.cc

  Lines 30 to 58 in f5efb5a

  	int world_size,local_rank;
  	std::map<int,std::string> rank2addr;
  	static void
  	initBootStrapNetRank(int argc,char*argv[]){
  	std::string host_file = argv[1];
  	std::string line;
  	std::ifstream file(host_file);
  	int rank = 0;
  	if(file.is_open()){
  	while (std::getline(file,line)){
  	/* code */
  	rank2addr[rank] = line;
  	rank++;
  	}
  	file.close();
  	} else{
  	std::cerr << "Failed to open the file" << std::endl;
  	}
  	}
  	void BootStrapNet(int argc, char *argv[]){
  	MPI_Init(&argc, &argv);
  	MPI_Comm_size(MPI_COMM_WORLD, &world_size);
  	MPI_Comm_rank(MPI_COMM_WORLD, &local_rank);
  	MPI_Barrier(MPI_COMM_WORLD);
  	initBootStrapNetRank(argc,argv);
  	}

3. SimAI_phynet 启动入口

• astra-sim-alibabacloud/astra-sim/network_frontend/phynet/SimAiMain.cc
• 重点位置：
  • 参数解析与默认值：
    

    SimAI/astra-sim-alibabacloud/astra-sim/network_frontend/phynet/SimAiMain.cc

    Lines 36 to 123 in f5efb5a

    	extern int local_rank;
    	extern AstraSim::Sys* global_sys;
    	extern FlowPhyRdma flow_rdma;
    	struct user_param {
    	int thread;
    	int gpus;
    	string workload;
    	int comm_scale;
    	GPUType gpu_type;
    	int nvswitch_num;
    	int gpus_per_server;
    	int gid_index;
    	user_param() {
    	thread = 1;
    	gpus = 8;
    	workload = "microAllReduce.txt";
    	comm_scale = 1;
    	gpu_type = GPUType::A100;
    	nvswitch_num = 1;
    	gpus_per_server = 8 ;
    	gid_index = 0;
    	};
    	~user_param(){};
    	user_param(int _thread, int _gpus, string _workload, int _comm_scale = 1)
    	: thread(_thread),
    	gpus(_gpus),
    	workload(_workload),
    	comm_scale(_comm_scale){};
    	};
    	static int user_param_prase(int argc,char * argv[],struct user_param* user_param){
    	int opt;
    	static struct option long_options[] = {
    	{"help", no_argument, 0, 'h'},
    	{"workloads", required_argument, 0, 'w'},
    	{"gpus", required_argument, 0, 'g'},
    	{"comm_scale", required_argument, 0, 's'},
    	{"gid_index", required_argument, 0, 'i'},
    	{0, 0, 0, 0}};
    	while ((opt = getopt(argc,argv,"ht:w:g:s:i:"))!=-1){
    	switch (opt)
    	{
    	case 'h':
    	/* code */
    	std::cout<<"-w workloads default microAllReduce.txt "<<std::endl;
    	std::cout<<"-g number of gpus,default 1"<<std::endl;
    	std::cout<<"-s comm_scale default 1"<<std::endl;
    	std::cout<<"-i rdma gid_indxe default 0" <<std::endl;
    	break;
    	case 't':
    	user_param->thread = stoi(optarg);
    	break;
    	case 'w':
    	user_param->workload = optarg;
    	break;
    	case 'g':
    	user_param->gpus = stoi(optarg);
    	if(user_param->gpus <= 8){
    	user_param->gpus =8;
    	}
    	break;
    	case 's':
    	user_param->comm_scale = stof(optarg);
    	break;
    	case 'i':
    	user_param->gid_index = stoi(optarg);
    	break;
    	default:
    	break;
    	}
    	}
    	return 0 ;
    	}
    	int main(int argc,char *argv[]){
    	BootStrapNet(argc,argv);
    	pid_t pid = getpid();
    	MockNcclLog::set_log_name("SimAi_"+to_string(local_rank)+".log");
    	MockNcclLog* NcclLog = MockNcclLog::getInstance();
    	NcclLog->writeLog(NcclLogLevel::DEBUG," Local rank %d PID %d ",local_rank,pid);
    	struct user_param user_param;
    	if(user_param_prase(argc,argv,&user_param)){
    	return -1;
    	}
    	#ifdef PHY_RDMA
    	flow_rdma = FlowPhyRdma(user_param.gid_index);

  • flow_rdma.ibv_init() 调用：
    

    SimAI/astra-sim-alibabacloud/astra-sim/network_frontend/phynet/SimAiMain.cc

    Lines 104 to 114 in f5efb5a

    	break;
    	default:
    	break;
    	}
    	}
    	return 0 ;
    	}
    	int main(int argc,char *argv[]){
    	BootStrapNet(argc,argv);
    	pid_t pid = getpid();

4. 物理网络使用文档

• docs/Tutorial.md
• 重点位置：
  • SimAI-Physical 编译与运行说明：
    

    SimAI/docs/Tutorial.md

    Lines 321 to 404 in f5efb5a

    	| msg size | Spectrum-X | DCN-SingleToR|
    	|----------|-------------|--------------|
    	| 16M | 33.095585 | 23.332048 |
    	| 32M | 38.572166 | 25.762846 |
    	| 64M | 42.049648 | 23.677116 |
    	| 128M | 44.036110 | 35.209461 |
    	| 256M | 45.101425 | 36.205692 |
    	| 512M | 45.653648 | 36.242008 |
    	# SimAI-Physical Usage
    	The current simulator is compatible with the ns3 discrete event simulator as the network backend, as well as the physical network backend for physical packet injection.
    	## Compile
    	SimAI-Phy currently uses the roceV2 protocol for traffic generation. The compilation process requires dependencies on libverbs related to the RDMA physical device, as well as the MPI program. Before compilation, please verify that your environment can successfully run the basic RDMA perfetest traffic generation tool and that it supports the related MPI program.
    	```bash
    	# Clone the repository
    	$ git clone https://github.com/aliyun/SimAI.git
    	$ cd ./SimAI/
    	# Clone submodules
    	$ git submodule update --init --recursive
    	# Make sure to use the newest commit
    	$ git submodule update --remote
    	# Compile SimAI-Analytical
    	$ ./scripts/build.sh -c analytical
    	# Compile SimAI-Simulation (ns3)
    	$ ./scripts/build.sh -c ns3
    	# Compile SimAI-phynet (phynet)
    	$ sudo yum install openmpi openmpi-devel
    	$ export MPI_INCLUDE_PATH=/usr/include/openmpi-x86_64/
    	$ export MPI_BIN_PATH=/usr/lib64/openmpi/bin/mpic++
    	$ ./scripts/build.sh -c phy
    	```
    	## Workload Generate
    	The workload required for SimAI-Phy physical traffic generation is the same as that for Sim-Simulation, and it is generated through AICB.
    	### Example workload
    	```bash
    	HYBRID_TRANSFORMER_FWD_IN_BCKWD model_parallel_NPU_group: 2 ep: 1 pp: 1 vpp: 8 ga: 1 all_gpus: 2 checkpoints: 0 checkpoint_initiates: 0
    	10
    	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
    	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
    	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
    	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
    	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
    	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
    	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
    	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
    	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
    	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
    	```
    	## Prepare the hostlist
    	Here the main task is to prepare the iplist required to start the MPI program, which is different from nccl-test. The number of IPs here should match the actual number of network cards involved in the physical traffic generation, rather than the number of nodes participating in the physical traffic generation.
    	```bash
    	33.255.199.130
    	33.255.199.129
    	```
    	## RUN
    	### MPI Run
    	```bash
    	/usr/lib64/openmpi/bin/mpirun -np 2 -host 33.255.199.130,33.255.199.129 --allow-run-as-root -x AS_LOG_LEVEL=0 ./bin/SimAI_phynet ./hostlist -g 2 -w ./example/microAllReduce.txt
    	```
    	The following output indicates that the program has finished running.
    	<img src="./images/Sim-phynet_finished.png" alt="Sim-phynet_finished" width="40%">
    	## Setting of MPI program parameters
    	| Parameter | Description | Default Value |
    	|------------------|-------------------------------------------------------|---------------|
    	| -np | The number of processes. | NULL |
    	| -host | IP list | NULL |
    	| --allow-run-as-root | Allow MPI programs to run with root privileges. | FALSE |
    	## Setting of SimAI-phynet parameters
    	| Parameter | Description | Default Value |
    	|------------------|------------------------------|----------------------------------------------------------|
    	| hostlist | Host IP list | NULL |
    	| -w --workload | Path to workload | ./microAllReduce.txt |
    	| -i --gid_index | Network topology path | 0 |
    	| -g --gpus | Number of GPUs | 8 (should be consistent with the number of IPs in the host IP list) |

参考仓库关系

• aliyun/SimAI：本 issue 的直接问题在这里，尤其是 SimAI_phynet 和 RDMA verbs 初始化。
• aliyun/aicb：主要用于生成 workload，和本次 ibv_open_device() 失败不是直接因果关系。
• aliyun/ns-3-alibabacloud：是仿真后端 RDMA/网络模型，和 SimAI_phynet 的物理 libibverbs 打开设备失败不是同一条路径。

---

3. 根因分析或实现思路

直接失败点

当前失败发生在：

g_ibv_ctx = ibv_open_device(ibv_dev);
assert(g_ibv_ctx != NULL);

也就是 ibv_open_device() 返回了 NULL。

可能原因有两类：

1. 代码选错或没有正确选中 RDMA 设备
2. 选中的 RDMA 设备确实无法打开，例如权限、驱动、容器设备挂载、/dev/infiniband/uverbs* 不可访问等

结合代码看，第一类问题的概率较高。

关键问题 1：ibdev2netdev() 只能处理很少的命名情况

当前逻辑：

if (strncmp(indev, "mlx5_bond_", 10) == 0) {
  netdev = "bond" + suffix;
} else {
  netdev = indev;
}

这意味着：

• mlx5_bond_0 会映射成 bond0
• 但 mlx5_0 会映射成 mlx5_0

问题是，在 Linux 上：

• mlx5_0 通常是 RDMA verbs device name
• 真正有 IP 的 Linux netdev 往往是 ethX、ensX、enpXsY、bond0 等
• 所以用 ioctl(SIOCGIFADDR) 查 mlx5_0 的 IPv4 地址通常会失败

这样会导致程序无法通过 hostlist 中的 IP 找到正确的 RDMA 设备。

关键问题 2：ibv_dev 未初始化

FlowPhyRdma::ibv_init() 里有：

struct ibv_device* ibv_dev;

但没有初始化为 NULL。

如果没有找到匹配设备，ibv_dev 可能是未定义值。后面的判断：

if (ibv_dev == NULL)

不可靠。最坏情况下可能拿一个未定义指针去调用 ibv_open_device()，然后在 assert(g_ibv_ctx != NULL) 崩溃。

这是一个明确的健壮性 bug，应最小修改为初始化：

struct ibv_device* ibv_dev = NULL;

关键问题 3：错误处理使用 assert，用户无法知道真实原因

当前逻辑直接：

assert(g_ibv_ctx != NULL);

这会隐藏真正原因，比如：

• 没找到设备
• 设备权限不足
• /dev/infiniband/uverbsX 不存在
• 容器没有挂载 RDMA 设备
• GID index 配置不匹配
• hostlist 里写的是管理网 IP 而不是 RDMA/RoCE 网卡 IP

更合理的是打印设备名、netdev、IP、errno/strerror(errno) 后返回错误码，让上层退出。

关键问题 4：main() 没检查 ibv_init() 返回值

SimAiMain.cc 中：

flow_rdma.ibv_init();

没有检查返回值。即使 ibv_init() 返回 -1，程序也会继续执行，可能在后面产生更难理解的问题。

建议最小修改为：

• 如果 ibv_init() 非 0，打印错误并退出
• 不继续进入 network callback / workload 执行阶段

额外注意：-g 2 当前会被强制改成 8

SimAiMain.cc 中：

user_param->gpus = stoi(optarg);
if(user_param->gpus <= 8){
  user_param->gpus =8;
}

所以用户传 -g 2 实际会变成 8。这不是当前 g_ibv_ctx assert 的直接原因，但会影响后续逻辑，因为 docs/Tutorial.md 里写了 -g 应该与 host IP list 数量一致。这个点建议单独确认或作为后续 issue 处理。

---

4. 最小修改方案

建议按最小改动分三步处理。

方案 A：先修复崩溃和错误提示

最小必要修改：

1. 在 ibv_init() 中初始化：

struct ibv_device* ibv_dev = NULL;

2. ibv_open_device() 失败时不要 assert，改为：

• 打印 ibv_dev->name
• 打印 ibv_dev->dev_name
• 打印 errno 和 strerror(errno)
• 返回 -1

3. ibv_get_device_list() 后增加失败处理：

• dev_lst == NULL
• nb_dev <= 0

不要直接 assert。

4. main() 中检查返回值：

• 如果 flow_rdma.ibv_init() 非 0，则打印错误并退出

这个修改不改变设备匹配策略，但能让用户看到明确错误。

方案 B：修复 RDMA device 到 netdev 的映射

建议把当前 ibdev2netdev() 扩展为：

1. 优先从 sysfs 查映射：

/sys/class/infiniband/<ibdev>/device/net/

该目录下通常能看到真正的 Linux netdev，例如：

/sys/class/infiniband/mlx5_0/device/net/ens5f0
/sys/class/infiniband/mlx5_bond_0/device/net/bond0

2. 如果 sysfs 查不到，再保留当前 fallback：

• mlx5_bond_X -> bondX
• 否则返回原名

这样不引入额外依赖，只需要标准 C/C++ 和 POSIX 目录读取能力，例如 opendir/readdir/closedir。

方案 C：增强日志，方便用户自查

在遍历 RDMA 设备时建议打印：

• RDMA device name
• sysfs 映射出的 netdev
• SIOCGIFADDR 是否成功
• netdev IP
• hostlist[local_rank] 期望 IP

这样用户可以直接看日志判断：

• hostlist 写错了
• 选错了网卡
• 没有 RDMA 设备
• 容器/权限问题
• GID index 不对

不建议作为本 issue 的最小修改

以下可以后续处理，不建议和本次最小修复混在一起：

1. 重构整个 phynet 参数解析
2. 修改 -g <= 8 强制变 8 的行为
3. 引入新的第三方依赖
4. 改动 ns-3 backend
5. 改动 AICB workload 生成逻辑

---

5. 测试前置条件

是否依赖额外数据？

分两类。

编译测试

不依赖 AICB workload，也不一定需要真实 RDMA 流量数据。

需要：

• 可编译 SimAI_phynet
• OpenMPI
• libibverbs 头文件和库
• 当前项目已有 phynet 构建路径

运行验证

需要真实或可访问的 RDMA/RoCE 环境，依赖额外环境条件：

1. 每个节点有 RDMA/RoCE 设备
2. ibv_devices 能看到设备
3. ibv_devinfo 能打开设备
4. /dev/infiniband/uverbs* 存在且当前用户可访问
5. 如果在容器中运行，需要把 /dev/infiniband 挂进去
6. hostlist 中写的是 RDMA/RoCE 网卡 IP，不是管理网 IP
7. -i gid_index 对应有效 RoCEv2 GID
8. MPI hostfile 和 SimAI hostlist 语义要区分：
   • MPI hostfile 用于 MPI 启动进程
   • SimAI hostlist 用于 SimAI_phynet 找本 rank 对应的物理发包 IP

建议用户在每个节点先执行：

ibv_devices
ibv_devinfo
ibdev2netdev
show_gids
ip addr

并确认 hostlist 中的 IP 能在 ibdev2netdev 对应的 netdev 上找到。

---

6. README 是否需要修改及建议内容

建议更新文档，但不一定只改根目录 README.md。

目前 phynet 的主要说明在 docs/Tutorial.md，所以建议优先补充：

• docs/Tutorial.md 的 SimAI-Physical Usage 部分
• 根目录 README.md 可增加一个简短链接，指向 docs/Tutorial.md 的 SimAI-Physical Usage

建议新增内容包括：

1. hostlist 必须填写 RDMA/RoCE 网卡 IP，而不是管理网 IP
2. ibdev2netdev / sysfs 映射检查方式
3. gid_index 如何检查，例如 show_gids
4. 运行前验证命令：
   • ibv_devices
   • ibv_devinfo
   • ibdev2netdev
   • show_gids
   • ib_write_bw / ib_read_bw
5. 容器运行时需要挂载 /dev/infiniband
6. 说明 -g 与 hostlist IP 数量需要一致；同时建议确认当前 -g <= 8 会被程序强制设为 8 的行为是否符合预期

---

针对 issue 中问题的直接回复建议

你看到的 g_ibv_ctx != NULL 失败不是 workload 本身的问题，而是在 SimAI_phynet 初始化 RDMA verbs 设备时没有成功打开设备。

当前代码里 ibdev2netdev() 的确比较可疑：它只特殊处理了 mlx5_bond_*，普通 mlx5_0 会被直接当作 Linux 网卡名去查 IP。但在大多数环境里 mlx5_0 是 RDMA device name，不是带 IP 的 netdev name，所以会导致设备匹配失败或选错设备。

建议先确认：

1. hostlist 里写的是 RoCE/RDMA 网卡 IP，不是管理网 IP；
2. 每台机器执行 ibdev2netdev，确认 mlx5_* 对应的真实 netdev；
3. 执行 show_gids，确认 -i 3 对应的 GID 是你期望的 RoCEv2 GID；
4. 执行 ibv_devinfo，确认当前用户能打开 verbs 设备；
5. 如果在容器里跑，需要确认 /dev/infiniband 已挂载进去。

代码侧建议做最小修复：

• 初始化 ibv_dev = NULL
• 不要在 ibv_open_device() 后直接 assert
• 打印 errno/strerror(errno) 并返回错误
• 通过 /sys/class/infiniband/<ibdev>/device/net/ 获取真实 netdev，而不是简单把 mlx5_0 当成网卡名
• 在 main() 中检查 flow_rdma.ibv_init() 的返回值，失败则退出

这样可以避免直接 abort，并且能明确告诉用户是 hostlist IP 不匹配、设备打不开、权限不足，还是 GID 配置问题。

---

EN

English issue reply draft

1. Requirement / Background Summary

This issue happens when running SimAI_phynet for physical network / RoCEv2 packet injection. The program aborts during RDMA verbs initialization:

FlowPhyRdma::ibv_init(): Assertion `g_ibv_ctx != NULL' failed.

The user runs SimAI_phynet with:

• MPI hostfile
• SimAI hostlist
• an AICB-generated workload
• -g 2
• -i 3 for RDMA gid_index

Based on the current implementation, SimAI_phynet first calls BootStrapNet() to read hostlist and map local_rank to an IP address. Then FlowPhyRdma::ibv_init() iterates over local RDMA devices, tries to map each RDMA device name to a Linux netdev name, reads the netdev IP address, and selects the RDMA device whose netdev IP matches hostlist[local_rank]. Finally, it calls ibv_open_device().

The user's suspicion is valid: the current ibdev2netdev() implementation is fragile. It only handles the special mlx5_bond_* -> bond* naming pattern. For normal devices such as mlx5_0 or mlx5_1, it treats the RDMA device name as if it were a Linux netdev name, which is usually incorrect.

---

2. Related Files / Modules

Main related code

1. RDMA initialization and device selection

• astra-sim-alibabacloud/astra-sim/system/SimAiFlowModelRdma.cc
• Important locations:
  • ibdev2netdev():
    

    SimAI/astra-sim-alibabacloud/astra-sim/system/SimAiFlowModelRdma.cc

    Lines 663 to 671 in f5efb5a

    	static void
    	ibdev2netdev(const char indev[64], char netdev[64]) {
    	if (strncmp(indev, "mlx5_bond_", 10) == 0) {
    	strcpy(netdev, "bond");
    	strcat(netdev, indev + 10);
    	} else {
    	strcpy(netdev, indev);
    	}
    	}

  • FlowPhyRdma::ibv_init():
    

    SimAI/astra-sim-alibabacloud/astra-sim/system/SimAiFlowModelRdma.cc

    Lines 673 to 708 in f5efb5a

    	int
    	FlowPhyRdma::ibv_init(void) {
    	int nb_dev = 0;
    	struct ibv_device **dev_lst = NULL;
    	struct ibv_device* ibv_dev;
    	in_addr_t src_addr = inet_addr(rank2addr[local_rank].c_str());
    	MockNcclLog* NcclLog = MockNcclLog::getInstance();
    	dev_lst = ibv_get_device_list(&nb_dev);
    	assert(nb_dev > 0);
    	assert(dev_lst != NULL);
    	for (int i = 0; i < nb_dev; ++i) {
    	in_addr_t ia = 0;
    	char netdev[64];
    	ibdev2netdev(dev_lst[i]->name,netdev);
    	int ret = zeta_util_ifname_to_inet_addr(netdev, &ia);
    	NcclLog->writeLog(
    	NcclLogLevel::DEBUG, "netdev %s ibdev %s",netdev,dev_lst[i]->name);
    	if (ret == 0 && ia == src_addr) {
    	ibv_dev = dev_lst[i];
    	break;
    	}
    	}
    	if (ibv_dev == NULL) {
    	NcclLog->writeLog(NcclLogLevel::ERROR,"ibv Device not found");
    	return -1;
    	} else{
    	NcclLog->writeLog(
    	NcclLogLevel::DEBUG, "ibv device %s init success ", ibv_dev->dev_name);
    	}
    	g_ibv_ctx = ibv_open_device(ibv_dev);
    	assert(g_ibv_ctx != NULL);
    	return 0;

2. hostlist loading and rank-to-IP mapping

• astra-sim-alibabacloud/astra-sim/system/BootStrapnet.cc
• Important location:
  

  SimAI/astra-sim-alibabacloud/astra-sim/system/BootStrapnet.cc

  Lines 30 to 58 in f5efb5a

  	int world_size,local_rank;
  	std::map<int,std::string> rank2addr;
  	static void
  	initBootStrapNetRank(int argc,char*argv[]){
  	std::string host_file = argv[1];
  	std::string line;
  	std::ifstream file(host_file);
  	int rank = 0;
  	if(file.is_open()){
  	while (std::getline(file,line)){
  	/* code */
  	rank2addr[rank] = line;
  	rank++;
  	}
  	file.close();
  	} else{
  	std::cerr << "Failed to open the file" << std::endl;
  	}
  	}
  	void BootStrapNet(int argc, char *argv[]){
  	MPI_Init(&argc, &argv);
  	MPI_Comm_size(MPI_COMM_WORLD, &world_size);
  	MPI_Comm_rank(MPI_COMM_WORLD, &local_rank);
  	MPI_Barrier(MPI_COMM_WORLD);
  	initBootStrapNetRank(argc,argv);
  	}

3. SimAI_phynet entry point

• astra-sim-alibabacloud/astra-sim/network_frontend/phynet/SimAiMain.cc
• Important locations:
  • Parameter parsing and defaults:
    

    SimAI/astra-sim-alibabacloud/astra-sim/network_frontend/phynet/SimAiMain.cc

    Lines 36 to 123 in f5efb5a

    	extern int local_rank;
    	extern AstraSim::Sys* global_sys;
    	extern FlowPhyRdma flow_rdma;
    	struct user_param {
    	int thread;
    	int gpus;
    	string workload;
    	int comm_scale;
    	GPUType gpu_type;
    	int nvswitch_num;
    	int gpus_per_server;
    	int gid_index;
    	user_param() {
    	thread = 1;
    	gpus = 8;
    	workload = "microAllReduce.txt";
    	comm_scale = 1;
    	gpu_type = GPUType::A100;
    	nvswitch_num = 1;
    	gpus_per_server = 8 ;
    	gid_index = 0;
    	};
    	~user_param(){};
    	user_param(int _thread, int _gpus, string _workload, int _comm_scale = 1)
    	: thread(_thread),
    	gpus(_gpus),
    	workload(_workload),
    	comm_scale(_comm_scale){};
    	};
    	static int user_param_prase(int argc,char * argv[],struct user_param* user_param){
    	int opt;
    	static struct option long_options[] = {
    	{"help", no_argument, 0, 'h'},
    	{"workloads", required_argument, 0, 'w'},
    	{"gpus", required_argument, 0, 'g'},
    	{"comm_scale", required_argument, 0, 's'},
    	{"gid_index", required_argument, 0, 'i'},
    	{0, 0, 0, 0}};
    	while ((opt = getopt(argc,argv,"ht:w:g:s:i:"))!=-1){
    	switch (opt)
    	{
    	case 'h':
    	/* code */
    	std::cout<<"-w workloads default microAllReduce.txt "<<std::endl;
    	std::cout<<"-g number of gpus,default 1"<<std::endl;
    	std::cout<<"-s comm_scale default 1"<<std::endl;
    	std::cout<<"-i rdma gid_indxe default 0" <<std::endl;
    	break;
    	case 't':
    	user_param->thread = stoi(optarg);
    	break;
    	case 'w':
    	user_param->workload = optarg;
    	break;
    	case 'g':
    	user_param->gpus = stoi(optarg);
    	if(user_param->gpus <= 8){
    	user_param->gpus =8;
    	}
    	break;
    	case 's':
    	user_param->comm_scale = stof(optarg);
    	break;
    	case 'i':
    	user_param->gid_index = stoi(optarg);
    	break;
    	default:
    	break;
    	}
    	}
    	return 0 ;
    	}
    	int main(int argc,char *argv[]){
    	BootStrapNet(argc,argv);
    	pid_t pid = getpid();
    	MockNcclLog::set_log_name("SimAi_"+to_string(local_rank)+".log");
    	MockNcclLog* NcclLog = MockNcclLog::getInstance();
    	NcclLog->writeLog(NcclLogLevel::DEBUG," Local rank %d PID %d ",local_rank,pid);
    	struct user_param user_param;
    	if(user_param_prase(argc,argv,&user_param)){
    	return -1;
    	}
    	#ifdef PHY_RDMA
    	flow_rdma = FlowPhyRdma(user_param.gid_index);

  • flow_rdma.ibv_init() call:
    

    SimAI/astra-sim-alibabacloud/astra-sim/network_frontend/phynet/SimAiMain.cc

    Lines 104 to 114 in f5efb5a

    	break;
    	default:
    	break;
    	}
    	}
    	return 0 ;
    	}
    	int main(int argc,char *argv[]){
    	BootStrapNet(argc,argv);
    	pid_t pid = getpid();

4. Physical network usage documentation

• docs/Tutorial.md
• Important location:
  • SimAI-Physical build and run instructions:
    

    SimAI/docs/Tutorial.md

    Lines 321 to 404 in f5efb5a

    	| msg size | Spectrum-X | DCN-SingleToR|
    	|----------|-------------|--------------|
    	| 16M | 33.095585 | 23.332048 |
    	| 32M | 38.572166 | 25.762846 |
    	| 64M | 42.049648 | 23.677116 |
    	| 128M | 44.036110 | 35.209461 |
    	| 256M | 45.101425 | 36.205692 |
    	| 512M | 45.653648 | 36.242008 |
    	# SimAI-Physical Usage
    	The current simulator is compatible with the ns3 discrete event simulator as the network backend, as well as the physical network backend for physical packet injection.
    	## Compile
    	SimAI-Phy currently uses the roceV2 protocol for traffic generation. The compilation process requires dependencies on libverbs related to the RDMA physical device, as well as the MPI program. Before compilation, please verify that your environment can successfully run the basic RDMA perfetest traffic generation tool and that it supports the related MPI program.
    	```bash
    	# Clone the repository
    	$ git clone https://github.com/aliyun/SimAI.git
    	$ cd ./SimAI/
    	# Clone submodules
    	$ git submodule update --init --recursive
    	# Make sure to use the newest commit
    	$ git submodule update --remote
    	# Compile SimAI-Analytical
    	$ ./scripts/build.sh -c analytical
    	# Compile SimAI-Simulation (ns3)
    	$ ./scripts/build.sh -c ns3
    	# Compile SimAI-phynet (phynet)
    	$ sudo yum install openmpi openmpi-devel
    	$ export MPI_INCLUDE_PATH=/usr/include/openmpi-x86_64/
    	$ export MPI_BIN_PATH=/usr/lib64/openmpi/bin/mpic++
    	$ ./scripts/build.sh -c phy
    	```
    	## Workload Generate
    	The workload required for SimAI-Phy physical traffic generation is the same as that for Sim-Simulation, and it is generated through AICB.
    	### Example workload
    	```bash
    	HYBRID_TRANSFORMER_FWD_IN_BCKWD model_parallel_NPU_group: 2 ep: 1 pp: 1 vpp: 8 ga: 1 all_gpus: 2 checkpoints: 0 checkpoint_initiates: 0
    	10
    	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
    	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
    	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
    	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
    	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
    	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
    	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
    	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
    	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
    	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
    	```
    	## Prepare the hostlist
    	Here the main task is to prepare the iplist required to start the MPI program, which is different from nccl-test. The number of IPs here should match the actual number of network cards involved in the physical traffic generation, rather than the number of nodes participating in the physical traffic generation.
    	```bash
    	33.255.199.130
    	33.255.199.129
    	```
    	## RUN
    	### MPI Run
    	```bash
    	/usr/lib64/openmpi/bin/mpirun -np 2 -host 33.255.199.130,33.255.199.129 --allow-run-as-root -x AS_LOG_LEVEL=0 ./bin/SimAI_phynet ./hostlist -g 2 -w ./example/microAllReduce.txt
    	```
    	The following output indicates that the program has finished running.
    	<img src="./images/Sim-phynet_finished.png" alt="Sim-phynet_finished" width="40%">
    	## Setting of MPI program parameters
    	| Parameter | Description | Default Value |
    	|------------------|-------------------------------------------------------|---------------|
    	| -np | The number of processes. | NULL |
    	| -host | IP list | NULL |
    	| --allow-run-as-root | Allow MPI programs to run with root privileges. | FALSE |
    	## Setting of SimAI-phynet parameters
    	| Parameter | Description | Default Value |
    	|------------------|------------------------------|----------------------------------------------------------|
    	| hostlist | Host IP list | NULL |
    	| -w --workload | Path to workload | ./microAllReduce.txt |
    	| -i --gid_index | Network topology path | 0 |
    	| -g --gpus | Number of GPUs | 8 (should be consistent with the number of IPs in the host IP list) |

Relation with the referenced repositories

• aliyun/SimAI: the direct issue is here, especially in SimAI_phynet and RDMA verbs initialization.
• aliyun/aicb: mainly used to generate workloads. It is not the direct cause of ibv_open_device() failure.
• aliyun/ns-3-alibabacloud: provides the simulation RDMA/network backend. It is a different path from SimAI_phynet physical libibverbs device opening.

---

3. Root Cause Analysis / Implementation Direction

Direct failure point

The failure occurs here:

g_ibv_ctx = ibv_open_device(ibv_dev);
assert(g_ibv_ctx != NULL);

This means ibv_open_device() returned NULL.

There are two likely categories of causes:

1. The code selected the wrong RDMA device, or failed to select the correct one.
2. The selected RDMA device cannot be opened due to permission, driver, container device mount, or /dev/infiniband/uverbs* access issues.

From the current code, the first category is highly likely.

Key issue 1: ibdev2netdev() only handles limited naming patterns

Current behavior:

if (strncmp(indev, "mlx5_bond_", 10) == 0) {
  netdev = "bond" + suffix;
} else {
  netdev = indev;
}

This means:

• mlx5_bond_0 becomes bond0
• But mlx5_0 remains mlx5_0

On Linux:

• mlx5_0 is usually an RDMA verbs device name
• The real Linux netdev that has an IP address is usually something like ethX, ensX, enpXsY, or bond0
• Therefore, calling ioctl(SIOCGIFADDR) on mlx5_0 normally fails

As a result, the program may fail to match the IP from hostlist to the correct RDMA device.

Key issue 2: ibv_dev is not initialized

Inside FlowPhyRdma::ibv_init():

struct ibv_device* ibv_dev;

This pointer is not initialized.

If no matching device is found, the later check:

if (ibv_dev == NULL)

is unreliable. In the worst case, an undefined pointer may be passed to ibv_open_device(), followed by the g_ibv_ctx != NULL assertion failure.

This is a clear robustness bug. The minimum fix is:

struct ibv_device* ibv_dev = NULL;

Key issue 3: assert hides the real reason

The current code directly does:

assert(g_ibv_ctx != NULL);

This hides the actual reason from the user, for example:

• no matching device
• insufficient permission
• missing /dev/infiniband/uverbsX
• container did not mount RDMA devices
• invalid or mismatched GID index
• hostlist contains a management IP instead of the RDMA/RoCE NIC IP

It would be better to log the RDMA device name, mapped netdev, IP address, errno, and strerror(errno), then return an error code.

Key issue 4: main() ignores the return value of ibv_init()

In SimAiMain.cc:

flow_rdma.ibv_init();

The return value is not checked. Even if ibv_init() returns -1, the program continues and may fail later with a less clear error.

The minimal fix is to exit early when ibv_init() fails.

Additional note: -g 2 is currently forced to 8

In SimAiMain.cc:

user_param->gpus = stoi(optarg);
if(user_param->gpus <= 8){
  user_param->gpus =8;
}

So passing -g 2 actually becomes 8. This is probably not the direct cause of the current g_ibv_ctx assertion, but it may cause follow-up mismatches because the documentation says -g should be consistent with the host IP list. This should be verified separately.

---

4. Minimal Viable Fix Plan

I suggest keeping the change minimal and splitting it into three parts.

Plan A: Fix crash and improve error reporting

Minimum required changes:

1. Initialize ibv_dev:

struct ibv_device* ibv_dev = NULL;

2. Do not directly assert after ibv_open_device() fails. Instead:

• log ibv_dev->name
• log ibv_dev->dev_name
• log errno and strerror(errno)
• return -1

3. Add error handling after ibv_get_device_list():

• dev_lst == NULL
• nb_dev <= 0

Avoid direct assert.

4. Check the return value in main():

• if flow_rdma.ibv_init() returns non-zero, log the failure and exit

This does not change the device matching logic yet, but it gives the user a clear error instead of an abort.

Plan B: Fix RDMA device to netdev mapping

Extend ibdev2netdev() as follows:

1. Prefer sysfs mapping:

/sys/class/infiniband/<ibdev>/device/net/

This directory usually contains the real Linux netdev, for example:

/sys/class/infiniband/mlx5_0/device/net/ens5f0
/sys/class/infiniband/mlx5_bond_0/device/net/bond0

2. If sysfs mapping fails, keep the current fallback:

• mlx5_bond_X -> bondX
• otherwise use the original name

This introduces no extra third-party dependency. POSIX directory APIs such as opendir/readdir/closedir are enough.

Plan C: Add diagnostic logs

When iterating through RDMA devices, log:

• RDMA device name
• mapped Linux netdev name
• whether SIOCGIFADDR succeeded
• netdev IP
• expected IP from hostlist[local_rank]

This helps users immediately identify whether:

• hostlist is wrong
• the selected NIC is wrong
• RDMA devices are missing
• permission/container setup is wrong
• GID index is wrong

Not recommended as part of the minimal fix

The following should be handled separately:

1. Refactoring all phynet argument parsing
2. Changing the -g <= 8 behavior
3. Adding new third-party dependencies
4. Modifying the ns-3 backend
5. Modifying AICB workload generation

---

5. Test Prerequisites

Does testing require extra data?

There are two levels.

Build test

No AICB workload is required, and real RDMA traffic data is not required.

Required:

• ability to build SimAI_phynet
• OpenMPI
• libibverbs headers and libraries
• the existing phynet build path

Runtime validation

Runtime validation requires a real or accessible RDMA/RoCE environment:

1. Each node has an RDMA/RoCE device
2. ibv_devices shows devices
3. ibv_devinfo can open the device
4. /dev/infiniband/uverbs* exists and is accessible by the current user
5. If running in a container, /dev/infiniband must be mounted
6. hostlist contains the RDMA/RoCE NIC IP, not the management IP
7. -i gid_index points to a valid RoCEv2 GID
8. MPI hostfile and SimAI hostlist have different purposes:
   • MPI hostfile is for launching MPI processes
   • SimAI hostlist is used by SimAI_phynet to find the physical packet-injection IP for each rank

Suggested checks on each node:

ibv_devices
ibv_devinfo
ibdev2netdev
show_gids
ip addr

Confirm that the IP in hostlist appears on the netdev mapped from the target RDMA device.

---

6. README Update Recommendation

Documentation should be updated, but not necessarily only the root README.md.

The detailed phynet documentation currently lives in docs/Tutorial.md, so I recommend:

• Add detailed troubleshooting to docs/Tutorial.md under SimAI-Physical Usage
• Optionally add a short pointer in the root README.md to the SimAI-Physical Usage section

Suggested documentation additions:

1. hostlist must contain RDMA/RoCE NIC IPs, not management IPs
2. How to check RDMA device to netdev mapping using ibdev2netdev or sysfs
3. How to choose gid_index, for example using show_gids
4. Pre-run validation commands:
   • ibv_devices
   • ibv_devinfo
   • ibdev2netdev
   • show_gids
   • ib_write_bw / ib_read_bw
5. Container requirement: mount /dev/infiniband
6. Clarify that -g should match the number of hostlist IPs, and verify whether the current -g <= 8 forced-to-8 behavior is intended

---

Direct reply suggestion for the issue

The g_ibv_ctx != NULL failure is not caused by the workload itself. It happens when SimAI_phynet initializes the RDMA verbs device and fails to open the selected device.

The current ibdev2netdev() logic is indeed suspicious. It only handles mlx5_bond_* specially. For a normal RDMA device like mlx5_0, it directly uses mlx5_0 as the Linux netdev name. In most environments, however, mlx5_0 is the RDMA device name, not the netdev that owns an IP address. The real netdev is usually something like ensX, ethX, enpXsY, or bondX. Therefore, the code may fail to match the IP from hostlist, or it may select/open the wrong device.

Please first check:

1. The IPs in hostlist are RDMA/RoCE NIC IPs, not management IPs.
2. Run ibdev2netdev on each node and confirm the real netdev behind each mlx5_* device.
3. Run show_gids and confirm that -i 3 points to the expected RoCEv2 GID.
4. Run ibv_devinfo and confirm the current user can open the verbs device.
5. If running inside a container, make sure /dev/infiniband is mounted into the container.

On the code side, the minimal fix should be:

• initialize ibv_dev = NULL
• avoid directly asserting after ibv_open_device()
• log errno/strerror(errno) and return an error
• map RDMA device to real netdev using /sys/class/infiniband/<ibdev>/device/net/
• check flow_rdma.ibv_init() return value in main() and exit early on failure

This would avoid the immediate abort and make it clear whether the problem is hostlist IP mismatch, device access failure, permission issue, or GID configuration.

[zxfszane]

这个问题我也碰到了。 mlx5_0 网卡 获取的 接口名也是 mlx5_0 导致无法获取ip。 我通过 ifconfig 直接把对应网卡 改名成 mlx5_0 。后续ip就能正常找到了。 同时还出现了一个奇怪的现象，完全相同的参数在 第一个主机上可以正常运行，在第二个主机上会获取 到 第一个主机的地址进行匹配的导致初始化失败。 因为一台服务器就一个网口，直接去取巧

int
FlowPhyRdma::ibv_init(void) {
    int nb_dev = 0;
    struct ibv_device **dev_lst = NULL;
    struct ibv_device* ibv_dev = NULL;
    in_addr_t src_addr = inet_addr(rank2addr[local_rank].c_str());
    MockNcclLog* NcclLog = MockNcclLog::getInstance();
    pid_t pid = getpid();

    dev_lst = ibv_get_device_list(&nb_dev);
    assert(nb_dev > 0);
    assert(dev_lst != NULL);
    fprintf(stdout, "[%d] Total IB devices found: %d src_addr: %x\n", pid, nb_dev, src_addr);
 
    for (int i = 0; i < nb_dev; ++i) {
        in_addr_t ia = 0;
        char netdev[64];
        ibdev2netdev(dev_lst[i]->name,netdev);
        int ret = zeta_util_ifname_to_inet_addr(netdev, &ia);

        fprintf(stdout, "[%d] i:%d dev:%s %s netdev:%s ia:%x ret=%d\n", pid, i, dev_lst[i]->name, dev_lst[i]->dev_name, netdev, ia, ret);
        NcclLog->writeLog(
        NcclLogLevel::DEBUG, "netdev %s ibdev %s",netdev,dev_lst[i]->name);
        if (ret == 0 && ia == src_addr) {
            ibv_dev = dev_lst[i];
            break;
        }
        else if (ret == 0) {
            ibv_dev = dev_lst[i];
        }
    }

    if (ibv_dev == NULL) {
        NcclLog->writeLog(NcclLogLevel::ERROR,"ibv Device not found");
        return -1;
    } else{
        NcclLog->writeLog(
            NcclLogLevel::DEBUG, "ibv device %s init success ", ibv_dev->dev_name);
    }
    g_ibv_ctx = ibv_open_device(ibv_dev);
    assert(g_ibv_ctx != NULL);

    return 0;
}

调试日志如下

+ mpirun -np 2 --wdir /opt/SimAI --host 192.168.22.1,192.168.22.2 --allow-run-as-root --mca plm_rsh_args '-p 2222' -x AS_LOG_LEVEL=3 -x UCX_IB_TRAFFIC_CLASS=96 /opt/SimAI/bin/SimAI_phynet /opt/SimAI/hostlist -g 2 -i 3 -w /opt/SimAI/example/work.txt
[535794] Total IB devices found: 4 src_addr: 116a8c0
[535794] i:0 dev:mlx5_0 uverbs0 netdev:mlx5_0 ia:216a8c0 ret=0
[1017216] Total IB devices found: 4 src_addr: 216a8c0
[1017216] i:0 dev:mlx5_0 uverbs0 netdev:mlx5_0 ia:116a8c0 ret=0
[535794] i:1 dev:mlx5_1 uverbs1 netdev:mlx5_1 ia:0 ret=-19
[1017216] i:1 dev:mlx5_1 uverbs1 netdev:mlx5_1 ia:0 ret=-19
[1017216] i:2 dev:mlx5_2 uverbs2 netdev:mlx5_2 ia:0 ret=-19
[535794] i:2 dev:mlx5_2 uverbs2 netdev:mlx5_2 ia:0 ret=-19
[535794] i:3 dev:mlx5_3 uverbs3 netdev:mlx5_3 ia:0 ret=-19
[1017216] i:3 dev:mlx5_3 uverbs3 netdev:mlx5_3 ia:0 ret=-19
The final active chunks per dimension 1 after allocating to queues is: 1
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 1total nodes in ring: 9
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 1total nodes in ring: 9
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 1total nodes in ring: 9
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 1total nodes in ring: 9
total nodes: 9
Success in opening workload file
model_parallel_NPU_group is 2
checkpoints layers are: 
layers initiating fwd_in_bckwd are: 

[tianhao909]

这个问题我也碰到了。 mlx5_0 网卡 获取的 接口名也是 mlx5_0 导致无法获取ip。 我通过 ifconfig 直接把对应网卡 改名成 mlx5_0 。后续ip就能正常找到了。 同时还出现了一个奇怪的现象，完全相同的参数在 第一个主机上可以正常运行，在第二个主机上会获取 到 第一个主机的地址进行匹配的导致初始化失败。 因为一台服务器就一个网口，直接去取巧

int
FlowPhyRdma::ibv_init(void) {
int nb_dev = 0;
struct ibv_device dev_lst = NULL;
struct ibv_device ibv_dev = NULL;
in_addr_t src_addr = inet_addr(rank2addr[local_rank].c_str());
MockNcclLog NcclLog = MockNcclLog::getInstance();
pid_t pid = getpid();

dev_lst = ibv_get_device_list(&nb_dev);
assert(nb_dev > 0);
assert(dev_lst != NULL);
fprintf(stdout, "[%d] Total IB devices found: %d src_addr: %x\n", pid, nb_dev, src_addr);

for (int i = 0; i < nb_dev; ++i) {
    in_addr_t ia = 0;
    char netdev[64];
    ibdev2netdev(dev_lst[i]->name,netdev);
    int ret = zeta_util_ifname_to_inet_addr(netdev, &ia);

    fprintf(stdout, "[%d] i:%d dev:%s %s netdev:%s ia:%x ret=%d\n", pid, i, dev_lst[i]->name, dev_lst[i]->dev_name, netdev, ia, ret);
    NcclLog->writeLog(
    NcclLogLevel::DEBUG, "netdev %s ibdev %s",netdev,dev_lst[i]->name);
    if (ret == 0 && ia == src_addr) {
        ibv_dev = dev_lst[i];
        break;
    }
    else if (ret == 0) {
        ibv_dev = dev_lst[i];
    }
}

if (ibv_dev == NULL) {
    NcclLog->writeLog(NcclLogLevel::ERROR,"ibv Device not found");
    return -1;
} else{
    NcclLog->writeLog(
        NcclLogLevel::DEBUG, "ibv device %s init success ", ibv_dev->dev_name);
}
g_ibv_ctx = ibv_open_device(ibv_dev);
assert(g_ibv_ctx != NULL);

return 0;

}
调试日志如下

+ mpirun -np 2 --wdir /opt/SimAI --host 192.168.22.1,192.168.22.2 --allow-run-as-root --mca plm_rsh_args '-p 2222' -x AS_LOG_LEVEL=3 -x UCX_IB_TRAFFIC_CLASS=96 /opt/SimAI/bin/SimAI_phynet /opt/SimAI/hostlist -g 2 -i 3 -w /opt/SimAI/example/work.txt
[535794] Total IB devices found: 4 src_addr: 116a8c0
[535794] i:0 dev:mlx5_0 uverbs0 netdev:mlx5_0 ia:216a8c0 ret=0
[1017216] Total IB devices found: 4 src_addr: 216a8c0
[1017216] i:0 dev:mlx5_0 uverbs0 netdev:mlx5_0 ia:116a8c0 ret=0
[535794] i:1 dev:mlx5_1 uverbs1 netdev:mlx5_1 ia:0 ret=-19
[1017216] i:1 dev:mlx5_1 uverbs1 netdev:mlx5_1 ia:0 ret=-19
[1017216] i:2 dev:mlx5_2 uverbs2 netdev:mlx5_2 ia:0 ret=-19
[535794] i:2 dev:mlx5_2 uverbs2 netdev:mlx5_2 ia:0 ret=-19
[535794] i:3 dev:mlx5_3 uverbs3 netdev:mlx5_3 ia:0 ret=-19
[1017216] i:3 dev:mlx5_3 uverbs3 netdev:mlx5_3 ia:0 ret=-19
The final active chunks per dimension 1 after allocating to queues is: 1
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 1total nodes in ring: 9
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 1total nodes in ring: 9
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 1total nodes in ring: 9
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 1total nodes in ring: 9
total nodes: 9
Success in opening workload file
model_parallel_NPU_group is 2
checkpoints layers are: 
layers initiating fwd_in_bckwd are: 

【人工回复】【Manual reply】

如果这两个文件的主机顺序不一致，rank 1 的主机就会拿到 rank 0 的 IP 去匹配本地设备

这样可能导致匹配失败？

或者 如果用户在 workaround 中改名后，两台主机的 RDMA 网卡 IP 配置有误（如未正确配置或使用了重复 IP），也可能导致此问题？

[tianhao909]

这个问题我也碰到了。 mlx5_0 网卡 获取的 接口名也是 mlx5_0 导致无法获取ip。 我通过 ifconfig 直接把对应网卡 改名成 mlx5_0 。后续ip就能正常找到了。 同时还出现了一个奇怪的现象，完全相同的参数在 第一个主机上可以正常运行，在第二个主机上会获取 到 第一个主机的地址进行匹配的导致初始化失败。 因为一台服务器就一个网口，直接去取巧

int
FlowPhyRdma::ibv_init(void) {
int nb_dev = 0;
struct ibv_device dev_lst = NULL;
struct ibv_device ibv_dev = NULL;
in_addr_t src_addr = inet_addr(rank2addr[local_rank].c_str());
MockNcclLog NcclLog = MockNcclLog::getInstance();
pid_t pid = getpid();

dev_lst = ibv_get_device_list(&nb_dev);
assert(nb_dev > 0);
assert(dev_lst != NULL);
fprintf(stdout, "[%d] Total IB devices found: %d src_addr: %x\n", pid, nb_dev, src_addr);

for (int i = 0; i < nb_dev; ++i) {
    in_addr_t ia = 0;
    char netdev[64];
    ibdev2netdev(dev_lst[i]->name,netdev);
    int ret = zeta_util_ifname_to_inet_addr(netdev, &ia);

    fprintf(stdout, "[%d] i:%d dev:%s %s netdev:%s ia:%x ret=%d\n", pid, i, dev_lst[i]->name, dev_lst[i]->dev_name, netdev, ia, ret);
    NcclLog->writeLog(
    NcclLogLevel::DEBUG, "netdev %s ibdev %s",netdev,dev_lst[i]->name);
    if (ret == 0 && ia == src_addr) {
        ibv_dev = dev_lst[i];
        break;
    }
    else if (ret == 0) {
        ibv_dev = dev_lst[i];
    }
}

if (ibv_dev == NULL) {
    NcclLog->writeLog(NcclLogLevel::ERROR,"ibv Device not found");
    return -1;
} else{
    NcclLog->writeLog(
        NcclLogLevel::DEBUG, "ibv device %s init success ", ibv_dev->dev_name);
}
g_ibv_ctx = ibv_open_device(ibv_dev);
assert(g_ibv_ctx != NULL);

return 0;

}
调试日志如下

+ mpirun -np 2 --wdir /opt/SimAI --host 192.168.22.1,192.168.22.2 --allow-run-as-root --mca plm_rsh_args '-p 2222' -x AS_LOG_LEVEL=3 -x UCX_IB_TRAFFIC_CLASS=96 /opt/SimAI/bin/SimAI_phynet /opt/SimAI/hostlist -g 2 -i 3 -w /opt/SimAI/example/work.txt
[535794] Total IB devices found: 4 src_addr: 116a8c0
[535794] i:0 dev:mlx5_0 uverbs0 netdev:mlx5_0 ia:216a8c0 ret=0
[1017216] Total IB devices found: 4 src_addr: 216a8c0
[1017216] i:0 dev:mlx5_0 uverbs0 netdev:mlx5_0 ia:116a8c0 ret=0
[535794] i:1 dev:mlx5_1 uverbs1 netdev:mlx5_1 ia:0 ret=-19
[1017216] i:1 dev:mlx5_1 uverbs1 netdev:mlx5_1 ia:0 ret=-19
[1017216] i:2 dev:mlx5_2 uverbs2 netdev:mlx5_2 ia:0 ret=-19
[535794] i:2 dev:mlx5_2 uverbs2 netdev:mlx5_2 ia:0 ret=-19
[535794] i:3 dev:mlx5_3 uverbs3 netdev:mlx5_3 ia:0 ret=-19
[1017216] i:3 dev:mlx5_3 uverbs3 netdev:mlx5_3 ia:0 ret=-19
The final active chunks per dimension 1 after allocating to queues is: 1
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 1total nodes in ring: 9
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 1total nodes in ring: 9
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 1total nodes in ring: 9
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 1total nodes in ring: 9
total nodes: 9
Success in opening workload file
model_parallel_NPU_group is 2
checkpoints layers are: 
layers initiating fwd_in_bckwd are: 

[Reply from Agent A] [For reference only]

cn

Issue #280 评论回复 — 针对 zxfszane (issuecomment-4534366623)

1. 需求背景总结

用户 zxfszane 在评论中报告了两个问题：

1. ibdev2netdev 映射错误（与原 issue 相同）：mlx5_0 网卡获取的接口名也是 mlx5_0，导致无法获取 IP。用户通过 ifconfig 直接把网卡改名成 mlx5_0 来绕过。
2. 第二台主机地址匹配错误（新问题）：完全相同的参数在第一台主机上可以正常运行，但第二台主机会获取到第一台主机的地址进行匹配，导致初始化失败。

---

2. 涉及文件/模块

文件	关键行	说明
astra-sim/system/SimAiFlowModelRdma.cc	L663-671	ibdev2netdev() IB 设备名到网口名映射错误
astra-sim/system/SimAiFlowModelRdma.cc	L677	ibv_dev 未初始化为 NULL
astra-sim/system/SimAiFlowModelRdma.cc	L678	rank2addr[local_rank] 地址查找逻辑
astra-sim/system/BootStrapnet.cc	L34-49	hostlist 读取逻辑（按行顺序映射 rank→IP）
astra-sim/network_frontend/phynet/SimAiMain.cc	L122-125	ibv_init() 返回值未检查

---

3. 根因分析

问题 1：ibdev2netdev() 映射错误（已确认）

static void
ibdev2netdev(const char indev[64], char netdev[64]) {
  if (strncmp(indev, "mlx5_bond_", 10) == 0) {
    strcpy(netdev, "bond");
    strcat(netdev, indev + 10);
  } else {
    strcpy(netdev, indev);  // BUG: 直接将 IB 设备名当作网口名
  }
}

根因：Linux 中 IB 设备名（mlx5_0、mlx5_1）与网络接口名（ens1f0、enp175s0f0、ib0 等）是不同的。代码仅处理了阿里云 bond 场景（mlx5_bond_X → bondX），对普通设备直接将 IB 设备名作为网口名传入 ioctl(SIOCGIFADDR)，导致 IP 查找失败。

zxfszane 的 workaround 确认：用户通过 ifconfig 把网卡改名为 mlx5_0，使得 ioctl 能通过这个名字找到 IP。这验证了上述根因分析。

问题 2：第二台主机获取到第一台主机的地址

代码流程：

initBootStrapNetRank() 读取 hostlist 文件:
  第 0 行 → rank2addr[0]
  第 1 行 → rank2addr[1]
  ...

MPI_Comm_rank() → local_rank

ibv_init() 中:
  src_addr = inet_addr(rank2addr[local_rank])  // 用 local_rank 索引 hostlist

根因分析：hostlist 文件的行序必须与 MPI rank 分配顺序完全一致。

具体来说：

• MPI 的 -hostfile ./hostfile 决定了哪台机器是 rank 0、哪台是 rank 1
• SimAI 的 ./hostlist（argv[1]）按行号映射 rank → RDMA IP
• 如果这两个文件的主机顺序不一致，rank 1 的主机就会拿到 rank 0 的 IP 去匹配本地设备

举例：

# MPI hostfile（决定 rank 分配）:
host_B    # MPI 分配为 rank 0
host_A    # MPI 分配为 rank 1

# SimAI hostlist（决定 rank→IP 映射）:
192.168.1.1   # rank2addr[0] = host_A 的 IP
192.168.1.2   # rank2addr[1] = host_B 的 IP

此时 host_B（rank 0）会用 192.168.1.1（host_A 的 IP）去匹配本地设备 → 匹配失败。

另一种可能：如果用户在 workaround 中改名后，两台主机的 RDMA 网卡 IP 配置有误（如未正确配置或使用了重复 IP），也会导致此问题。

---

4. 最小修改方案

修复 1（核心）：ibdev2netdev() 使用 sysfs 精确查询网口名

#include <dirent.h>

static int
ibdev2netdev(const char indev[64], char netdev[64], int port, int gid_idx) {
  char path[256];
  // 优先通过 GID 属性精确查询（考虑 RoCE 场景下 GID index 对应特定接口）
  snprintf(path, sizeof(path),
    "/sys/class/infiniband/%s/ports/%d/gid_attrs/ndevs/%d", indev, port, gid_idx);

  FILE* fp = fopen(path, "r");
  if (fp) {
    if (fscanf(fp, "%63s", netdev) == 1) {
      fclose(fp);
      return 0;
    }
    fclose(fp);
  }

  // 回退：通过 device/net/ 目录查找
  snprintf(path, sizeof(path),
    "/sys/class/infiniband/%s/device/net", indev);
  DIR* dir = opendir(path);
  if (dir != NULL) {
    struct dirent* entry;
    while ((entry = readdir(dir)) != NULL) {
      if (entry->d_name[0] == '.') continue;
      strncpy(netdev, entry->d_name, 63);
      netdev[63] = '\0';
      closedir(dir);
      return 0;
    }
    closedir(dir);
  }

  // 回退：bond 兼容
  if (strncmp(indev, "mlx5_bond_", 10) == 0) {
    strcpy(netdev, "bond");
    strcat(netdev, indev + 10);
    return 0;
  }

  // 最后回退
  strcpy(netdev, indev);
  return -1;
}

调用处也需修改：

// 原: ibdev2netdev(dev_lst[i]->name, netdev);
// 改为:
ibdev2netdev(dev_lst[i]->name, netdev, IB_PORT, gid_index);

修复 2：ibv_dev 初始化为 NULL

// L677: 将
struct ibv_device* ibv_dev;
// 改为
struct ibv_device* ibv_dev = NULL;

修复 3：ibv_init() 返回值检查

// SimAiMain.cc L122-125
#ifdef PHY_RDMA
  flow_rdma = FlowPhyRdma(user_param.gid_index);
  if (flow_rdma.ibv_init() != 0) {
    NcclLog->writeLog(NcclLogLevel::ERROR,
      "ibv_init failed, please check IB device and hostlist configuration");
    MPI_Abort(MPI_COMM_WORLD, -1);
  }
#endif

修复 4：增加诊断日志帮助排查地址匹配问题

在 ibv_init() 中增加日志：

NcclLog->writeLog(NcclLogLevel::DEBUG,
  "local_rank=%d, expected IP=%s", local_rank, rank2addr[local_rank].c_str());

以及在循环中：

NcclLog->writeLog(NcclLogLevel::DEBUG,
  "dev[%d] ibdev=%s netdev=%s ip=%s match=%s",
  i, dev_lst[i]->name, netdev, inet_ntoa(*(struct in_addr*)&ia),
  (ia == src_addr) ? "YES" : "NO");

---

5. 针对 zxfszane 的具体建议

关于 workaround（改名网卡）

你的 workaround 虽然有效，但有几个风险：

• 重启后改名会丢失（除非配置 udev 规则）
• 如果有多块网卡，可能引起混乱
• 不解决根本问题

推荐方案：直接修改 ibdev2netdev() 函数使用 sysfs 查询（见修复 1），这是一劳永逸的方案。

关于第二台主机地址匹配错误

请检查以下几点：

# 1. 确认 MPI hostfile 的主机顺序
cat ./hostfile
# 输出示例：
# host_A slots=1
# host_B slots=1
# 这意味着 host_A 是 rank 0, host_B 是 rank 1

# 2. 确认 hostlist 的 IP 顺序与上面一致
cat ./hostlist
# 输出应为：
# <host_A 的 RDMA IP>
# <host_B 的 RDMA IP>

# 3. 在每台主机上验证 RDMA IP
# 在 host_A 上:
ip addr show <rdma_interface>
# 确认 IP 与 hostlist 第 1 行一致

# 在 host_B 上:
ip addr show <rdma_interface>
# 确认 IP 与 hostlist 第 2 行一致

关键规则：hostlist 中第 N 行的 IP，必须是 MPI rank N 所在主机的 RDMA 网口 IP。

---

6. 测试前置条件

1. 硬件要求：至少 2 台有 RDMA 网卡（Mellanox ConnectX）的物理服务器
2. 驱动要求：ibverbs 库和 mlx5_core 驱动已安装
3. sysfs 路径可用：/sys/class/infiniband/ 目录存在且可读
4. hostlist 顺序正确：文件中 IP 顺序与 MPI hostfile 主机顺序一致
5. 不依赖额外测试数据：使用已有 workload 文件即可

---

7. README 是否需要修改及建议内容

建议修改 docs/Tutorial.md，补充以下内容：

1. hostlist 文件说明：

   • 明确说明 hostlist 中 IP 的顺序必须与 MPI hostfile 中主机的顺序一致
   • 第 N 行对应 MPI rank N 的 RDMA 网口 IP

2. 多机部署验证步骤：

   # 验证 IB 设备与网口映射
   cat /sys/class/infiniband/mlx5_0/ports/1/gid_attrs/ndevs/0
   
   # 验证该网口有正确 IP
   ip addr show <netdev_name>
   
   # 验证 hostlist 顺序与 MPI rank 对应
   mpirun -np 2 -hostfile ./hostfile hostname
   # 输出的主机名顺序应与 hostlist 的 IP 顺序一致

3. 常见错误及排查：

   • Assertion 'g_ibv_ctx != NULL' failed → 检查 ibdev2netdev 映射和 hostlist 顺序
   • 第二台主机初始化失败 → 检查 hostlist 与 MPI hostfile 的顺序一致性

en

Issue #280 Comment Reply — Response to zxfszane (issuecomment-4534366623)

1. Background Summary

User zxfszane reported two problems in their comment:

1. ibdev2netdev mapping error (same as original issue): The mlx5_0 NIC gets interface name mlx5_0, which cannot be used to retrieve the IP. The user worked around this by renaming the network interface to mlx5_0 via ifconfig.
2. Second host address mismatch (new issue): With identical parameters, the program runs normally on the first host but on the second host it matches against the first host's address, causing initialization failure.

---

2. Affected Files/Modules

File	Key Lines	Description
astra-sim/system/SimAiFlowModelRdma.cc	L663-671	ibdev2netdev() IB device to netdev mapping error
astra-sim/system/SimAiFlowModelRdma.cc	L677	ibv_dev not initialized to NULL
astra-sim/system/SimAiFlowModelRdma.cc	L678	rank2addr[local_rank] address lookup logic
astra-sim/system/BootStrapnet.cc	L34-49	hostlist reading logic (sequential rank-to-IP mapping)
astra-sim/network_frontend/phynet/SimAiMain.cc	L122-125	ibv_init() return value unchecked

---

3. Root Cause Analysis

Issue 1: ibdev2netdev() Mapping Error (Confirmed)

static void
ibdev2netdev(const char indev[64], char netdev[64]) {
  if (strncmp(indev, "mlx5_bond_", 10) == 0) {
    strcpy(netdev, "bond");
    strcat(netdev, indev + 10);
  } else {
    strcpy(netdev, indev);  // BUG: uses IB device name as network interface name
  }
}

Root Cause: In Linux, IB device names (mlx5_0, mlx5_1) and network interface names (ens1f0, enp175s0f0, ib0, etc.) are different entities. The code only handles Alibaba Cloud's bond scenario (mlx5_bond_X -> bondX). For regular devices, it directly passes the IB device name to ioctl(SIOCGIFADDR), causing IP lookup failure.

zxfszane's workaround confirms this: The user renamed their network interface to mlx5_0 so that ioctl can find the IP via this name. This validates the root cause analysis.

Issue 2: Second Host Gets First Host's Address

Code Flow:

initBootStrapNetRank() reads hostlist file:
  line 0 -> rank2addr[0]
  line 1 -> rank2addr[1]
  ...

MPI_Comm_rank() -> local_rank

ibv_init():
  src_addr = inet_addr(rank2addr[local_rank])  // indexes hostlist by local_rank

Root Cause: The line order in hostlist MUST match the MPI rank assignment order exactly.

Specifically:

• MPI's -hostfile ./hostfile determines which machine is rank 0, which is rank 1
• SimAI's ./hostlist (argv[1]) maps line numbers to RDMA IPs
• If the host order in these two files is inconsistent, rank 1's host will try to match rank 0's IP against its local devices

Example:

# MPI hostfile (determines rank assignment):
host_B    # MPI assigns as rank 0
host_A    # MPI assigns as rank 1

# SimAI hostlist (determines rank->IP mapping):
192.168.1.1   # rank2addr[0] = host_A's IP
192.168.1.2   # rank2addr[1] = host_B's IP

In this case, host_B (rank 0) will try to match 192.168.1.1 (host_A's IP) against its local devices -> match fails.

Alternative possibility: If the user's workaround (renaming interface) resulted in incorrect RDMA NIC IP configuration on both hosts (e.g., duplicate IPs), this would also cause the issue.

---

4. Minimal Fix Proposal

Fix 1 (Core): ibdev2netdev() using sysfs for accurate netdev lookup

#include <dirent.h>

static int
ibdev2netdev(const char indev[64], char netdev[64], int port, int gid_idx) {
  char path[256];
  // Primary: query via GID attributes (precise for RoCE scenarios)
  snprintf(path, sizeof(path),
    "/sys/class/infiniband/%s/ports/%d/gid_attrs/ndevs/%d", indev, port, gid_idx);

  FILE* fp = fopen(path, "r");
  if (fp) {
    if (fscanf(fp, "%63s", netdev) == 1) {
      fclose(fp);
      return 0;
    }
    fclose(fp);
  }

  // Fallback: enumerate device/net/ directory
  snprintf(path, sizeof(path),
    "/sys/class/infiniband/%s/device/net", indev);
  DIR* dir = opendir(path);
  if (dir != NULL) {
    struct dirent* entry;
    while ((entry = readdir(dir)) != NULL) {
      if (entry->d_name[0] == '.') continue;
      strncpy(netdev, entry->d_name, 63);
      netdev[63] = '\0';
      closedir(dir);
      return 0;
    }
    closedir(dir);
  }

  // Fallback: bond compatibility
  if (strncmp(indev, "mlx5_bond_", 10) == 0) {
    strcpy(netdev, "bond");
    strcat(netdev, indev + 10);
    return 0;
  }

  // Last resort fallback
  strcpy(netdev, indev);
  return -1;
}

Call site change:

// Before: ibdev2netdev(dev_lst[i]->name, netdev);
// After:
ibdev2netdev(dev_lst[i]->name, netdev, IB_PORT, gid_index);

Fix 2: Initialize ibv_dev to NULL

// L677: Change
struct ibv_device* ibv_dev;
// To:
struct ibv_device* ibv_dev = NULL;

Fix 3: Check ibv_init() return value

// SimAiMain.cc L122-125
#ifdef PHY_RDMA
  flow_rdma = FlowPhyRdma(user_param.gid_index);
  if (flow_rdma.ibv_init() != 0) {
    NcclLog->writeLog(NcclLogLevel::ERROR,
      "ibv_init failed, please check IB device and hostlist configuration");
    MPI_Abort(MPI_COMM_WORLD, -1);
  }
#endif

Fix 4: Add diagnostic logging for address matching

Add to ibv_init():

NcclLog->writeLog(NcclLogLevel::DEBUG,
  "local_rank=%d, expected IP=%s", local_rank, rank2addr[local_rank].c_str());

And inside the device iteration loop:

NcclLog->writeLog(NcclLogLevel::DEBUG,
  "dev[%d] ibdev=%s netdev=%s ip=%s match=%s",
  i, dev_lst[i]->name, netdev, inet_ntoa(*(struct in_addr*)&ia),
  (ia == src_addr) ? "YES" : "NO");

---

5. Specific Recommendations for zxfszane

Regarding the Workaround (Renaming Interface)

Your workaround is effective but has several risks:

• The rename is lost after reboot (unless udev rules are configured)
• May cause confusion if there are multiple NICs
• Does not address the root cause

Recommended approach: Directly modify the ibdev2netdev() function to use sysfs queries (see Fix 1 above). This is a permanent solution.

Regarding the Second Host Address Mismatch

Please verify the following:

# 1. Check MPI hostfile host order
cat ./hostfile
# Example output:
# host_A slots=1
# host_B slots=1
# This means host_A is rank 0, host_B is rank 1

# 2. Verify hostlist IP order matches the above
cat ./hostlist
# Should be:
# <host_A's RDMA IP>
# <host_B's RDMA IP>

# 3. Verify RDMA IP on each host
# On host_A:
ip addr show <rdma_interface>
# Confirm IP matches line 1 of hostlist

# On host_B:
ip addr show <rdma_interface>
# Confirm IP matches line 2 of hostlist

Key Rule: Line N in hostlist must be the RDMA NIC IP of the host that MPI assigns as rank N.

---

6. Test Prerequisites

1. Hardware: At least 2 physical servers with RDMA NICs (Mellanox ConnectX series)
2. Drivers: ibverbs library and mlx5_core driver installed
3. sysfs available: /sys/class/infiniband/ directory exists and is readable
4. hostlist order correct: IPs in hostlist match MPI hostfile host order
5. No extra test data needed: Existing workload files are sufficient

---

7. README Changes Recommendation

Suggested changes to docs/Tutorial.md:

1. hostlist file documentation:

   • Clearly state that IP order in hostlist must match host order in MPI hostfile
   • Line N corresponds to MPI rank N's RDMA NIC IP

2. Multi-host deployment verification steps:

   # Verify IB device to netdev mapping
   cat /sys/class/infiniband/mlx5_0/ports/1/gid_attrs/ndevs/0
   
   # Verify the interface has the correct IP
   ip addr show <netdev_name>
   
   # Verify hostlist order matches MPI rank assignment
   mpirun -np 2 -hostfile ./hostfile hostname
   # Output host order should match IP order in hostlist

3. Common errors and troubleshooting:

   • Assertion 'g_ibv_ctx != NULL' failed -> Check ibdev2netdev mapping and hostlist order
   • Second host initialization failure -> Verify hostlist and MPI hostfile order consistency

[tianhao909]

这个问题我也碰到了。 mlx5_0 网卡 获取的 接口名也是 mlx5_0 导致无法获取ip。 我通过 ifconfig 直接把对应网卡 改名成 mlx5_0 。后续ip就能正常找到了。 同时还出现了一个奇怪的现象，完全相同的参数在 第一个主机上可以正常运行，在第二个主机上会获取 到 第一个主机的地址进行匹配的导致初始化失败。 因为一台服务器就一个网口，直接去取巧

int
FlowPhyRdma::ibv_init(void) {
int nb_dev = 0;
struct ibv_device dev_lst = NULL;
struct ibv_device ibv_dev = NULL;
in_addr_t src_addr = inet_addr(rank2addr[local_rank].c_str());
MockNcclLog NcclLog = MockNcclLog::getInstance();
pid_t pid = getpid();

dev_lst = ibv_get_device_list(&nb_dev);
assert(nb_dev > 0);
assert(dev_lst != NULL);
fprintf(stdout, "[%d] Total IB devices found: %d src_addr: %x\n", pid, nb_dev, src_addr);

for (int i = 0; i < nb_dev; ++i) {
    in_addr_t ia = 0;
    char netdev[64];
    ibdev2netdev(dev_lst[i]->name,netdev);
    int ret = zeta_util_ifname_to_inet_addr(netdev, &ia);

    fprintf(stdout, "[%d] i:%d dev:%s %s netdev:%s ia:%x ret=%d\n", pid, i, dev_lst[i]->name, dev_lst[i]->dev_name, netdev, ia, ret);
    NcclLog->writeLog(
    NcclLogLevel::DEBUG, "netdev %s ibdev %s",netdev,dev_lst[i]->name);
    if (ret == 0 && ia == src_addr) {
        ibv_dev = dev_lst[i];
        break;
    }
    else if (ret == 0) {
        ibv_dev = dev_lst[i];
    }
}

if (ibv_dev == NULL) {
    NcclLog->writeLog(NcclLogLevel::ERROR,"ibv Device not found");
    return -1;
} else{
    NcclLog->writeLog(
        NcclLogLevel::DEBUG, "ibv device %s init success ", ibv_dev->dev_name);
}
g_ibv_ctx = ibv_open_device(ibv_dev);
assert(g_ibv_ctx != NULL);

return 0;

}
调试日志如下

+ mpirun -np 2 --wdir /opt/SimAI --host 192.168.22.1,192.168.22.2 --allow-run-as-root --mca plm_rsh_args '-p 2222' -x AS_LOG_LEVEL=3 -x UCX_IB_TRAFFIC_CLASS=96 /opt/SimAI/bin/SimAI_phynet /opt/SimAI/hostlist -g 2 -i 3 -w /opt/SimAI/example/work.txt
[535794] Total IB devices found: 4 src_addr: 116a8c0
[535794] i:0 dev:mlx5_0 uverbs0 netdev:mlx5_0 ia:216a8c0 ret=0
[1017216] Total IB devices found: 4 src_addr: 216a8c0
[1017216] i:0 dev:mlx5_0 uverbs0 netdev:mlx5_0 ia:116a8c0 ret=0
[535794] i:1 dev:mlx5_1 uverbs1 netdev:mlx5_1 ia:0 ret=-19
[1017216] i:1 dev:mlx5_1 uverbs1 netdev:mlx5_1 ia:0 ret=-19
[1017216] i:2 dev:mlx5_2 uverbs2 netdev:mlx5_2 ia:0 ret=-19
[535794] i:2 dev:mlx5_2 uverbs2 netdev:mlx5_2 ia:0 ret=-19
[535794] i:3 dev:mlx5_3 uverbs3 netdev:mlx5_3 ia:0 ret=-19
[1017216] i:3 dev:mlx5_3 uverbs3 netdev:mlx5_3 ia:0 ret=-19
The final active chunks per dimension 1 after allocating to queues is: 1
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 1total nodes in ring: 9
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 1total nodes in ring: 9
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 1total nodes in ring: 9
ring of node 0, id: 0 dimension: local total nodes in ring: 9 index in ring: 0 offset: 1total nodes in ring: 9
total nodes: 9
Success in opening workload file
model_parallel_NPU_group is 2
checkpoints layers are: 
layers initiating fwd_in_bckwd are: 

[Reply from Agent B] [For reference only]

以下是基于 issue、评论和当前代码的分析与最小修改建议，不涉及直接改代码。结论：这个问题主要集中在 SimAI_phynet 的 RDMA 设备选择逻辑，尤其是 RDMA device name 到 Linux netdev name 的映射不可靠，以及 MPI rank 与 hostlist IP 的对应关系可能不符合预期。

说明：我通过 GitHub code search / 文件读取看了相关代码。code search 返回结果数量有限，可能不完整；如需继续查看可在 GitHub UI 搜索：
https://github.com/search?q=repo%3Aaliyun%2FSimAI+ibdev2netdev+OR+g_ibv_ctx+OR+ibv_init&type=code

---

中文版 MD

1. 需求背景总结

这个 issue 的背景是：用户使用 SimAI_phynet 做物理网络 / RoCEv2 发包时，程序在 RDMA verbs 初始化阶段失败：

FlowPhyRdma::ibv_init(): Assertion 'g_ibv_ctx != NULL' failed.

从 issue 和后续评论看，核心现象有两个：

1. mlx5_0 这类 RDMA 设备名被直接当成 Linux 网卡接口名使用，导致无法正确获取 IP。
2. 用户临时通过 ifconfig 把对应网卡重命名成 mlx5_0 后，IP 匹配可以继续往下走。
3. 新评论里还提到一个额外现象：相同参数下，第一个主机可以跑，第二个主机却拿到了第一个主机的地址进行匹配，导致初始化失败。

从代码看，这个问题大概率不是 AICB workload 本身导致的，也不是 ns-3 后端路径导致的，而是在 SimAI_phynet 物理 RDMA 初始化时，选择本机 RDMA 设备的逻辑存在环境假设。

---

2. 涉及文件/模块

主要涉及 aliyun/SimAI

1. RDMA 设备选择与初始化

文件：

astra-sim-alibabacloud/astra-sim/system/SimAiFlowModelRdma.cc

关键位置：

• zeta_util_ifname_to_inet_addr()：通过 Linux netdev name 查询 IPv4 地址
  

  SimAI/astra-sim-alibabacloud/astra-sim/system/SimAiFlowModelRdma.cc

  Lines 634 to 661 in f5efb5a

  	static int
  	zeta_util_ifname_to_inet_addr(const char *ifname, in_addr_t *addr) {
  	int ret = 0;
  	int fd;
  	fd = socket(AF_INET, SOCK_DGRAM, 0);
  	if (fd == -1) {
  	return -errno;
  	}
  	if (addr != NULL) {
  	struct ifreq ifr = {};
  	strncpy(ifr.ifr_name, ifname, IFNAMSIZ - 1);
  	ifr.ifr_addr.sa_family = AF_INET;
  	ret = ioctl(fd, SIOCGIFADDR, &ifr);
  	if (ret != 0) {
  	ret = -errno;
  	goto out;
  	}
  	*addr = ((struct sockaddr_in*)&ifr.ifr_addr)->sin_addr.s_addr;
  	}
  	out:
  	close(fd);
  	return ret;
  	}

• ibdev2netdev()：当前 RDMA device name 到 netdev name 的映射逻辑
  

  SimAI/astra-sim-alibabacloud/astra-sim/system/SimAiFlowModelRdma.cc

  Lines 663 to 671 in f5efb5a

  	static void
  	ibdev2netdev(const char indev[64], char netdev[64]) {
  	if (strncmp(indev, "mlx5_bond_", 10) == 0) {
  	strcpy(netdev, "bond");
  	strcat(netdev, indev + 10);
  	} else {
  	strcpy(netdev, indev);
  	}
  	}

• FlowPhyRdma::ibv_init()：遍历 RDMA 设备、匹配 hostlist IP、调用 ibv_open_device()
  

  SimAI/astra-sim-alibabacloud/astra-sim/system/SimAiFlowModelRdma.cc

  Lines 673 to 708 in f5efb5a

  	int
  	FlowPhyRdma::ibv_init(void) {
  	int nb_dev = 0;
  	struct ibv_device **dev_lst = NULL;
  	struct ibv_device* ibv_dev;
  	in_addr_t src_addr = inet_addr(rank2addr[local_rank].c_str());
  	MockNcclLog* NcclLog = MockNcclLog::getInstance();
  	dev_lst = ibv_get_device_list(&nb_dev);
  	assert(nb_dev > 0);
  	assert(dev_lst != NULL);
  	for (int i = 0; i < nb_dev; ++i) {
  	in_addr_t ia = 0;
  	char netdev[64];
  	ibdev2netdev(dev_lst[i]->name,netdev);
  	int ret = zeta_util_ifname_to_inet_addr(netdev, &ia);
  	NcclLog->writeLog(
  	NcclLogLevel::DEBUG, "netdev %s ibdev %s",netdev,dev_lst[i]->name);
  	if (ret == 0 && ia == src_addr) {
  	ibv_dev = dev_lst[i];
  	break;
  	}
  	}
  	if (ibv_dev == NULL) {
  	NcclLog->writeLog(NcclLogLevel::ERROR,"ibv Device not found");
  	return -1;
  	} else{
  	NcclLog->writeLog(
  	NcclLogLevel::DEBUG, "ibv device %s init success ", ibv_dev->dev_name);
  	}
  	g_ibv_ctx = ibv_open_device(ibv_dev);
  	assert(g_ibv_ctx != NULL);
  	return 0;

2. hostlist 读取与 rank 到 IP 的映射

文件：

astra-sim-alibabacloud/astra-sim/system/BootStrapnet.cc

关键位置：

• MPI_Comm_rank() 得到当前进程 rank
• 从 argv[1] 读取 hostlist
• 按行填充 rank2addr[rank]

链接：

SimAI/astra-sim-alibabacloud/astra-sim/system/BootStrapnet.cc

Lines 30 to 58 in f5efb5a

	int world_size,local_rank;
	std::map<int,std::string> rank2addr;
	static void
	initBootStrapNetRank(int argc,char*argv[]){
	std::string host_file = argv[1];
	std::string line;
	std::ifstream file(host_file);
	int rank = 0;
	if(file.is_open()){
	while (std::getline(file,line)){
	/* code */
	rank2addr[rank] = line;
	rank++;
	}
	file.close();
	} else{
	std::cerr << "Failed to open the file" << std::endl;
	}
	}
	void BootStrapNet(int argc, char *argv[]){
	MPI_Init(&argc, &argv);
	MPI_Comm_size(MPI_COMM_WORLD, &world_size);
	MPI_Comm_rank(MPI_COMM_WORLD, &local_rank);
	MPI_Barrier(MPI_COMM_WORLD);
	initBootStrapNetRank(argc,argv);
	}

这里需要注意：代码使用的是 MPI global rank 去索引 rank2addr[local_rank]，变量名叫 local_rank，但实际上来自 MPI_Comm_rank(MPI_COMM_WORLD, &local_rank)，所以它是 MPI world rank，不是节点内 local rank。

3. SimAI_phynet 启动入口

文件：

astra-sim-alibabacloud/astra-sim/network_frontend/phynet/SimAiMain.cc

关键位置：

• 参数解析，尤其是 -g 和 -i
• flow_rdma.ibv_init() 调用后没有检查返回值

链接：

SimAI/astra-sim-alibabacloud/astra-sim/network_frontend/phynet/SimAiMain.cc

Lines 68 to 125 in f5efb5a

	static int user_param_prase(int argc,char * argv[],struct user_param* user_param){
	int opt;
	static struct option long_options[] = {
	{"help", no_argument, 0, 'h'},
	{"workloads", required_argument, 0, 'w'},
	{"gpus", required_argument, 0, 'g'},
	{"comm_scale", required_argument, 0, 's'},
	{"gid_index", required_argument, 0, 'i'},
	{0, 0, 0, 0}};
	while ((opt = getopt(argc,argv,"ht:w:g:s:i:"))!=-1){
	switch (opt)
	{
	case 'h':
	/* code */
	std::cout<<"-w workloads default microAllReduce.txt "<<std::endl;
	std::cout<<"-g number of gpus,default 1"<<std::endl;
	std::cout<<"-s comm_scale default 1"<<std::endl;
	std::cout<<"-i rdma gid_indxe default 0" <<std::endl;
	break;
	case 't':
	user_param->thread = stoi(optarg);
	break;
	case 'w':
	user_param->workload = optarg;
	break;
	case 'g':
	user_param->gpus = stoi(optarg);
	if(user_param->gpus <= 8){
	user_param->gpus =8;
	}
	break;
	case 's':
	user_param->comm_scale = stof(optarg);
	break;
	case 'i':
	user_param->gid_index = stoi(optarg);
	break;
	default:
	break;
	}
	}
	return 0 ;
	}
	int main(int argc,char *argv[]){
	BootStrapNet(argc,argv);
	pid_t pid = getpid();
	MockNcclLog::set_log_name("SimAi_"+to_string(local_rank)+".log");
	MockNcclLog* NcclLog = MockNcclLog::getInstance();
	NcclLog->writeLog(NcclLogLevel::DEBUG," Local rank %d PID %d ",local_rank,pid);
	struct user_param user_param;
	if(user_param_prase(argc,argv,&user_param)){
	return -1;
	}
	#ifdef PHY_RDMA
	flow_rdma = FlowPhyRdma(user_param.gid_index);
	flow_rdma.ibv_init();
	#endif

4. 文档位置

文件：

docs/Tutorial.md

关键位置：

• SimAI-Physical 编译、hostlist、运行参数说明

链接：

SimAI/docs/Tutorial.md

Lines 330 to 404 in f5efb5a

	# SimAI-Physical Usage
	The current simulator is compatible with the ns3 discrete event simulator as the network backend, as well as the physical network backend for physical packet injection.
	## Compile
	SimAI-Phy currently uses the roceV2 protocol for traffic generation. The compilation process requires dependencies on libverbs related to the RDMA physical device, as well as the MPI program. Before compilation, please verify that your environment can successfully run the basic RDMA perfetest traffic generation tool and that it supports the related MPI program.
	```bash
	# Clone the repository
	$ git clone https://github.com/aliyun/SimAI.git
	$ cd ./SimAI/
	# Clone submodules
	$ git submodule update --init --recursive
	# Make sure to use the newest commit
	$ git submodule update --remote
	# Compile SimAI-Analytical
	$ ./scripts/build.sh -c analytical
	# Compile SimAI-Simulation (ns3)
	$ ./scripts/build.sh -c ns3
	# Compile SimAI-phynet (phynet)
	$ sudo yum install openmpi openmpi-devel
	$ export MPI_INCLUDE_PATH=/usr/include/openmpi-x86_64/
	$ export MPI_BIN_PATH=/usr/lib64/openmpi/bin/mpic++
	$ ./scripts/build.sh -c phy
	```
	## Workload Generate
	The workload required for SimAI-Phy physical traffic generation is the same as that for Sim-Simulation, and it is generated through AICB.
	### Example workload
	```bash
	HYBRID_TRANSFORMER_FWD_IN_BCKWD model_parallel_NPU_group: 2 ep: 1 pp: 1 vpp: 8 ga: 1 all_gpus: 2 checkpoints: 0 checkpoint_initiates: 0
	10
	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
	```
	## Prepare the hostlist
	Here the main task is to prepare the iplist required to start the MPI program, which is different from nccl-test. The number of IPs here should match the actual number of network cards involved in the physical traffic generation, rather than the number of nodes participating in the physical traffic generation.
	```bash
	33.255.199.130
	33.255.199.129
	```
	## RUN
	### MPI Run
	```bash
	/usr/lib64/openmpi/bin/mpirun -np 2 -host 33.255.199.130,33.255.199.129 --allow-run-as-root -x AS_LOG_LEVEL=0 ./bin/SimAI_phynet ./hostlist -g 2 -w ./example/microAllReduce.txt
	```
	The following output indicates that the program has finished running.
	<img src="./images/Sim-phynet_finished.png" alt="Sim-phynet_finished" width="40%">
	## Setting of MPI program parameters
	| Parameter | Description | Default Value |
	|------------------|-------------------------------------------------------|---------------|
	| -np | The number of processes. | NULL |
	| -host | IP list | NULL |
	| --allow-run-as-root | Allow MPI programs to run with root privileges. | FALSE |
	## Setting of SimAI-phynet parameters
	| Parameter | Description | Default Value |
	|------------------|------------------------------|----------------------------------------------------------|
	| hostlist | Host IP list | NULL |
	| -w --workload | Path to workload | ./microAllReduce.txt |
	| -i --gid_index | Network topology path | 0 |
	| -g --gpus | Number of GPUs | 8 (should be consistent with the number of IPs in the host IP list) |

参考仓库关系

• aliyun/SimAI：本 issue 的直接代码位置。
• aliyun/aicb：主要用于生成 workload。当前 RDMA 设备打不开的问题不直接由 AICB 产生。
• aliyun/ns-3-alibabacloud：ns-3 仿真后端。搜索结果显示 ns-3 构建中会排除 SimAiFlowModelRdma.cc 这类 phynet 物理 RDMA 文件，因此该问题不属于 ns-3 仿真路径。

---

3. 根因分析或实现思路

3.1 mlx5_0 被当成 Linux netdev name 是当前代码的明确问题

当前 ibdev2netdev() 的逻辑是：

• 如果 RDMA device name 以 mlx5_bond_ 开头，则映射成 bondX
• 否则直接把 RDMA device name 复制为 netdev name

也就是说：

• mlx5_bond_0 会变成 bond0
• mlx5_0 会保持为 mlx5_0

但在多数 Linux / Mellanox / NVIDIA RDMA 环境里：

• mlx5_0 是 verbs/RDMA device name
• 真正带 IP 的 Linux netdev 通常是 ethX、ensX、enpXsY、bondX 等

所以用户说的“mlx5_0 网卡获取的接口名也是 mlx5_0 导致无法获取 IP”，这个判断是对的。

用户用 ifconfig 把真实网卡改名成 mlx5_0 后可以继续运行，本质上是让 Linux netdev 名字强行匹配当前代码的错误假设。这可以作为临时验证，但不建议作为长期方案，因为它可能影响系统网络配置、udev 命名、运维脚本和其他 RDMA 工具。

---

3.2 ibv_dev 未初始化，会放大问题

FlowPhyRdma::ibv_init() 中有：

struct ibv_device* ibv_dev;

它没有初始化为 NULL。

如果遍历所有 RDMA 设备都没有找到匹配 IP 的设备，后面的 if (ibv_dev == NULL) 判断是不可靠的，因为 ibv_dev 可能是未定义值。

这会导致两类问题：

1. 明明没找到设备，却没有进入 ibv Device not found 分支。
2. 可能拿未定义指针去调用 ibv_open_device()，最终触发 g_ibv_ctx != NULL 的 assert。

所以即使先不改映射逻辑，也应该最小修复为初始化 ibv_dev = NULL。

---

3.3 ibv_open_device() 失败后直接 assert，不利于定位

当前逻辑是：

• 找到一个 ibv_dev
• 调用 ibv_open_device(ibv_dev)
• 如果返回空，直接 assert

这个问题是：用户看不到真实失败原因。

ibv_open_device() 返回 NULL 可能有多种原因：

• 没有选中正确的 RDMA 设备
• /dev/infiniband/uverbsX 不存在或权限不足
• 容器没有挂载 /dev/infiniband
• 驱动 / rdma-core / libibverbs 环境异常
• GID index 与当前网卡 / RoCE 配置不匹配
• hostlist 里写的是管理网 IP，不是 RoCE/RDMA 网卡 IP

建议改为打印 ibv_dev->name、ibv_dev->dev_name、errno、strerror(errno)，然后返回错误码。

---

3.4 新评论中的“第二台机器拿到第一台机器地址”更像 rank 与 hostlist 对应关系问题

评论日志里有类似现象：

• 一个进程的 src_addr 是 192.168.22.1，但本机 mlx5_0 查到的 IP 是 192.168.22.2
• 另一个进程的 src_addr 是 192.168.22.2，但本机 mlx5_0 查到的 IP 是 192.168.22.1

这说明很可能出现了：

MPI rank 顺序与 hostlist 行顺序不一致。

当前代码假设：

• MPI rank 0 一定对应 hostlist 第 0 行
• MPI rank 1 一定对应 hostlist 第 1 行

但实际 mpirun --host 192.168.22.1,192.168.22.2 -np 2 下，rank 到主机的分配可能受 OpenMPI mapping / ranking 策略、host slot、运行目录、SSH 启动顺序等影响。即使很多环境默认看起来是按顺序，也不建议代码完全依赖这个隐式假设。

另外也要排查：

1. 两台机器上的 /opt/SimAI/hostlist 内容是否完全一致。
2. mpirun --wdir /opt/SimAI 后，每个节点是否读到同一个 hostlist 文件。
3. 是否使用了不同的容器 / 挂载路径，导致 hostlist 内容不同。
4. inet_addr() 用 %x 打印时会受到字节序显示影响，不过这里两个地址互换的现象仍然指向 rank/IP 对应关系问题。

建议在日志里补充打印：

• MPI world rank
• hostname / MPI_Get_processor_name()
• rank2addr[local_rank]
• 当前遍历到的 RDMA device
• 映射出的 netdev
• netdev IP

这样可以直接判断“哪个 rank 跑在哪台机器上、它期望哪个 IP、实际本机网卡 IP 是什么”。

---

3.5 -g 2 会被强制改成 8，不是当前 assert 的直接原因，但值得单独说明

SimAiMain.cc 中：

• 用户传 -g 2
• 如果 gpus <= 8，代码会强制设为 8

所以 -g 2 实际会变成 8。这不是 ibv_open_device() 初始化失败的直接原因，因为 RDMA 初始化发生在后续网络模拟构建前；但它会影响后续 ring / node 数量，例如日志中出现 total nodes in ring: 9。

建议这个点不要和本 issue 的最小修复混在一起，可以单独确认是否符合预期。

---

4. 最小修改方案

建议按“最小可行修改”分三层处理。

方案 A：先修复崩溃和错误信息

这是最低风险、最小改动。

建议：

1. 在 FlowPhyRdma::ibv_init() 中初始化 ibv_dev 为 NULL。
2. ibv_get_device_list() 后不要只依赖 assert，增加正常错误处理：
   • dev_lst == NULL
   • nb_dev <= 0
3. 如果没有匹配到设备，打印：
   • 当前 MPI rank
   • rank2addr[local_rank]
   • 遍历过的 RDMA device / netdev / IP 查询结果
4. ibv_open_device() 返回 NULL 时，不要直接 assert：
   • 打印设备名
   • 打印 errno / strerror(errno)
   • 返回 -1
5. SimAiMain.cc 中检查 flow_rdma.ibv_init() 返回值，失败时退出，不继续运行后续逻辑。

这一步不改变设备选择策略，但可以避免无信息 abort。

---

方案 B：修复 RDMA device 到 Linux netdev 的映射

建议不要再用“mlx5_0 直接等于 netdev”这个 fallback 作为主要逻辑。

更稳妥的最小方案：

1. 优先用 gid-aware sysfs 路径查真实 netdev：

   /sys/class/infiniband/<ibdev>/ports/<IB_PORT>/gid_attrs/ndevs/<gid_index>

   例如：

   /sys/class/infiniband/mlx5_0/ports/1/gid_attrs/ndevs/3

   这个路径的优点是和 -i gid_index 对齐，适合 RoCE 多 GID、多网口、bond 或 VLAN 的场景。

2. 如果上面查不到，再 fallback 到：

   /sys/class/infiniband/<ibdev>/device/net/

   从该目录中取真实 Linux netdev。

3. 如果 sysfs 都失败，再保留旧逻辑：

   • mlx5_bond_X -> bondX
   • 否则保留原名，只作为最后 fallback

这样不需要引入第三方依赖，只需要使用现有 C/C++ 标准库或 POSIX 接口。

---

方案 C：处理 MPI rank 与 hostlist 关系

针对评论中“第二台机器拿到第一台机器地址”的问题，建议先不大改架构，而是最小增强诊断和文档。

建议：

1. 在 BootStrapNet() 或 ibv_init() 中打印：
   • MPI rank
   • processor name / hostname
   • rank2addr[rank]
2. 运行时建议用户加 OpenMPI 参数确认 rank 分配：
   • --display-map
   • 必要时使用 --map-by ppr:1:node --rank-by node
3. 文档明确说明：
   • hostlist 的第 N 行必须对应 MPI world rank N，而不只是“主机列表顺序”。
   • MPI --host / hostfile 和 SimAI 的 hostlist 是两个概念。
4. 如果后续要更健壮，可以考虑通过 hostname 或本机网卡 IP 反向匹配 hostlist，而不是完全依赖 rank2addr[world_rank]。但这属于更大改动，不建议放入本次最小修复。

---

不建议本次一起做的改动

为了保持最小改动，不建议本 issue 中同时处理：

1. 重构所有 phynet 参数解析。
2. 修改 -g <= 8 被强制改成 8 的行为。
3. 改 AICB workload 生成逻辑。
4. 改 ns-3 backend。
5. 引入新的第三方依赖。
6. 做复杂的多网卡策略选择配置。

---

5. 测试前置条件

编译测试

编译测试不需要额外 workload 数据，也不需要真实跨节点 RDMA 通信。

需要：

1. 能编译 SimAI phynet。
2. OpenMPI / mpic++。
3. libibverbs / rdma-core 头文件和库。
4. 当前项目已有构建路径。

可验证：

• phynet 能否编译通过。
• ibv_init() 新增日志和错误处理是否编译通过。
• 不影响 ns-3 / analytical 构建路径。

运行测试

运行验证需要真实或可访问的 RDMA/RoCE 环境。

需要：

1. 每台机器有 RDMA/RoCE 设备。
2. ibv_devices 能看到设备。
3. ibv_devinfo 能打开设备。
4. /dev/infiniband/uverbs* 存在且当前用户可访问。
5. 如果在容器里运行，需要挂载 /dev/infiniband。
6. hostlist 中写的是 RDMA/RoCE 网卡 IP，不是管理网 IP。
7. -i gid_index 对应有效 GID。
8. MPI rank 与 hostlist 行顺序一致。

建议用户在每台机器上执行：

• ibv_devices
• ibv_devinfo
• show_gids
• ip addr
• ls /sys/class/infiniband/<ibdev>/ports/1/gid_attrs/ndevs/
• cat /sys/class/infiniband/<ibdev>/ports/1/gid_attrs/ndevs/<gid_index>

如果要验证真实 RDMA 通信，还需要 ib_write_bw / ib_read_bw 这类 perftest 工具。

AICB workload 只需要一个能触发 phynet 初始化和简单通信的最小 workload，不需要额外业务数据集。

---

6. README 是否需要修改及建议内容

建议修改文档，但优先不是根目录 README，而是 docs/Tutorial.md 的 SimAI-Physical Usage 部分。

建议新增内容：

1. hostlist 语义说明

   • hostlist 中应填写 RDMA/RoCE 网卡 IP。
   • 不应填写管理网 IP，除非管理网就是 RDMA/RoCE 发包网卡。
   • hostlist 第 N 行需要对应 MPI world rank N。

2. MPI hostfile 与 SimAI hostlist 的区别

   • MPI --host / hostfile 用于启动进程。
   • SimAI hostlist 用于 SimAI_phynet 根据 rank 选择本机 RDMA 设备。

3. RDMA device 与 Linux netdev 不是同一个名字

   • mlx5_0 通常是 RDMA device name。
   • 实际带 IP 的 netdev 可能是 ensX / ethX / enpXsY / bondX。
   • 不建议通过重命名系统网卡来适配程序。

4. GID index 检查方法

   • 使用 show_gids。
   • 或查看 /sys/class/infiniband/<ibdev>/ports/1/gid_attrs/ndevs/<gid_index>。

5. 运行前检查命令

   • ibv_devices
   • ibv_devinfo
   • show_gids
   • ip addr
   • ib_write_bw / ib_read_bw

6. 容器运行注意事项

   • 需要挂载 /dev/infiniband。
   • 需要确保容器内能访问对应 RDMA 设备和 sysfs。

7. -g 行为说明

   • 当前代码中 -g <= 8 会被强制设成 8。
   • 这可能导致用户传 -g 2 但后续 ring 节点数按 8 GPU + NVSwitch 处理。

---

English MD Version

1. Requirement / Background Summary

This issue happens when running SimAI_phynet for physical network / RoCEv2 packet injection. The program fails during RDMA verbs initialization:

FlowPhyRdma::ibv_init(): Assertion 'g_ibv_ctx != NULL' failed.

From the issue description and the later comment, there are two main symptoms:

1. RDMA device names such as mlx5_0 are used directly as Linux network interface names, so the program fails to get the IP address.
2. The user temporarily renamed the corresponding Linux netdev to mlx5_0 via ifconfig, after which IP lookup worked.
3. The later comment also reports another symptom: with the same parameters, the first host can run, while the second host seems to use the first host's address for matching, causing RDMA initialization to fail.

Based on the current code, this is most likely not caused by the AICB workload itself, and it is not part of the ns-3 backend path. The direct problem is in SimAI_phynet physical RDMA initialization and device selection.

---

2. Related Files / Modules

Main repository: aliyun/SimAI

1. RDMA device selection and initialization

File:

astra-sim-alibabacloud/astra-sim/system/SimAiFlowModelRdma.cc

Important locations:

• zeta_util_ifname_to_inet_addr(): queries IPv4 address from a Linux netdev name
  

  SimAI/astra-sim-alibabacloud/astra-sim/system/SimAiFlowModelRdma.cc

  Lines 634 to 661 in f5efb5a

  	static int
  	zeta_util_ifname_to_inet_addr(const char *ifname, in_addr_t *addr) {
  	int ret = 0;
  	int fd;
  	fd = socket(AF_INET, SOCK_DGRAM, 0);
  	if (fd == -1) {
  	return -errno;
  	}
  	if (addr != NULL) {
  	struct ifreq ifr = {};
  	strncpy(ifr.ifr_name, ifname, IFNAMSIZ - 1);
  	ifr.ifr_addr.sa_family = AF_INET;
  	ret = ioctl(fd, SIOCGIFADDR, &ifr);
  	if (ret != 0) {
  	ret = -errno;
  	goto out;
  	}
  	*addr = ((struct sockaddr_in*)&ifr.ifr_addr)->sin_addr.s_addr;
  	}
  	out:
  	close(fd);
  	return ret;
  	}

• ibdev2netdev(): current mapping from RDMA device name to Linux netdev name
  

  SimAI/astra-sim-alibabacloud/astra-sim/system/SimAiFlowModelRdma.cc

  Lines 663 to 671 in f5efb5a

  	static void
  	ibdev2netdev(const char indev[64], char netdev[64]) {
  	if (strncmp(indev, "mlx5_bond_", 10) == 0) {
  	strcpy(netdev, "bond");
  	strcat(netdev, indev + 10);
  	} else {
  	strcpy(netdev, indev);
  	}
  	}

• FlowPhyRdma::ibv_init(): iterates RDMA devices, matches hostlist IP, and calls ibv_open_device()
  

  SimAI/astra-sim-alibabacloud/astra-sim/system/SimAiFlowModelRdma.cc

  Lines 673 to 708 in f5efb5a

  	int
  	FlowPhyRdma::ibv_init(void) {
  	int nb_dev = 0;
  	struct ibv_device **dev_lst = NULL;
  	struct ibv_device* ibv_dev;
  	in_addr_t src_addr = inet_addr(rank2addr[local_rank].c_str());
  	MockNcclLog* NcclLog = MockNcclLog::getInstance();
  	dev_lst = ibv_get_device_list(&nb_dev);
  	assert(nb_dev > 0);
  	assert(dev_lst != NULL);
  	for (int i = 0; i < nb_dev; ++i) {
  	in_addr_t ia = 0;
  	char netdev[64];
  	ibdev2netdev(dev_lst[i]->name,netdev);
  	int ret = zeta_util_ifname_to_inet_addr(netdev, &ia);
  	NcclLog->writeLog(
  	NcclLogLevel::DEBUG, "netdev %s ibdev %s",netdev,dev_lst[i]->name);
  	if (ret == 0 && ia == src_addr) {
  	ibv_dev = dev_lst[i];
  	break;
  	}
  	}
  	if (ibv_dev == NULL) {
  	NcclLog->writeLog(NcclLogLevel::ERROR,"ibv Device not found");
  	return -1;
  	} else{
  	NcclLog->writeLog(
  	NcclLogLevel::DEBUG, "ibv device %s init success ", ibv_dev->dev_name);
  	}
  	g_ibv_ctx = ibv_open_device(ibv_dev);
  	assert(g_ibv_ctx != NULL);
  	return 0;

2. hostlist loading and rank-to-IP mapping

File:

astra-sim-alibabacloud/astra-sim/system/BootStrapnet.cc

Important behavior:

• MPI_Comm_rank() gets the current process rank.
• argv[1] is read as the SimAI hostlist file.
• Each line is stored as rank2addr[rank].

Link:

SimAI/astra-sim-alibabacloud/astra-sim/system/BootStrapnet.cc

Lines 30 to 58 in f5efb5a

	int world_size,local_rank;
	std::map<int,std::string> rank2addr;
	static void
	initBootStrapNetRank(int argc,char*argv[]){
	std::string host_file = argv[1];
	std::string line;
	std::ifstream file(host_file);
	int rank = 0;
	if(file.is_open()){
	while (std::getline(file,line)){
	/* code */
	rank2addr[rank] = line;
	rank++;
	}
	file.close();
	} else{
	std::cerr << "Failed to open the file" << std::endl;
	}
	}
	void BootStrapNet(int argc, char *argv[]){
	MPI_Init(&argc, &argv);
	MPI_Comm_size(MPI_COMM_WORLD, &world_size);
	MPI_Comm_rank(MPI_COMM_WORLD, &local_rank);
	MPI_Barrier(MPI_COMM_WORLD);
	initBootStrapNetRank(argc,argv);
	}

Note: the variable is named local_rank, but it is assigned from MPI_Comm_rank(MPI_COMM_WORLD, &local_rank). So it is actually the MPI world rank, not the per-node local rank.

3. SimAI_phynet entry point

File:

astra-sim-alibabacloud/astra-sim/network_frontend/phynet/SimAiMain.cc

Important locations:

• Parameter parsing, especially -g and -i
• flow_rdma.ibv_init() is called without checking its return value

Link:

SimAI/astra-sim-alibabacloud/astra-sim/network_frontend/phynet/SimAiMain.cc

Lines 68 to 125 in f5efb5a

	static int user_param_prase(int argc,char * argv[],struct user_param* user_param){
	int opt;
	static struct option long_options[] = {
	{"help", no_argument, 0, 'h'},
	{"workloads", required_argument, 0, 'w'},
	{"gpus", required_argument, 0, 'g'},
	{"comm_scale", required_argument, 0, 's'},
	{"gid_index", required_argument, 0, 'i'},
	{0, 0, 0, 0}};
	while ((opt = getopt(argc,argv,"ht:w:g:s:i:"))!=-1){
	switch (opt)
	{
	case 'h':
	/* code */
	std::cout<<"-w workloads default microAllReduce.txt "<<std::endl;
	std::cout<<"-g number of gpus,default 1"<<std::endl;
	std::cout<<"-s comm_scale default 1"<<std::endl;
	std::cout<<"-i rdma gid_indxe default 0" <<std::endl;
	break;
	case 't':
	user_param->thread = stoi(optarg);
	break;
	case 'w':
	user_param->workload = optarg;
	break;
	case 'g':
	user_param->gpus = stoi(optarg);
	if(user_param->gpus <= 8){
	user_param->gpus =8;
	}
	break;
	case 's':
	user_param->comm_scale = stof(optarg);
	break;
	case 'i':
	user_param->gid_index = stoi(optarg);
	break;
	default:
	break;
	}
	}
	return 0 ;
	}
	int main(int argc,char *argv[]){
	BootStrapNet(argc,argv);
	pid_t pid = getpid();
	MockNcclLog::set_log_name("SimAi_"+to_string(local_rank)+".log");
	MockNcclLog* NcclLog = MockNcclLog::getInstance();
	NcclLog->writeLog(NcclLogLevel::DEBUG," Local rank %d PID %d ",local_rank,pid);
	struct user_param user_param;
	if(user_param_prase(argc,argv,&user_param)){
	return -1;
	}
	#ifdef PHY_RDMA
	flow_rdma = FlowPhyRdma(user_param.gid_index);
	flow_rdma.ibv_init();
	#endif

4. Documentation

File:

docs/Tutorial.md

Important section:

• SimAI-Physical compile, hostlist, run command, and parameters

Link:

SimAI/docs/Tutorial.md

Lines 330 to 404 in f5efb5a

	# SimAI-Physical Usage
	The current simulator is compatible with the ns3 discrete event simulator as the network backend, as well as the physical network backend for physical packet injection.
	## Compile
	SimAI-Phy currently uses the roceV2 protocol for traffic generation. The compilation process requires dependencies on libverbs related to the RDMA physical device, as well as the MPI program. Before compilation, please verify that your environment can successfully run the basic RDMA perfetest traffic generation tool and that it supports the related MPI program.
	```bash
	# Clone the repository
	$ git clone https://github.com/aliyun/SimAI.git
	$ cd ./SimAI/
	# Clone submodules
	$ git submodule update --init --recursive
	# Make sure to use the newest commit
	$ git submodule update --remote
	# Compile SimAI-Analytical
	$ ./scripts/build.sh -c analytical
	# Compile SimAI-Simulation (ns3)
	$ ./scripts/build.sh -c ns3
	# Compile SimAI-phynet (phynet)
	$ sudo yum install openmpi openmpi-devel
	$ export MPI_INCLUDE_PATH=/usr/include/openmpi-x86_64/
	$ export MPI_BIN_PATH=/usr/lib64/openmpi/bin/mpic++
	$ ./scripts/build.sh -c phy
	```
	## Workload Generate
	The workload required for SimAI-Phy physical traffic generation is the same as that for Sim-Simulation, and it is generated through AICB.
	### Example workload
	```bash
	HYBRID_TRANSFORMER_FWD_IN_BCKWD model_parallel_NPU_group: 2 ep: 1 pp: 1 vpp: 8 ga: 1 all_gpus: 2 checkpoints: 0 checkpoint_initiates: 0
	10
	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
	mlp_norm -1 1055000 ALLGATHER 1073741824 1055000 NONE 0 1055000 NONE 0 100
	```
	## Prepare the hostlist
	Here the main task is to prepare the iplist required to start the MPI program, which is different from nccl-test. The number of IPs here should match the actual number of network cards involved in the physical traffic generation, rather than the number of nodes participating in the physical traffic generation.
	```bash
	33.255.199.130
	33.255.199.129
	```
	## RUN
	### MPI Run
	```bash
	/usr/lib64/openmpi/bin/mpirun -np 2 -host 33.255.199.130,33.255.199.129 --allow-run-as-root -x AS_LOG_LEVEL=0 ./bin/SimAI_phynet ./hostlist -g 2 -w ./example/microAllReduce.txt
	```
	The following output indicates that the program has finished running.
	<img src="./images/Sim-phynet_finished.png" alt="Sim-phynet_finished" width="40%">
	## Setting of MPI program parameters
	| Parameter | Description | Default Value |
	|------------------|-------------------------------------------------------|---------------|
	| -np | The number of processes. | NULL |
	| -host | IP list | NULL |
	| --allow-run-as-root | Allow MPI programs to run with root privileges. | FALSE |
	## Setting of SimAI-phynet parameters
	| Parameter | Description | Default Value |
	|------------------|------------------------------|----------------------------------------------------------|
	| hostlist | Host IP list | NULL |
	| -w --workload | Path to workload | ./microAllReduce.txt |
	| -i --gid_index | Network topology path | 0 |
	| -g --gpus | Number of GPUs | 8 (should be consistent with the number of IPs in the host IP list) |

Relationship with referenced repositories

• aliyun/SimAI: the direct issue is here.
• aliyun/aicb: mainly generates workload files. It is not the direct cause of this RDMA device opening failure.
• aliyun/ns-3-alibabacloud: the ns-3 simulation backend. The physical RDMA file SimAiFlowModelRdma.cc is not on the same backend path.

---

3. Root Cause Analysis / Implementation Direction

3.1 Treating mlx5_0 as a Linux netdev name is a real bug

The current ibdev2netdev() behavior is:

• If the RDMA device name starts with mlx5_bond_, map it to bondX.
• Otherwise, copy the RDMA device name as the Linux netdev name.

That means:

• mlx5_bond_0 becomes bond0
• mlx5_0 remains mlx5_0

However, in most Linux / Mellanox / NVIDIA RDMA environments:

• mlx5_0 is an RDMA verbs device name.
• The real Linux netdev that has an IP address is usually ethX, ensX, enpXsY, bondX, etc.

So the user's observation is correct: resolving mlx5_0 to mlx5_0 causes IP lookup failure.

Renaming the actual Linux netdev to mlx5_0 via ifconfig only works because it makes the system conform to the current code's incorrect assumption. It can be used as a temporary verification workaround, but it is not a good long-term solution.

---

3.2 ibv_dev is not initialized

Inside FlowPhyRdma::ibv_init(), the code declares:

struct ibv_device* ibv_dev;

It is not initialized to NULL.

If no device matches the expected IP, the later if (ibv_dev == NULL) check is unreliable because ibv_dev may contain an undefined value.

This can cause two problems:

1. The code may fail to enter the ibv Device not found branch even when no device was found.
2. The code may pass an undefined pointer to ibv_open_device(), then fail at g_ibv_ctx != NULL.

So even before changing the mapping logic, ibv_dev should be initialized to NULL.

---

3.3 assert hides the real failure reason

The current flow is:

• Select an ibv_dev
• Call ibv_open_device(ibv_dev)
• Assert if it returns NULL

This hides the actual reason from the user.

ibv_open_device() may fail because of:

• wrong RDMA device selection
• missing or inaccessible /dev/infiniband/uverbsX
• container missing /dev/infiniband mount
• driver / rdma-core / libibverbs issue
• mismatched GID index
• hostlist contains management IP instead of RDMA/RoCE NIC IP

The code should log ibv_dev->name, ibv_dev->dev_name, errno, and strerror(errno), then return an error code.

---

3.4 The “second host uses first host address” symptom likely points to rank/hostlist mismatch

The comment logs suggest a pattern like:

• One process expects 192.168.22.1, but the local netdev IP is 192.168.22.2.
• Another process expects 192.168.22.2, but the local netdev IP is 192.168.22.1.

This strongly suggests:

MPI rank order does not match the SimAI hostlist line order.

The current code assumes:

• MPI rank 0 corresponds to hostlist line 0.
• MPI rank 1 corresponds to hostlist line 1.

But in practice, mpirun --host 192.168.22.1,192.168.22.2 -np 2 may be affected by OpenMPI mapping/ranking policy, host slots, SSH launch order, and runtime configuration.

Also verify:

1. /opt/SimAI/hostlist is identical on both machines.
2. Each node reads the same hostlist file under --wdir /opt/SimAI.
3. Containers or mounts are not causing different hostlist contents.
4. %x printing of inet_addr() may look byte-order dependent, but the observed address swap still points to a rank/IP mapping issue.

Suggested diagnostic logs:

• MPI world rank
• hostname / MPI_Get_processor_name()
• rank2addr[local_rank]
• RDMA device name
• mapped netdev name
• netdev IP

This will show exactly which rank runs on which host and which IP it expects.

---

3.5 -g 2 is forced to 8

In SimAiMain.cc, if the user passes -g 2, the code changes it to 8 because values <= 8 are forced to 8.

This is probably not the direct cause of ibv_open_device() failure, but it affects later topology/ring construction. For example, it may explain logs showing a larger ring than expected.

This should be clarified separately, but it should not be mixed into the minimal fix for this issue.

---

4. Minimal Viable Fix Plan

Plan A: Fix crash and improve diagnostics

Minimum changes:

1. Initialize ibv_dev to NULL.
2. Add normal error handling after ibv_get_device_list():
   • dev_lst == NULL
   • nb_dev <= 0
3. If no matching device is found, log:
   • MPI rank
   • expected rank2addr[local_rank]
   • all scanned RDMA devices / netdevs / IP lookup results
4. If ibv_open_device() returns NULL, do not directly assert:
   • log device name
   • log errno / strerror(errno)
   • return -1
5. In SimAiMain.cc, check the return value of flow_rdma.ibv_init() and exit early on failure.

This does not change selection behavior yet, but it prevents an unhelpful abort.

---

Plan B: Fix RDMA device to Linux netdev mapping

Do not use “mlx5_0 equals netdev name” as the primary mapping logic.

Recommended minimal mapping order:

1. Prefer the gid-aware sysfs path:

   /sys/class/infiniband/<ibdev>/ports/<IB_PORT>/gid_attrs/ndevs/<gid_index>

   Example:

   /sys/class/infiniband/mlx5_0/ports/1/gid_attrs/ndevs/3

   This is better aligned with the selected -i gid_index and works better for RoCE multi-GID, bond, VLAN, or multi-NIC environments.

2. If that fails, fallback to:

   /sys/class/infiniband/<ibdev>/device/net/

3. If sysfs mapping fails, keep the old fallback:

   • mlx5_bond_X -> bondX
   • otherwise keep the original name only as a last fallback

No new third-party dependency is needed.

---

Plan C: Clarify MPI rank and hostlist mapping

For the address-swap symptom in the comment, start with diagnostics and documentation instead of large refactoring.

Suggested minimal work:

1. Log MPI rank, hostname, and rank2addr[rank].
2. Ask users to run OpenMPI with --display-map to confirm mapping.
3. If necessary, use mapping options such as --map-by ppr:1:node --rank-by node.
4. Document that SimAI hostlist line N must correspond to MPI world rank N.
5. Longer term, consider matching the local interface IP against hostlist instead of relying only on rank order. But that is a larger behavior change and should not be part of the minimal fix.

---

Changes not recommended in this issue

To keep the patch minimal, do not include:

1. Full phynet argument parser refactoring.
2. Changing the -g <= 8 behavior.
3. AICB workload changes.
4. ns-3 backend changes.
5. New third-party dependencies.
6. Complex multi-NIC selection policies.

---

5. Test Prerequisites

Build test

Build testing does not require an AICB dataset or real RDMA traffic.

Required:

1. SimAI phynet build environment.
2. OpenMPI / mpic++.
3. libibverbs / rdma-core headers and libraries.
4. Existing phynet build path.

Validate:

• SimAI_phynet builds successfully.
• New logging and error handling compile.
• ns-3 and analytical paths are not affected.

Runtime validation

Runtime testing requires a real or accessible RDMA/RoCE environment.

Required:

1. RDMA/RoCE device on each node.
2. ibv_devices can list devices.
3. ibv_devinfo can open the device.
4. /dev/infiniband/uverbs* exists and is accessible.
5. If running inside a container, /dev/infiniband must be mounted.
6. hostlist contains RDMA/RoCE NIC IPs, not management IPs.
7. -i gid_index points to a valid GID.
8. MPI rank order matches hostlist line order.

Suggested checks on each node:

• ibv_devices
• ibv_devinfo
• show_gids
• ip addr
• ls /sys/class/infiniband/<ibdev>/ports/1/gid_attrs/ndevs/
• cat /sys/class/infiniband/<ibdev>/ports/1/gid_attrs/ndevs/<gid_index>

For real RDMA communication validation, ib_write_bw / ib_read_bw are also useful.

Only a minimal workload is needed to trigger phynet initialization and simple communication. No extra business dataset is required.

---

6. README / Documentation Update Recommendation

Documentation should be updated, preferably in docs/Tutorial.md under SimAI-Physical Usage.

Suggested additions:

1. hostlist semantics

   • hostlist should contain RDMA/RoCE NIC IPs.
   • Do not use management IPs unless the management NIC is also the RDMA/RoCE NIC.
   • hostlist line N should correspond to MPI world rank N.

2. Difference between MPI hostfile and SimAI hostlist

   • MPI --host / hostfile controls process launch.
   • SimAI hostlist controls RDMA device selection by rank.

3. RDMA device name is not Linux netdev name

   • mlx5_0 is usually an RDMA device name.
   • The IP is usually on ensX, ethX, enpXsY, or bondX.
   • Renaming the Linux netdev to match mlx5_0 is only a workaround, not recommended.

4. GID index validation

   • Use show_gids.
   • Or check /sys/class/infiniband/<ibdev>/ports/1/gid_attrs/ndevs/<gid_index>.

5. Pre-run checks

   • ibv_devices
   • ibv_devinfo
   • show_gids
   • ip addr
   • ib_write_bw / ib_read_bw

6. Container notes

   • Mount /dev/infiniband.
   • Ensure the container can access RDMA devices and sysfs.

7. Clarify -g behavior

   • Current code forces -g <= 8 to 8.
   • Passing -g 2 may still create an 8-GPU logical setup plus NVSwitch behavior later.

[ZeyuJ9]

@zxfszane 请问你的测试命令mpirun -np 2 --wdir /opt/SimAI --host 192.168.22.1,192.168.22.2 --allow-run-as-root --mca plm_rsh_args '-p 2222' -x AS_LOG_LEVEL=3 -x UCX_IB_TRAFFIC_CLASS=96 /opt/SimAI/bin/SimAI_phynet /opt/SimAI/hostlist -g 2 -i 3 -w /opt/SimAI/example/work.txt能正常运行结束吗？后面不会有MPI_ERR_RANK: invalid rank的问题吗