pfc触发段错误

[xyh522]

我将配置文件中的PACKET_PAYLOAD_SIZE 9000 改为1000，将buffer_size从32改为9，其余不变相同的一个workload，执行后者报错如图，AI告诉我是pfc resume的时候输入的队列id不对导致的，存在这个问题吗。

PFC 触发链路是：

SwitchNode::SendToDev()
-> CheckAndSendPfc(inDev, qIndex)
-> device->SendPfc(qIndex, 0) // pause

恢复链路是：

SwitchNode::SwitchNotifyDequeue()
-> CheckAndSendResume(inDev, qIndex)
-> device->SendPfc(qIndex, 1) // resume

GPU/NIC 收到 resume 后：

QbbNetDevice::Receive()
-> Resume(qIndex)

问题就在 Resume()：

Ptr lastQp = m_rdmaEQ->GetQp(qIndex);

这里的 qIndex 是 PFC 队列号/优先级号，比如 3，不是 RDMA QP 数组下标。
你这次 gdb 里已经看到：

qIndex = 3
node id = 10
node type = 0
RDMA QP count = 2

也就是只有 2 个 QP，却访问 GetQp(3)，直接越界段错误。

为什么 BUFFER_SIZE=9 更容易触发？

PFC 阈值这样算：

(buffer_size - total_hdrm - total_rsrv - shared_used_bytes) >> pfc_a_shift[port]

BUFFER_SIZE 从 32MB 改成 9MB，buffer_size 变小，PFC threshold 明显变低，更容易 pause，也更频繁 resume。于是更容易走到这个错误的 Resume(qIndex)。

[tianhao909]

【人工回复】【Manual reply】

“AI告诉我是pfc resume的时候输入的队列id不对导致的，存在这个问题吗”
存在

“为什么 BUFFER_SIZE=9 更容易触发？”
有可能是因为 BUFFER_SIZE 从 32MB 降到 9MB 后，可用共享 buffer 变小，PFC threshold 变低，因此更容易触发 pause/resume，

ai 应该说的对

[tianhao909]

我将配置文件中的PACKET_PAYLOAD_SIZE 9000 改为1000，将buffer_size从32改为9，其余不变相同的一个workload，执行后者报错如图，AI告诉我是pfc resume的时候输入的队列id不对导致的，存在这个问题吗。

PFC 触发链路是：

SwitchNode::SendToDev() -> CheckAndSendPfc(inDev, qIndex) -> device->SendPfc(qIndex, 0) // pause

恢复链路是：

SwitchNode::SwitchNotifyDequeue() -> CheckAndSendResume(inDev, qIndex) -> device->SendPfc(qIndex, 1) // resume

GPU/NIC 收到 resume 后：

QbbNetDevice::Receive() -> Resume(qIndex)

问题就在 Resume()：

Ptr lastQp = m_rdmaEQ->GetQp(qIndex);

这里的 qIndex 是 PFC 队列号/优先级号，比如 3，不是 RDMA QP 数组下标。 你这次 gdb 里已经看到：

qIndex = 3 node id = 10 node type = 0 RDMA QP count = 2

也就是只有 2 个 QP，却访问 GetQp(3)，直接越界段错误。

为什么 BUFFER_SIZE=9 更容易触发？

PFC 阈值这样算：

(buffer_size - total_hdrm - total_rsrv - shared_used_bytes) >> pfc_a_shift[port]

BUFFER_SIZE 从 32MB 改成 9MB，buffer_size 变小，PFC threshold 明显变低，更容易 pause，也更频繁 resume。于是更容易走到这个错误的 Resume(qIndex)。

[Reply from Agent A] [For reference only]

CN

Issue #286 回复 — PFC 触发段错误根因分析与最小修复方案（中文版）

仓库：aliyun/SimAI（仿真后端 aliyun/ns-3-alibabacloud）
触发现象：将 PACKET_PAYLOAD_SIZE 从 9000 改为 1000、BUFFER_SIZE 从 32 改为 9（MB）后，相同 workload 出现段错误（SIGSEGV）。
gdb 现场：qIndex = 3，RDMA QP count = 2，访问 m_rdmaEQ->GetQp(qIndex) 时崩溃。

---

一、是否真的存在该 bug

结论：是，确实存在。

定位：ns-3-alibabacloud/simulation/src/point-to-point/model/qbb-net-device.cc 的 QbbNetDevice::Resume(unsigned qIndex)，把同一个 qIndex 先当 PFC 优先级号、又当 RDMA QP 数组下标，语义不一致：

void QbbNetDevice::Resume(unsigned qIndex)
{
    NS_LOG_FUNCTION(this << qIndex);
    NS_ASSERT_MSG(m_paused[qIndex], "Must be PAUSEd");
    m_paused[qIndex] = false;                                  // (1) 当"优先级号"用,正确
    NS_LOG_INFO(...);
    Ptr<RdmaQueuePair> lastQp = m_rdmaEQ->GetQp(qIndex);       // (2) 当"QP 数组下标"用,错误
    if(lastQp->nvls_enable == 1 && m_node->GetNodeType() == 2)
        SwitchAsHostSend();
    else
        DequeueAndTransmit();
}

继续往下追：

// qbb-net-device.cc
Ptr<RdmaQueuePair> RdmaEgressQueue::GetQp(uint32_t i) {
    return m_qpGrp->Get(i);
}

// rdma-queue-pair.cc
Ptr<RdmaQueuePair> RdmaQueuePairGroup::Get(uint32_t idx) {
    return m_qps[idx];   // std::vector::operator[],无越界检查
}

std::vector::operator[] 在 idx >= size() 时是 未定义行为（UB, Undefined Behavior）——可能返回随机字节构成的"伪 Ptr<>"，紧接着访问 lastQp->nvls_enable 即触发 SIGSEGV。这与您 gdb 中观察到的行为一一吻合。

---

二、PFC 触发链路（中英对照）

术语展开：

• PFC（Priority-based Flow Control，基于优先级的流控）：链路级反压机制。
• PG（Priority Group，优先级组）：0..7 共 8 个优先级。
• QP（Queue Pair，队列对）：RDMA 中一对发送/接收队列。
• MMU（Memory Management Unit，交换芯片缓冲管理单元）：负责 ingress/egress 计费与 pause/resume 阈值决策。
• NVLS（NVLink SHARP）：NVSwitch 上的 in-network reduction 路径。
• UB（Undefined Behavior，未定义行为）：C++ 中越界访问 vector 的语言级后果。

flowchart TB
    A[SwitchNode SendToDev<br/>qIndex = ch.udp.pg<br/>0..7 优先级号]
    A --> B[CheckAndSendPfc inDev qIndex]
    B --> C[device SendPfc qIndex 0<br/>PauseHeader.qindex = 优先级]
    C --> D[对端 GPU 或 NIC<br/>QbbNetDevice Receive]
    D --> E[ch.l3Prot 0xFE 解析 PFC]
    E --> F[m_paused 优先级 = true]

    G[SwitchNode SwitchNotifyDequeue] --> H[CheckAndSendResume inDev qIndex]
    H --> I[device SendPfc qIndex 1<br/>PauseHeader.qindex = 优先级]
    I --> J[对端 QbbNetDevice Receive]
    J --> K[Resume qIndex]

    K --> L[m_paused 优先级 = false 正确]
    K --> M[m_rdmaEQ GetQp qIndex<br/>把优先级当 QP 数组下标]
    M --> N{越界?}
    N -- "qIndex 大于等于 m_qps.size" --> X[UB 段错误]
    N -- 否 --> O[读 lastQp.nvls_enable<br/>选调度路径]

Loading

红色分支即您观察到的 crash 路径。

---

三、根因分析

1. 两个数组、两套语义：

   • bool m_paused[qCnt]（qCnt = 8）按 优先级 索引，下标 0..7 永远合法；
   • m_qpGrp->m_qps 是按 QP 创建顺序 增长的 std::vector<Ptr<RdmaQueuePair>>，元素数取决于 workload 实际建立的 QP 数，常常远小于 8。

2. SwitchNode::SendToDev → SendPfc → PauseHeader.qindex → 对端 Receive 的整条链路上，qIndex 自始至终都是 优先级号，对端用它去置 m_paused[] 完全正确；问题只出在 Resume() 内多写了一行 m_rdmaEQ->GetQp(qIndex)。

3. 这一行 GetQp 的唯一用途只是判断"应该走 SwitchAsHostSend() 还是 DequeueAndTransmit()"。它对 QP 状态既不读也不写。也就是说，根本不需要按 qIndex 去查具体某条 QP——真正的 QP 调度由 DequeueAndTransmit() → GetNextQindex() 内部全量遍历 m_qps[] 完成。

---

四、为什么 BUFFER_SIZE=9MB + PACKET_PAYLOAD_SIZE=1000 必现，而默认配置很少出现

PFC pause 阈值大致为：

$$ T_{pause} = \frac{\text{BUFFER_SIZE} - \text{total_hdrm} - \text{total_rsrv} - \text{shared_used}}{2^{\text{pfc_a_shift}}} $$

将 BUFFER_SIZE 由 32 MB 缩到 9 MB，分子直接缩水到约原值的 28%，$T_{pause}$ 大幅下降，pause/resume 频率成倍上升；同时 payload 由 9000 B 降到 1000 B，每秒报文数变多，进一步抬升触发率。

而 m_qps.size() 由 workload 通信对决定，常常仅 2~4 条；SimAI 默认数据通道走 PG = 3。三者叠加：

$$ \underbrace{\text{优先级} \geq \text{QP 数}}_{\text{下标越界条件}} \times \underbrace{\text{高频 resume}}_{\text{触发频率}} ;\Longrightarrow; \text{必踩 UB} $$

默认配置下 pause/resume 几乎到不了数据 PG，UB 即便存在也几乎不被触发。

---

五、最小修复方案（保持现有代码风格、不引入新依赖）

5.1 首选方案（推荐）

在 Resume() 中先做边界检查再访问 QP，并且把"按 qIndex 查 QP"明确收敛到 NVSwitch + NVLS 的判断意图。

目标文件： simulation/src/point-to-point/model/qbb-net-device.cc
目标函数： QbbNetDevice::Resume

void QbbNetDevice::Resume(unsigned qIndex)
{
    NS_LOG_FUNCTION(this << qIndex);
    NS_ASSERT_MSG(m_paused[qIndex], "Must be PAUSEd");
    m_paused[qIndex] = false;
    NS_LOG_INFO("Node " << m_node->GetId() << " dev " << m_ifIndex
        << " queue " << qIndex << " resumed at "
        << Simulator::Now().GetSeconds());

    bool nvlsHostPath = false;
    // 仅 NVSwitch (nodeType==2) 才需要按优先级试取一条 QP 用于判断 NVLS 路径;
    // 普通 NIC (nodeType==0) 直接走 DequeueAndTransmit 即可。
    if (m_node->GetNodeType() == 2 &&
        m_rdmaEQ != nullptr &&
        m_rdmaEQ->m_qpGrp != nullptr &&
        qIndex < m_rdmaEQ->m_qpGrp->GetN()) {
        Ptr<RdmaQueuePair> lastQp = m_rdmaEQ->GetQp(qIndex);
        if (lastQp != nullptr && lastQp->nvls_enable == 1)
            nvlsHostPath = true;
    }

    if (nvlsHostPath) SwitchAsHostSend();
    else              DequeueAndTransmit();
}

要点：

• 仅当 nodeType==2 且 qIndex < GetN() 才解引用 QP，杜绝越界。
• 普通 NIC（nodeType==0，您的场景）始终走 DequeueAndTransmit()，与 NewQp() 中 nvls_enable!=1 分支语义一致。
• 不改 PFC 协议字段含义，不动 SwitchNode / SendPfc 任何代码。
• 改动量约 8 行，符合"最小改动"。

5.2 额外安全网（可选，建议同时落地）

在 RdmaQueuePairGroup::Get(idx) 与 RdmaEgressQueue::GetQp(i) 处加：

NS_ASSERT_MSG(idx < m_qps.size(), "RdmaQueuePairGroup::Get: idx out of range");

风格与同文件 RdmaEgressQueue::RecoverQueue / GetNBytes 既有断言完全一致；release 模式不会拖慢仿真，但出现类似越界时给出明确报错而不是 UB。

---

六、可选清理（非本次最小修复，建议作为后续 refactor）

QbbNetDevice 自身已经有 device 级属性 NVLS_enable：

.AddAttribute ("NVLS_enable", "enable NVLS",
        UintegerValue (0),
        MakeUintegerAccessor (&QbbNetDevice::nvls_enable),
        MakeUintegerChecker<uint32_t>())

理论上挂在该 device 上的 QP 的 nvls_enable 与 device 上的 nvls_enable 在 NVLS 场景下应一致。若团队确认这一不变量，可一行替换：

if (nvls_enable == 1 && m_node->GetNodeType() == 2) SwitchAsHostSend();
else                                                DequeueAndTransmit();

更彻底，但需先做一致性 review，因此不放入本次最小修复。

---

七、测试与回归建议

测试项	配置	期望
复现（修复前）	您的 workload + PACKET_PAYLOAD_SIZE=1000 + BUFFER_SIZE=9	段错误必现
修复回归 1	同上	跑到结束、无 SIGSEGV
修复回归 2	默认 BUFFER_SIZE=32 / PACKET_PAYLOAD_SIZE=9000	数值结果与修复前一致
修复回归 3	NVSwitch (nodeType==2 && NVLS_enable=1)	恢复后正确进入 SwitchAsHostSend()
构造性单测（可选）	强制 m_qpGrp->GetN()==2，触发 priority=3 的 PFC pause→resume	修复前 crash、修复后正常返回

• 不需要额外测试数据，使用现有 example/microAllReduce.txt 等 workload 即可。
• 不依赖真实 IB 硬件，全程纯 ns-3 仿真路径。

---

八、是否需要更新 README

建议更新（属稳定性修复，影响 PFC 高频触发场景）。

修改位置：

• SimAI/README.md 的 "Known Issues / Recent Fixes" 节；
• ns-3-alibabacloud/README.md 顶部已有 dev/qp 分支说明，可在 "Notes / Recently fixed" 补一条。

建议补充内容：

Issue #286 fixed：在小 BUFFER_SIZE（如 9 MB）+ 小 PACKET_PAYLOAD_SIZE（如 1000 B）配置下，PFC pause/resume 高频触发，QbbNetDevice::Resume() 误把 PFC 优先级号当作 RDMA QP 数组下标导致越界段错误，已修复。

配置使用建议：常规仿真保留默认 BUFFER_SIZE=32MB / PACKET_PAYLOAD_SIZE=9000；若需调小，请使用包含本修复的版本。

设计澄清：m_paused[] 按优先级管理；QP 调度由 DequeueAndTransmit() → GetNextQindex() 内部遍历 m_qps[] 完成，不应按 PFC 的 qIndex 直接查 QP。

---

九、协议与兼容性影响

• PFC wire format 不变：PauseHeader.qindex 在源端、链路、对端的语义全程一致——优先级号。
• 不改发送端 PFC 逻辑：SwitchNode::CheckAndSendPfc/Resume、SendPfc()、SendToDev() 一概不动。
• 不影响其他 CC（Congestion Control，拥塞控制）模式：DCQCN / HPCC / TIMELY / DCTCP / HPCC-PINT 的发送端调度不受影响。
• 公共接口 / 数据布局未改动，二进制兼容。

---

十、缺失信息与显式假设

1. 假设 1： PauseHeader.qindex 在所有调用方一律承载"优先级号"。
   已通过 SwitchNode::SendToDev、QbbNetDevice::SendPfc、PauseHeader 的源码直接证实。
2. 假设 2： device 级 NVLS_enable 与该 device 上 QP 的 nvls_enable 在 NVLS 场景下一致。
   本最小修复不依赖该假设（仍走 GetQp + 边界检查）；只有"§6 可选清理"才依赖它。
3. 待确认： 是否需同步在 dev/qp 分支落地相同修复。若同名函数结构相同，可直接镜像该补丁。

---

十一、TL;DR

• 您的判断正确，是 bug，不是配置问题。
• 根因：QbbNetDevice::Resume() 把"PFC 优先级号"误用作"RDMA QP 数组下标"。
• 在 m_qps.size() < 优先级 时触发 std::vector 越界 UB → SIGSEGV。
• 小 BUFFER_SIZE + 小 payload 让 PFC 高频触发，把潜在 UB 变成必现 crash。
• 最小修复：Resume() 内加 qIndex < m_qpGrp->GetN() 边界检查 + 限定 nodeType==2 才查 QP；约 8 行，不改协议、不改接口、不引依赖。
• 建议同时为 RdmaQueuePairGroup::Get 加断言，防御未来类似越界。

EN

Issue #286 Reply — Root cause analysis & minimal fix for the PFC segfault (English version)

Repo: aliyun/SimAI (network backend: aliyun/ns-3-alibabacloud)
Symptom: With the same workload, changing PACKET_PAYLOAD_SIZE from 9000 to 1000 and BUFFER_SIZE from 32 to 9 (MB) reliably triggers a segfault (SIGSEGV).
gdb evidence: qIndex = 3, RDMA QP count = 2, crash inside m_rdmaEQ->GetQp(qIndex).

---

1. Does this bug really exist?

Yes.

Location: ns-3-alibabacloud/simulation/src/point-to-point/model/qbb-net-device.cc, function QbbNetDevice::Resume(unsigned qIndex). The same qIndex value is used first as a PFC Priority Group index (correct) and then as an RDMA QP array index (wrong):

void QbbNetDevice::Resume(unsigned qIndex)
{
    NS_LOG_FUNCTION(this << qIndex);
    NS_ASSERT_MSG(m_paused[qIndex], "Must be PAUSEd");
    m_paused[qIndex] = false;                                  // (1) priority index, OK
    NS_LOG_INFO(...);
    Ptr<RdmaQueuePair> lastQp = m_rdmaEQ->GetQp(qIndex);       // (2) used as QP slot, WRONG
    if(lastQp->nvls_enable == 1 && m_node->GetNodeType() == 2)
        SwitchAsHostSend();
    else
        DequeueAndTransmit();
}

The chain underneath:

// qbb-net-device.cc
Ptr<RdmaQueuePair> RdmaEgressQueue::GetQp(uint32_t i) {
    return m_qpGrp->Get(i);
}

// rdma-queue-pair.cc
Ptr<RdmaQueuePair> RdmaQueuePairGroup::Get(uint32_t idx) {
    return m_qps[idx];   // std::vector::operator[], no bounds check
}

std::vector::operator[] with idx >= size() is Undefined Behavior (UB). It can return a "fake Ptr<>" pointing at random bytes; the next dereference (lastQp->nvls_enable) then produces SIGSEGV — exactly what you observed under gdb.

---

2. PFC trigger / resume call chain

Glossary:

• PFC — Priority-based Flow Control, link-level back-pressure.
• PG — Priority Group, 0..7 (8 priorities).
• QP — RDMA Queue Pair (one send/recv pair).
• MMU — switch buffer/admission control unit; owns pause/resume thresholds.
• NVLS — NVLink SHARP, in-network reduction path on NVSwitch.
• UB — Undefined Behavior in C++; the language-level consequence of out-of-range vector access.

flowchart TB
    A[SwitchNode SendToDev<br/>qIndex = ch.udp.pg<br/>0..7 priority]
    A --> B[CheckAndSendPfc inDev qIndex]
    B --> C[device SendPfc qIndex 0<br/>PauseHeader.qindex = priority]
    C --> D[Peer GPU or NIC<br/>QbbNetDevice Receive]
    D --> E[parse PFC l3Prot 0xFE]
    E --> F[m_paused priority = true]

    G[SwitchNode SwitchNotifyDequeue] --> H[CheckAndSendResume inDev qIndex]
    H --> I[device SendPfc qIndex 1<br/>PauseHeader.qindex = priority]
    I --> J[Peer QbbNetDevice Receive]
    J --> K[Resume qIndex]

    K --> L[m_paused priority = false correct]
    K --> M[m_rdmaEQ GetQp qIndex<br/>priority used as QP index]
    M --> N{out of range?}
    N -- "qIndex greater or equal m_qps.size" --> X[UB - segfault]
    N -- no --> O[read lastQp.nvls_enable<br/>pick scheduling path]

Loading

The red branch is your crash.

---

3. Root cause

1. Two arrays, two semantics:

   • bool m_paused[qCnt] (qCnt=8) is indexed by priority; bounds 0..7 are always valid.
   • m_qpGrp->m_qps is a std::vector<Ptr<RdmaQueuePair>> indexed by QP creation order; its length depends on the workload and is often far smaller than 8.

2. SwitchNode::SendToDev → SendPfc → PauseHeader.qindex → peer Receive — across this chain qIndex consistently means priority, and the peer using it as m_paused[] index is correct. The defect is the single extra line m_rdmaEQ->GetQp(qIndex) inside Resume().

3. That GetQp call's only purpose is to choose between SwitchAsHostSend() and DequeueAndTransmit(). It does not read or mutate any QP state. In other words, a per-qIndex QP lookup is not actually needed — real per-QP scheduling happens inside DequeueAndTransmit() → GetNextQindex(), which iterates the entire m_qps[].

---

4. Why BUFFER_SIZE=9MB + PACKET_PAYLOAD_SIZE=1000 makes it deterministic

The PFC pause threshold is roughly:

$$ T_{pause} = \frac{\text{BUFFER_SIZE} - \text{total_hdrm} - \text{total_rsrv} - \text{shared_used}}{2^{\text{pfc_a_shift}}} $$

Shrinking BUFFER_SIZE from 32 MB to 9 MB cuts the numerator to roughly 28% of its original value, lowering $T_{pause}$ substantially, so PFC pause/resume fires far more often. Smaller payload (1000 B vs 9000 B) further increases packet density and thus event rate.

m_qps.size() is determined by workload communication pairs — often only 2 to 4. SimAI's default data PG is 3. The combination

$$ \underbrace{\text{priority} \geq \text{QP count}}_{\text{out-of-range condition}} \times \underbrace{\text{frequent resume}}_{\text{trigger rate}} ;\Longrightarrow; \text{deterministic UB hit} $$

means the latent UB becomes a hard crash. With defaults, pause/resume rarely reaches the data PG, so the UB stays latent.

---

5. Minimal fix (style-preserving, no new dependencies)

5.1 Preferred fix

Add a bounds check inside Resume(), and confine "QP lookup by qIndex" to the only place that needs it — the NVSwitch + NVLS path.

File: simulation/src/point-to-point/model/qbb-net-device.cc
Function: QbbNetDevice::Resume

void QbbNetDevice::Resume(unsigned qIndex)
{
    NS_LOG_FUNCTION(this << qIndex);
    NS_ASSERT_MSG(m_paused[qIndex], "Must be PAUSEd");
    m_paused[qIndex] = false;
    NS_LOG_INFO("Node " << m_node->GetId() << " dev " << m_ifIndex
        << " queue " << qIndex << " resumed at "
        << Simulator::Now().GetSeconds());

    bool nvlsHostPath = false;
    // Only NVSwitch (nodeType==2) needs to peek a QP to decide the NVLS path;
    // a regular NIC (nodeType==0) always uses DequeueAndTransmit.
    if (m_node->GetNodeType() == 2 &&
        m_rdmaEQ != nullptr &&
        m_rdmaEQ->m_qpGrp != nullptr &&
        qIndex < m_rdmaEQ->m_qpGrp->GetN()) {
        Ptr<RdmaQueuePair> lastQp = m_rdmaEQ->GetQp(qIndex);
        if (lastQp != nullptr && lastQp->nvls_enable == 1)
            nvlsHostPath = true;
    }

    if (nvlsHostPath) SwitchAsHostSend();
    else              DequeueAndTransmit();
}

Key points:

• The QP is only dereferenced when nodeType==2 AND qIndex < GetN() — the UB is gone.
• A regular NIC (nodeType==0, your scenario) always falls through to DequeueAndTransmit(), matching the nvls_enable!=1 branch already used in NewQp().
• No change to PFC wire format or to SwitchNode / SendPfc.
• About 8 lines — truly minimal.

5.2 Optional safety net (recommended)

Add an assertion in RdmaQueuePairGroup::Get(idx) and RdmaEgressQueue::GetQp(i):

NS_ASSERT_MSG(idx < m_qps.size(), "RdmaQueuePairGroup::Get: idx out of range");

This matches the existing assertion style in RdmaEgressQueue::RecoverQueue / GetNBytes. No release-mode cost, but turns any future regression of this kind into a clear assertion instead of UB.

---

6. Optional cleanup (follow-up refactor, NOT part of the minimal fix)

QbbNetDevice already exposes a device-level attribute NVLS_enable:

.AddAttribute ("NVLS_enable", "enable NVLS",
        UintegerValue (0),
        MakeUintegerAccessor (&QbbNetDevice::nvls_enable),
        MakeUintegerChecker<uint32_t>())

If the team confirms that QPs attached to a given device share the device's nvls_enable value in NVLS scenarios, the function can collapse to one line:

if (nvls_enable == 1 && m_node->GetNodeType() == 2) SwitchAsHostSend();
else                                                DequeueAndTransmit();

Cleaner, but it depends on that invariant being true — so it is intentionally kept out of the minimal fix.

---

7. Test & regression plan

Test	Config	Expected
Pre-fix repro	Your workload + PACKET_PAYLOAD_SIZE=1000 + BUFFER_SIZE=9	Deterministic segfault
Post-fix regression 1	Same as above	Run to completion, no SIGSEGV
Post-fix regression 2	Default BUFFER_SIZE=32 / PACKET_PAYLOAD_SIZE=9000	Numerical results unchanged
Post-fix regression 3	NVSwitch (nodeType==2 && NVLS_enable=1)	Correctly enters SwitchAsHostSend() after resume
Optional unit-style test	Force m_qpGrp->GetN()==2 and drive a priority=3 PFC pause→resume	Pre-fix crash, post-fix clean run

• No extra test data required; existing workloads such as example/microAllReduce.txt are sufficient.
• No real IB hardware required — the entire path is simulated by ns-3.

---

8. Should the README be updated?

Yes, recommended — this is a stability fix that affects PFC-heavy configurations.

Suggested locations:

• SimAI/README.md under "Known Issues / Recent Fixes";
• ns-3-alibabacloud/README.md already has a top section about the dev/qp branch — add a "Notes / Recently fixed" entry.

Suggested wording:

Issue #286 fixed: With small BUFFER_SIZE (e.g. 9 MB) and small PACKET_PAYLOAD_SIZE (e.g. 1000 B), PFC pause/resume fires very frequently and QbbNetDevice::Resume() mistakenly used the PFC priority as an RDMA QP array index, causing an out-of-bounds segfault. Now fixed.

Configuration guidance: prefer the defaults (BUFFER_SIZE=32MB / PACKET_PAYLOAD_SIZE=9000); if you must reduce them, please use a build that contains this fix.

Design clarification: m_paused[] is priority-indexed; QP scheduling is performed inside DequeueAndTransmit() → GetNextQindex() over the entire m_qps[]. PFC's qIndex should not be used as a QP lookup key.

---

9. Protocol & compatibility impact

• PFC wire format unchanged. PauseHeader.qindex semantics are consistent end to end — always priority.
• Sender-side PFC logic unchanged. SwitchNode::CheckAndSendPfc/Resume, SendPfc(), SendToDev() are not touched.
• No impact on other CC (Congestion Control) modes — DCQCN / HPCC / TIMELY / DCTCP / HPCC-PINT sender-side scheduling is untouched.
• No public API or data layout change — binary compatible.

---

10. Open assumptions & questions

1. Assumption 1: PauseHeader.qindex represents priority across all callers. Confirmed by reading SwitchNode::SendToDev, QbbNetDevice::SendPfc, and PauseHeader.
2. Assumption 2: Device-level NVLS_enable matches per-QP nvls_enable in NVLS-enabled deployments. The minimal fix does not depend on this (it still goes through GetQp with a bounds check); only the optional cleanup (§6) does.
3. TBD: Should the same fix land on the dev/qp branch as well? If the function is structurally identical there, the patch can be mirrored as is.

---

11. TL;DR

• Your suspicion is correct — this is a real bug, not a configuration issue.
• Root cause: QbbNetDevice::Resume() reuses the same qIndex first as a PFC priority and then as an RDMA QP array index.
• When m_qps.size() < priority, std::vector::operator[] is UB → SIGSEGV.
• A small BUFFER_SIZE plus small payload makes PFC pause/resume frequent enough to turn the latent UB into a deterministic crash.
• Minimal fix: in Resume(), gate the QP dereference behind nodeType==2 && qIndex < m_qpGrp->GetN(); about 8 lines, no protocol / API / dependency changes.
• Recommended companion: add NS_ASSERT_MSG bounds checks to RdmaQueuePairGroup::Get to harden against future regressions.

[tianhao909]

[Reply from Agent B] [For reference only]

CN

从代码看，这个问题是存在的，BUFFER_SIZE=9 更像是触发条件，不是根因。

PFC 报文里的 qIndex 表示的是 PFC priority / traffic class，例如 0~7 的优先级队列；但 QbbNetDevice::Resume(unsigned qIndex) 里当前会执行：

• m_paused[qIndex] = false
• m_rdmaEQ->GetQp(qIndex)

这里第二步有问题。m_rdmaEQ->GetQp(i) 的 i 是 RDMA QP group 中 m_qps 的数组下标，范围应该是 0..m_qpGrp->GetN()-1，不是 PFC 队列号。因此当 PFC qIndex=3，但当前 NIC 上只有 2 个 QP 时，就会访问 GetQp(3)，导致越界和段错误。

相关代码位置：

• QbbNetDevice::Receive() 收到 PFC resume 后调用 Resume(qIndex)
• QbbNetDevice::Resume() 中把 PFC qIndex 用作 QP index
• RdmaQueuePairGroup::Get(idx) 直接访问 m_qps[idx]，没有边界检查
• SwitchMmu::GetPfcThreshold() 中 BUFFER_SIZE 会影响 PFC threshold

BUFFER_SIZE 从 32MB 降到 9MB 后，可用共享 buffer 变小，PFC threshold 变低，因此更容易触发 pause/resume；PACKET_PAYLOAD_SIZE 从 9000 改到 1000 后，同样 workload 会产生更多 packet，也会增加队列变化和 PFC 触发概率。所以这两个配置更容易暴露该问题，但不是根因。

建议的最小修复是：在 QbbNetDevice::Resume(unsigned qIndex) 中不要再使用 PFC qIndex 调用 m_rdmaEQ->GetQp(qIndex)。Resume() 只需要解除对应 priority 的 pause 状态，然后根据当前 device/node 是否是 NVLS switch-as-host 场景决定调用 SwitchAsHostSend() 或 DequeueAndTransmit()。也就是说，应基于设备级状态判断发送路径，而不是通过 PFC 队列号索引某个 QP。

同时可以考虑给 RdmaEgressQueue::GetQp() 增加边界断言，作为防御性检查；但最小必要改动是在 Resume() 中修正 qIndex 语义混用。

测试上建议使用 issue 中的 workload、PACKET_PAYLOAD_SIZE=1000、BUFFER_SIZE=9 复现；修复后应不再段错误，并确认默认 PACKET_PAYLOAD_SIZE=9000、BUFFER_SIZE=32 的场景没有回归。若涉及 NVLS，还需要额外验证 node type 为 2 且 NVLS 开启时 resume 后仍走 SwitchAsHostSend()。

==========

==========

==========

结论：issue 中指出的问题是存在的，置信度较高。QbbNetDevice::Resume(unsigned qIndex) 当前把 PFC 队列/优先级编号 qIndex 当成 RDMA QP 数组下标去调用 m_rdmaEQ->GetQp(qIndex)，当 qIndex=3 而当前 NIC 上只有 2 个 QP 时，会越界并触发段错误。BUFFER_SIZE=9 不是根因，但会更容易触发 PFC pause/resume，从而更容易暴露这个越界问题。

说明：我查了 issue 描述和评论，当前 issue 没有评论；同时检索了 aliyun/SimAI 和 aliyun/ns-3-alibabacloud 相关代码。GitHub code search 返回数量有限，结果可能不完整；如需继续人工检索可在 GitHub UI 中查看：
https://github.com/search?q=repo%3Aaliyun%2Fns-3-alibabacloud+%22m_rdmaEQ-%3EGetQp%22+OR+%22void+QbbNetDevice%3A%3AResume%22&type=code

---

中文版回复

1. 需求背景总结

用户在相同 workload 下修改配置：

• PACKET_PAYLOAD_SIZE：从 9000 改为 1000
• BUFFER_SIZE：从 32 改为 9
• 其他配置保持不变

之后仿真在 PFC 相关路径触发段错误。issue 中怀疑：

• PFC pause/resume 的 qIndex 表示的是 PFC 队列/优先级号；
• 但在 NIC 收到 PFC resume 后，QbbNetDevice::Resume(qIndex) 内部把该 qIndex 作为 RDMA QP 数组下标 使用；
• 当 PFC 队列号大于当前 RDMA QP 数量时，例如 qIndex=3 且 QP count=2，就会越界访问并崩溃。

从代码看，这个判断基本成立。

---

2. 涉及文件/模块

主要涉及 aliyun/ns-3-alibabacloud 中的网络模拟模块，aliyun/SimAI 主要是使用该 ns-3 后端及配置入口。

关键文件

1. simulation/src/point-to-point/model/qbb-net-device.cc

   关键位置：

   • RdmaEgressQueue::GetQp(uint32_t i)
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/qbb-net-device.cc#L154-L156

   • QbbNetDevice::Resume(unsigned qIndex)
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/qbb-net-device.cc#L455-L466

   • QbbNetDevice::Receive(Ptr<Packet> packet) 中处理 PFC pause/resume
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/qbb-net-device.cc#L491-L500

   • RDMA 发送队列按 QP 下标调度
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/qbb-net-device.cc#L291-L331

2. simulation/src/point-to-point/model/qbb-net-device.h

   关键位置：

   • m_paused[qCnt] 是按 PFC 优先级/队列维护的暂停状态
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/qbb-net-device.h#L166-L172

   • RdmaEgressQueue 中 m_qpGrp 是 RDMA QP 组，不是 PFC 优先级队列数组
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/qbb-net-device.h#L38-L60

3. simulation/src/point-to-point/model/switch-node.cc

   关键位置：

   • pause/resume 由交换机 MMU 判断后发送 PFC
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/switch-node.cc#L88-L100

   • SendToDev() 中计算数据包的 qIndex，RDMA UDP 使用 ch.udp.pg 作为优先级
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/switch-node.cc#L102-L126

4. simulation/src/point-to-point/model/switch-mmu.cc

   关键位置：

   • PFC pause/resume 判断
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/switch-mmu.cc#L76-L88

   • 动态 PFC threshold 计算
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/switch-mmu.cc#L93-L95

5. simulation/src/point-to-point/model/rdma-queue-pair.cc

   关键位置：

   • RdmaQueuePairGroup::GetN() 返回当前 QP 数量
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/rdma-queue-pair.cc#L245-L254

   • RdmaQueuePairGroup::Get(uint32_t idx) 直接访问 m_qps[idx]，没有边界检查
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/rdma-queue-pair.cc#L252-L254

6. aliyun/SimAI 中的配置入口

   • 默认 PACKET_PAYLOAD_SIZE 9000、BUFFER_SIZE 32
     

     SimAI/astra-sim-alibabacloud/inputs/config/SimAI.conf

     Lines 1 to 65 in f5efb5a

     	ENABLE_QCN 1
     	USE_DYNAMIC_PFC_THRESHOLD 1
     	PACKET_PAYLOAD_SIZE 9000
     	FLOW_FILE /etc/astra-sim/simulation/flow1.txt
     	TRACE_FILE /etc/astra-sim/simulation/trace1.txt
     	TRACE_OUTPUT_FILE /etc/astra-sim/simulation/llama_hpn7_mix.tr
     	FCT_OUTPUT_FILE /etc/astra-sim/simulation/llama_hpn7_fct.txt
     	PFC_OUTPUT_FILE /etc/astra-sim/simulation/llama_hpn7_pfc.txt
     	SIMULATOR_STOP_TIME 40000000000000.00
     	CC_MODE 1
     	ALPHA_RESUME_INTERVAL 1
     	RATE_DECREASE_INTERVAL 4
     	CLAMP_TARGET_RATE 0
     	RP_TIMER 900
     	EWMA_GAIN 0.00390625
     	FAST_RECOVERY_TIMES 1
     	RATE_AI 50Mb/s
     	RATE_HAI 100Mb/s
     	MIN_RATE 100Mb/s
     	DCTCP_RATE_AI 1000Mb/s
     	ERROR_RATE_PER_LINK 0.0000
     	L2_CHUNK_SIZE 4000
     	L2_ACK_INTERVAL 1
     	L2_BACK_TO_ZERO 0
     	HAS_WIN 1
     	GLOBAL_T 0
     	VAR_WIN 1
     	FAST_REACT 1
     	U_TARGET 0.95
     	MI_THRESH 0
     	INT_MULTI 1
     	MULTI_RATE 0
     	SAMPLE_FEEDBACK 0
     	PINT_LOG_BASE 1.05
     	PINT_PROB 1.0
     	RATE_BOUND 1
     	ACK_HIGH_PRIO 0
     	LINK_DOWN 0 0 0
     	ENABLE_TRACE 1
     	KMAX_MAP 6 25000000000 400 50000000000 800 100000000000 1600 200000000000 1200 400000000000 3200 1600000000000 2400
     	KMIN_MAP 6 25000000000 100 50000000000 200 100000000000 400 200000000000 300 400000000000 800 1600000000000 600
     	PMAX_MAP 6 25000000000 0.2 50000000000 0.2 100000000000 0.2 200000000000 0.8 400000000000 0.2 1600000000000 0.2
     	BUFFER_SIZE 32
     	QLEN_MON_FILE /etc/astra-sim/simulation/llama_hpn7_qlen.txt
     	BW_MON_FILE /etc/astra-sim/simulation/llama_hpn7_bw.txt
     	RATE_MON_FILE /etc/astra-sim/simulation/llama_hpn7_rate.txt
     	CNP_MON_FILE /etc/astra-sim/simulation/llama_hpn7_cnp.txt
     	MON_START 0
     	MON_END 20000
     	QP_MON_INTERVAL 100
     	QLEN_MON_INTERVAL 10000
     	BW_MON_INTERVAL 10000

   • buffer_size 默认值在 ns3 frontend common 中定义
     

     SimAI/astra-sim-alibabacloud/astra-sim/network_frontend/ns3/common.h

     Lines 26 to 111 in f5efb5a

     	#include "ns3/applications-module.h"
     	#include "ns3/core-module.h"
     	#include "ns3/error-model.h"
     	#include "ns3/global-route-manager.h"
     	#include "ns3/internet-module.h"
     	#include "ns3/ipv4-static-routing-helper.h"
     	#include "ns3/packet.h"
     	#include "ns3/point-to-point-helper.h"
     	#include "ns3/qbb-helper.h"
     	#include <ns3/rdma-client-helper.h>
     	#include <ns3/rdma-client.h>
     	#include <ns3/rdma-driver.h>
     	#include <ns3/rdma.h>
     	#include <ns3/sim-setting.h>
     	#include <ns3/switch-node.h>
     	#include <ns3/nvswitch-node.h>
     	#include <atomic>
     	using namespace ns3;
     	using namespace std;
     	NS_LOG_COMPONENT_DEFINE("GENERIC_SIMULATION");
     	uint32_t cc_mode = 1;
     	bool enable_qcn = true, use_dynamic_pfc_threshold = true;
     	uint32_t packet_payload_size = 1000, l2_chunk_size = 0, l2_ack_interval = 0;
     	double pause_time = 5, simulator_stop_time = 3.01;
     	std::string data_rate, link_delay, topology_file, flow_file, trace_file,
     	trace_output_file;
     	std::string fct_output_file = "fct.txt";
     	std::string pfc_output_file = "pfc.txt";
     	std::string send_output_file = "send.txt";
     	double alpha_resume_interval = 55, rp_timer, ewma_gain = 1 / 16;
     	double rate_decrease_interval = 4;
     	uint32_t fast_recovery_times = 5;
     	std::string rate_ai, rate_hai, min_rate = "100Mb/s";
     	std::string dctcp_rate_ai = "1000Mb/s";
     	bool clamp_target_rate = false, l2_back_to_zero = false;
     	double error_rate_per_link = 0.0;
     	uint32_t has_win = 1;
     	uint32_t global_t = 1;
     	uint32_t mi_thresh = 5;
     	bool var_win = false, fast_react = true;
     	bool multi_rate = true;
     	bool sample_feedback = false;
     	double pint_log_base = 1.05;
     	double pint_prob = 1.0;
     	double u_target = 0.95;
     	uint32_t int_multi = 1;
     	bool rate_bound = true;
     	int nic_total_pause_time =
     	0;
     	uint32_t ack_high_prio = 0;
     	uint64_t link_down_time = 0;
     	uint32_t link_down_A = 0, link_down_B = 0;
     	uint32_t enable_trace = 1;
     	uint32_t buffer_size = 16;
     	uint32_t node_num, switch_num, link_num, trace_num, nvswitch_num, gpus_per_server;
     	GPUType gpu_type;
     	std::vector<int>NVswitchs;
     	uint32_t qp_mon_interval = 100;
     	uint32_t bw_mon_interval = 10000;
     	uint32_t qlen_mon_interval = 10000;
     	uint64_t mon_start = 0, mon_end = 2100000000;
     	string qlen_mon_file;
     	string bw_mon_file;
     	string rate_mon_file;
     	string cnp_mon_file;
     	string total_flow_file = "/root/astra-sim/extern/network_backend/ns3-interface/simulation/monitor_output/";
     	FILE* total_flow_output = nullptr;
     	unordered_map<uint64_t, uint32_t> rate2kmax, rate2kmin;
     	unordered_map<uint64_t, double> rate2pmax;
     	std::ifstream topof, flowf, tracef;

---

3. 根因分析或实现思路

3.1 直接崩溃点

QbbNetDevice::Resume(unsigned qIndex) 中有如下逻辑：

• 先断言该 PFC 优先级处于 pause 状态；
• 将 m_paused[qIndex] 设为 false；
• 然后调用 m_rdmaEQ->GetQp(qIndex)；
• 根据该 QP 的 nvls_enable 和 node type 决定调用 SwitchAsHostSend() 还是 DequeueAndTransmit()。

问题在于：

• qIndex 在 PFC 报文里表示 priority group / traffic class / PFC queue index；
• m_rdmaEQ->GetQp(i) 的 i 表示 当前 RDMA QP group 中 vector 的下标；
• 这两个编号空间不是同一个东西。

因此，qIndex=3 合法地表示 PFC 优先级 3，但如果当前 NIC 上只有 2 个 QP，则 GetQp(3) 越界。

3.2 为什么这个问题不是 BUFFER_SIZE=9 本身导致的？

BUFFER_SIZE=9 更像是触发条件，不是根因。

SwitchMmu::GetPfcThreshold() 计算逻辑为：

• (buffer_size - total_hdrm - total_rsrv - shared_used_bytes) >> pfc_a_shift[port]

当 BUFFER_SIZE 从 32MB 降到 9MB：

• 可用共享 buffer 变小；
• PFC threshold 变低；
• 更容易触发 pause；
• 队列消耗下降后也更容易进入 resume 路径；
• 因此更高概率走到 QbbNetDevice::Resume(qIndex) 中的错误访问。

所以：小 buffer 只是让 PFC resume 更频繁，从而更容易暴露已有 bug。

3.3 PACKET_PAYLOAD_SIZE=1000 的影响

PACKET_PAYLOAD_SIZE 从 9000 改为 1000 会让同样 message size 拆成更多 packet：

• 单个流产生更多包；
• 队列入队/出队事件更多；
• buffer 占用变化更细碎、更频繁；
• PFC pause/resume 状态切换机会增加。

它也不是直接根因，但会增加触发 PFC 路径的概率。

3.4 当前代码中两个 qIndex 的语义混用

需要区分：

1. PFC / BEgressQueue / switch MMU 的 qIndex

   • 范围通常是 0..7
   • 表示优先级或 traffic class
   • 对应 m_paused[qCnt]

2. RDMA QP group 的 qIndex

   • 范围是 0..m_qpGrp->GetN()-1
   • 表示 m_qps vector 下标
   • 由 RdmaEgressQueue::GetNextQindex() 选择

Resume() 收到的是第 1 类 qIndex，但代码把它传给了第 2 类接口 GetQp()，因此语义错误。

---

4. 最小修改方案

推荐最小修复

在 QbbNetDevice::Resume(unsigned qIndex) 中，不要再使用 PFC qIndex 去访问 m_rdmaEQ->GetQp(qIndex)。

当前逻辑里 lastQp 的唯一用途是判断：

• lastQp->nvls_enable == 1 && m_node->GetNodeType() == 2

然后决定：

• NVLS switch-as-host 场景调用 SwitchAsHostSend()
• 其他场景调用 DequeueAndTransmit()

更小、更直接的修改方式是基于 device/node 状态判断，而不是通过某个 QP 下标判断。例如：

• 如果当前节点是 NVSwitch / switch-as-host 且设备级 nvls_enable == 1，调用 SwitchAsHostSend()
• 否则调用 DequeueAndTransmit()

建议的实现思路：

1. 保留 qIndex < qCnt 的语义检查，避免 PFC header 异常导致 m_paused 越界。
2. 在 Resume() 中删除或避免：
   • Ptr<RdmaQueuePair> lastQp = m_rdmaEQ->GetQp(qIndex);
3. 使用 m_node->GetNodeType() 和 nvls_enable 判断发送路径。
4. 不引入额外依赖，不改变数据结构，不修改调度算法。

为什么这是最小改动？

因为：

• 不需要改变 PFC 报文格式；
• 不需要改变 RDMA QP group 结构；
• 不需要改变 RdmaEgressQueue::GetNextQindex()；
• 不需要改变 switch MMU 的 pause/resume 逻辑；
• 只修复 Resume() 中语义错误的索引使用。

可选增强，但不是最小必要项

可以考虑给 RdmaEgressQueue::GetQp(uint32_t i) 增加边界断言，类似已有的 GetNBytes() / RecoverQueue()：

• 好处：后续如果再误用 QP index，会更早暴露为明确断言，而不是随机段错误；
• 风险：可能改变 release/assert 行为；
• 建议：作为防御性改动可以接受，但若坚持最小改动，可先不做。

不建议的方案

不建议简单把：

• m_rdmaEQ->GetQp(qIndex)

改成：

• m_rdmaEQ->GetQp(0)

原因：

• 这只是规避越界；
• 仍然没有解决 PFC 队列号与 QP 下标语义不一致的问题；
• 在多个 QP、多个 PG、NVLS 混合场景下可能仍然逻辑错误。

---

5. 测试前置条件

是否依赖额外数据？

需要依赖能触发 PFC 的 workload 和拓扑/配置，但不需要引入新的第三方依赖。

最小复现测试需要：

1. 原 issue 中相同 workload；
2. PACKET_PAYLOAD_SIZE=1000；
3. BUFFER_SIZE=9；
4. ENABLE_QCN=1；
5. USE_DYNAMIC_PFC_THRESHOLD=1；
6. 能产生 RDMA traffic 且触发 PFC pause/resume 的拓扑。

建议测试项

1. 复现测试

   • 使用 issue 中的配置修改复现；
   • 修改前应能观察到 resume 后越界或段错误；
   • 修改后不再段错误。

2. 默认配置回归

   • 使用默认 PACKET_PAYLOAD_SIZE=9000、BUFFER_SIZE=32；
   • 确认原有 workload 行为不变。

3. PFC 功能检查

   • 查看 PFC 输出文件，例如 PFC_OUTPUT_FILE；
   • 确认 pause/resume 仍然正常记录。

4. 多 QP / 少 QP 场景

   • 特别验证 QP 数量小于 PFC qIndex 的场景，例如 QP count=1/2，PFC qIndex=3；
   • 修改后不应访问 GetQp(3)。

5. NVLS 场景，如果该 workload 涉及

   • 如果 m_node->GetNodeType()==2 且 nvls_enable==1，确认 resume 后仍走 SwitchAsHostSend()；
   • 如果不是 NVLS，确认 resume 后走 DequeueAndTransmit()。

缺失信息和假设

目前缺少：

• issue 截图中的完整 backtrace 文本；
• 触发问题的完整 workload 文件；
• 拓扑文件；
• 修改后的完整配置文件；
• gdb 中 m_rdmaEQ->m_qpGrp->GetN()、qIndex、node type 的完整上下文；
• 是否启用 NVLS。

当前分析基于以下假设：

• PFC 报文中的 qIndex 来自 PauseHeader，语义是优先级队列；
• RdmaEgressQueue::GetQp() 的参数语义是 QP vector index；
• 段错误发生在或接近 QbbNetDevice::Resume() 中 GetQp(qIndex)；
• issue 中 qIndex=3、QP count=2 的 gdb 信息准确。

---

6. README 是否需要修改及建议内容

是否需要修改 README？

不需要作为本次 bugfix 的必要部分修改 README。

原因：

• 这是实现层面的索引语义 bug；
• 用户配置本身并非非法；
• BUFFER_SIZE=9 只是更容易触发 PFC，不应该要求用户通过 README 避免该配置；
• 最小修复应在代码层面保证 PFC resume 不会错误访问 QP vector。

可选文档建议

如果维护者希望补充说明，可以在开发文档或网络配置说明中增加一小段：

• BUFFER_SIZE 影响 PFC threshold；
• 较小 buffer 会更频繁触发 PFC pause/resume；
• PACKET_PAYLOAD_SIZE 较小时，同样 workload 会产生更多 packet，可能增加队列波动和 PFC 触发频率；
• 但这些配置不应导致段错误。

建议不是 README 必改项，可放在网络模拟配置文档或 troubleshooting 中。

---

建议给 issue 的中文回复草稿

从代码看，这个问题是存在的，`BUFFER_SIZE=9` 更像是触发条件，不是根因。

PFC 报文里的 `qIndex` 表示的是 PFC priority / traffic class，例如 0~7 的优先级队列；但 `QbbNetDevice::Resume(unsigned qIndex)` 里当前会执行：

- `m_paused[qIndex] = false`
- `m_rdmaEQ->GetQp(qIndex)`

这里第二步有问题。`m_rdmaEQ->GetQp(i)` 的 `i` 是 RDMA QP group 中 `m_qps` 的数组下标，范围应该是 `0..m_qpGrp->GetN()-1`，不是 PFC 队列号。因此当 PFC `qIndex=3`，但当前 NIC 上只有 2 个 QP 时，就会访问 `GetQp(3)`，导致越界和段错误。

相关代码位置：

- `QbbNetDevice::Receive()` 收到 PFC resume 后调用 `Resume(qIndex)`
- `QbbNetDevice::Resume()` 中把 PFC qIndex 用作 QP index
- `RdmaQueuePairGroup::Get(idx)` 直接访问 `m_qps[idx]`，没有边界检查
- `SwitchMmu::GetPfcThreshold()` 中 `BUFFER_SIZE` 会影响 PFC threshold

`BUFFER_SIZE` 从 32MB 降到 9MB 后，可用共享 buffer 变小，PFC threshold 变低，因此更容易触发 pause/resume；`PACKET_PAYLOAD_SIZE` 从 9000 改到 1000 后，同样 workload 会产生更多 packet，也会增加队列变化和 PFC 触发概率。所以这两个配置更容易暴露该问题，但不是根因。

建议的最小修复是：在 `QbbNetDevice::Resume(unsigned qIndex)` 中不要再使用 PFC `qIndex` 调用 `m_rdmaEQ->GetQp(qIndex)`。`Resume()` 只需要解除对应 priority 的 pause 状态，然后根据当前 device/node 是否是 NVLS switch-as-host 场景决定调用 `SwitchAsHostSend()` 或 `DequeueAndTransmit()`。也就是说，应基于设备级状态判断发送路径，而不是通过 PFC 队列号索引某个 QP。

同时可以考虑给 `RdmaEgressQueue::GetQp()` 增加边界断言，作为防御性检查；但最小必要改动是在 `Resume()` 中修正 qIndex 语义混用。

测试上建议使用 issue 中的 workload、`PACKET_PAYLOAD_SIZE=1000`、`BUFFER_SIZE=9` 复现；修复后应不再段错误，并确认默认 `PACKET_PAYLOAD_SIZE=9000`、`BUFFER_SIZE=32` 的场景没有回归。若涉及 NVLS，还需要额外验证 node type 为 2 且 NVLS 开启时 resume 后仍走 `SwitchAsHostSend()`。

---

English Version

1. Requirement / Background Summary

The user changed the same workload configuration as follows:

• PACKET_PAYLOAD_SIZE: from 9000 to 1000
• BUFFER_SIZE: from 32 to 9
• Other settings remained unchanged

After that, the simulation crashed with a segmentation fault around the PFC path. The issue suspects that:

• the qIndex carried by PFC pause/resume is a PFC priority / traffic class index;
• however, after the NIC receives a PFC resume frame, QbbNetDevice::Resume(qIndex) uses that qIndex as an RDMA QP vector index;
• if qIndex=3 while the current NIC only has 2 QPs, GetQp(3) accesses out of bounds and crashes.

Based on the code, this diagnosis is very likely correct.

---

2. Related Files / Modules

The relevant code is mainly in aliyun/ns-3-alibabacloud. aliyun/SimAI provides the ns-3 frontend/configuration and uses this backend.

Key Files

1. simulation/src/point-to-point/model/qbb-net-device.cc

   Key locations:

   • RdmaEgressQueue::GetQp(uint32_t i)
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/qbb-net-device.cc#L154-L156

   • QbbNetDevice::Resume(unsigned qIndex)
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/qbb-net-device.cc#L455-L466

   • QbbNetDevice::Receive(Ptr<Packet> packet) handling PFC pause/resume
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/qbb-net-device.cc#L491-L500

2. simulation/src/point-to-point/model/qbb-net-device.h

   Key locations:

   • m_paused[qCnt] is indexed by PFC priority/queue
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/qbb-net-device.h#L166-L172

   • RdmaEgressQueue::m_qpGrp is an RDMA QP group, not an array of PFC priorities
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/qbb-net-device.h#L38-L60

3. simulation/src/point-to-point/model/switch-node.cc

   Key locations:

   • switch MMU sends PFC pause/resume
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/switch-node.cc#L88-L100

   • data packet qIndex is derived from TCP/UDP priority group
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/switch-node.cc#L102-L126

4. simulation/src/point-to-point/model/switch-mmu.cc

   Key locations:

   • PFC pause/resume checks
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/switch-mmu.cc#L76-L88

   • dynamic PFC threshold calculation
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/switch-mmu.cc#L93-L95

5. simulation/src/point-to-point/model/rdma-queue-pair.cc

   Key locations:

   • RdmaQueuePairGroup::GetN() returns the current QP count
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/rdma-queue-pair.cc#L245-L254

   • RdmaQueuePairGroup::Get(uint32_t idx) directly accesses m_qps[idx] without bounds checking
     https://github.com/aliyun/ns-3-alibabacloud/blob/7e3cb5b88c99abcb582c5abc3919484a4805111b/simulation/src/point-to-point/model/rdma-queue-pair.cc#L252-L254

6. SimAI configuration

   • default PACKET_PAYLOAD_SIZE 9000 and BUFFER_SIZE 32
     

     SimAI/astra-sim-alibabacloud/inputs/config/SimAI.conf

     Lines 1 to 65 in f5efb5a

     	ENABLE_QCN 1
     	USE_DYNAMIC_PFC_THRESHOLD 1
     	PACKET_PAYLOAD_SIZE 9000
     	FLOW_FILE /etc/astra-sim/simulation/flow1.txt
     	TRACE_FILE /etc/astra-sim/simulation/trace1.txt
     	TRACE_OUTPUT_FILE /etc/astra-sim/simulation/llama_hpn7_mix.tr
     	FCT_OUTPUT_FILE /etc/astra-sim/simulation/llama_hpn7_fct.txt
     	PFC_OUTPUT_FILE /etc/astra-sim/simulation/llama_hpn7_pfc.txt
     	SIMULATOR_STOP_TIME 40000000000000.00
     	CC_MODE 1
     	ALPHA_RESUME_INTERVAL 1
     	RATE_DECREASE_INTERVAL 4
     	CLAMP_TARGET_RATE 0
     	RP_TIMER 900
     	EWMA_GAIN 0.00390625
     	FAST_RECOVERY_TIMES 1
     	RATE_AI 50Mb/s
     	RATE_HAI 100Mb/s
     	MIN_RATE 100Mb/s
     	DCTCP_RATE_AI 1000Mb/s
     	ERROR_RATE_PER_LINK 0.0000
     	L2_CHUNK_SIZE 4000
     	L2_ACK_INTERVAL 1
     	L2_BACK_TO_ZERO 0
     	HAS_WIN 1
     	GLOBAL_T 0
     	VAR_WIN 1
     	FAST_REACT 1
     	U_TARGET 0.95
     	MI_THRESH 0
     	INT_MULTI 1
     	MULTI_RATE 0
     	SAMPLE_FEEDBACK 0
     	PINT_LOG_BASE 1.05
     	PINT_PROB 1.0
     	RATE_BOUND 1
     	ACK_HIGH_PRIO 0
     	LINK_DOWN 0 0 0
     	ENABLE_TRACE 1
     	KMAX_MAP 6 25000000000 400 50000000000 800 100000000000 1600 200000000000 1200 400000000000 3200 1600000000000 2400
     	KMIN_MAP 6 25000000000 100 50000000000 200 100000000000 400 200000000000 300 400000000000 800 1600000000000 600
     	PMAX_MAP 6 25000000000 0.2 50000000000 0.2 100000000000 0.2 200000000000 0.8 400000000000 0.2 1600000000000 0.2
     	BUFFER_SIZE 32
     	QLEN_MON_FILE /etc/astra-sim/simulation/llama_hpn7_qlen.txt
     	BW_MON_FILE /etc/astra-sim/simulation/llama_hpn7_bw.txt
     	RATE_MON_FILE /etc/astra-sim/simulation/llama_hpn7_rate.txt
     	CNP_MON_FILE /etc/astra-sim/simulation/llama_hpn7_cnp.txt
     	MON_START 0
     	MON_END 20000
     	QP_MON_INTERVAL 100
     	QLEN_MON_INTERVAL 10000
     	BW_MON_INTERVAL 10000

---

3. Root Cause Analysis / Implementation Direction

3.1 Immediate Crash Point

QbbNetDevice::Resume(unsigned qIndex) currently does the following:

• asserts the corresponding queue is paused;
• sets m_paused[qIndex] = false;
• calls m_rdmaEQ->GetQp(qIndex);
• uses that QP’s nvls_enable flag and node type to choose between SwitchAsHostSend() and DequeueAndTransmit().

The problem is that:

• the qIndex from a PFC frame means priority group / traffic class / PFC queue index;
• the argument of m_rdmaEQ->GetQp(i) means the index of a QP in the current RDMA QP vector;
• these are different namespaces.

Therefore, qIndex=3 is a valid PFC priority, but if the NIC only has 2 QPs, GetQp(3) is out of bounds.

3.2 Why BUFFER_SIZE=9 Is Not the Root Cause

BUFFER_SIZE=9 is more likely a trigger, not the root cause.

SwitchMmu::GetPfcThreshold() computes the threshold roughly as:

• (buffer_size - total_hdrm - total_rsrv - shared_used_bytes) >> pfc_a_shift[port]

When BUFFER_SIZE decreases from 32MB to 9MB:

• available shared buffer becomes smaller;
• the PFC threshold becomes lower;
• pause is triggered more easily;
• resume is also reached more frequently after queue occupancy drops;
• therefore the faulty QbbNetDevice::Resume(qIndex) path is exercised more often.

So the smaller buffer exposes an existing bug.

3.3 Effect of PACKET_PAYLOAD_SIZE=1000

Changing PACKET_PAYLOAD_SIZE from 9000 to 1000 means the same message is split into more packets:

• more enqueue/dequeue events;
• more queue occupancy fluctuation;
• more opportunities to trigger PFC pause/resume.

It is also not the direct root cause, but it increases the probability of hitting the problematic path.

3.4 Semantic Mismatch of Two qIndex Concepts

There are two different meanings of qIndex here:

1. PFC / BEgressQueue / switch MMU qIndex

   • usually ranges from 0..7
   • represents priority or traffic class
   • maps to m_paused[qCnt]

2. RDMA QP group qIndex

   • ranges from 0..m_qpGrp->GetN()-1
   • represents an index in the m_qps vector
   • selected by RdmaEgressQueue::GetNextQindex()

Resume() receives the first kind of qIndex, but calls an API that expects the second kind. That is the core bug.

---

4. Minimal Viable Fix Plan

Recommended Minimal Fix

In QbbNetDevice::Resume(unsigned qIndex), do not use the PFC qIndex to call m_rdmaEQ->GetQp(qIndex).

Currently, lastQp is only used to decide whether to call:

• SwitchAsHostSend() for the NVLS switch-as-host case; or
• DequeueAndTransmit() for normal NIC transmission.

A smaller and cleaner fix is to make this decision based on device/node state instead of indexing a QP using the PFC queue id. For example:

• if the current node is switch-as-host / NVLS and device-level nvls_enable == 1, call SwitchAsHostSend();
• otherwise call DequeueAndTransmit().

Suggested implementation direction:

1. Keep or add a sanity check that qIndex < qCnt.
2. Remove or avoid:
   • Ptr<RdmaQueuePair> lastQp = m_rdmaEQ->GetQp(qIndex);
3. Use m_node->GetNodeType() and device-level nvls_enable to decide the transmit path.
4. Do not introduce new dependencies.
5. Do not change PFC packet format, RDMA QP layout, or scheduling algorithm.

Why This Is Minimal

This fix does not require:

• changing PFC headers;
• changing RDMA QP group data structures;
• changing RdmaEgressQueue::GetNextQindex();
• changing switch MMU pause/resume logic;
• changing workload or configuration format.

It only fixes the incorrect index usage inside Resume().

Optional Defensive Improvement

Optionally, add bounds checking or an assertion in RdmaEgressQueue::GetQp(uint32_t i), similar to GetNBytes() and RecoverQueue().

This is useful for catching future misuse earlier, but it is not strictly required for the minimal fix.

Not Recommended

Do not simply replace:

• m_rdmaEQ->GetQp(qIndex)

with:

• m_rdmaEQ->GetQp(0)

That would only hide the out-of-bounds access and would still mix PFC priority indices with QP vector indices incorrectly.

---

5. Test Preconditions

Does Testing Depend on Extra Data?

Testing requires a workload/topology/configuration that can trigger PFC, but it does not require any new third-party dependency.

The minimal reproduction needs:

1. the same workload from the issue;
2. PACKET_PAYLOAD_SIZE=1000;
3. BUFFER_SIZE=9;
4. ENABLE_QCN=1;
5. USE_DYNAMIC_PFC_THRESHOLD=1;
6. a topology that generates RDMA traffic and triggers PFC pause/resume.

Suggested Tests

1. Reproduction test

   • Run the issue workload with PACKET_PAYLOAD_SIZE=1000 and BUFFER_SIZE=9;
   • before the fix, it should be able to reproduce the crash;
   • after the fix, it should no longer crash.

2. Default configuration regression

   • Run with PACKET_PAYLOAD_SIZE=9000 and BUFFER_SIZE=32;
   • confirm existing behavior is unchanged.

3. PFC behavior validation

   • Check the configured PFC_OUTPUT_FILE;
   • confirm pause/resume events are still emitted.

4. Low QP count / higher PFC queue index case

   • Verify cases where QP count is smaller than the PFC qIndex, e.g. QP count = 1 or 2 and PFC qIndex = 3;
   • after the fix, Resume() should not access GetQp(3).

5. NVLS case, if applicable

   • If m_node->GetNodeType()==2 and NVLS is enabled, confirm resume still calls SwitchAsHostSend();
   • otherwise confirm it calls DequeueAndTransmit().

Missing Information and Assumptions

Missing information:

• full text backtrace from the screenshot;
• exact workload file;
• exact topology file;
• full modified config;
• complete gdb context for qIndex, QP count, and node type;
• whether NVLS is enabled in the failing run.

Assumptions:

• the PFC qIndex comes from PauseHeader and represents traffic class / priority;
• RdmaEgressQueue::GetQp() expects a QP vector index;
• the crash happens at or near QbbNetDevice::Resume() when calling GetQp(qIndex);
• the reported qIndex=3 and QP count = 2 are accurate.

---

6. README Update Recommendation

Is a README Update Required?

No, a README update is not required for the minimal bug fix.

Reason:

• this is an implementation-level index semantic bug;
• the user configuration is not inherently invalid;
• BUFFER_SIZE=9 only makes PFC more likely to trigger;
• the code should handle PFC resume safely regardless of the number of active QPs.

Optional Documentation Note

If maintainers want to improve documentation, a short note can be added to the network configuration or troubleshooting docs:

• BUFFER_SIZE affects the dynamic PFC threshold;
• smaller buffer sizes can cause more frequent PFC pause/resume;
• smaller PACKET_PAYLOAD_SIZE creates more packets for the same workload and can increase queue fluctuations;
• these settings may expose PFC-related bugs, but should not cause segmentation faults.

This is optional and not necessary for the fix itself.