多个模型同时训练问题

[zzy999-win]

请问该仿真器支持在拓扑上同时训练多个大小不同的模型吗？谢谢!

[tianhao909]

【人工回复】【Manual reply】
目前应该不支持好像

[tianhao909]

[Reply from Agent A] [For reference only]

cn

Issue #282 回复：多个模型同时训练问题

问题

请问该仿真器支持在拓扑上同时训练多个大小不同的模型吗？谢谢!

回复

感谢您的提问！

当前状态

SimAI 当前版本不支持在同一拓扑上同时训练多个不同大小的模型。 当前设计假设一次模拟运行中，所有 GPU 节点执行同一个训练任务（workload）。

架构原因

SimAI 当前的系统设计存在以下约束，导致无法原生支持多 Job 并发模拟：

1. 单一 Workload 绑定：每个 Sys 实例（对应一个 GPU 节点）只能绑定一个 Workload 对象，该对象定义了模型的所有层和通信模式。

2. 全局共享 Workload 文件：在 NS3 模拟模式（AstraSimNetwork.cc）和 Analytical 模式（AnalyticalAstra.cc）的入口函数中，所有 GPU 节点都使用同一个 workload 文件初始化。

3. 全局统一通信组：MockNcclGroup 假设所有 GPU 属于一个统一的全局通信组，TP/DP/EP/PP 的分组逻辑覆盖所有节点。

4. 命令行参数设计：当前只接受单个 -w 参数指定一个 workload 文件路径。

Workaround（当前可行方案）

虽然不支持单次运行中的多模型并发模拟，您可以通过以下方式近似分析：

1. 分别运行多次模拟：

   • 为每个模型单独准备 workload 文件（使用 AICB 生成）
   • 修改 GPU 数量参数（-g）匹配该模型使用的 GPU 子集
   • 分别运行 SimAI 获取各模型的通信时间

2. 示例：假设有 128 GPU 集群，模型 A 使用 64 GPU，模型 B 使用 64 GPU：

   # 模型 A（64 GPU）
   ./bin/SimAI_analytical -w workload_modelA.txt -g 64 -g_p_s 8 -r result_A-
   
   # 模型 B（64 GPU）
   ./bin/SimAI_analytical -w workload_modelB.txt -g 64 -g_p_s 8 -r result_B-

注意：此方法无法捕获多个作业之间的网络带宽争用效应。如果您的分析场景要求精确模拟多作业竞争网络资源的情况，目前还没有直接支持的方式。

未来方向

如果社区对此功能有较强需求，支持多 Job 并发模拟的核心改动方向大致为：

1. 入口参数扩展：支持传入多个 workload 文件 + GPU 分配映射（如 -w job1.txt:0-63 -w job2.txt:64-127）
2. Sys 实例按 Job 分配 Workload：不同 GPU ID 区间绑定不同的 workload 文件
3. 通信组隔离：每个 Job 创建独立的 MockNcclGroup，仅包含该 Job 的 GPU
4. 网络层天然共享：NS3 网络模拟层本身支持多流竞争，无需改动

欢迎在此 Issue 中进一步讨论您的具体使用场景，或者考虑提交 PR 来贡献此功能。

en

Issue #282 Reply: Training Multiple Models Simultaneously

Question

Does this simulator support training multiple models of different sizes on the same topology simultaneously? Thanks!

Reply

Thank you for the question!

Current Status

SimAI does not currently support training multiple different models simultaneously on the same topology. The current design assumes that in a single simulation run, all GPU nodes execute the same training task (workload).

Architectural Reasons

The following architectural constraints in SimAI's current design prevent native support for multi-job concurrent simulation:

1. Single Workload Binding: Each Sys instance (corresponding to one GPU node) can only bind to a single Workload object, which defines all layers and communication patterns for the model.

2. Globally Shared Workload File: In both the NS3 simulation mode entry (AstraSimNetwork.cc) and the Analytical mode entry (AnalyticalAstra.cc), all GPU nodes are initialized with the same workload file.

3. Globally Unified Communication Group: MockNcclGroup assumes all GPUs belong to a single global communication group, with TP/DP/EP/PP partitioning logic spanning all nodes.

4. Command-Line Parameter Design: Currently only a single -w parameter is accepted to specify one workload file path.

Workaround (Currently Feasible Approach)

Although multi-model concurrent simulation within a single run is not supported, you can approximate the analysis as follows:

1. Run multiple simulations separately:

   • Prepare a separate workload file for each model (generated using AICB)
   • Adjust the GPU count parameter (-g) to match the GPU subset used by each model
   • Run SimAI separately for each model to obtain communication times

2. Example: Suppose you have a 128-GPU cluster, with Model A using 64 GPUs and Model B using 64 GPUs:

   # Model A (64 GPUs)
   ./bin/SimAI_analytical -w workload_modelA.txt -g 64 -g_p_s 8 -r result_A-
   
   # Model B (64 GPUs)
   ./bin/SimAI_analytical -w workload_modelB.txt -g 64 -g_p_s 8 -r result_B-

Note: This approach cannot capture network bandwidth contention effects between multiple jobs. If your analysis scenario requires precise simulation of multi-job competition for network resources, there is currently no direct support for this.

Future Direction

If there is strong community demand for this feature, the core changes needed to support multi-job concurrent simulation would roughly be:

1. Entry Parameter Extension: Support passing multiple workload files + GPU allocation mapping (e.g., -w job1.txt:0-63 -w job2.txt:64-127)
2. Job-Based Workload Assignment for Sys Instances: Different GPU ID ranges bind to different workload files
3. Communication Group Isolation: Each job creates an independent MockNcclGroup containing only that job's GPUs
4. Natural Network Sharing: The NS3 network simulation layer inherently supports multi-flow contention and requires no modification

We welcome further discussion about your specific use case in this Issue, or consider submitting a PR to contribute this feature.

[tianhao909]

[Reply from Agent B] [For reference only]

说明：我已基于 issue、已有评论以及 aliyun/SimAI、aliyun/aicb、aliyun/ns-3-alibabacloud 做了只读分析，未修改代码。
另外，GitHub code search 工具返回结果有数量限制，下面的代码位置可能不是完整全集；可在 GitHub UI 继续搜索：
https://github.com/search?q=repo%3Aaliyun%2FSimAI+Workload+Sys+workload&type=code

1. 需求背景总结

中文

Issue #282 的问题是：

该仿真器是否支持在同一个拓扑上同时训练多个大小不同的模型？

结合代码和仓库结构，目前可以拆成两个层面的需求：

1. 已有能力

   • SimAI 可以基于 AICB 生成的 workload 文件模拟单个训练/推理任务。
   • AICB 支持生成不同模型大小、不同并行配置的 workload 文件。
   • ns-3-alibabacloud 负责底层网络拓扑和流量仿真，本身不关心“模型大小”，只处理由上层转换出来的通信流。
   • vidur-alibabacloud 支持多请求推理仿真，但这是推理请求级别的多请求，不等同于多个训练模型同时跑在同一训练拓扑上。

2. Issue 关注的能力

   • 用户希望在同一份网络拓扑中，同时放置多个训练 job。
   • 每个 job 可能对应不同模型大小、不同 workload 文件、不同 GPU 数量、不同 TP/PP/EP/DP 配置。
   • 多个 job 的通信应在同一个 ns-3 网络里并发执行，从而观察链路竞争、拥塞、总耗时等结果。

基于当前代码判断：目前 SimAI 训练仿真默认不直接支持“多个不同训练模型在同一个拓扑中同时运行”的一键配置。
当前主要入口仍是单 workload、单 job 语义；ns-3 frontend 虽然会为多个 node 创建 Sys，但所有 Sys 使用同一个 workload 文件。

English

Issue #282 asks:

Does the simulator support training multiple models of different sizes simultaneously on the same topology?

From the current repositories, this can be separated into two parts:

1. Existing capability

   • SimAI can simulate one training/inference workload generated by AICB.
   • AICB can generate workload files for different model sizes and parallel configurations.
   • ns-3-alibabacloud simulates the network flows and topology; it does not directly understand “model size”.
   • vidur-alibabacloud supports multi-request inference simulation, but that is not the same as running multiple concurrent training jobs on one training topology.

2. Capability requested by the issue

   • Run multiple training jobs on the same network topology.
   • Each job may use a different model size, workload file, GPU partition, and parallelism configuration.
   • These jobs should inject traffic into the same ns-3 simulation so that link contention and congestion can be observed.

Based on the current code, SimAI does not appear to provide out-of-the-box one-command support for concurrently training multiple different models on the same topology.
The current training simulation path is mostly single-workload/single-job oriented.

---

2. 涉及文件/模块

中文

重点相关模块如下：

1. SimAI 主入口 / 参数解析

   • astra-sim-alibabacloud/astra-sim/system/AstraParamParse.hh
     • UserParam::workload 是单个字符串。
     • -w/--workload 当前只解析一个 workload 文件。
     • -g/--gpus 虽然是 std::vector<int>，但在当前 analytical 入口中主要被当作单个 job 的维度/总 GPU 配置使用。

2. Analytical 入口

   • astra-sim-alibabacloud/astra-sim/network_frontend/analytical/AnalyticalAstra.cc
   • 关键逻辑：
     • physical_dims = {param->gpus};
     • 只创建一个 AstraSim::Sys
     • my_workload 参数为 WORKLOAD_PATH + param->workload
     • 只调用一次 systems->workload->fire()
   • 因此 analytical 模式当前明显是单 workload 入口。

3. ns-3 Simulation 入口

   • astra-sim-alibabacloud/astra-sim/network_frontend/ns3/AstraSimNetwork.cc
   • 关键逻辑：
     • 参数结构体中只有 string workload
     • -w 只接收一个 workload 文件
     • 对每个 node 创建一个 AstraSim::Sys
     • 但每个 Sys 都使用同一个 user_param.workload
   • 这意味着当前 ns-3 frontend 是“同一个 workload 在所有节点上运行”，而不是多个 job 使用不同 workload 文件。

4. Sys / Workload 生命周期

   • astra-sim-alibabacloud/astra-sim/system/Sys.hh
   • astra-sim-alibabacloud/astra-sim/system/Sys.cc
   • astra-sim-alibabacloud/astra-sim/workload/Workload.hh
   • astra-sim-alibabacloud/astra-sim/workload/Workload.cc
   • 每个 Sys 持有一个 Workload* workload。
   • Workload::initialize_workload() 读取一个 workload 文件。
   • Workload::fire() 启动该 workload 的状态机。
   • Workload::check_for_sim_end() 当前以单 workload 完成作为完成条件的一部分。

5. AICB workload 生成

   • aliyun/aicb/scripts/megatron_workload_with_aiob.sh
   • aliyun/aicb/workload_generator/SimAI_training_workload_generator.py
   • AICB 可以生成不同模型大小的 workload 文件，是实现多模型训练仿真的输入来源。

6. ns-3-alibabacloud

   • 网络层主要负责流量调度、收发、trace、flow monitor 等。
   • 如果上层能正确为不同 job 创建不同 Sys、不同 node offset、不同 workload，ns-3 层理论上可以承载这些并发流量。
   • 但当前 frontend 没有把“多个 job”配置暴露出来。

English

Relevant modules:

1. SimAI main parameter parsing

   • astra-sim-alibabacloud/astra-sim/system/AstraParamParse.hh
   • UserParam::workload is a single string.
   • -w/--workload currently parses only one workload file.
   • -g/--gpus is a std::vector<int>, but in the current analytical path it is used as the dimension / GPU count for one job.

2. Analytical frontend

   • astra-sim-alibabacloud/astra-sim/network_frontend/analytical/AnalyticalAstra.cc
   • It creates only one AstraSim::Sys.
   • The workload argument is WORKLOAD_PATH + param->workload.
   • It calls systems->workload->fire() once.
   • Therefore the analytical mode is currently single-workload oriented.

3. ns-3 simulation frontend

   • astra-sim-alibabacloud/astra-sim/network_frontend/ns3/AstraSimNetwork.cc
   • The local user_param has only one string workload.
   • -w accepts only one workload file.
   • One AstraSim::Sys is created per node, but every Sys receives the same user_param.workload.
   • So the current ns-3 frontend runs the same workload across nodes, instead of multiple different workloads for different jobs.

4. Sys / Workload lifecycle

   • astra-sim-alibabacloud/astra-sim/system/Sys.hh
   • astra-sim-alibabacloud/astra-sim/system/Sys.cc
   • astra-sim-alibabacloud/astra-sim/workload/Workload.hh
   • astra-sim-alibabacloud/astra-sim/workload/Workload.cc
   • Each Sys owns one Workload* workload.
   • Workload::initialize_workload() reads one workload file.
   • Workload::fire() starts the workload state machine.
   • Workload::check_for_sim_end() is also written around one workload instance.

5. AICB workload generation

   • aliyun/aicb/scripts/megatron_workload_with_aiob.sh
   • aliyun/aicb/workload_generator/SimAI_training_workload_generator.py
   • AICB can generate workload files for different model sizes. These files are the natural input for a multi-model simulation.

6. ns-3-alibabacloud

   • The network layer mainly handles flows, receive/send events, traces, and network behavior.
   • If the upper layer creates different Sys instances with different offsets, GPU partitions, and workloads, the ns-3 layer should be able to carry concurrent traffic.
   • However, the current frontend does not expose this as a multi-job configuration.

---

3. 根因分析或实现思路

中文

结论

当前直接回答 issue：

目前训练仿真路径不支持开箱即用地在同一拓扑上一键同时训练多个不同大小的模型。
现有代码更接近“一个仿真入口加载一个 workload 文件”的设计。
但从架构上看，可以通过较小改动支持：把多个模型分别生成为多个 workload 文件，然后在同一 ns-3 simulation 中为每个 job 创建独立的 Sys 集合，并用不同 npu_offset 映射到同一拓扑中的不同 GPU 分区。

根因

1. CLI 只支持单 workload

   • AstraParamParse.hh 和 ns-3 frontend 的本地 user_param 都只维护单个 workload 字符串。
   • 没有 --jobs、--workloads、--job_config 这类多 job 配置。

2. frontend 创建 Sys 时没有 job 维度

   • Analytical 入口只创建一个 Sys。
   • ns-3 入口虽然创建多个 Sys，但它们都使用同一个 workload。
   • 没有把 node/GPU 划分成多个 job partition。

3. Sys / Workload 是一对一关系

   • 每个 Sys 只有一个 Workload* workload。
   • 这本身不是问题；多 job 可以通过多个 Sys 集合实现。
   • 问题在于 frontend 没有根据多 job 配置创建多个 Sys 集合。

4. 完成条件偏单 job

   • Sys::call_events() 中存在 finished_workloads == 1 的单 workload 完成判断。
   • 如果多个 job 并行运行，需要避免第一个 job 完成后过早释放或退出。

实现思路

优先建议支持 ns-3 Simulation 模式的多 job，因为用户问的是“在拓扑上同时训练”，通常关注的是共享拓扑的网络竞争。Analytical 模式可后续再支持，或者明确说明 analytical 只适合快速估算，不能准确反映多 job 链路竞争。

推荐实现模型：

Job 0:
  workload = model_a.txt
  gpu_count = 64
  npu_offset = 0
  run_name = job0_model_a

Job 1:
  workload = model_b.txt
  gpu_count = 128
  npu_offset = 64
  run_name = job1_model_b

Same ns-3 topology:
  total GPUs >= 64 + 128

每个 job 的 GPU ID 区间不重叠，但所有 job 共享同一个 ns-3 网络拓扑和链路资源。

English

Conclusion

The direct answer to the issue is:

The current training simulation path does not provide out-of-the-box support for running multiple different model training jobs simultaneously on the same topology.
The current design is mostly “one simulation entry loads one workload file”.
However, the architecture can be extended with relatively small changes: generate one workload file per model, then create separate Sys groups for each job in the same ns-3 simulation, using different npu_offset values to map them to different GPU partitions on the same topology.

Root cause

1. The CLI supports only one workload

   • AstraParamParse.hh and the ns-3 frontend local user_param both keep only one workload string.
   • There is no --jobs, --workloads, or --job_config option.

2. The frontend has no job dimension

   • The analytical frontend creates only one Sys.
   • The ns-3 frontend creates multiple Sys objects, but all of them use the same workload.
   • There is no partitioning of nodes/GPUs into multiple jobs.

3. Sys and Workload are one-to-one

   • Each Sys has one Workload* workload.
   • This is acceptable; multi-job support can be implemented by creating multiple groups of Sys.
   • The missing part is frontend orchestration.

4. Completion logic is single-job oriented

   • Sys::call_events() has logic based on finished_workloads == 1.
   • For multiple jobs, the simulation should not exit or delete state after the first job finishes.

Implementation idea

Prioritize multi-job support in ns-3 Simulation mode, because the user asks about training on a shared topology, which usually implies link contention and network-level interaction. Analytical mode can be added later, or documented as an approximation.

Recommended model:

Job 0:
  workload = model_a.txt
  gpu_count = 64
  npu_offset = 0
  run_name = job0_model_a

Job 1:
  workload = model_b.txt
  gpu_count = 128
  npu_offset = 64
  run_name = job1_model_b

Same ns-3 topology:
  total GPUs >= 64 + 128

Each job uses a non-overlapping GPU ID range, while all jobs share the same ns-3 topology and link resources.

---

4. 最小修改方案

中文

建议采用最小侵入式方案：不改 Workload 文件格式，不改 AICB，不引入新依赖，只扩展 frontend 的参数和编排逻辑。

方案 A：先支持 ns-3 Simulation 多 job，推荐

1. 新增 job 配置结构

在 AstraSimNetwork.cc 中添加类似结构：

struct job_param {
  std::string workload;
  int gpus;
  int npu_offset;
  std::string run_name;
};

2. 扩展 CLI

两种轻量方式：

方式 1：逗号分隔参数，改动最小

./bin/SimAI_simulator \
  -t 16 \
  -w model_a.txt,model_b.txt \
  -j_gpus 64,128 \
  -n topo \
  -c SimAI.conf

自动计算：

job0 offset = 0
job1 offset = 64

方式 2：重复 job 参数，可读性更好但解析稍复杂

./bin/SimAI_simulator \
  -t 16 \
  --job model_a.txt:64 \
  --job model_b.txt:128 \
  -n topo \
  -c SimAI.conf

为了最小改动，建议先用方式 1。

3. 在 ns-3 frontend 中按 job 创建 Sys

当前逻辑是：

for (int j = 0; j < nodes_num; j++) {
  systems[j] = new AstraSim::Sys(..., user_param.workload, ..., npu_offset = 0, ...);
}

建议改为概念上如下：

for each job:
  for local_rank in [0, job.gpus):
    global_rank = job.npu_offset + local_rank
    networks[global_rank] = new ASTRASimNetwork(global_rank, job.npu_offset)
    systems[global_rank] = new AstraSim::Sys(
      networks[global_rank],
      nullptr,
      local_rank,
      job.npu_offset,
      1,
      {job.gpus},
      {1},
      "",
      job.workload,
      ...
      job.run_name,
      ...
      {job.gpus},
      ...
    )

注意点：

• id 建议使用 job 内局部 rank，便于 workload 内部 group 计算。
• npu_offset 用于把 job 内 rank 映射到全局拓扑 node。
• ASTRASimNetwork::sim_send() 已经对 dst += npu_offset 做映射；sim_recv() 也对 src += npu_offset 做映射，因此这个方向是可行的。
• 每个 job 的 run_name 应不同，避免输出文件互相覆盖。

4. 完成条件修改

当前 Sys::call_events() 中有类似单 job 判断：

if ((finished_workloads == 1 && event_queue.size() == 0 && pending_sends.size() == 0) ||
    initialized == false) {
  delete this;
}

多 job 下不能简单依赖 finished_workloads == 1。最小修改建议：

• 保持每个 Sys 自己的 workload 完成状态。
• frontend 或全局状态维护 expected_finished_workloads。
• 只有所有 job 的所有相关 Sys 都完成，且 event queue / pending sends 为空时，才结束 simulation。

如果只想做第一版，可以先避免在第一个 workload 完成时提前 delete this，让 Simulator::Run() 自然跑到全局无事件为止；但需要确认不会导致资源泄漏或悬空指针。

5. 输出文件隔离

当前 Workload 输出包含：

• EndToEnd.csv
• detailed_<total_nodes>.csv
• <run_name>_dimension_utilization_<npu_offset>.csv

多 job 下建议：

• path 或 run_name 带 job 名。
• 最小方案：run_name = job0_<workload_basename>、job1_<workload_basename>。
• 更稳妥：每个 job 一个子目录，例如 results/job0/、results/job1/。

6. 不建议第一版改 AICB

AICB 已经可以分别生成不同模型 workload。
最小方案中，用户需要先生成多个 workload 文件，然后传给 SimAI。

方案 B：Analytical 模式支持多 job，不优先

Analytical 模式当前网络模型是抽象 busbw，不一定能真实体现同拓扑多 job 链路竞争。可以后续支持：

./bin/SimAI_analytical \
  -w model_a.txt,model_b.txt \
  -j_gpus 64,128

但需要在文档中说明：该模式更适合快速估算，不等价于 ns-3 多 job 竞争仿真。

English

Recommended minimal approach: do not change the workload file format, do not change AICB, and do not add dependencies. Only extend the frontend orchestration logic.

Option A: Add multi-job support to ns-3 Simulation first, recommended

1. Add a job config structure

In AstraSimNetwork.cc, add a structure like:

struct job_param {
  std::string workload;
  int gpus;
  int npu_offset;
  std::string run_name;
};

2. Extend the CLI

Two lightweight options:

Option 1: comma-separated arguments, minimal parsing change

./bin/SimAI_simulator \
  -t 16 \
  -w model_a.txt,model_b.txt \
  -j_gpus 64,128 \
  -n topo \
  -c SimAI.conf

Offsets are computed automatically:

job0 offset = 0
job1 offset = 64

Option 2: repeated job arguments, cleaner but more parsing work

./bin/SimAI_simulator \
  -t 16 \
  --job model_a.txt:64 \
  --job model_b.txt:128 \
  -n topo \
  -c SimAI.conf

For minimal changes, option 1 is preferable.

3. Create Sys instances per job in the ns-3 frontend

Current logic:

for (int j = 0; j < nodes_num; j++) {
  systems[j] = new AstraSim::Sys(..., user_param.workload, ..., npu_offset = 0, ...);
}

Conceptually change it to:

for each job:
  for local_rank in [0, job.gpus):
    global_rank = job.npu_offset + local_rank
    networks[global_rank] = new ASTRASimNetwork(global_rank, job.npu_offset)
    systems[global_rank] = new AstraSim::Sys(
      networks[global_rank],
      nullptr,
      local_rank,
      job.npu_offset,
      1,
      {job.gpus},
      {1},
      "",
      job.workload,
      ...
      job.run_name,
      ...
      {job.gpus},
      ...
    )

Important details:

• Use local rank as Sys::id so workload group logic remains job-local.
• Use npu_offset to map job-local ranks to global topology node IDs.
• ASTRASimNetwork::sim_send() already does dst += npu_offset; sim_recv() also does src += npu_offset, so this direction looks compatible.
• Use unique run_name values per job to avoid output file conflicts.

4. Update completion logic

Current Sys::call_events() contains single-job logic like:

if ((finished_workloads == 1 && event_queue.size() == 0 && pending_sends.size() == 0) ||
    initialized == false) {
  delete this;
}

For multi-job support, the simulation must not finish or delete state after the first workload completes. Minimal change:

• Keep per-Sys workload completion state.
• Maintain a global expected_finished_workloads.
• End the simulation only when all jobs’ related Sys objects have completed and no events / pending sends remain.

For a first version, another approach is to avoid deleting state after the first workload finishes and let Simulator::Run() drain all events naturally. But this must be checked carefully to avoid leaks or dangling pointers.

5. Isolate output files

Current outputs include:

• EndToEnd.csv
• detailed_<total_nodes>.csv
• <run_name>_dimension_utilization_<npu_offset>.csv

For multi-job runs:

• Include the job name in run_name, or
• Write each job into a separate result subdirectory.

Minimal approach:

run_name = job0_<workload_basename>
run_name = job1_<workload_basename>

More robust approach:

results/job0/
results/job1/

6. Do not change AICB in the first version

AICB can already generate workload files for different model sizes.
The first version should require users to generate multiple workload files first, then pass them to SimAI.

Option B: Add Analytical multi-job later

Analytical mode uses an abstract bus bandwidth model and may not accurately capture shared-topology contention. It can be added later:

./bin/SimAI_analytical \
  -w model_a.txt,model_b.txt \
  -j_gpus 64,128

But the documentation should clearly state that this is an approximation and not equivalent to ns-3 multi-job network contention simulation.

---

5. 测试前置条件

中文

是否依赖额外数据？

分两类：

1. 最小功能测试

   • 不需要额外真实模型数据。
   • 可以复用仓库已有的 workload 示例，例如 example/microAllReduce.txt。
   • 为了模拟“不同模型”，可以准备两个不同 workload 文件：
     • 一个小通信量文件。
     • 一个大通信量文件。
   • 这两个文件可以来自 AICB，也可以临时复制并调整示例 workload 的通信大小。

2. 真实多模型训练场景验证

   • 需要 AICB 生成的多个 workload 文件。
   • 每个 workload 对应一个模型和并行配置。
   • 需要 ns-3 topo 和 config：
     • topo 文件，例如由 gen_Topo_Template.py 生成。
     • astra-sim-alibabacloud/inputs/config/SimAI.conf。
   • 如果使用 analytical，还需要 busbw 或相关网络带宽参数。

建议测试用例

1. 单 job 回归

   • 原有命令不变：

     ./bin/SimAI_simulator -t 16 -w ./example/microAllReduce.txt -n <topo> -c <conf>

   • 结果应与修改前一致或在可接受误差内。

2. 两个 job，使用相同 workload

   • 验证多 job orchestration 是否正确。
   • 例如：

     -w microAllReduce.txt,microAllReduce.txt -j_gpus 8,8

3. 两个 job，使用不同 workload

   • 验证 issue 的核心场景。
   • 例如：

     -w small_model.txt,large_model.txt -j_gpus 8,16

4. GPU 数量校验

   • sum(job.gpus) 大于 topology GPU 数时应报错。
   • job GPU 数为 0 或 workload 数量与 j_gpus 数量不一致时应报错。

5. 输出隔离

   • 检查两个 job 的结果文件不会互相覆盖。

6. 共享拓扑竞争

   • 构造两个 job 同时运行，检查 ns-3 链路利用率、完成时间、flow trace 是否体现并发流量。

English

Does testing require extra data?

There are two levels:

1. Minimal functional tests

   • No real model data is required.
   • Reuse existing workload examples such as example/microAllReduce.txt.
   • To emulate different models, prepare two workload files:
     • One with smaller communication sizes.
     • One with larger communication sizes.
   • These can be generated by AICB or derived from existing examples.

2. Real multi-model training validation

   • Requires multiple workload files generated by AICB.
   • Each workload corresponds to one model and one parallelism configuration.
   • Requires ns-3 topology and config:
     • A topology file, for example generated by gen_Topo_Template.py.
     • astra-sim-alibabacloud/inputs/config/SimAI.conf.
   • Analytical mode may also require bus bandwidth configuration or related bandwidth parameters.

Suggested tests

1. Single-job regression

   • Existing command should still work:

     ./bin/SimAI_simulator -t 16 -w ./example/microAllReduce.txt -n <topo> -c <conf>

   • Results should match the previous behavior or remain within acceptable variance.

2. Two jobs using the same workload

   • Validates orchestration:

     -w microAllReduce.txt,microAllReduce.txt -j_gpus 8,8

3. Two jobs using different workloads

   • Validates the core issue:

     -w small_model.txt,large_model.txt -j_gpus 8,16

4. GPU count validation

   • Error if sum(job.gpus) exceeds topology GPU count.
   • Error if job GPU count is zero.
   • Error if the number of workloads and GPU counts does not match.

5. Output isolation

   • Ensure results from different jobs do not overwrite each other.

6. Shared-topology contention

   • Run two jobs concurrently and verify that ns-3 link utilization, completion time, and flow traces reflect concurrent traffic.

---

6. README 是否需要修改及建议内容

中文

是否需要修改 README？

如果只是回复 issue：不需要立即修改 README。
如果实现多 job 功能：需要修改 README / Tutorial，否则用户不知道如何配置多 workload、多 GPU 分区和结果输出。

建议更新：

1. README_CN.md
2. README.md
3. docs/Tutorial.md
4. 如支持 ns-3 only，应在文档中明确说明 analytical 与 ns-3 simulation 的差异。

建议加入的中文文档内容

### 多训练任务共享拓扑仿真

SimAI 支持在同一个 ns-3 网络拓扑中同时运行多个训练任务。每个任务需要提供独立的 workload 文件和 GPU 数量。不同任务会映射到同一拓扑中的不同 GPU 分区，并共享底层网络链路资源。

示例：

```bash
AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator \
  -t 16 \
  -w ./workloads/model_a.txt,./workloads/model_b.txt \
  -j_gpus 64,128 \
  -n ./Spectrum-X_192g_8gps_100Gbps_A100 \
  -c astra-sim-alibabacloud/inputs/config/SimAI.conf
```

说明：

- `-w` 接收逗号分隔的 workload 文件列表。
- `-j_gpus` 接收每个 job 使用的 GPU 数量。
- 每个 job 的 GPU 分区按顺序自动分配：
  - job0 使用 `[0, 63]`
  - job1 使用 `[64, 191]`
- `sum(j_gpus)` 不能超过拓扑中的 GPU 总数。
- 建议使用 AICB 为每个模型分别生成 workload 文件。
- 当前该功能主要面向 ns-3 Simulation 模式，用于观察共享拓扑下的链路竞争和通信影响。

可直接回复 Issue 的中文 Markdown

您好，目前 SimAI 的训练仿真路径还不支持开箱即用地在同一个拓扑中同时训练多个大小不同的模型。

当前 SimAI/AICB 的常规流程是：AICB 为一个模型和一组并行配置生成一个 workload 文件，SimAI 的训练仿真入口一次加载一个 workload 文件进行仿真。ns-3 frontend 虽然会在拓扑中的多个节点上创建 `Sys`，但当前所有 `Sys` 使用的是同一个 workload，因此还不是“多个不同模型 job 共享同一拓扑并发运行”的配置方式。

不过从架构上看，这个能力是可以支持的：可以为不同模型分别生成 workload 文件，然后在同一个 ns-3 simulation 中为每个 job 创建独立的 `Sys` 集合，并通过不同的 `npu_offset` 映射到同一拓扑中的不同 GPU 分区。这样不同 job 的通信流会进入同一个 ns-3 网络，从而可以观察共享链路下的竞争和完成时间变化。

一个最小实现方案是：

1. 扩展 ns-3 frontend 参数，例如支持：
   ```bash
   -w model_a.txt,model_b.txt -j_gpus 64,128
   ```
2. 为每个 job 自动分配 GPU offset：
   - job0: GPU `[0, 63]`
   - job1: GPU `[64, 191]`
3. 每个 job 创建自己的 `Sys`/`Workload`，但共享同一个 ns-3 topology。
4. 修改完成条件，避免第一个 job 完成后提前结束仿真。
5. 输出文件按 job 名或 job 目录隔离，避免覆盖。

目前建议的 workaround 是：先用 AICB 分别生成不同模型的 workload 文件，然后分别运行 SimAI；如果需要真实的共享拓扑竞争，需要做上述 frontend 层面的多 job 扩展。

English

Should README be updated?

If we only answer the issue: README does not need to be changed immediately.
If multi-job support is implemented: README / Tutorial should be updated, otherwise users will not know how to configure multiple workloads, GPU partitions, or output directories.

Suggested files to update:

1. README_CN.md
2. README.md
3. docs/Tutorial.md
4. If only ns-3 mode is supported, clearly document the difference between analytical and ns-3 simulation modes.

Suggested English documentation

### Multi-training-job simulation on a shared topology

SimAI supports running multiple training jobs simultaneously in the same ns-3 network topology. Each job should provide its own workload file and GPU count. Jobs are mapped to different GPU partitions in the same topology and share the underlying network links.

Example:

```bash
AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator \
  -t 16 \
  -w ./workloads/model_a.txt,./workloads/model_b.txt \
  -j_gpus 64,128 \
  -n ./Spectrum-X_192g_8gps_100Gbps_A100 \
  -c astra-sim-alibabacloud/inputs/config/SimAI.conf
```

Notes:

- `-w` accepts a comma-separated list of workload files.
- `-j_gpus` specifies the number of GPUs used by each job.
- GPU partitions are assigned sequentially:
  - job0 uses `[0, 63]`
  - job1 uses `[64, 191]`
- `sum(j_gpus)` must not exceed the total number of GPUs in the topology.
- It is recommended to use AICB to generate one workload file per model.
- This feature is primarily intended for ns-3 Simulation mode, where shared-topology link contention can be modeled.

English Markdown reply for the issue

Hi, the current SimAI training simulation path does not provide out-of-the-box support for training multiple different-sized models simultaneously on the same topology.

The current SimAI/AICB workflow is mostly: AICB generates one workload file for one model and one parallel configuration, and the SimAI training frontend loads one workload file for simulation. Although the ns-3 frontend creates `Sys` objects for multiple nodes in the topology, they currently all use the same workload file. Therefore, this is not yet equivalent to running multiple different model jobs concurrently on a shared topology.

Architecturally, this can be supported. Different models can first be converted into different workload files by AICB. Then, in the same ns-3 simulation, SimAI can create a separate group of `Sys`/`Workload` objects for each job and use different `npu_offset` values to map them to different GPU partitions in the same topology. In that case, traffic from all jobs would enter the same ns-3 network, allowing link contention and completion time impact to be observed.

A minimal implementation plan would be:

1. Extend the ns-3 frontend CLI, for example:
   ```bash
   -w model_a.txt,model_b.txt -j_gpus 64,128
   ```
2. Automatically assign GPU offsets:
   - job0: GPU `[0, 63]`
   - job1: GPU `[64, 191]`
3. Create one `Sys`/`Workload` group per job while sharing the same ns-3 topology.
4. Update the completion logic so the simulation does not stop after the first job finishes.
5. Isolate output files by job name or job directory to avoid overwriting results.

For now, the recommended workaround is to generate workload files for each model with AICB and run SimAI separately. If shared-topology contention is required, the frontend-level multi-job extension described above is needed.