added systems jcahpc-miyabi-g by yoshifuminakamura · Pull Request #1436 · llnl/benchpark

yoshifuminakamura · 2026-07-07T15:27:54Z

Description

Adds a Benchpark system configuration for JCAHPC Miyabi-G, the GH200 GPU partition of the Miyabi supercomputer.
Includes PBS queue definitions, JCAHPC group accounting support, NVHPC/CUDA/HPC-X external package configuration, and compiler configuration for the Miyabi-G module environment.
Updates PBS allocation handling so systems can request a site-specific per-node core count via n_cores_per_node; Miyabi-G uses this to request the full GH200 CPU cpuset and avoid MPI/hwloc issues with too-small ncpus values.
Also comments out the PBS n_gpus chunk-resource emission in the allocation modifier; on Miyabi-G, GPU allocation is already represented by n_nodes/select, and adding a separate gpus chunk resource is unnecessary for this site configuration.
Dependencies: existing Miyabi-G site modules and the existing Benchpark hardware description for NVIDIA GH200.
Relevant issues: none.

Adding/modifying a system (docs: Adding a System)

Add/modify systems/jcahpc-miyabi-g/system.py file
Reuse existing systems/all_hardware_descriptions/NVIDIA-neoverse-GH200-Infiniband/hardware_description.yaml, which appears in the docs catalogue

Adding/modifying core functionality, CI, or documentation:

Update docs
Update .github/workflows and .gitlab/tests unit tests (if needed)
Ensure rules.yml resource_rules_* change rules match shared_*_clusters.yml run_test_* job matricies. (if needed)

michaelmckinsey1

I think the allocation modifier changes could be breaking.

michaelmckinsey1 · 2026-07-07T18:53:36Z

+        # No GPU chunk sub-resource is emitted here.
+        # On Miyabi-G, GPU allocation is already reflected in n_nodes/select, so
+        # adding a separate GPU chunk resource is unnecessary.


Is this commented out because it breaks Miyabi-G? Should gpus= be unset for a "full node"?

I suppose since there is only 1 gpu the only option is 0 or 1.

Looking at this again, the issue is not necessarily that gpus= itself breaks Miyabi-G. The pre-existing PBS code path appears to call self.gpus_as_gpus_per_rank(v.n_gpus), but gpus_as_gpus_per_rank() expects the allocation object v, not the integer v.n_gpus.

So this block may fail before we even test whether Miyabi-G accepts gpus=1. Since each Miyabi-G node has one GPU, gpus=1 may be fine. I should not have commented this out globally; I will restore the PBS gpus= emission and fix the call to use self.gpus_as_gpus_per_rank(v).

michaelmckinsey1 · 2026-07-07T18:57:16Z

+        # implied by the rank/thread layout. This keeps site policy out of the
+        # shared modifier: a system that wants a different value (MIG slices,
+        # partial-node sharing, ...) simply sets `n_cores_per_node` accordingly.
+        n_cpus_per_node = v.n_cores_per_node or (


We avoid setting the number of cpus/tasks based on the number of cores on a node, because the number of tasks is set per-experiment.

This is essentially a full-node setting @pearce8, which we have avoided in the past.

@yoshifuminakamura Is it not possible to run using less than 72 cores on the system?

It is possible to run with fewer than 72 cores on Miyabi-G. The issue I hit is more specific: for example, saxpy requests a single thread/rank, and the current PBS modifier then generates ncpus=1. On Miyabi-G that gives the job a one-CPU cpuset, and HPC-X/Open MPI's hwloc fails immediately at program startup.

So the requirement is not "always use all 72 cores"; it is "avoid a too-small cpuset such as ncpus=1". Allocating a larger CPU cpuset avoids the hwloc startup failure. I can rework this so it is expressed as a Miyabi-G-specific minimum/requested PBS ncpus value rather than framing it as a full-node setting.

michaelmckinsey1 · 2026-07-07T19:36:26Z

+    variant(
+        "group",
+        default="none",
+        multi=False,
+        description=(
+            "JCAHPC project/group code for accounting, emitted as "
+            "'#PBS -W group_list=<group>' (e.g. group=gz00). Required to "
+            "actually submit a job; 'none' omits the directive."
+        ),
+    )


Is a group different from a bank here? Would -A <group> fail?

qsub with -A fails on Miyabi.
I don't fully understand the meaning of “bank,” but in Miyabi, a “group” refers to the ID of a project group to which multiple people can belong. A budget (available node time) is linked to this group. Since a single user can belong to multiple project groups with the same uid, you must specify which project group the job belongs to using the -W option when submitting a job.

michaelmckinsey1 · 2026-07-07T19:37:58Z

+    # Verified against the live login node (miyabi-g1) on 2026-07-06:
+    #   NVHPC root : /work/opt/local/aarch64/cores/nvidia/<ver>/Linux_aarch64/<ver>
+    #   HPC-X MPI  : <nvhpc-root>/comm_libs/<cuda>/hpcx/latest/ompi
+    #   Std. CUDA  : /work/opt/local/aarch64/cores/cuda/<ver>
+    # nvhpc release -> (bundled default CUDA, HPC-X OpenMPI version)
+    nvhpc_to_cuda_mpi = {
+        "24.5": ("12.4", "4.1.7"),
+        "24.9": ("12.6", "4.1.7"),
+        "24.11": ("12.6", "4.1.7"),
+        "25.1": ("12.6", "4.1.7"),
+        "25.3": ("12.8", "4.1.7"),
+        "25.5": ("12.9", "4.1.9"),
+        "25.9": ("13.0", "4.1.9"),
+        "25.11": ("13.0", "4.1.9"),
+        "26.3": ("13.1", "4.1.9"),
+    }


@yoshifuminakamura Can we specify this in same way you did for riken-gh200 for consistency?

benchpark/systems/riken-gh200/system.py

Lines 66 to 76 in 72ee531

self.nvhpc_version = Version(self.spec.variants["nvhpc"][0])

if str(self.nvhpc_version) == "26.3":

self.cuda_version = "13.1"

if str(self.nvhpc_version) == "25.9":

self.cuda_version = "13.0"

if str(self.nvhpc_version) == "25.7":

self.cuda_version = "12.9"

if str(self.nvhpc_version) == "24.9":

self.cuda_version = "12.6"

if str(self.nvhpc_version) == "24.3":

self.cuda_version = "12.3"

Yes, I can change this to match the riken-gh200 style for consistency.

michaelmckinsey1 · 2026-07-07T19:44:21Z

+        variables = {
+            "cuda_arch": "90",
+            # Miyabi-G nodes are allocated whole (one GH200 per node, no MIG
+            # sharing), so request the node's full core count. Otherwise the
+            # generated `ncpus` collapses to the rank/thread count (e.g. 1 for a
+            # single-rank job); PBS then pins the job to that narrow cpuset and
+            # HPC-X/Open MPI's hwloc segfaults building the topology on GH200.
+            "n_cores_per_node": self.sys_cores_per_node,
+        }


These variables should be set in the id_to_resources section. I didn't catch this with the riken-gh200 PR

I will move variables into id_to_resources.

added systems jcahpc-miyabi-g

566775d

github-actions Bot added feature New feature or request system New or modified system config labels Jul 7, 2026

yoshifuminakamura added 2 commits July 8, 2026 00:55

fix lint for jcahpc miyabi-g system

70604e2

fix miyabi-g import ordering

79b8081

yoshifuminakamura requested a review from michaelmckinsey1 July 7, 2026 16:18

michaelmckinsey1 requested changes Jul 7, 2026

View reviewed changes

address miyabi-g review comments

ea6fbd9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

added systems jcahpc-miyabi-g#1436

added systems jcahpc-miyabi-g#1436
yoshifuminakamura wants to merge 4 commits into
llnl:developfrom
yoshifuminakamura:feature/jcahpc-miyabi-g

yoshifuminakamura commented Jul 7, 2026

Uh oh!

michaelmckinsey1 left a comment

Uh oh!

michaelmckinsey1 Jul 7, 2026

Uh oh!

yoshifuminakamura Jul 9, 2026

Uh oh!

michaelmckinsey1 Jul 7, 2026

Uh oh!

yoshifuminakamura Jul 9, 2026

Uh oh!

michaelmckinsey1 Jul 7, 2026

Uh oh!

yoshifuminakamura Jul 9, 2026

Uh oh!

michaelmckinsey1 Jul 7, 2026

Uh oh!

yoshifuminakamura Jul 9, 2026

Uh oh!

michaelmckinsey1 Jul 7, 2026

Uh oh!

yoshifuminakamura Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	self.nvhpc_version = Version(self.spec.variants["nvhpc"][0])
	if str(self.nvhpc_version) == "26.3":
	self.cuda_version = "13.1"
	if str(self.nvhpc_version) == "25.9":
	self.cuda_version = "13.0"
	if str(self.nvhpc_version) == "25.7":
	self.cuda_version = "12.9"
	if str(self.nvhpc_version) == "24.9":
	self.cuda_version = "12.6"
	if str(self.nvhpc_version) == "24.3":
	self.cuda_version = "12.3"

Uh oh!

Conversation

yoshifuminakamura commented Jul 7, 2026

Description

Adding/modifying a system (docs: Adding a System)

Adding/modifying core functionality, CI, or documentation:

Uh oh!

michaelmckinsey1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants