Add clutter placement by zhx06 · Pull Request #1242 · isaac-sim/IsaacLab-Arena

zhx06 · 2026-09-10T18:02:01Z

Summary

Add offline physics-settled clutter placement

Detailed description

Add a standalone example to settle ordinary Arena assets, preserving base rotations and generating independent layouts.
Reject invalid layouts and save one exact-pose scene YAML per generated environment.
Restore cached object and robot poses on reset without changing the relation solver.
Validate output filesystem support before settling and publish caches without overwriting existing files.
Validation: focused PhysX, graph, and CLI tests; pre-commit --all-files. Newton and CAP/Berkeley reproduction remain unverified.

Signed-off-by: zhx06 <zihaox@nvidia.com>

greptile-apps · 2026-09-10T21:48:42Z

Confidence Score: 4/5

The PR is not yet safe to merge because environments without relation solving no longer reset embodiment roots to their declared initial poses.

Findings

Embodiment Pose Stops Resetting ▶
Cache Requires Hard Links ▶

Summary

Adds drop-pose sampling, settling, containment validation, and atomic per-file cache publication.
Extends graph conversion to support explicit initial poses and retain graph-node-to-asset mappings.
Adds a CLI, example scene, documentation, and broad simulation and unit-test coverage.
The embodiment initial-pose integration drops its existing reset behavior, and cache publication assumes hard-link support.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Spec[Concrete scene YAML] --> Build[Build Arena environment]
  Build --> Sample[Sample noninterpenetrating drop poses]
  Sample --> Sim[Run physics settling]
  Sim --> Validate{Rest and containment valid?}
  Validate -->|No| Retry[Retry with deterministic attempt seed]
  Retry --> Sample
  Validate -->|Yes| Cache[Export exact poses to scene YAML]
  Cache --> Runtime[Load ordinary runtime environment]
  Runtime --> Reset[Replay cached asset reset poses]

_{Reviews (1) · Last reviewed commit: "keep offline"}

Signed-off-by: zhx06 <zihaox@nvidia.com>

alexmillane

This is a good start.

However I have a few opinions on the design:

Output: Right now the system outputs a single environment.yaml per placement. Generating lots of graphs in the case that we want many placements. I would suggest instead that we introduce a new type of yaml file that is an optional companion to our usual environment files. If such a file is passed to the run-time (or specified in the environment.yaml) poses are taken from this companion file. This file would be a mapping between object names and a list of poses-per-object. With this design we can get 1000s of layouts with just two files. It also separates specification of the environment from the specification of the placement. Wdyt?
Settling specification. Right now we specify which objects are to be settled via command line parameters. Which means that they're not captured in the environment.yaml. What do you think about instead adding a new relation-type called ClutterOn which is specified in the environment.yaml and captures that this object is the target of the clutter placement. This way the placement of the object travels with the environment spec, rather than being captured on a users CLI (and then immediately lost).
Before drop initialization: Right now we use a parallel system to our ObjectPlacer to arrange the object prior to dropping them. Could we not implement this inside the object placer. ClutterOn could translate to an object being above a support surface, as low as possible, without colliding with other (ClutterOn) objects. This would leave to the objects forming a vertical collumn prior to the settling. It also would mean that ClutterOn would produce valid results even without running through generate_clutter_scene.py. They would just drop at the start of simulation.

alexmillane · 2026-09-11T09:16:03Z

+    """Apply a fixed YAML pose at construction and on every reset."""
+    if value is None:
+        return
+    assert isinstance(value, dict), "initial_pose must be a mapping"


Can we correct the type annotation above. At the input we say Any but then check that it's None | dict

alexmillane · 2026-09-11T09:18:43Z

+def _apply_initial_pose(asset: Asset, value: Any) -> None:
+    """Apply a fixed YAML pose at construction and on every reset."""
+    if value is None:
+        return
+    assert isinstance(value, dict), "initial_pose must be a mapping"
+    assert not (set(value) - {"position_xyz", "rotation_xyzw"}), "Unknown initial_pose fields"
+    if "position_xyz" not in value or "rotation_xyzw" not in value:
+        declared = asset.get_initial_pose()
+        assert declared is None or isinstance(declared, Pose), "Partial initial_pose requires a fixed default pose"
+        default = declared if declared is not None else Pose.identity()
+        position = value.get("position_xyz", default.position_xyz)
+        rotation = value.get("rotation_xyzw", default.rotation_xyzw)
+    else:
+        position, rotation = value["position_xyz"], value["rotation_xyzw"]
+    for name, values, size in (("position_xyz", position, 3), ("rotation_xyzw", rotation, 4)):
+        assert isinstance(values, (list, tuple)) and len(values) == size, f"{name} needs {size} numbers"
+        assert all(
+            isinstance(v, Real) and not isinstance(v, bool) and math.isfinite(v) for v in values
+        ), f"{name} must contain finite numbers"
+    assert math.isclose(sum(v * v for v in rotation), 1.0, abs_tol=1e-4), "rotation_xyzw must be a unit quaternion"
+    asset.set_initial_pose(Pose(tuple(float(v) for v in position), tuple(float(v) for v in rotation)))


99% of this function is extracting a Pose from a dict.

Then we set it in half the final line.

Suggestion to break this function up. The first function should be

def _get_pose_from_dict(pose_dict: dict) -> Pose | None:

This

def _apply_initial_pose(pose: Pose | None):

alexmillane · 2026-09-11T13:47:33Z

+
+
+def scene_with_cached_poses(
+    spec: ArenaEnvGraphSpec, poses: Mapping[str, Pose], assets_by_node_id: Mapping[str, Asset]


I'm interested why the agent wants to use Mapping. Do we expect something other than a dict here?

alexmillane · 2026-09-11T14:02:44Z

This is a test/example file correct?

I would suggest that this goes into isaaclab_arena_environments or isaaclab_arena_examples.

Generally we try to keep examples out of the core framework.

alexmillane · 2026-09-11T14:18:00Z

+def write_scene_cache(spec: ArenaEnvGraphSpec, path: Path) -> None:
+    """Publish a complete YAML file atomically without overwriting; requires hard-link support."""
+    path.parent.mkdir(parents=True, exist_ok=True)
+    with tempfile.TemporaryDirectory(dir=path.parent, prefix=".clutter-") as directory:
+        temporary = Path(directory) / path.name
+        spec.write_yaml(temporary)
+        # A same-filesystem hard link publishes the complete file without overwriting a raced writer.
+        os.link(temporary, path)


I'm wondering why we have to go through all this effort to write a file? What's wrong with just writing a file normally?

zhx06 added 2 commits September 10, 2026 14:42

add support for cluttered scene

a8627c8

Signed-off-by: zhx06 <zihaox@nvidia.com>

keep offline

f25f5e4

Signed-off-by: zhx06 <zihaox@nvidia.com>

zhx06 force-pushed the zxiao/feature/clutter-placement branch from 2155f20 to f25f5e4 Compare September 10, 2026 21:43

zhx06 marked this pull request as ready for review September 10, 2026 21:44

zhx06 requested review from alexmillane, cvolkcvolk, peterd-NV, qianl-nv, viiik-inside and xyao-nv as code owners September 10, 2026 21:44

greptile-apps Bot reviewed Sep 10, 2026

View reviewed changes

Comment thread isaaclab_arena/environment_spec/arena_env_graph_conversion_utils.py Outdated

Comment thread isaaclab_arena_environments/isaac_cap/clutter/cache.py

Fix cached pose resets and output validation

2d321a6

Signed-off-by: zhx06 <zihaox@nvidia.com>

zhx06 changed the title ~~Add support for cluttered scene~~ Add offline physics-settled clutter placement Sep 10, 2026

address comment from review bot

f2f25c1

Signed-off-by: zhx06 <zihaox@nvidia.com>

zhx06 changed the title ~~Add offline physics-settled clutter placement~~ Add clutter placement Sep 10, 2026

move to cap folder

40df67d

Signed-off-by: zhx06 <zihaox@nvidia.com>

alexmillane reviewed Sep 11, 2026

View reviewed changes

Navigation Menu

Uh oh!

Add clutter placement - #1242

Add clutter placement#1242
zhx06 wants to merge 5 commits into
mainfrom
zxiao/feature/clutter-placement

zhx06 commented Sep 10, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Sep 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

alexmillane left a comment

Uh oh!

alexmillane Sep 11, 2026

Uh oh!

alexmillane Sep 11, 2026

Uh oh!

alexmillane Sep 11, 2026

Uh oh!

alexmillane Sep 11, 2026

Uh oh!

alexmillane Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants



		def scene_with_cached_poses(
		spec: ArenaEnvGraphSpec, poses: Mapping[str, Pose], assets_by_node_id: Mapping[str, Asset]

Uh oh!

Conversation

zhx06 commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Detailed description

Uh oh!

greptile-apps Bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Diagram

Uh oh!

Uh oh!

Uh oh!

alexmillane left a comment

Choose a reason for hiding this comment

Uh oh!

alexmillane Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

alexmillane Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

alexmillane Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

alexmillane Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

alexmillane Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zhx06 commented Sep 10, 2026 •

edited

Loading

greptile-apps Bot commented Sep 10, 2026 •

edited

Loading