Add PhysX deformable pick-and-place support

[qianl-nv]

Summary

Add PhysX deformable pick-and-place support

Detailed description

• Introduces DeformableObject with backend-specific spawner configs, nodal reset events, and placement integration on top of the existing ObjectBase. DeformableObject is supported for placement using rest shape bbox from USD or from spawner size.
• Extends PickAndPlaceTask, spatial predicates, and ArenaWorld so deformable pick objects use placement-only success (no contact sensor).
• Adds the droid_deformable_pick_and_place environment with selectable deformable assets (cube, surface, teddy bear) and PhysX smoke/regression tests. Verified cube pick and place with teleop. Works with OpenPi but the picking is not always reliable.

Command to run:

python submodules/IsaacLab/scripts/environments/teleoperation/teleop_se3_agent.py --viz kit --task droid_deformable_pick_and_place --embodiment droid_differential_ik --external_callback isaaclab_arena.environments.isaaclab_interop.environment_registration_callback

Known issues to be addressed to follow up PRs:

• Switching deformable objects and pnp to Newton (dependent on Newton-compatible Droid support)
• Improve geometry-based object_supported_by predicate for deformable+rigid to use rigid vertices . This requires extension of ArenaWorld to hold cache of rooted object mesh/vertices.
• Deformable material / simulation cfg optimization for the task is deferred until we support Newton.

[qianl-nv]

self review 1

[qianl-nv]

self review 2

[greptile-apps]

Confidence Score: 4/5

The PR is not yet safe to merge because deformable tasks can report successful placement without the destination actually supporting the object's low nodes.

Findings

1.  Support ignores horizontal overlap ▶

Summary

• Introduces primitive and USD-backed deformable object configurations with backend validation.
• Adds live nodal position, velocity, centroid, and speed access through ArenaWorld.
• Uses geometric support instead of contact sensors for deformable pick-and-place success.
• Adds cube, surface, and teddy-bear assets plus PhysX smoke and reset tests.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Deformable nodal state] --> B[ArenaWorld live centroid and vertices]
  B --> C{Centroid over destination footprint?}
  B --> D{Low nodes near destination top Z?}
  B --> E{Maximum nodal speed below threshold?}
  C --> F[Pick-and-place success]
  D --> F
  E --> F
  G[Missing check: low-node XY overlap] -.-> D

Loading

_{Reviews (1) · Last reviewed commit: "Remove redundant unit test"}

[arena-review-bot]

🤖 Isaac Lab-Arena Review Bot

Summary

Adds a DeformableObject asset, nodal-state reset events, deformable-aware reads on ArenaWorld, a geometric support check so PickAndPlaceTask can score deformables without a contact sensor, and a DROID example environment. The layering is sensible — the asset lives in assets/, the example in isaaclab_arena_environments/, and the generalized ArenaWorld reads keep the predicates type-agnostic. My concerns are about the tuning surface for the new success criterion and about whether the end-to-end tests actually execute.

Design, Boundaries & Scope

The deformable success path introduces three tolerances (support_tolerance, low_point_tolerance, minimum_support_fraction) that are hardcoded in object_supported_by and unreachable from PickAndPlaceTask, while force_threshold and support_cone_half_angle_rad remain on the task but silently do nothing for deformables. For an eval library, a success threshold that cannot be configured (and one that can be configured but is ignored) is worth fixing before this lands. Smaller items: physics_preset is a bare string compared against literals in the builder, and PlaceableAsset.write_layout_pose_to_sim largely re-implements RootedObjectBase.set_object_pose.

Findings

🟡 isaaclab_arena/tasks/predicates/spatial.py:180 — hardcoded support tolerances; force_threshold / support_cone_half_angle_rad ignored on the deformable path.
🟡 isaaclab_arena/tests/test_deformable_object.py:403 — the only two live-sim deformable tests are skipped unless pytetwild is installed; it appears nowhere else in the repo.
🔵 isaaclab_arena/environments/arena_world.py:124 — getattr(self._scene, "deformable_objects", {}) in three methods vs direct access in four others.
🔵 isaaclab_arena/environments/arena_world.py:196 — get_average_speed_w docstring says "mean nodal speed" but computes the norm of the mean nodal velocity; this feeds ObjectMovedRateMetric.
🔵 isaaclab_arena/terms/events.py:41 — the UsdFileCfg branch looks equivalent to the else for centered mesh primitives; can it be collapsed?
🔵 isaaclab_arena/tasks/pick_and_place_task.py:84 — _uses_deformable duplicates contact_sensor_name is None; the resolved SceneEntityCfg ternary repeats twice.
🔵 isaaclab_arena/assets/deformable_object.py:54 — physics_preset as a string; an enum would remove the "default" alias handling in the builder.
🔵 isaaclab_arena/relations/placement_asset.py:136 — write_layout_pose_to_sim overlaps RootedObjectBase.set_object_pose.

Test Coverage

Coverage is good at the config and unit level: test_deformable_object.py covers backend inference, bounding-box inference for all three library assets, preset validation, and the nodal reset transform against fakes; test_object_on_destination.py adds the deformable branch (mixed true/false cases) and the deformable-destination rejection; test_arena_world_scene_access.py adds the deformable reads. All new tests land in phase 1 and carry no markers, which matches what they do. The gap is the two tests that actually run a sim — test_deformable_reset_and_initial_pose and test_deformable_pick_and_place_success — which are skipped without pytetwild; the latter is also success-only, so a negative case (cube off the plate → False) would be worth adding.

Verdict

Minor fixes needed