Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,3 +97,6 @@ and `logo_dark`.

Keep cache-removal documentation focused on migration impact. Omit asset-fetch
implementation walkthroughs and alternative override instructions.

When rebasing example package moves, check upstream-added `:code-file:` references
against the new paths; Git can merge these references without detecting stale paths.
31 changes: 16 additions & 15 deletions docs/source/references/camera_streaming.rst
Original file line number Diff line number Diff line change
Expand Up @@ -344,7 +344,7 @@ environment with ``isaaccapture``) and run::
python -m isaaccapture.cloudxr.service status
python -m isaaccapture.cloudxr.service stop

Run ``camera_viz.py --help`` for the rest (install dir, env-config file, WSS proxy toggle).
Run ``camera_viz.sh run CONFIG --help`` for the rest (install dir, env-config file, WSS proxy toggle).

Runtime settings themselves are declared in the config rather than exported. The viewer ships
defaults (pose wait off, runtime foveation on), and ``display.cloudxr`` overrides them per
Expand Down Expand Up @@ -376,7 +376,7 @@ runtime resolved, read the settings dump at the top of the newest
Split mode — robot → workstation over RTP
-----------------------------------------

Split mode runs the capture side on the robot (``camera_streamer.py``) and ships RTP H.264 to
Split mode runs the capture side on the robot (the ``camera_streamer`` module) and ships RTP H.264 to
the viewer on the workstation (``source: rtp``).

.. warning::
Expand Down Expand Up @@ -521,28 +521,29 @@ Televiz as the compositor at the end of the chain:

camera_viz/
├── camera_viz.sh — CLI: setup / loopback / run / deploy / service-*
├── camera_viz.py — receiver / viewer (drives a Televiz VizSession)
├── camera_streamer.py — robot-side RTP sender (per-camera supervisor)
├── pipeline/ — source ABC + threaded runner
├── placements/ — XR lock-mode strategies (world / head / lazy / gimbal)
├── sources/ — V4L2 / OAK-D / ZED / video replay / synthetic / rtp_h264
├── transports/ — RTP sender + receiver (native + GStreamer)
├── codec/ — native NVENC / NVDEC pybind module
├── configs/ — one YAML per source kind
├── test_data/ — sample replay clip (Git LFS)
├── python/isaaccapture_examples/camera_viz/
│ ├── camera_viz.py — receiver / viewer (drives a Televiz VizSession)
│ ├── camera_streamer.py — robot-side RTP sender (per-camera supervisor)
│ ├── pipeline/ — source ABC + threaded runner
│ ├── placements/ — XR lock-mode strategies (world / head / lazy / gimbal)
│ ├── sources/ — V4L2 / OAK-D / ZED / video replay / synthetic / rtp_h264
│ ├── transports/ — RTP sender + receiver (native + GStreamer)
│ └── codec/ — native NVENC / NVDEC pybind module
└── scripts/ — installer + systemd unit template

- **Sources** (:code-dir:`sources/ <examples/camera_viz/sources>`) implement a common source ABC and
hand frames to a threaded runner in :code-dir:`pipeline/ <examples/camera_viz/pipeline>`. Each
- **Sources** (:code-dir:`sources/ <examples/camera_viz/python/isaaccapture_examples/camera_viz/sources>`) implement a common source ABC and
hand frames to a threaded runner in :code-dir:`pipeline/ <examples/camera_viz/python/isaaccapture_examples/camera_viz/pipeline>`. Each
source produces GPU frames where possible — e.g. the ZED source uses ``retrieve_image(MEM.GPU)`` so
BGRA8 stays in VRAM and a CUDA kernel channel-swaps into contiguous RGBA with no host round-trip.
- **The viewer** (:code-file:`camera_viz.py <examples/camera_viz/camera_viz.py>`) creates a
- **The viewer** (:code-file:`camera_viz.py <examples/camera_viz/python/isaaccapture_examples/camera_viz/camera_viz.py>`) creates a
``VizSession`` and adds one ``QuadLayer`` per enabled camera, then submits each frame to its layer
and calls ``render()`` once per frame. Stereo cameras submit both eyes.
- **Transport** (:code-dir:`transports/ <examples/camera_viz/transports>`) carries the split mode:
- **Transport** (:code-dir:`transports/ <examples/camera_viz/python/isaaccapture_examples/camera_viz/transports>`) carries the split mode:
an RTP H.264 sender on the robot and a receiver on the workstation, with a native NVENC/NVDEC codec
module (:code-dir:`codec/ <examples/camera_viz/codec>`) or a GStreamer fallback.
- **Placement** (:code-dir:`placements/ <examples/camera_viz/placements>`) holds the XR lock-mode
module (:code-dir:`codec/ <examples/camera_viz/python/isaaccapture_examples/camera_viz/codec>`) or a GStreamer fallback.
- **Placement** (:code-dir:`placements/ <examples/camera_viz/python/isaaccapture_examples/camera_viz/placements>`) holds the XR lock-mode
strategies. Placement is application policy — Televiz only renders a layer at whatever pose the app
sets each frame.

Expand Down
27 changes: 14 additions & 13 deletions examples/camera_viz/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ SPDX-License-Identifier: Apache-2.0
| `zed` | ZED 2 / Mini / X One; mono or `stereo: true` (per-eye SDK retrieve, zero-copy GPU) |
| `video` | Video-file replay (anything OpenCV/FFmpeg reads) — preview / testing without a camera. Loops by default; `stereo: true` splits side-by-side files into eyes (viewer only) |

In XR mode the viewer **attaches to the CloudXR runtime + WSS proxy**, starting a background service if none is serving — nothing to start separately (`--accept-eula` for the first run; CloudXR.js is hosted at `https://<host>:48322/client/` by default when this run starts the service — `--no-host-client` only applies then; stop and restart the service to change hosting; `camera_viz.py --help` for the rest). Output: XR headset (default) or desktop window (`run CONFIG --mode window`); one surface per camera — a flat plane (default), a cylinder arc, or an equirect sphere (`placements.<name>.shape`, XR only for the curved shapes). Stereo cameras render true SBS in XR; window mode shows the left eye. XR placements: `world` / `head` / `lazy` / `gimbal`.
In XR mode the viewer **attaches to the CloudXR runtime + WSS proxy**, starting a background service if none is serving — nothing to start separately (`--accept-eula` for the first run; CloudXR.js is hosted at `https://<host>:48322/client/` by default when this run starts the service — `--no-host-client` only applies then; stop and restart the service to change hosting; `camera_viz.sh run CONFIG --help` for the rest). Output: XR headset (default) or desktop window (`run CONFIG --mode window`); one surface per camera — a flat plane (default), a cylinder arc, or an equirect sphere (`placements.<name>.shape`, XR only for the curved shapes). Stereo cameras render true SBS in XR; window mode shows the left eye. XR placements: `world` / `head` / `lazy` / `gimbal`.

---

Expand Down Expand Up @@ -323,20 +323,21 @@ camera_viz xr · local · 1 camera
```
camera_viz/
├── camera_viz.sh — CLI: setup / loopback / run / deploy / service-*
├── camera_viz.py — receiver / viewer (entrypoint + wiring)
├── camera_streamer.py — robot-side RTP sender (per-camera supervisor)
├── config.py — YAML → SourceEntry: parse + validate, no allocation
├── display.py — VizSession + one layer per surface
├── cloudxr_env.py — display.cloudxr → the runtime's env file
├── dashboard.py — terminal status panel
├── controls/ — XR controller bindings, shapes, stereo geometry, HUD
├── pipeline/ — source ABC + threaded runner
├── placements/ — XR lock-mode strategies
├── sources/ — V4L2 / OAK-D / ZED / synthetic / video replay / rtp_h264
├── transports/ — RTP sender + receiver, native + GStreamer
├── codec/ — native NVENC/NVDEC pybind module
├── configs/ — one YAML per camera kind
├── test_data/ — sample replay clip (Git LFS)
├── python/isaaccapture_examples/camera_viz/
│ ├── camera_viz.py — receiver / viewer (entrypoint + wiring)
│ ├── camera_streamer.py — robot-side RTP sender (per-camera supervisor)
│ ├── config.py — YAML → SourceEntry: parse + validate, no allocation
│ ├── display.py — VizSession + one layer per surface
│ ├── cloudxr_env.py — display.cloudxr → the runtime's env file
│ ├── dashboard.py — terminal status panel
│ ├── controls/ — XR controller bindings, shapes, stereo geometry, HUD
│ ├── pipeline/ — source ABC + threaded runner
│ ├── placements/ — XR lock-mode strategies
│ ├── sources/ — V4L2 / OAK-D / ZED / synthetic / video replay / rtp_h264
│ ├── transports/ — RTP sender + receiver, native + GStreamer
│ └── codec/ — native NVENC/NVDEC pybind module
└── scripts/
├── _install_deps.sh — installer (setup + deploy)
└── camera-streamer.service.in — systemd unit template
Expand Down
18 changes: 9 additions & 9 deletions examples/camera_viz/camera_viz.sh
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ PY
}

# ──────────────────────────────────────────────────────────────────────
# run (the viewer; args after CONFIG forward to camera_viz.py, e.g. --mode xr)
# run (the viewer; args after CONFIG forward to the viewer, e.g. --mode xr)
# ──────────────────────────────────────────────────────────────────────

cmd_run() {
Expand All @@ -209,7 +209,7 @@ cmd_run() {
_require_rtp_deps
fi
log_step "Starting camera_viz — Ctrl-C to exit"
"$LOCAL_VENV/bin/python" "$HERE/camera_viz.py" "$@"
"$LOCAL_VENV/bin/python" -m isaaccapture_examples.camera_viz "$@"
}

# ──────────────────────────────────────────────────────────────────────
Expand Down Expand Up @@ -240,11 +240,11 @@ cmd_loopback() {
trap '_loopback_cleanup' EXIT

log_step "Starting camera_streamer → 127.0.0.1 (background)"
"$LOCAL_VENV/bin/python" "$HERE/camera_streamer.py" "$1" --host 127.0.0.1 &
"$LOCAL_VENV/bin/python" -m isaaccapture_examples.camera_viz.camera_streamer "$1" --host 127.0.0.1 &
_LOOPBACK_SENDER_PID=$!

log_step "Starting camera_viz (foreground) — Ctrl-C to exit"
"$LOCAL_VENV/bin/python" "$HERE/camera_viz.py" "$_LOOPBACK_RECV_CONFIG"
"$LOCAL_VENV/bin/python" -m isaaccapture_examples.camera_viz "$_LOOPBACK_RECV_CONFIG"
}

# ──────────────────────────────────────────────────────────────────────
Expand All @@ -256,7 +256,7 @@ cmd_deploy() {
# anything we don't recognize would land in REMOTE_REST (where the
# positional CONFIG lives).
local no_service=false
# ``--streaming-host`` → injected as ``--host`` on camera_streamer.py's
# ``--streaming-host`` → injected as ``--host`` on the streamer's
# CLI inside the rendered systemd unit. Lets you keep the YAML at
# 127.0.0.1 for loopback and override only when deploying to a robot.
local streaming_host="${STREAMING_HOST:-}"
Expand Down Expand Up @@ -304,10 +304,10 @@ cmd_deploy() {
local manual_host_flag=""
if [[ -n "$streaming_host" ]]; then
manual_host_flag=" --host $streaming_host"
log_info "(--streaming-host has no effect in --no-service mode — pass it to camera_streamer.py yourself.)"
log_info "(--streaming-host has no effect in --no-service mode — pass it to the streamer yourself.)"
fi
log_info "Run manually with:"
log_info " ssh $REMOTE_USER@$REMOTE_HOST 'cd ~/camera_viz && .venv/bin/python camera_streamer.py $config$manual_host_flag'"
log_info " ssh $REMOTE_USER@$REMOTE_HOST 'cd ~/camera_viz && .venv/bin/python -m isaaccapture_examples.camera_viz.camera_streamer $config$manual_host_flag'"
log_info "Re-run without --no-service when you're ready to install the systemd unit."
return 0
fi
Expand Down Expand Up @@ -441,9 +441,9 @@ REMOTE (Jetson robot)
deploy [--host H --user U [--password P]]
[--streaming-host IP] [--no-service] CONFIG
rsync source, install deps, install + start
systemd user service running camera_streamer.py.
systemd user service running the streamer.
--no-service stops after deps so you can run
camera_streamer.py by hand first.
the streamer by hand first.
--streaming-host injects ``--host IP`` into the
unit's ExecStart so the sender streams there
regardless of streaming.host in the YAML. Same
Expand Down
26 changes: 22 additions & 4 deletions examples/camera_viz/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,14 @@
# ``camera_viz.sh setup``; pass ``setup --wheel <path>`` to use a locally
# built wheel instead.

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "camera-viz"
# The dist name mirrors the import path, so an installed example claims no bare
# top-level name in site-packages.
name = "isaaccapture-examples-camera-viz"
version = "0.0.0"
description = "Televiz camera-feed visualizer"
requires-python = ">=3.11,<3.14"
Expand Down Expand Up @@ -38,13 +44,25 @@ rtp = [
# setup script apt-installs them and uses --system-site-packages),
# not pip — PyGObject from PyPI requires a source-build with cairo
# + girepository dev headers. pybind11 stays in the venv as a build-
# time dep for the codec at ./codec/.
# time dep for the native codec.
"pybind11>=2.11",
]

# Native NVENC/NVDEC codec lives in ./codec/. Build with
# ``codec/build.sh``; ``./camera_viz.sh setup`` does this automatically.
# Native NVENC/NVDEC codec lives in ./python/isaaccapture_examples/camera_viz/codec/.
# Build with
# its build.sh; ``./camera_viz.sh setup`` does this automatically.
# Requires CUDA toolkit + NVIDIA driver (libnvidia-encode + libnvcuvid).

# `isaaccapture_examples` is a PEP 420 namespace shared by every example dist and
# must stay without an __init__.py. `only-include` + `sources`, not `packages`:
# the latter keeps only the last path component and would root the wheel at a
# bare top-level `camera_viz/`.
#
# The native codec's .so is built in place by codec/build.sh, so it is picked up
# from the source tree rather than shipped by this build.
[tool.hatch.build.targets.wheel]
only-include = ["python/isaaccapture_examples/camera_viz"]
sources = ["python"]

[tool.uv]
python-preference = "only-managed"
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

import sys

from .camera_viz import main

sys.exit(main())
Original file line number Diff line number Diff line change
Expand Up @@ -29,14 +29,14 @@

import yaml

from pipeline import FrameSource
from sources import (
from .pipeline import FrameSource
from .sources import (
PairedFrameSource,
build_local_camera,
resolve_video_paths,
set_verbose,
)
from transports import RtpH264Sender, make_encoder
from .transports import RtpH264Sender, make_encoder

logger = logging.getLogger("isaaccapture.camera_streamer")

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,18 +30,16 @@
from isaaccapture import logging_config
from isaaccapture.cloudxr import CloudXRLauncher

import cloudxr_env
import config
import display
from controls import (
from . import cloudxr_env, config, display
from .controls import (
ControllerControls,
ControlTarget,
controls_config_from_yaml,
make_hud,
)
from dashboard import Dashboard
from pipeline import VizRunner
from sources import resolve_video_paths, set_notify_sink, set_verbose
from .dashboard import Dashboard
from .pipeline import VizRunner
from .sources import resolve_video_paths, set_notify_sink, set_verbose


def _parse_args(argv: Optional[list[str]]):
Expand Down Expand Up @@ -390,7 +388,3 @@ def _on_signal(signum, frame):
# actually in force inspectable while the runtime is up.
settings_stack.close()
return 0


if __name__ == "__main__":
sys.exit(main())
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@
from dataclasses import dataclass
from typing import List, Optional, Tuple

from pipeline import FrameSource
from placements import PlacementConfig, PlacementStrategy, build as build_placement
from sources import PairedFrameSource, RtpH264Source, build_local_camera
from .pipeline import FrameSource
from .placements import PlacementConfig, PlacementStrategy, build as build_placement
from .sources import PairedFrameSource, RtpH264Source, build_local_camera


@dataclass
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@

from . import shapes, stereo
from .hud import split_message
from placements import (
from ..placements import (
PlacementConfig,
PlacementStrategy,
build as build_placement,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
import numpy as np
from PIL import Image, ImageDraw, ImageFont

from placements import HeadLocked, PlacementConfig
from ..placements import HeadLocked, PlacementConfig

_W, _H = 1280, 160
_PAD, _BAR = 36, 12 # left text inset, accent-bar width
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
from dataclasses import replace
from typing import List, Optional, Tuple

from placements import yaw_quat
from ..placements import yaw_quat


def _clamp(value: float, limits: Tuple[float, float]) -> float:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@

import isaaccapture.viz as viz

from config import SourceEntry, VALID_SHAPES
from placements import yaw_quat
from .config import SourceEntry, VALID_SHAPES
from .placements import yaw_quat

# ImageLayerBase::kSlotCount (kMaxFramesInFlight + 2). Only used to
# report the VRAM that shape switching adds.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@

import isaaccapture.viz as viz

from dashboard import CameraRow, Dashboard, Snapshot
from ..dashboard import CameraRow, Dashboard, Snapshot

from .interface import FrameSource

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
from pathlib import Path
from typing import List

from pipeline import FrameSource
from ..pipeline import FrameSource

from ._helpers import PairedFrameSource, set_notify_sink, set_verbose
from .oakd import OakdSource
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@

import numpy as np

from pipeline import Frame, FrameSource, SourceSpec
from ..pipeline import Frame, FrameSource, SourceSpec


_SINK = None
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ def _ensure_initialized(self) -> None:
if self._decoder is not None:
return
try:
import codec
from .. import codec
except ImportError as e:
raise RuntimeError(
"NvH264Decoder requires the native codec. Run "
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@

import numpy as np

from pipeline import Frame, FrameSource, SourceSpec
from ..pipeline import Frame, FrameSource, SourceSpec
from ._helpers import alloc_pinned_host, notify, notify_verbose

logger = logging.getLogger("isaaccapture.camera_viz.sources.oakd")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@
import time
from typing import Optional

from pipeline import Frame, FrameSource, SourceSpec
from transports import RtpH264Receiver
from ..pipeline import Frame, FrameSource, SourceSpec
from ..transports import RtpH264Receiver

from ._nv_decode import NvH264Decoder

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
import time
from typing import Optional

from pipeline import Frame, FrameSource, SourceSpec
from ..pipeline import Frame, FrameSource, SourceSpec


class SyntheticSource(FrameSource):
Expand Down
Loading
Loading