diff --git a/skills/ncore-data-conversion/SKILL.md b/skills/ncore-data-conversion/SKILL.md new file mode 100644 index 00000000..2a7d01c4 --- /dev/null +++ b/skills/ncore-data-conversion/SKILL.md @@ -0,0 +1,343 @@ +--- +name: ncore-data-conversion +description: >- + Use when getting sensor data into an NCore V4 store: running a built-in + converter (PAI, Waymo, COLMAP/ScanNet++, KITTI, nuScenes, Argoverse 2), + adapting the nearest built-in converter for a dataset NCore does not ship + one for, or checking a converted store before something downstream reads + it. Covers V4 conventions for poses, camera and lidar models, cuboids and + timestamps. Do NOT use to train reconstructions or to extract per-object + 3D assets. +license: Apache-2.0 +metadata: + author: NVIDIA NCore + version: "0.3.0" + tags: ncore, data-conversion, sensors, v4, zarr, itar + upstream: https://github.com/NVIDIA/ncore +--- + + + +# NCore V4 data conversion + +Get sensor data into an **NCore V4** store, and check it before anything +downstream consumes it. A V4 store is a general-purpose sensor-data format: +NuRec is one consumer, so are `ncore_vis`, gsplat-based research code, and +your own tooling. Nothing here assumes a NuRec pipeline. + +Three jobs, in order of preference: run a built-in converter; adapt the +nearest built-in converter when none fits; check the result either way. + +This skill ships inside the ncore repository, so paths like +`docs/data/conventions.rst` refer to files you already have. Prefer them over +the rendered site, since they match your checkout. + +## Prerequisites + +- **Bazel**, via `bazelisk`. Not optional: converters and inspection tools are + Bazel targets. The `nvidia-ncore` wheel packages only the `ncore` library and + declares no console entry points, so `ncore_vis` and friends are reachable + only from a checkout. See `CONTRIBUTING.md`. +- **A GitHub PAT with `read:packages` in `~/.netrc`**, per `CONTRIBUTING.md`. + External test-data and docs archives are fetched from + `maven.pkg.github.com`; without it the first `bazel build` fails in a way + that looks unrelated to what you were doing. +- **An NVIDIA GPU for the PAI converter.** Its camera path decodes H.264 on the + GPU through PyNvVideoCodec, imported at module scope, so the binary needs the + GPU stack even to start. The other five converters have no such requirement. +- **For PAI on a gated clip**: `HF_TOKEN` set, on an account that has accepted + the `nvidia/PhysicalAI-Autonomous-Vehicles` license. A valid token without + accepted access fails as an opaque HTTP error, not a helpful message. + +## Choose a converter + +Six converters under `tools/data_converter/`. Pick by dataset; if none matches, +see [references/adapting-a-converter.md](references/adapting-a-converter.md). + +| Dataset | Target | Subcommand | +|---|---|---| +| PAI clip (HuggingFace or local) | `//tools/data_converter/pai:convert` | `pai-stream-v4`, `pai-v4` | +| Waymo `.tfrecord` | `//tools/data_converter/waymo:convert` | `waymo-v4` | +| COLMAP scene, ScanNet++ DSLR | `//tools/data_converter/colmap:convert` | `colmap-v4`, `scannetpp-v4` | +| KITTI raw | `//tools/data_converter/kitti:convert` | `kitti-v4` | +| nuScenes | `//tools/data_converter/nuscenes:convert` | `nuscenes-v4` | +| Argoverse 2 Sensor | `//tools/data_converter/argoverse2:convert` | `argoverse2-v4` | + +Flag documentation is split between two places, and which one is fuller +depends on the converter. Prefer `--help` over both: + +| Converter | Fuller flag reference | +|---|---| +| Waymo, COLMAP, PAI | `docs/conversions//.rst` | +| KITTI, nuScenes, Argoverse 2 | `tools/data_converter//README.md` (the `.rst` defers to it) | + +Behavior worth knowing before you pick or adapt one: + +- **nuScenes** treats cameras as global shutter: one capture timestamp per + image, no rolling-shutter metadata, so frame start equals frame end. Its + lidar model is derived from the data by default (`--lidar-model-source`, + default `empirical`) from 1085 native firing columns, upsampled 4x. Cuboid + positions are already geometric centers. Spin direction is hard-coded `cw`. +- **Argoverse 2** aggregates two stacked units that spin oppositely in their + own frames, so `spinning_direction` is derived per unit and can be `ccw`. + `--lidar-model-source none` stores raw ray bundles with no structured model + and no lidar intrinsics. +- **COLMAP** timestamps are synthetic, one second per frame. It never infers a + frame rate from filenames. +- **PAI** uses Hyperion 8 sensor ids (`camera_front_wide_120fov`, + `lidar_top_360fov`, and so on) and needs the GPU noted above. + +## Run a converter + +Each `convert` binary takes shared base flags (`--root-dir`, `--output-dir`, +`--verbose`, and `--no-cameras` / `--camera-id` / `--no-lidars` / `--lidar-id` +/ `--no-radars` / `--radar-id` to restrict which sensors are processed), +followed by a subcommand carrying format-specific flags. The sensor-restriction +flags are for narrowing a run during development, not for making an unsupported +host work. + +Common subcommand flags, though not every subcommand offers every one: + +| Flag | Default | Meaning | +|---|---|---| +| `--store-type {itar,directory}` | `itar` | `itar` is compact and self-contained; `directory` is easier to inspect | +| `--profile {default,separate-sensors,separate-all}` | `separate-sensors` | Component group layout. Every converter defaults to `separate-sensors`, including Waymo | +| `--sequence-meta` / `--no-sequence-meta` | enabled | Writes `//.json`, the file that expands to every shard. `scannetpp-v4` always writes it and offers no switch | +| `--world-global-mode` | `none` | Only `waymo-v4` and `colmap-v4` take it; `localized` is Waymo-only. The other four converters store a `world` to `world_global` pose unconditionally | + +Assign real values before running; never paste angle-bracket placeholders into +a shell, which reads them as redirections. + +Waymo: + +```bash +OUT_DIR= # FILL IN +TFRECORD_DIR= # FILL IN +: "${OUT_DIR:?set OUT_DIR}" "${TFRECORD_DIR:?set TFRECORD_DIR}" + +bazel run //tools/data_converter/waymo:convert -- \ + --root-dir "$TFRECORD_DIR" \ + --output-dir "$OUT_DIR" \ + waymo-v4 +``` + +nuScenes, one scene: + +```bash +NUSCENES_ROOT= # FILL IN +OUT_DIR= # FILL IN +: "${NUSCENES_ROOT:?set NUSCENES_ROOT}" "${OUT_DIR:?set OUT_DIR}" + +bazel run //tools/data_converter/nuscenes:convert -- \ + --root-dir "$NUSCENES_ROOT" \ + --output-dir "$OUT_DIR" \ + nuscenes-v4 \ + --version v1.0-trainval \ + --scene-name scene-0001 # or --scene-token; omit for all scenes +``` + +PAI, streaming, which needs no `--root-dir`. The converter reads the token from +`HF_TOKEN`; `--hf-token` exists on `pai-stream-v4` and defaults to that +variable, so do not pass it explicitly. Doing so puts the secret in the +process's argv, readable from `/proc//cmdline` for the whole run, and in +your shell history. + +```bash +OUT_DIR= # FILL IN +CLIP_ID= # FILL IN +: "${OUT_DIR:?set OUT_DIR}" "${CLIP_ID:?set CLIP_ID}" + +bazel run //tools/data_converter/pai:convert -- \ + --output-dir "$OUT_DIR" \ + --camera-id camera_front_wide_120fov \ + pai-stream-v4 \ + --clip-id "$CLIP_ID" +``` + +Different sequences coexist safely, since converters write under +`//`. Re-running the *same* sequence to an existing +`itar` store truncates it before producing anything, so write to a fresh path +or back up first. The `directory` store type does not have this behavior. + +## Check the result + +There is no validator, and no single check establishes correctness. What +follows is a progression from cheap to thorough. + +Reading a store back is itself a check: opening a sequence enforces version +agreement, rejects duplicate component groups, and verifies that sequence ids +and timestamp intervals agree across shards. + +Under the default `separate-sensors` profile a sequence is split into one shard +per sensor plus a default shard holding poses, intrinsics, masks and cuboids. +Only the sequence `.json` expands to all of them. A single bare `.zarr` or +`.zarr.itar` path works only if that one store holds every component you need; +otherwise pass the JSON, or repeat `--component-group` once per shard. + +**1. Dump the metadata.** Fastest way to see what actually landed: + +```bash +SEQ_META= # FILL IN: path to the sequence .json +OUT_DIR= # FILL IN +: "${SEQ_META:?set SEQ_META}" "${OUT_DIR:?set OUT_DIR}" + +bazel run //tools:ncore_sequence_meta -- \ + --output-dir="$OUT_DIR" \ + v4 --component-group="$SEQ_META" +``` + +**2. Look at it.** `ncore_vis` makes wrong extrinsics and rotated cameras +obvious. It defaults to `--host=0.0.0.0` and its Viser server is +unauthenticated, which on a shared or cloud host exposes your sensor imagery +and geometry to the network. Bind loopback unless you mean otherwise. + +```bash +bazel run //tools/ncore_vis -- \ + --host=127.0.0.1 \ + v4 --component-group="$SEQ_META" +``` + +See `docs/tools/ncore_vis.rst` for the viewer's own options. + +**3. Project lidar onto camera**, where the sequence has both. This exercises +extrinsics, intrinsics, per-ray timestamps and pose density together, which is +what makes it useful and also what makes a failure ambiguous. See +[projections that do not line up](#when-projections-do-not-line-up) for +separating the causes. + +```bash +SOURCE_ID= # FILL IN, e.g. lidar_top +CAMERA_ID= # FILL IN, e.g. camera_front +: "${SOURCE_ID:?set SOURCE_ID}" "${CAMERA_ID:?set CAMERA_ID}" + +bazel run //tools:ncore_project_pc_to_img -- \ + --source-id="$SOURCE_ID" \ + --camera-id="$CAMERA_ID" \ + --output-dir="$OUT_DIR" \ + --device=cpu \ + v4 --component-group="$SEQ_META" +``` + +Use ids your store actually contains. No converter emits `lidar00` or +`camera01`; KITTI and nuScenes write `lidar_top`, nuScenes cameras are +`camera_front`, `camera_back_left` and so on, while PAI writes +`lidar_top_360fov` and `camera_rear_left_70fov`. Read them off +`ncore_sequence_meta` if unsure. + +Two defaults to keep in mind: `--no-lidar-model` means the projection uses +stored ray directions and does not exercise the structured lidar model, and +`--no-external-distortion` means a camera carrying external distortion can look +misaligned until you pass `--external-distortion`. `--device` defaults to +`cuda` here, but to `cpu` in `ncore_evaluate_lidar_model`. + +## When projections do not line up + +Projected lidar points that do not sit on image features are the common +failure. It is usually inherited from the source data rather than introduced by +conversion, and three independent causes produce the same picture: extrinsics, +intrinsics, and timestamps. The appearance alone does not tell you which, so +separate them by choosing what to project. The lists below are common cases, +not an exhaustive set. + +### Separate calibration from timing: project at standstill + +Find frames where the rig is not moving and project only those, using +`--start-frame` and `--stop-frame`. Without ego motion, timestamp errors have +no geometric effect, so any misalignment that survives is calibration. + +- Unaligned at standstill: extrinsics or intrinsics. +- Aligned at standstill, unaligned once moving: timing. + +`--pose {start,end,mean,rolling-shutter}` is the second lever. If switching +between `start` and `end` visibly changes the projection, the frame interval is +doing real work and its endpoints are worth checking. + +### Unaligned at standstill + +- The whole cloud is offset or rotated rigidly against otherwise consistent + image content. Look at the lidar-to-rig static extrinsic, and at the source + data's own lidar frame convention. The source is the more common cause, and + conversion cannot correct it. +- Alignment is good near the image center and degrades toward the edges. + Intrinsics: distortion coefficients, or the FTheta principal-point + convention, which is stored pixel-centered with the runtime adding half a + pixel. +- Model-predicted ray directions disagree with the stored native ones. Measure + it rather than guessing: `//tools:ncore_evaluate_lidar_model` reports angular + error and any systematic azimuth shift, and + `docs/tools/lidar_model_eval.rst` gives expected magnitudes. Remember the + projector does not use the model unless you pass `--lidar-model`. + +### Aligned at standstill, unaligned in motion + +- Misalignment grows with vehicle speed. Per-ray timestamps or the frame + interval. A sweep midpoint stored as the frame start shifts every ray by half + a sweep; `frame_timestamps_us[0]` is the real start of frame. +- Misalignment grows with rotation rate rather than speed, or points appear + drawn out along the motion direction. The pose trajectory is too sparse to + interpolate through. Combine every available pose source and densify from + IMU or odometry. Adding more cameras does not help: synchronized cameras + share a trigger time, so N cameras and M frames still dedupe to about M + unique waypoints. +- A rolling-shutter camera shows internal inconsistency rather than a uniform + offset. Either one global timestamp is being stored for the whole frame, or + `ShutterType.GLOBAL` is set on a rolling sensor. Store a real + `[start, end]` interval per camera and the matching enum. + +### The writer rejected the data + +- `Frame start timestamp must be contained in the sequence time range`. The + frame falls outside the half-open sequence interval. Trim the frame or widen + the interval; the writer does not clamp. +- An assertion that row elevations must be sorted descending. Sort + `row_elevations_rad` descending and separate duplicates by about 1e-6 rad. +- Dynamic poses must cover the full sequence time range. The pose track starts + after the sequence start or ends before its last microsecond. Extend it, or + pass `require_sequence_time_coverage=False` if per-sensor pose tracks are + legitimately ragged, as the COLMAP converter does. + +### Other symptoms + +- Cuboids sit half-buried in the ground. A bottom-centered origin was stored as + a centroid. Add `dim_z / 2`, after checking the sign: a ground-resting box + gives `mean(z) - mean(h)/2 approx. -mean(h)/2` when bottom-centered and + approx. 0 when already centered. +- The ego vehicle is reconstructed as part of the scene. No ego mask. Store a + per-camera binary mask through `MasksComponent.store_camera_masks`. +- Most rays in a Livox-style scan have range 0. A structured spinning model was + forced onto a non-repetitive scan pattern. Keep `LidarSensorComponent` and + pass `model_element=None`. +- `RuntimeError: double != float` in a downstream consumer. That consumer wants + float32. V4 stores either, so cast to what the failing consumer expects + rather than assuming the format requires it. +- Sub-centimetre detail is lost at scene scale. Large global coordinates + (UTM, ECEF) narrowed to float32. Re-reference poses to the first pose, but + only while preserving `T_world_world_global`; otherwise keep float64. See + `docs/data/conventions.rst` on the local `world` frame. + +## Scope and constraints + +- Conversion is one way. No tool reconstructs the original dataset from a V4 + store, so keep the source. +- Cuboid dimensions come from source annotations. Nothing here estimates them. +- Checks here are inspection, not verification. They catch common structural + and calibration errors; they do not establish every schema invariant. + +## Reference + +- V4 facts not covered by the repo docs: + [references/v4-invariants.md](references/v4-invariants.md) +- Writing a converter for an unsupported dataset: + [references/adapting-a-converter.md](references/adapting-a-converter.md) +- Format specification: `docs/data/conventions.rst`, `docs/data/formats.rst` +- Camera and lidar models: `docs/data/sensor_models.rst` +- Store types and performance: `docs/data/storage_and_access.rst` +- Tools: `docs/tools/` (`ncore_vis`, `data_vis`, `ncore_sequence_meta`, + `lidar_model_eval`) +- Per-converter guides: `docs/conversions/`, `tools/data_converter/*/README.md` +- Contributing, build setup and repo gates: `CONTRIBUTING.md` +- Published docs: +- Library only, without the tools: `pip install nvidia-ncore` diff --git a/skills/ncore-data-conversion/evals/evals.json b/skills/ncore-data-conversion/evals/evals.json new file mode 100644 index 00000000..034bf032 --- /dev/null +++ b/skills/ncore-data-conversion/evals/evals.json @@ -0,0 +1,79 @@ +[ + { + "id": "ncore-data-conversion-001", + "question": "I have a directory of Waymo Open Dataset .tfrecord files and I need them in NCore V4 format. How do I do that?", + "expected_skill": "ncore-data-conversion", + "expected_script": null, + "ground_truth": "The agent used the skill to route the user to the built-in Waymo converter, gave the bazel run //tools/data_converter/waymo:convert invocation with the waymo-v4 subcommand and the base --root-dir/--output-dir flags, and did not suggest writing a converter.", + "expected_behavior": [ + "The agent identified that NCore ships a built-in Waymo converter", + "The agent gave the correct Bazel target and the waymo-v4 subcommand", + "The agent noted that re-running the same sequence over an existing itar store truncates it, while different sequences coexist under //", + "The agent did not propose authoring a new converter or a template" + ] + }, + { + "id": "ncore-data-conversion-002", + "question": "NCore doesn't ship a converter for my dataset. It's per-frame files with a spinning LiDAR and a few calibrated cameras. What's the right way to get it into V4?", + "expected_skill": "ncore-data-conversion", + "expected_script": null, + "ground_truth": "The agent used the skill to explain that no built-in converter exists and that the correct approach is to adapt the nearest in-tree converter, chosen by input layout and sensor model rather than by name, working on a feature branch and checking the resulting store.", + "expected_behavior": [ + "The agent said no built-in converter exists for the dataset", + "The agent directed the user to adapt the nearest in-tree converter, naming a plausible candidate such as nuScenes or Argoverse 2 for a per-frame spinning-lidar layout", + "The agent pointed at that converter's own source as the reference implementation", + "The agent did not offer a generic scaffold or template to copy" + ] + }, + { + "id": "ncore-data-conversion-003", + "question": "I converted a sequence, and when I project the LiDAR points onto the camera images they don't land on the right things. How do I work out what's wrong?", + "expected_skill": "ncore-data-conversion", + "expected_script": null, + "ground_truth": "The agent used the skill to separate the possible causes rather than guessing among them. It explained that extrinsics, intrinsics and timestamps all produce misalignment, and that projecting frames where the rig is stationary removes timing as a variable, since timestamp errors have no geometric effect without ego motion. It noted that the cause is often in the source data rather than introduced by conversion.", + "expected_behavior": [ + "The agent proposed projecting frames at standstill to separate calibration faults from timing faults, rather than guessing at a single cause", + "The agent named extrinsics, intrinsics and timestamps as distinct causes that produce the same appearance", + "The agent did not reference a validate.py or any validator, which do not exist in the repository", + "The agent used ncore_vis or ncore_sequence_meta to inspect the store, and ncore_project_pc_to_img only where the sequence has both a camera and a lidar", + "The agent did not claim that conversion can correct a calibration error present in the source dataset" + ] + }, + { + "id": "ncore-data-conversion-004", + "question": "My lidar is a Livox-style sensor with a non-repetitive scan pattern, so it has no fixed row and column grid. How should I store its returns in V4?", + "expected_skill": "ncore-data-conversion", + "expected_script": null, + "ground_truth": "The agent used the skill to explain that a structured spinning lidar model is optional: the sensor should still use LidarSensorComponent, passing model_element=None so that raw ray bundles with per-ray timestamps are stored without a row and column model.", + "expected_behavior": [ + "The agent said LidarSensorComponent is still the right component", + "The agent said to pass model_element=None rather than forcing a structured spinning model", + "The agent noted that per-ray timestamps are preserved without a structured model", + "The agent did not propose PointCloudsComponent as the primary answer solely on the grounds that timestamps would otherwise be lost" + ] + }, + { + "id": "ncore-data-conversion-005", + "question": "I already have my scene in NCore V4 format and I want to train a neural radiance field reconstruction. What learning rate and batch size should I use, and how do I launch the training?", + "expected_skill": null, + "expected_script": null, + "ground_truth": "The agent correctly identified this as a reconstruction training task rather than a data conversion task, and declined to answer from this skill, since the data is already in V4 format and training is a downstream consumer's concern.", + "expected_behavior": [ + "The agent did not invoke the skill for this training-related question", + "The agent stated that training and rendering are outside this skill's scope", + "The agent did not invent NCore flags or converter options to answer it" + ] + }, + { + "id": "ncore-data-conversion-006", + "question": "I have a converted V4 sequence with cars and pedestrians in it. How do I extract each object as a separate 3D asset I can drop into a simulator?", + "expected_skill": null, + "expected_script": null, + "ground_truth": "The agent correctly identified per-object 3D asset extraction as outside this skill's scope, which covers getting data into V4 and checking it, not deriving assets from a reconstructed scene.", + "expected_behavior": [ + "The agent did not invoke the skill for this asset-extraction question", + "The agent stated that extracting per-object 3D assets is outside this skill's scope", + "The agent did not invent an NCore tool or converter flag that performs asset extraction" + ] + } +] diff --git a/skills/ncore-data-conversion/references/adapting-a-converter.md b/skills/ncore-data-conversion/references/adapting-a-converter.md new file mode 100644 index 00000000..e4a14015 --- /dev/null +++ b/skills/ncore-data-conversion/references/adapting-a-converter.md @@ -0,0 +1,90 @@ + + +# Adapting a converter for an unsupported dataset + +NCore does not ship a converter for every dataset; PandaSet is a common +example. There is no scaffold or template, and no porting guide elsewhere in +the repository. The in-tree converters are the reference implementations, so +adapt the nearest one rather than starting from scratch. + +## 1. Pick the nearest by shape, not by name + +Match on input layout and sensor model: + +| Your data looks like | Start from | +|---|---| +| Per-frame files, spinning lidar, calibrated cameras, per-frame ego poses | nuScenes or Argoverse 2 | +| A single aggregated sweep from stacked lidar units | Argoverse 2 | +| Poses and images from structure-from-motion, no lidar | COLMAP | +| Sequential logs with an OXTS-style GPS/IMU stream | KITTI | +| Record-oriented container files | Waymo | +| Streamed from a remote object store rather than a local tree | PAI | + +## 2. Read it + +For your chosen converter: + +- `tools/data_converter//converter.py` is the substance in every case. +- `tools/data_converter//main.py` exists for KITTI, nuScenes and + Argoverse 2, and is a three-line registration shim. PAI, Waymo and COLMAP + are driven from `converter.py` directly. +- `tools/data_converter/cli.py` defines the shared base flags for all six. +- `ncore/impl/data_converter/base.py` holds the config dataclasses, including + the `--root-dir` requirement that file-based converters inherit and + streaming ones do not. +- The matching page under `docs/conversions/`, plus + `tools/data_converter//README.md`. + +Only KITTI, nuScenes and Argoverse 2 ship converter tests, and those are +dataset-gated: they carry the `manual` Bazel tag and skip without their +dataset environment variable, so CI never runs them. Argoverse 2 additionally +ships a data-free unit test for its lidar-model derivation which does run in +CI, and which is the closest thing to an executable example of the geometry +code. + +## 3. Keep the structure, change the edges + +What varies between converters is narrower than it looks. In practice you are +replacing: + +- **The reader.** How frames, calibration and annotations are enumerated from + the source layout. +- **Sensor-model construction.** Which camera model parameter class fits, and + whether a structured lidar model can be derived at all. +- **Timestamp derivation.** Where per-frame intervals and per-ray times come + from, or how they are synthesised when the source has none. + +What should not vary is the writer sequence, the component-group profile +handling, and the sequence-meta output. If you find yourself changing those, +check whether the dataset really is closest to the converter you picked. + +Two decisions worth making early: + +- **A structured lidar model is optional.** If the scan pattern is not a + repeating row and column grid, pass `model_element=None` and store raw ray + bundles with per-ray timestamps. Forcing a spinning model onto a + non-repetitive pattern produces rays at range 0. Argoverse 2's + `--lidar-model-source none` is the in-tree precedent. +- **Derive `spinning_direction` from the data** rather than assuming, if the + dataset has more than one lidar unit or you have not verified it. nuScenes + hard-codes `cw`; Argoverse 2 derives per unit because its two stacked units + spin oppositely in their own frames. + +## 4. Work on a branch, then check the gates + +Adapt on a feature branch in your own fork. Before proposing anything: + +```bash +bazel run //:format.check +bazel test //tools/data_converter/... # dataset-gated tests will skip +``` + +Then check the store itself, which is the part that applies regardless of +which converter you started from. See the "Check the result" section of +`SKILL.md`. + +`CONTRIBUTING.md` covers the rest of the repository's expectations: conventional +commits, GPG signing, SPDX headers, rebase-only history. diff --git a/skills/ncore-data-conversion/references/v4-invariants.md b/skills/ncore-data-conversion/references/v4-invariants.md new file mode 100644 index 00000000..a22f1de1 --- /dev/null +++ b/skills/ncore-data-conversion/references/v4-invariants.md @@ -0,0 +1,137 @@ + + +# V4 invariants not covered by the repo docs + +Enough to read a failure and to know what an adapted converter must satisfy. +Everything here is either absent from `docs/` or stated only in code. For +anything else, prefer the specification: `docs/data/conventions.rst` for frames +and transformations, `docs/data/formats.rst` for the component layout, +`docs/data/sensor_models.rst` for camera and lidar models. + +## Time + +- The sequence interval is **half-open**, `[start, stop)`, in microseconds. The + inclusive last timestamp is therefore `stop - 1`. `docs/` gives the field + name but not the semantics; see `HalfClosedInterval` in + `ncore/impl/common/transformations.py`. +- `store_dynamic_pose(..., require_sequence_time_coverage=True)` is the + default, and asserts the first timestamp equals the sequence start and the + last equals `stop - 1`. Setting it `False` relaxes only that endpoint + exactness. Containment, strict increase, and a minimum of two poses are + enforced either way. The COLMAP converter passes `False` because per-camera + pose tracks are ragged. +- Frames are stored as `[start-of-frame, end-of-frame]` per frame, and keyed by + the end-of-frame timestamp. The format does not define these as exposure or + sweep bounds; a global-shutter camera legitimately stores `start == end`, as + COLMAP does. +- Out-of-range frames are **not clamped**. The writer asserts both endpoints + lie inside the sequence interval and fails with `Frame start timestamp must + be contained in the sequence time range`. Trim frames or widen the interval + yourself. Converters generally skip such frames rather than adjusting them. + +## Poses + +- Pose dtype is not fixed by the format. Both `store_static_pose` and + `store_dynamic_pose` accept and round-trip float32 and float64. If a + downstream consumer raises `RuntimeError: double != float`, match that + consumer rather than assuming V4 requires float32. Note the singular getters + return float64 regardless of what was stored, while the plural generators + honour the stored dtype. +- Pose **density** drives motion compensation, independently of the coverage + rule above. Synchronized cameras share one trigger time, so N cameras and M + frames dedupe to about M unique waypoints. Adding cameras does not densify + the trajectory. +- Re-referencing poses to the first pose is a precision measure, not a V4 rule. + KITTI, PAI, nuScenes and Argoverse 2 do it unconditionally; Waymo only under + `--world-global-mode localized`; COLMAP never. The rationale and the + `T_world_world_global` edge are in `docs/data/conventions.rst`. +- The ego edge is typical, not required. The COLMAP converter stores + `source_frame_id=` against the camera's own reference frame. + +## Cameras + +- FTheta stores its principal point in **pixel-center** convention, where an + index names the center of a pixel. The runtime adds half a pixel to reach the + corner-origin image convention that the rest of the library uses, and + subtracts it again on the way out. So for a 1920x1080 image whose optical + center is the image center, the **stored** value is `[959.5, 539.5]` and the + **runtime** value is `[960.0, 540.0]`. Getting this backwards is a + half-pixel error in exactly the direction the convention exists to prevent. + See `docs/data/sensor_models.rst` for the convention and + `ncore/impl/sensors/camera.py` for the offset. +- No camera model carries a per-row shutter-delay field. Rolling-shutter timing + comes entirely from the frame's `[start, end]` interval, with per-row times + interpolated from the row index. PAI's source data has a `shutter_delay_us`, + but the converter consumes it to derive the frame start and does not persist + it. + +## Lidar + +- `model_element` is optional on `LidarSensorComponent.store_frame`: pass + `None` for a sensor without a row and column grid, and per-ray `timestamp_us` + still works. It has no default, so it must be passed explicitly. +- `row_elevations_rad` must be **strictly decreasing**, row 0 highest. The + check is expressed as a clockwise relative-angle comparison, so the failure + message talks about descending order. +- `spinning_direction` controls azimuth ordering and sweep interpretation only. + Ray `z` is `sin(elevation)`, so a wrong value cannot flip the cloud + vertically. Derive it per unit from the data where units may differ, as + Argoverse 2 does. +- Per-column firing time uses two deliberately different conventions. The + runtime lidar model maps a column onto the **closed** frame interval, so + column `N-1` lands exactly on the frame end and the divisor is `n_columns - 1`. + The converter helper treats a revolution as **half-open**, with the frame end + belonging to the next frame's column 0, so its divisor is `n_columns`. Each + is correct for its consumer. Follow whichever the code you are working in + already uses, and do not port the expression between them. + +## Components + +- `MasksComponent` stores one untimestamped named mask set per camera and + cannot represent anything varying per frame. For per-frame typed labels such + as depth or semantic segmentation, prefer `CameraLabelsComponent`, which is + independently timestamped. That is a recommendation, not the only route: the + COLMAP converter carries per-image masks as per-frame `CameraSensor` + `generic_data["mask"]` and writes an empty static mask set. +- `PointCloudsComponent` can carry per-point `timestamp_us` as a + schema-declared `INVARIANT` attribute, so it does not lose timing. Prefer + `LidarSensorComponent` for ray-bundle semantics and automatic per-ray motion + compensation, not because PointClouds cannot hold timestamps. +- Cuboid `BBox3.centroid` is the **geometric center**, with `dim` a symmetric + extent about it and `rot` XYZ Euler angles. If a source is bottom-centered, + add `dim_z / 2`. Check the sign first: a ground-resting box gives + `mean(z) - mean(h)/2 approx. -mean(h)/2` when bottom-centered and approx. 0 + when already centered. +- IMU is not a first-class component. Raw IMU samples are not SE(3) poses: + integrate them to estimate or densify the ego trajectory, and keep the raw + GPS/IMU stream in the Poses component's `generic_data`, which the format + explicitly supports and KITTI already uses for OXTS. + +## Motion compensation + +Not documented outside docstrings; `MotionCompensator` lives in +`ncore/impl/common/transformations.py` and is not exported from a public +package. + +`motion_decompensate_points` expects `xyz_reftime` **already in the sensor +frame** at one reference timestamp, namely the timestamp the data was +compensated to. Transform world or ego-frame points into that frame first, and +pass the matching `reference_timestamp_us` and `anchor_frame_id`, where the +anchor must match the one used for compensation. Feeding world-frame points +straight in silently corrupts both range and direction rather than raising. + +## Sensor ids differ between converters + +Ids are not portable, so a `--source-id` copied from another dataset's example +will not resolve: + +| Converter | Lidar | Cameras | +|---|---|---| +| KITTI, nuScenes | `lidar_top` | nuScenes: `camera_front`, `camera_front_left`, `camera_back_left`, and so on | +| PAI | `lidar_top_360fov` | `camera_front_wide_120fov`, `camera_rear_left_70fov`, and so on | + +Note nuScenes says `back` where PAI says `rear`, and PAI's lidar carries a +field-of-view suffix. Read the actual ids off `//tools:ncore_sequence_meta`. diff --git a/skills/ncore-data-conversion/skill-card.md b/skills/ncore-data-conversion/skill-card.md new file mode 100644 index 00000000..2ec628a2 --- /dev/null +++ b/skills/ncore-data-conversion/skill-card.md @@ -0,0 +1,59 @@ + + +## Description:
+Converts sensor data into an NCore V4 store using NCore's built-in dataset converters, guides adaptation of the nearest converter when no built-in one fits, and checks the resulting store.
+ +This skill is ready for commercial/non-commercial use.
+ +## Owner +NVIDIA NCore
+ +### License/Terms of Use:
+Apache-2.0, per the [repository LICENSE](../../LICENSE).
+ +Adapted from NVIDIA-authored material in [NVIDIA/nurec-skills](https://github.com/NVIDIA/nurec-skills) at commit `5b9d287`, narrowed to the built-in converters and re-verified against this repository.
+ +## Use Case:
+Engineers converting autonomous-vehicle and scene-capture datasets (PAI, Waymo, COLMAP/ScanNet++, KITTI, nuScenes, Argoverse 2) into the NCore V4 sensor-data format, adapting an existing converter for a dataset NCore does not support, or diagnosing calibration and timing problems in a converted store.
+ +### Deployment Geography for Use:
+Global
+ +## Requirements / Dependencies:
+**Requires API Key or External Credential:** [Yes]
+**Credential Type(s):** [API key]
+ +A GitHub personal access token with `read:packages` is required to build the repository. A Hugging Face token is required only for the PAI converter, on an account that has accepted the `nvidia/PhysicalAI-Autonomous-Vehicles` dataset license. The skill instructs reading tokens from the environment rather than passing them as command-line arguments, so they are not exposed through process arguments or shell history.
+ +Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate.
+ +## Known Risks and Mitigations:
+Risk: Re-running a converter over an existing indexed-tar store truncates it before writing, so an interrupted re-run can leave neither the previous store nor a complete new one.
+Mitigation: The skill directs output to a fresh path or a backup, and notes that the directory store type does not behave this way.
+ +Risk: The `ncore_vis` viewer binds all interfaces by default and its server is unauthenticated, exposing sensor imagery and geometry on shared or cloud hosts.
+Mitigation: The skill's invocation binds loopback explicitly and states the reason.
+ +Risk: Conversion cannot correct calibration or timing errors present in the source dataset, and a plausible-looking store may still be wrong.
+Mitigation: The skill states that its checks are inspection rather than verification, and gives a procedure that separates calibration faults from timing faults instead of guessing.
+ +Risk: Review before execution, as generated commands could be incorrect for a given dataset layout.
+Mitigation: Review commands before running; the skill uses explicit fill-in variables with guard checks rather than emitting ready-to-paste paths.
+ +## Reference(s):
+- [NVIDIA NCore](https://github.com/NVIDIA/ncore)
+- [NCore documentation](https://nvidia.github.io/ncore/)
+- [V4 invariants](references/v4-invariants.md)
+- [Adapting a converter](references/adapting-a-converter.md)
+ +## Skill Output:
+**Output Type(s):** [Analysis, Shell commands, Configuration instructions]
+**Output Format:** [Markdown with inline bash code blocks]
+**Output Parameters:** [1D]
+**Other Properties Related to Output:** [None]
+ +## Evaluation Tasks:
+6 evaluation tasks (4 positive, 2 negative) defined in [evals/evals.json](evals/evals.json).