Rebase to introduce latest bugfixes - #6
Open
7jep7 wants to merge 815 commits into
Open
Conversation
…#3606) * fix(gr00t): fix gr00t config dataclass init TypeError * fix(groot): guard strict config decorator without transformers for passing CI --------- Co-authored-by: Pepijn <138571049+pkooij@users.noreply.github.com>
* fix(deps): cap placo below 0.9.16 and harden kinematics import placo 0.9.16 links against liburdfdom_sensor.so.4, which is unavailable on Ubuntu 24.04 (noble ships urdfdom 3.x). Importing placo on that base crashes with: ImportError: liburdfdom_sensor.so.4.0: cannot open shared object file This broke nightly Latest Deps tests (CPU and GPU) when the lockfile upgrade picked placo 0.9.16, since lerobot.model.kinematics unconditionally imports placo when _placo_available is true, and that check (importlib.util.find_spec) cannot detect dlopen failures of transitive shared libraries — so unrelated subsystems (RL actor, gym_manipulator) became unimportable. Two changes: 1. Pin placo to <0.9.16 in pyproject.toml + regenerate uv.lock (0.9.16 → 0.9.15). Short-term unblock for nightly CI until system urdfdom 4.x is broadly available. 2. Harden the import guard in src/lerobot/model/kinematics.py: wrap 'import placo' in try/except ImportError so a missing transitive .so no longer crashes module import. RobotKinematics instantiation now raises an informative ImportError citing the underlying dlopen failure via _raise_if_placo_unusable(). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(kinematics): hoist _placo_runtime_error to module scope for mypy Mypy walks the TYPE_CHECKING branch in which the runtime else-block is not executed, so _placo_runtime_error was only defined at runtime and mypy reported 'Name "_placo_runtime_error" is not defined' on the three references inside _raise_if_placo_unusable. Declare the symbol unconditionally at module scope with a default of None; the runtime import-failure branch still assigns to it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style(kinematics): drop verbose comments around placo import guard Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #3145 added YAML support for policy.path but left two bugs: 1. extract_path_fields_from_config only deleted config_data[field] when no sibling overrides existed. With siblings, the dict stayed in place and draccus crashed decoding it as PreTrainedConfig (no 'type' key). Sibling overrides go into _config_yaml_overrides and are applied later by from_pretrained(), so the field can always be removed. 2. wrap() updated config_path_cli to the cleaned temp file path but never propagated it to the draccus.parse fallback branch. cli_args still contained --config_path=<original>, so draccus read the original YAML with path: still present. Tests passed because they (a) called extract_path_fields_from_config directly and (b) included type: alongside path: in the YAML, sidestepping both bugs. Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
…5.4.0, <5.6.0 (#3652) Co-authored-by: Steven Palma <imstevenpmwork@ieee.org>
* feat(rewards): add TOPReward reward model * refactor(rewards): clean up TOPReward processor/model * fix(rewards/topreward): add missing input keys mm_token_type_ids * fix(rewards/topreward): fix pyproject extra typo and simplify processor (#3653) Add lerobot[topreward] extra to all in pyproject.toml, drop the redundant labels arg in scoring, and collapse the dead-branch shape check in the encoder processor. * optmize topreward input processing (#3660) --------- Co-authored-by: Cole <91766445+jcoleharrison@users.noreply.github.com> Co-authored-by: Haoming Song <haomingsong24@gmail.com>
* add molmoact2 policy * add apache headers to molmoact2 files * simplify molmoact2 package imports * align molmoact2 feature validation with eo pattern * remove molmoact2 processor override from factory * guard molmoact2 transformers imports * guard molmoact2 processor transformers import * add scipy dependency to molmoact2 extra * use a single molmoact2 action queue * move molmoact2 config logic into config * fix molmoact2 hf image key resolution * load molmoact2 without remote code * lazy import molmoact2 scipy * format molmoact2 files * skip molmoact2 tests without optional deps * fix molmoact2 pre-commit checks * validate molmoact2 gripper range
* feat/add ROBOMETER reward model * feat(rewards): add Robometer offline progress labeling script * fix(rewards/robometer): add missing input keys mm_token_type_ids * chore(rewards/robometer): default to lerobot/Robometer-4b model * doc(rewards/robometer): update citation and original github link * feat(rewards/robometer): add image key argument to compute Robometer progress
…t Hub org visibility settings (#3713)
#3711) * fix(train): enable relative action overrides for pretrained processors Keep pretrained processor pipelines when use_relative_actions is enabled and apply relative/absolute action processor settings through overrides. Rename the relative action processor registry key to relative_actions_processor. * fix(config): reject rename_map without pretrained checkpoint Fail fast when rename_map is set during fresh initialization, since fresh configs derive feature names from the current dataset and no rename is applied. --------- Co-authored-by: Pepijn <138571049+pkooij@users.noreply.github.com>
* first text draft (no images) * simplified docs * fix formatting * add youtube video * add a tip about compatibility * fix broken link
The docs pointed at src/lerobot/datasets/v30/, which does not exist. Both scripts actually live in src/lerobot/scripts/: - convert_dataset_v21_to_v30.py - augment_dataset_quantile_stats.py Updated the four references (one python -m module path and three file-path invocations) to the correct location, matching each script's own usage docstring.
* first commit * feat(policies): add VLA-JEPA * feat(policies): add VLA-JEPA * support vla_jepa * (feat)policies: add VLA-JEPA * linting * adding deps to pyproject.toml * updating uv lock * adding guards to avoid needing transformers and diffusers for type checking and basic tests * fixing action and state dim * fix warnings with qwen processor kwargs * fixing wm_loss not propagating * adjusting obs steps, tublets size to match original implementation * some more fixes to be closer to the original implem * adding more tests to ensure good coverage * align VLA-JEPA architecture with original checkpoint - Remove stale `action_num_heads` / `action_attention_head_dim` config fields; DiT head dimensions are now always derived from the preset (DiT-B/L/test). - Add `num_target_vision_tokens` and `action_max_seq_len` config fields required by the action head's future-token embedding and positional embedding tables. - Fix default `qwen_model_name` to 2B (matches all released checkpoints). - Rename `ActionEncoder` attrs w1/w2/w3 → layer1/layer2/layer3 to match checkpoint key names; replace `nn.Sequential` decoder/state-encoder with `_MLP2` (layer1/layer2 naming). - Fix `VLAJEPAActionHead` to size ActionEncoder and StateEncoder at `inner_dim` (DiT input width) rather than `action_hidden_size` (DiT output width). - Rename `DiT.blocks` → `transformer_blocks` and `attn` → `attn1` to match checkpoint; add alternating cross/self attention (even blocks cross-attend to Qwen context, odd blocks self-attend). - Add `DiT-test` preset for unit tests. - Rewrite `ActionConditionedVideoPredictor` with explicit ViT-style blocks (`_PredictorBlock` with fused qkv) to match checkpoint structure; rename `encoder`/`norm`/`proj` → `predictor_blocks`/`predictor_norm`/`predictor_proj`. * propagate action_is_pad masking through VLA-JEPA policy pipeline Pass the `action_is_pad` tensor from the batch through to the action head so padded timesteps are excluded from the flow-matching loss. * update VLA-JEPA tests for arch changes and action_is_pad - Switch conftest to use `action_model_type="DiT-test"` now that `action_num_heads` / `action_attention_head_dim` have been removed. - Add action_head tests covering fully-padded loss (zero) and equivalence of action_is_pad=None vs all-zeros mask. - Remove obsolete `test_native_to_lerobot_wm_only` test. * add VLA-JEPA documentation Covers architecture overview, pretrained checkpoints, config reference, training/eval commands for LIBERO-10, and guidance on fine-tuning for single-camera datasets. * add one-shot script to convert ginwind/VLA-JEPA checkpoints to safetensors (will remove once migrated) * make default params more aligned with paper and pretrained models - adding possibility of freezing qwen backbone and world model - added tests for weight loading * trying out to re-init the action head to avoid pretraining dimension mismatch * allow different state dim and action dim * removing missleading future_action_window_size to just use chunk_size * lots of changes to make existing weights work, need to massively refactor the pre and post processing * refactoring into using pre and post processor * pre-commit cleanup * fixing doc defaults args Signed-off-by: Maxime Ellerbach <maxime@ellerbach.net> * adressing dtype zeros issue * adding guard for diffusers * fixing training and exal examples * trying to close success rate gap * fix qwen norm layer output libero eval is now as expected * adding instructions for different embodiement + fixing some tests * smol fix to avoid having default CPU device when training * fixing misconception about multiview / singleview handling * removing conversion script * adding licences * adding .mdx docs and shortening polivy_vla_jepa_README.md * removing useless pre-processor * cleanup * removing swish in favor of silu * adding configuration gripper index and threshold * fixing simlink --------- Signed-off-by: Maxime Ellerbach <maxime@ellerbach.net> Co-authored-by: ginwind <ginwind@mail.ustc.edu.cn>
* feat(rollout): adding legacy strategy * adding legacy to existing tests * updating docs and docstring * changing misleading docstring Signed-off-by: Maxime Ellerbach <maxime@ellerbach.net> * adding extra guard like dagged with try except finally * Potential fix for pull request finding Signed-off-by: Maxime Ellerbach <maxime@ellerbach.net> * adding reset to initial position * moving smooth teleop handover to control_utils and adding this behavior to legacy strategy * reducing duration of the handover * * renaming to episodic * changing semantics of the docstring * fixing leader - follower handover disable torque * adding optionnal config to disable handover * wiring the smooth_leader_follower_handover config * renaming config smooth_leader_to_follower_handover --------- Signed-off-by: Maxime Ellerbach <maxime@ellerbach.net>
* feat(processor): add in-memory pipeline serialization Expose processor pipeline config and tensor state without requiring temporary files, so processors can be transported, compared, or hashed directly in memory. * feat(processor): enhance DataProcessorPipeline with registry support - Added a new RegisteredLazyTensorStateStep for registry-based serialization tests. - Improved state filename handling in _get_state_filename method. - Refactored validation logic in _validate_loaded_config to simplify parameter types. - Updated tests to verify registry step functionality and ensure correct state loading. * refactor(processor): update state handling in DataProcessorPipeline - Introduced a new static method _get_state_key to derive in-memory state keys from serialized filenames. - Updated state_dict and load_state_dict methods to use suffixless state keys instead of filenames. - Adjusted related tests to reflect changes in state key handling, ensuring consistency in state management * fix(processor): update loaded_config argument description in DataProcessorPipeline - Clarified the documentation for the loaded_config parameter to indicate that it may be a non-dictionary value, enhancing understanding for future developers.
* fix(pyproject): adding ceiling bound on mujoco (<3.9.0) * chore(uv.lock): updating uv.lock * fix(linux): adding missing linux dependencies * chore(uv.lock): updating uv.lock
…d gate dataset download per node (#3768) * fix(datasets): expose a generator on EpisodeAwareSampler for distributed shuffle sync In distributed training, accelerate can only synchronize the shuffle permutation across ranks when the sampler exposes a generator attribute. EpisodeAwareSampler shuffled via the global torch RNG, so disjoint batch shards relied on every rank's global CPU RNG staying in lockstep forever; any rank-asymmetric RNG consumption (e.g. eval rollouts on the main process only) silently desynced the permutations and ranks trained on overlapping/missing samples. * fix(train): seed sampler generator and gate dataset download per node - Pass a generator seeded with cfg.seed to EpisodeAwareSampler so accelerator.prepare registers it as the synchronized RNG and the shuffle order is reproducible. - Gate the initial make_dataset call on is_local_main_process instead of is_main_process: the global main process only exists on node 0, so on every other node all local ranks were downloading the dataset and building the Arrow cache concurrently.
* chore: update readme * chore: update authors in project readme
…rics in a multi rank setup (#3773) * feat(training): bump accelerate + use reduction types for tracked metrics in a multi rank setup * chore: address feedback
Signed-off-by: Steven Palma <imstevenpmwork@ieee.org>
* feat(trim): adding optional trimming option in reencode_video * tests(trim): add triming test --------- Co-authored-by: Pepijn <138571049+pkooij@users.noreply.github.com>
…mpler (#3769) * fix(datasets): expose a generator on EpisodeAwareSampler for distributed shuffle sync In distributed training, accelerate can only synchronize the shuffle permutation across ranks when the sampler exposes a generator attribute. EpisodeAwareSampler shuffled via the global torch RNG, so disjoint batch shards relied on every rank's global CPU RNG staying in lockstep forever; any rank-asymmetric RNG consumption (e.g. eval rollouts on the main process only) silently desynced the permutations and ranks trained on overlapping/missing samples. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(train): seed sampler generator and gate dataset download per node - Pass a generator seeded with cfg.seed to EpisodeAwareSampler so accelerator.prepare registers it as the synchronized RNG and the shuffle order is reproducible. - Gate the initial make_dataset call on is_local_main_process instead of is_main_process: the global main process only exists on node 0, so on every other node all local ranks were downloading the dataset and building the Arrow cache concurrently. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(datasets): add DeterministicEpisodeAwareSampler with O(1) memory and sample-exact resume Add a sampler that never materializes frame indices: it stores only per-episode boundaries (numpy, a few bytes per episode) and maps logical positions to frame indices on the fly with searchsorted. Shuffling uses a seeded Feistel permutation over [0, num_frames) (cycle-walking to the exact domain), so the data order is a pure function of (seed, epoch): - no RNG state to synchronize across distributed ranks, - constant memory and zero epoch-boundary cost at any dataset size, - O(1) seek to any position, enabling sample-exact resume. Opt in with --deterministic_sampler=true. On resume, lerobot-train maps the checkpointed step back to (epoch, start_index) via compute_sampler_state and continues at the exact sample where the run left off (up to accelerate's even_batches padding at epoch boundaries). The shuffle is pseudo-random rather than a true uniform permutation, the standard trade-off in large-scale training loaders. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(datasets): fold deterministic mode into EpisodeAwareSampler Instead of a parallel DeterministicEpisodeAwareSampler class, extend the existing EpisodeAwareSampler with a deterministic=True mode (seeded Feistel permutation, epoch auto-advance, state_dict/load_state_dict). The default mode is behavior-identical: same torch.randperm consumption and the same generator contract accelerate synchronizes; the O(N) Python index list is replaced by O(num_episodes) boundary arrays in both modes, with `indices` kept as a back-compat property. Passing a generator together with deterministic=True is rejected, and the state/seek methods raise outside deterministic mode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(train): enable deterministic_sampler by default Deterministic data order (sample-exact resume, no cross-rank RNG sync, O(1) sampler memory) is now the default for map-style training; set deterministic_sampler=false to restore the legacy RNG-based shuffle. Streaming datasets ignore the flag (the sampler path only applies to map-style datasets), replacing the previous hard validation error so streaming configs keep working with the new default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(datasets): default EpisodeAwareSampler to deterministic mode and trim comments deterministic=True is now the class default as well as the training default; the legacy RNG path requires an explicit deterministic=False (the train script's non-deterministic branch passes it). Docstrings and inline comments slimmed down across the changed files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(sampler): drain resumed trillion-frame sampler via iter() to avoid list() prealloc list(sampler) calls PyObject_LengthHint -> __len__ (the full 10**12 epoch length) and preallocates that many slots before iterating, OOMing even though the resumed epoch only yields 3 frames. Collect through the iterator (no length hint) so the test exercises the real O(1) seek/drain instead of CPython's list growth heuristic. * fix(datasets): guard Feistel cycle-walking loop against non-convergence Replace the unbounded while True in EpisodeAwareSampler._permute with a bounded for loop capped at _MAX_CYCLE_WALK_STEPS (100) and raise RuntimeError if the cycle-walk fails to land in [0, num_frames). The loop is expected to converge in <4 steps on the chosen power-of-two domain, so the bound is a safety net that should never trip in practice but prevents a pathological infinite loop. https://claude.ai/code/session_01HQ15tFrBsHYScjGWosEv22 * fix(datasets): make deterministic-sampler resume robust to world-size changes compute_sampler_state mapped a checkpointed step back to (epoch, start_index) using the *current* num_processes, but the number of sampler positions a step consumes scales with the world size that produced it. Resuming on a different GPU count therefore landed on the wrong epoch/offset, silently re-seeing or skipping data. Record num_processes in training_step.json at checkpoint time and feed the checkpoint's value into compute_sampler_state on resume, so the data order resumes at the right position regardless of the new world size. Warn when the world size changed (the global offset is correct, but per-rank sample-exactness needs the same topology). Old checkpoints without the field fall back to the current world size. Also document compute_sampler_state's assumptions explicitly: num_processes / batch_size must match the checkpointing run, and accelerate's even_batches=True padding is mirrored by the ceil(... / num_processes) term. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com> * style: apply ruff-format to lerobot_train.py Collapse the compute_sampler_state(...) call onto one line so the ruff-format pre-commit hook passes (fixes the failing CI check). Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(datasets): use seeded torch.randperm instead of Feistel in EpisodeAwareSampler Drop the Feistel permutation (and its SplitMix64 hash / cycle-walking) in favor of a torch.randperm seeded from (seed, epoch). The deterministic mode keeps its key properties - data order is a pure function of (seed, epoch), so it reproduces on every rank with no global-RNG synchronization, and - state_dict / load_state_dict still resume sample-exactly, now by regenerating the epoch's permutation and slicing from the saved offset. Construction stays O(num_episodes) (only episode boundaries are stored, never a per-frame index list). The trade-off vs Feistel: the per-epoch shuffle is again O(num_frames) memory (the randperm tensor) and no longer O(1)-seekable, in exchange for ~30 fewer LOC and a truly uniform shuffle. Tests updated: the trillion-frame O(1) test is replaced with a boundary-storage check and a scale resume-exactness test. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(datasets): make EpisodeAwareSampler always deterministic With Feistel gone, deterministic and legacy modes were both just torch.randperm and the deterministic path strictly dominated (reproducible across ranks via the (seed, epoch) seed, no accelerate generator sync, resumable). Collapse to a single path and drop the redundant flag: - remove the `deterministic` and `generator` constructor args, `_iter_default`, and `_require_deterministic`; `set_epoch` / `state_dict` / `load_state_dict` are now unconditional - remove the `deterministic_sampler` train config field and the legacy generator branch in lerobot_train.py (non-streaming map datasets always use the sampler) - drop the now-obsolete generator/legacy tests Note: removes the `generator` kwarg from EpisodeAwareSampler (back-compat break vs main); the order is now a pure function of (seed, epoch), so no cross-rank RNG sync is needed. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(datasets): address sampler review (batch_size resume guard + docs) - Record batch_size in training_step.json alongside num_processes and feed the checkpoint's value into compute_sampler_state on resume; warn when it differs (per-rank sample-exactness needs the same batch size). - Document the set_epoch vs __iter__ auto-advance coupling on EpisodeAwareSampler (callers should rely on exactly one mechanism per run). - Note the broadened (reproducibility-breaking) sampler guard and the no-generator distributed sharding correctness in lerobot_train.py. - Add load_training_batch_size + parallel tests. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(train): download dataset once on the global main process Gate the training dataset download on the global is_main_process (download once to the shared dataset root, barrier, then every other rank reads the already-populated copy) instead of per-node is_local_main_process. LeRobotDataset skips its snapshot_download when try_load() succeeds, so no rank re-downloads. Assumes the dataset root / HF cache is on storage shared across nodes. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(datasets): trim sampler comment and drop duplicate tests Remove the verbose dataloader-guard comment and the two EpisodeAwareSampler tests that duplicated existing validation/warning coverage (no coverage loss). Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>
* update policy deployment instruction with rollout * add port and fix formatting * add more base models to generate model card * updated and extended model descriptions * fix bug * improved and extended structure * exclude the templates from config * add images and visualize dataset button * add all policies we have docs for * remove policies without the docs * new fields, improved examples
Steerable annotation pipeline (lerobot-annotate) that populates the language_persistent and language_events columns introduced in PR 1 (#3467) directly into data/chunk-*/file-*.parquet. This is PR 2 of the three-PR plan: PR 1 (Add extensive language support #3467): schema + DSL + rendering, base of this PR PR 2 (this PR): annotation pipeline writing into PR 1's columns PR 3: model with language prediction and runtime A VLM (Qwen-VL family, served on vLLM) watches each episode's video and emits grounded language annotations: subtasks, plans, memory, task rephrasings, interjections + speech, and per-camera VQA. The pipeline is built for production annotation at scale — single-camera grounding, embedded-frame inputs, a describe-then-segment grounding flow, and a deterministic full-episode coverage guarantee — informed by Scale's dense-captioning findings (representation > sampling, rules > reasoning, model capacity is the biggest lever, two-pass systems compound errors)
* feat(edit-dataset): add `concatenate_videos` opt-out to merge
When merging datasets, source mp4s are concatenated into shards capped at
`video_files_size_in_mb` (default 200 MB). This is great for dataloader
throughput but destroys per-episode (or per-source) video boundaries,
which is undesirable when you want to inspect, ship, or reuse the
individual mp4s.
Add a `concatenate_videos: bool = True` knob plumbed through
`MergeConfig` → `merge_datasets` → `aggregate_datasets` → `aggregate_videos`.
When False, each source mp4 is copied 1:1 to its own destination mp4 with
no re-muxing, so the merge preserves source video boundaries.
Usage:
lerobot-edit-dataset \
--new_repo_id user/merged \
--operation.type=merge \
--operation.repo_ids "['user/a', 'user/b']" \
--operation.concatenate_videos=false
Defaults are unchanged; the dataloader path is unaffected because the
`episodes.parquet` `from_timestamp`/`to_timestamp` index keeps working
regardless of whether each mp4 holds one or many episodes.
* feat(edit-dataset): extend concatenate opt-out to data files
Following review, add a concatenate_data flag mirroring concatenate_videos,
threaded through MergeConfig, merge_datasets, aggregate_datasets, aggregate_data
and append_or_create_parquet_file. Metadata index files still always concatenate.
Also trim the verbose docstrings and comments since the names are
self-explanatory, and extend the existing merge test to cover data files.
* fix(datasets): avoid uint8 overflow in image stats * fix(datasets): promote stats batches dynamically
* chore(robots): homogenize bi setups * feat(robots): split openarm mini into single and bi * refactor(robots): mixin for bi classes * docs: update docs
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* add third party hardware section * fix links to main * fix formatting
`send_action` keys `arm_goal_pos` as "<motor>.pos" but a Present_Position read is keyed by bare motor name, so pairing the two raised KeyError as soon as `max_relative_target` was set, making the safety cap unusable. SOFollower strips the suffix before the same lookup.
* feat(data): add recipe-driven language supervision * test(collate): expect preserved language columns * Address PR review feedback * Address Claude review feedback
…mulation, and DCP checkpoints (#4010) * feat(train): parallel training engine with FSDP2, HSDP, and DCP checkpoints Replace the FSDP1 training path with a config-owned parallel-training engine: - Topology and runtime configs (--parallelism.*, --accelerator.*): dp_replicate x dp_shard degrees select single-process, DDP (unchanged default), FSDP2, or HSDP; mixed precision, first-class gradient accumulation, and FSDP/DDP tuning knobs are mirrored as plain dataclasses that build the accelerate objects at runtime, so every run is reproducible from its train_config.json alone. Accelerate env vars are guarded against configuring the engine behind the config system's back. - Declarative policy surface: policies declare FSDP2 wrap units (_fsdp_wrap_modules) and non-forward entry points (_fsdp_forward_methods); a shared engine resolves them around accelerator.prepare(). Context-parallel fields are reserved and validated to 1. - Checkpoints: selectable --checkpoint_format (safetensors | dcp | safetensors_dcp); the sharded optimizer channel is always DCP; two-phase resume (step+RNG before prepare, DCP model/optimizer after) reshards across GPU-topology changes; lerobot-convert-dcp merges DCP shards into a distributable model.safetensors offline. - Publishing: PreTrainedPolicy.push_model_to_hub is replaced by the free publish_trained_model (model + processors + card + train config, all-ranks gather with main-rank writes); PreTrainedPolicy._save_pretrained gathers state dicts internally, removing the state_dict= threading from save_pretrained. - lerobot_train is restructured around the engine: optimizer built before the single prepare() call, deferred weight load on DCP resumes, collective save_checkpoint with no call-site rank branches, dp-world-size-based sample accounting. Breaking changes: FSDP checkpoints from lerobot <= 0.6.x are not resumable (weights stay loadable via from_pretrained; pin lerobot==0.6.x to finish old runs); the `accelerate launch --config_file` yaml flow is superseded by the config flags; training autocast is owned exclusively by --accelerator.mixed_precision (policy.dtype only casts parameters). Also fixes: reward-model hub publishing crash (TypeError on extra kwargs). Verified by ~200 new CPU tests (config round-trips, checkpoint round-trips per format, two-phase resume, publisher contracts, converter equivalence, accelerate canaries), a 5-test 4-GPU suite (FSDP2 save/resume bit-exactness, HSDP/DDP loss parity, changed-topology resume, all-ranks save_pretrained, grad-accum equivalence), and end-to-end ACT (1/4/8 GPUs) + FastWAM 6B (FSDP2 + HSDP) training runs.
…d of incorrect weighted mean (#3804) * fix(stats): use conservative bounds for quantile aggregation instead of incorrect weighted mean * docs: add --overwrite/--skip-images/--root options to augment_dataset_quantile_stats usage * fix(dataset): clarify quantile aggregation semantics * fix(augment): handle quantile stats edge cases
…ssert (#4347) Two post-merge CI failures on main, both from #4010. Benchmark Integration Tests (Libero) — `accelerate launch` exports whole groups of variables unconditionally (the five ACCELERATE_DYNAMO_* it writes default the backend to "no"), so matching on prefixes refused launches that configure nothing, contradicting the documented flow where accelerate is supported as a plain launcher. The guard now watches only the three switches that hand a subsystem to the environment. GPU Tests — `test_metrics_tracker_reduce_across_ranks_invokes_all_reduce` compared the captured reduction buffer against a CPU tensor, so the assert raised "Expected all tensors to be on the same device" wherever CUDA is available. The expected tensor is built on the buffer's device instead.
…#4323) * feat(train): add opt-in EMA of the policy weights (--ema.enable=true) Maintain an EMA shadow via diffusers' EMAModel (lazy import, no new dependency) with the reference Diffusion Policy schedule. Saves the shadow for exact resume plus a loadable pretrained_model_ema/ per checkpoint, evaluates the EMA weights during env eval, and pushes them to a sibling <repo_id>-ema repo. Fixes #4259. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(diffusion): document the --ema.enable training flag Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(tests): skip EMA training tests when accelerate/diffusers are missing * feat(train): support constant EMA decay (--ema.decay) for openpi-style policies * fix(train): gate EMA step on sync_gradients; use parallel_dims.is_sharded guard --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
TruffleHog's Lob API-key detector matches `\b((live|test)_[a-zA-Z0-9_]{35})\b`.
One EMA config test was named `test_` plus exactly 35 word characters, so it
matched, and Lob's verifier treats the 403/422 that api.lob.com returns for junk
keys as proof of a live key. That failed the Security workflow on main after
#4323 with "Found verified Lob result" and exit code 183.
Add one word to the name so the suffix is 39 characters rather than 35. No
behavior change. Note that the offending string is deliberately not spelled out
here: TruffleHog scans commit messages as well as added lines, so quoting it
would retrigger the very detector this commit is working around.
Refs #4323
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…e YAML flow (#4370) The FSDP multi-GPU test still generated an `accelerate launch --config_file` FSDP1 YAML, which exports ACCELERATE_USE_FSDP into the workers. Since the FSDP2/parallelism rewrite, `guard_against_env_interference()` hard-errors on exactly that variable, so the test failed on every rank. Its assertions were stale too: sharded runs now write DCP optimizer shards, not a gathered `optimizer_state.safetensors`. Drop the YAML generation entirely and use `accelerate launch` as the plain launcher the docs describe, with the topology coming from `--parallelism.*`. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* docs: add API documentation infrastructure
LeRobot's documentation build passes `--not_python_module`, which tells
doc-builder there is no importable Python package and disables `[[autodoc]]`
entirely. The result is that all 90+ pages are hand-written guides and there is
no generated API reference at all.
This is the machinery to change that. It deliberately contains no docstring
changes of its own — every docstring edit lives in the follow-up PR, so this
one can be reviewed as tooling and configuration alone.
**The standard.** `docs/source/writing_docstrings.mdx` is the contract: Google
section headers with Hugging Face type formatting, the machine-checked argument
line, `**Attributes**:`, doc-builder cross-references, fenced doctest examples.
It also records three behaviours that are not discoverable from the source and
were verified against a local build: `[[autodoc]]` silently skips members with
no docstring; doc-builder does not inherit docstrings from base classes, so a
registered config shim whose body is `pass` renders every field with no
description; and module-level aliases resolve to the canonical class.
**Autodoc turned on**, with two changes that are not obvious:
- `--version main` on the main-docs job. Without `--not_python_module`,
doc-builder resolves the version from `lerobot.__version__` and only maps it
to the default branch when it contains "dev". transformers relies on that;
our main carries 0.6.2. Verified by building both ways — dropping the flag
alone would publish the main docs to /lerobot/v0.6.2/ instead of
/lerobot/main/ and disable notebook building.
- `pre_command` on both jobs. doc-builder ships a mock-deps registry entry for
lerobot, so the reusable workflow takes its light-install path, which cannot
import the package. The heavy dependencies cannot be mocked either: draccus
runs `register_subclass` at import time and `processor/converters.py` calls
`functools.singledispatch.register(torch.Tensor)`, which needs a real class.
`[dataset]` is the only extra required.
Workflow triggers gain `src/**`, since the reference is now generated from
docstrings. `docs/source/api/` is excluded from the prettier hook, which reads
`[[autodoc]]` member lists as lazy paragraph continuations and joins a ten-entry
list onto one line.
Nine API reference pages, scaffolded with each module's base class.
**Doctests.** `LeRobotDocTestParser` is mandatory rather than optional here:
ruff's `docstring-code-format = true` drops the blank line before a closing
fence, after which stdlib's `_EXAMPLE_RE` reads the fence as expected output and
every example with output fails. It is written against the installed pytest
rather than copied from transformers, whose version predates pytest 9's
`import_path` signature and its own fix for the `@property` line-number bug.
`preprocess_string` also diverges: the upstream fenced-block split puts a
single-line example's code in a chunk with no `>>>` in it, so neither the CUDA
skip nor the `+IGNORE_RESULT` injection fires for it.
**Checkers.** `utils/check_docstrings.py` is the ~300-line core of the
2203-line transformers original; the `@auto_docstring` system, modular
propagation, GitPython and `checkers.py` are not ported.
`utils/check_config_docstrings.py` checks that every registered robot config
documents its port and calibration semantics.
**Gates**, all set to values that pass today: ruff `D` with per-file-ignores
per unconverted module, `interrogate` at `fail-under = 52` against a measured
52.1%, and Makefile targets wired into the quality workflow. The doctest
allowlist ships empty and the `doctest` target handles that, because the files
carrying runnable examples arrive with the docstring PR.
* ci: build the docs on Python 3.12
The shared doc-builder workflows create their virtualenv with the runner's
system Python, which is 3.10.12 on ubuntu-22.04. lerobot requires >=3.12, so
the build died during "Setup environment":
× No solution found when resolving dependencies:
╰─▶ Because the current Python version (3.10.12) does not satisfy
Python>=3.12 and lerobot==0.6.2 depends on Python>=3.12 ...
That step runs before `pre_command`, so the real install this workflow already
performs never got the chance to run. There was no fix available on the caller
side either: `env:` does not propagate into a reusable workflow, so `UV_PYTHON`
is unavailable, and `uv venv` runs in the runner workspace root rather than the
checkout, so a `.python-version` file cannot reach it. The non-light fallback
(`uv pip install "./pkg[dev]"`) fails identically, so this is not specific to
the mock-deps path — it blocks any package requiring 3.12+.
huggingface/doc-builder#808 adds a `python_version` input to both build
workflows, which this passes. Pins move to that merge commit, picking up three
unrelated fixes in the same range (#810, #811, #812); the upload workflow is
unchanged there and is bumped only to keep all three pins on one SHA.
* chore: sort imports in vla_jepa tests
Enabling pydocstyle in the previous commit changes how ruff determines where a
module's import block ends, which makes I001 fire on three vla_jepa tests that
were clean before. The blank line between the `conftest` and `lerobot` imports
is the trigger: both are first-party, so isort wants them in one contiguous
block, and the docstring-aware analysis is what makes it notice.
These files are unrelated to the API reference, so the fix is only to satisfy
the new gate.
Pre-existing `I001` violations that `pre-commit run --all-files` auto-fixes. They are unrelated to the documentation work but block the green run the rest of this branch is verified against; confirmed present on main before this branch.
* fix(datasets): Resolve only recipe-referenced bindings Eagerly resolving every DEFAULT_BINDINGS entry made rendering fail on frames whose events a default binding cannot disambiguate, e.g. the camera-less vqa default on multi-camera frames, even when the recipe never references that binding. Add TrainingRecipe.referenced_binding_names() and skip bindings the recipe does not consume. Split out of #4183 so the data-layer fix lands independently of the language runtime. Refs #4183 * fix(robots): Use module logger in ensure_safe_goal_position Route the clamping warning through a module-level logger instead of the root logger so applications can filter it by name. Split out of #4183. Refs #4183
* feat(pi05): add optional MEM observation history * fix(pi05): harden MEM inference contracts * test(pi05): gate MEM dataset dependency * fix(pi05): align MEM with the paper and harden history plumbing Follow-up on review feedback for the MEM short-horizon memory. Paper fidelity (MEM appendix C): - Compose the spatial and temporal attention weights per eq. 3 instead of summing two attention sub-blocks, deriving q/k/v for both stages from the shared z + e(t) of eq. 1. Applying the composition is spatial attention over temporally mixed values, so both stages stay on SDPA and `T == 1` reduces to stock SigLIP attention exactly rather than by a special case. - Drop past-frame tokens once the last temporal layer has run; the layers above it can no longer mix across frames. - Represent proprioception once when proprioceptive memory is on: the state leaves the discretized prompt and is carried only by the linear projection (section III-D). - Default `memory_stride` to 30, one second at the usual 30 fps LeRobot recording rate. 10 fps datasets such as `lerobot/robomme` need 10. Correctness and performance: - Resolve image history for `observation.image*`, not just `observation.images*`. Singular-key datasets silently trained on single frames before, with the state history still stacked. Raise when a requested image history matches no key. - Stop forcing eager attention on the whole vision tower. The composed path calls SDPA directly, so the spatial layers keep their configured kernel and no attention matrix is materialized. - Address `sample_observation_history` by age from the newest frame, so the current observation is the last frame for any queue length, and reject a queue that is too short. Document why the ring buffer must span the whole horizon. - Share one clone for the pre-episode queue fill instead of `maxlen` copies. - Train and save `proprio_history_proj` in full under PEFT; it has no pretrained weights to adapt and would otherwise stay frozen and be absent from adapters. Docs and hygiene: - State that the long-horizon language memory is out of scope and that finetuning from a memory-free checkpoint is the paper's weaker `MEM-Posttrain-Only` ablation. - Align `robomme.mdx` with the `camera1`/`camera2` keys the code now uses. - Complete the Apache headers and drop the duplicate `tokenizer_max_length`. * fix(pi05): address MEM review follow-ups * fix(pi05): validate MEM temporal interval
…tion_for_inference (#4433) Transfer camera frames to the compute device as compact uint8 (a 4x smaller copy than float32) and run the /255 + permute there, instead of converting on the CPU control thread. Outputs are bit-identical. In a 30 FPS sync-inference rollout (2 cameras, MPS) this cuts the per-tick observation prep on the control thread from 9.1 ms to 0.6 ms: the conversion kernels run asynchronously on the accelerator, overlapped with the loop's pacing sleep, raising the achieved control rate from 27.5 Hz to 28.8 Hz.
…odebase (#4428) * fix(legacy depth): fixing legacy depth makers when aggregating datasets and unifying legacy markers handling across the codebase * chore(test): updating tests * refactor(datasets): centralize depth-map detection in is_depth_map utility Add a pure ``is_depth_map`` predicate in ``configs.video`` that handles the canonical marker and both legacy forms, and reuse it in ``depth_keys``, ``encoder_config_from_video_info`` and ``canonicalize_depth_marker``. * fix(datasets): never leave an empty video_info entry after canonicalizing When the legacy depth marker lived in a separate ``video_info`` dict, popping it could leave an empty ``video_info``. Drop it in ``canonicalize_depth_marker`` so merged metadata stays clean, and simplify the merge-equality normalization. * fix(non-depth markers): fixing canonical non-depth markers to always be info.is_depth_map: False * fix(stats): use is_depth_map util for depth detection
Co-authored-by: root <kinsonnee@gmail.com>
* refactor(rollout): cadence cycle * feat(rollout): add Cadence Reporting * feat(scripts): use cycletimer + move it to utils * test(rollout): simplify tests
…utils (#3423) Co-authored-by: Caroline Pascal <caroline8.pascal@gmail.com>
…rame leakage (#4381) * fix(vla_jepa): add opt-in causal world-model context to stop future-frame leakage (#4153) * style: fix prettier formatting in vla_jepa.mdx table * test(vla_jepa): strengthen #4381 causal-context regression coverage Co-authored-by: vdmaas98 <159767407+vdmaas98@users.noreply.github.com> * style(vla_jepa): trim causal_world_model_context comment to one line --------- Co-authored-by: vdmaas98 <159767407+vdmaas98@users.noreply.github.com>
… rollout control (#4380) * feat(rollout): interactive v1 Co-authored-by: Pepijn <pepijn@huggingface.co> * feat(rollout): mute logs in interactive mode Co-authored-by: Pepijn <pepijn@huggingface.co> * feat(rollout): add subtask command Co-authored-by: Pepijn <pepijn@huggingface.co> * refactor(rollout): integrate feedback -> api, recording, log mut and keyboard Co-authored-by: Pepijn <pepijn@huggingface.co> * fix(rollout): correct label for recording * feat(rollout): vqa command draft * feat(rollout): autosteer command draft * fix(rollout): contract for policy * refactor(rollout): add policy contract and fix _generate_text * feat(rollout): print autosteer subtasks * chore(rollout): demote pump_query in engine * chore(rollout): interactive strategy clarification * docs(rollout): update docstrings + comments * fix(utils): harden stdin listener EOF and error paths - Blocking reader: detect EOF with a falsy check so bytes streams (b"") terminate instead of busy-spinning forever without firing on_eof. - Drop the str->bytes->str round-trip: _emit_line now takes str and the select path decodes at its two emit sites, removing an uncaught encode failure path. - Guard the initial fileno() in _run_select so a stream closed before the thread's first instruction still reports a dead channel via on_eof. - Document the sole-consumer constraint of the descriptor-level reader. - Tests: blocking bytes-stream EOF, un-newlined final command flush at EOF, and partial-line accumulation across reads. * fix(rollout): close query-channel races in the engine ABC - Stale autosteer apply: a NEXT_SUBTASK generation finishing after /reset, /subtask, or /autosteer off no longer overwrites the operator's newer instruction — the check-and-apply now runs atomically under _query_lock and stale turns are discarded without an announcement. - Autosteer double-queue on async backends: a _query_in_flight flag (set at claim, cleared at publish) stops _poll_autosteer from queueing a duplicate turn during the whole generation latency. - Undelivered answers are no longer overwritten: the single _ready_answer slot becomes a small queue drained fully by each pump, so two /vqa answered between pumps (e.g. during torch.compile warmup) both arrive. - generate_text output is validated centrally in _service_query (non-empty str) and the str() coercion is dropped in both backends, so None/tensor returns become error answers instead of the live task 'None'. - Segment cleanup drains undelivered NEXT_SUBTASK answers (new engine.drop_ready_subtask_answers), so the idle pump cannot announce a subtask from a stopped sequencer; VQA answers stay deliverable. - pump_query/_service_query now report whether a query was served inline, for overrun-warning suppression and post-query observation refresh. * fix(rollout): close RTC reset TOCTOU and stale post-query chunks - reset() clears the action queue inside the same _obs_lock section as the epoch bump, and _rtc_loop checks the epoch and merges inside one _obs_lock section: a reset can no longer interleave between check and merge and leak a pre-reset chunk into the freshly cleared queue (lock order stays _obs_lock -> queue.lock on both sides). - After _service_query serves a blocking text query, the RTC loop re-reads the observation and the epoch before the refill branch, so the next chunk is conditioned on the freshest pose instead of a seconds-old snapshot and the discard guard also covers resets that landed during the query. - Reword the task-changed log: the switch is applied from the next *merged* chunk (the epoch guard may still discard the current one). - ActionQueue.get_with_task raises a self-describing RuntimeError when the task-label queue desyncs from the action queue, instead of a bare IndexError on the control thread's hot path (+ test). * fix(rollout): make controller lifecycle terminal and failure surface truthful - One-shot semantics made explicit: serve() latches a terminal stopped state on exit and raises on re-entry; start/reset/set_task refuse with False once stopped (or when a stop is pending / a failure is latched) instead of acknowledging commands nothing will ever service. - strategy.run() exceptions are captured into a controller-level failure slot, surfaced through failed/failure_traceback and a new STRATEGY_FAILED event before STOPPED — instead of unwinding through serve() as a clean-looking STOPPED with the traceback lost to the thread excepthook. - Segment startup re-checks engine.failed after clearing the LinkedEvent's local flag, closing the window where a dying RTC thread's shutdown signal was wiped and a segment started against a dead engine. - return_to_initial_position() now returns whether the move completed; a failed move emits the new RESET_FAILED event (RESET_DONE promises the robot is home) and the session words the failure explicitly. - The constructor rejects one-shot strategies (supports_interactive=False), enforcing on the library path the contract the CLI already enforces. * fix(rollout): make sentry tail saves safe, counted, and non-silent - Tail-save recovery no longer masks a partially-committed save_episode: DatasetWriter commits parquet rows before its failure-prone steps, so a mid-save failure now latches the dataset as poisoned (further segments raise, background/teardown Hub pushes are refused) and re-raises to the controller's failure surface instead of silently growing corruption. - Empty-tail segments (e.g. /start then /reset during warmup) skip the save via dataset.has_pending_frames() instead of taking the exception path with a muting-piercing WARNING traceback on a normal path. - Failure cleanup uses the public dataset.clear_episode_buffer( delete_images=False) instead of poking writer internals. - Tail saves count toward upload_every_n_episodes: post-save bookkeeping is factored into _register_saved_episode and called from both save sites, so interactive sessions (segments shorter than one rotation) background-push on the documented cadence instead of only at teardown. - Both save sites warn (WARNING pierces the interactive log muting) before blocking on the episode lock while a Hub upload is in flight, so /reset and /stop no longer freeze the robot with zero feedback. * fix(rollout): console truthfulness and operator-facing polish - Skip the FPS-overrun warning for ticks whose overrun is an expected inline text generation (pump_query's served flag), in base and sentry loops — the warning channel stays reserved for genuine slowdowns. - warn_loop_overrun takes the effective target rate (1/control_interval, i.e. fps x interpolation_multiplier) instead of the base fps, so the message names the budget actually missed; all call sites updated. - _mute_system_output nests warnings.catch_warnings() instead of hand-restoring warnings.filters (which missed the mutation-counter bump and could leave warnings suppressed after the session). - _print emits one write per message so listener-thread acknowledgements and serve-thread events cannot interleave mid-line. - /subtask "" and /vqa "" no longer apply/queue the empty string: quotes are stripped before the emptiness check, matching /autosteer. - Exhaustive AskResult handling in /vqa and /autosteer: unknown variants log an error instead of being mislabeled busy/success. - Docs: note that the transcript's '>' is typed input (no prompt is rendered), and that --duration bounds each /start segment in interactive mode (also noted on the config field). * refactor(rollout): trim top-level exports, document the architecture - Drop parse_command, InteractiveCommand (terminal front-end internals) and PolicyQuery (engine-internal; observers only ever see QueryAnswer) from the package's top-level surface; they remain importable from their defining modules, and tests now import them from there. - Add an Architecture section to the package docstring: the four components with their responsibilities/lifetimes/threads, the downward call DAG, the control-plane vs data-plane split of engine callers, the two upward channels, and the record-intent rule that answers 'when do things happen'. - Document the fourth interactive-strategy requirement: the mandatory end-of-tick engine.pump_query(obs_processed) call, in both the RolloutStrategy contract and the run() docstring — a strategy satisfying every previously documented bullet still silently degraded text queries. * feat(policies): make the interactive rollout contracts enforceable - drop_queued_actions() resolves the action queue through a declared ClassVar (_action_queue_attrs, mirroring _fsdp_forward_methods) instead of hard-coded duck-typing: a chunking policy whose queue lives under a different name now has an explicit knob, and the convention is documented next to populate_queues. - __init_subclass__ raises when generate_text() is overridden without supports_text_generation(): the rollout stack consults the flag before accepting /vqa //autosteer, so a text head without it is unreachable and the operator is told the policy has none — the mismatch now fails at class-definition time. Checkpoint-conditional overrides stay possible. - Conformance tests against a real chunking policy (tiny ACT): a fresh forward after drop_queued_actions(), the queue attribute being one the base class declares, custom-attr extension, and both tandem-override directions. * test(rollout): RTC engine coverage, deterministic clocks, asserted joins - New RTC engine test group driving a real RTCInferenceEngine with a gate-released stub chunk policy: dispatched-task provenance across a set_task with queued leftovers, reset() discarding a chunk from an in-flight inference (and recovering after), /vqa answered on the RTC thread but delivered only by the control thread's pump, and get_action failing loudly on a provenance-less queue entry. - Autosteer interval tests run on a fake clock injected into the engine module (no global time patch), removing the reproduced wall-clock race with the 100 ms budget and the real sleep. - _join_session helper joins and asserts thread exit everywhere: a hung session thread can no longer silently leak process-wide log muting into the rest of the suite; the first logging test also restores logging.disable in its finally. - The mock strategy is now create_autospec(RolloutStrategy, instance=True) with a real BaseStrategyConfig, so method-signature drift fails in tests instead of on hardware during /reset. - Negative and zero-boundary tests for the autosteer_interval_s config validation. * chore(rollout): appease ruff format and B023 in new tests * fix(rollout): close gaps found by adversarial verification of the review fixes A 12-agent verification pass over the full diff confirmed 46 item implementations and surfaced five gaps, all fixed here: - Sentry: rotation-site save_episode failures now take the same poison path as tail saves (shared _checked_save_episode); previously a rotation failure bypassed the latch, and has_pending_frames() then KeyError'd on the popped 'size' key inside run()'s finally, masking the original error and letting teardown upload the partially-committed dataset. The tail save also returns early when the poison is already latched, and has_pending_frames() treats a mid-save-abandoned buffer as empty. - Sentry: the background _push closure re-checks _dataset_poisoned under _episode_lock, so a push queued behind an in-flight one cannot upload a dataset poisoned while it waited. - Engine: the NEXT_SUBTASK failure path gets the same liveness rule as the success path (_fail_subtask): a stale turn's failure no longer kills a goal the operator has since set, nor announces 'could not plan' for a sequencer that segment cleanup already stopped. - Policies: the generate_text/supports_text_generation tandem guard resolves through the MRO instead of cls.__dict__ — no false positive on subclasses of conforming parents, and mixin-supplied text heads are caught. - Session: /start//subtask//reset refusals from a stopping or failed controller are worded truthfully instead of 'Already running' / 'Task unchanged'. All fixes pinned by new tests. * fix(rollout): reconcile interactive rollout with #4423's unified pacing The merge commit before this one resolved every conflicting hunk to upstream, so this commit is the whole of "what the interactive-rollout branch had to change to comply with main". Reviewing it in isolation shows the full cost of #4423 for this branch. `warn_loop_overrun` is deleted: `CycleTimer.wait()` now owns the slow-loop warning for all nine control loops. Its one non-redundant behaviour — do not warn about a tick that ran slow because it served an inline /vqa or /autosteer generation — is re-expressed as `timer.restart()` on a truthy `engine.pump_query(...)`, before `timer.wait()`. That re-arms the start-up exemption so the group `wait()` closes is not judged, which is the idiom #4423 already uses after the blocking `save_episode` and after DAgger's handover ramps. The guard must stay conditional: an unconditional `restart()` exempts every group and silently disables the warning (and the effective-cadence line) for the whole run — a regression that leaves every existing assertion green, so both directions are now pinned by tests. Re-grafted onto upstream's loops: - `configs.py`: `interactive` / `autosteer_interval_s` were deleted by the merge while the `__post_init__` validators that read them survived — every `RolloutConfig(...)` raised `AttributeError`. Restored, with the cadence caveats documented on both fields. - `base.py`, `sentry.py`: the end-of-tick query pump, inside a `timer.section("query")` so an inline generation is attributed rather than silently inflating unaccounted work. - `sentry.py`: restartability restored on top of upstream's cadence loop — no `VideoEncodingManager` in `run()`, the four save/push helpers, the poison latch, and `engine.dispatched_task` frame labelling now read inside upstream's `emitted_policy_action` gate. `log_run_summary()` runs before the tail save, which unlike the suppressed save it replaces can re-raise. `_episodes_since_push` moved to `__init__` so a strategy driven without `setup()` still has a counter. - `episodic.py`, `highlight.py`: dropped the now-unused import only. Both keep the merge's hybrid state deliberately — normalising them to upstream would reinstate `self._return_to_initial_position`, which `core.py` no longer defines. `dagger.py` is byte-identical to upstream. Tests: sentry's `send_next_action` stub now drives the interpolator, since upstream gates recording on `emitted_policy_action` and a stub that only returns a dict records nothing. Upstream's `_make_loop_ctx` pins `pump_query` to `False` — a bare `MagicMock` returns truthy and would restart the timer every tick, neutering cadence judging for every strategy it drives. The query-exemption test asks mid-run instead of before the loop (asking first landed on the start-up group, so it passed vacuously), and a positive control asserts a slow tick *without* a query still warns. A new sentry case covers `dispatched_task` labelling at `interpolation_multiplier=2`, where the recording tick is no longer the tick that ran inference — the one claim the reconciliation makes that multiplier 1 cannot exercise. Three mutants (no `restart()`, unconditional `restart()`, recording gate removed) were each verified to fail exactly the intended test and no other. Known limitation, documented rather than fixed: an interactive session mutes below WARNING, so #4423's per-episode and whole-run cadence summaries (INFO) are withheld for the session. Surfacing them through the session's own channel needs a report sink on `CycleTimer` and is a follow-up, not a merge fix. 402 passed, 3 skipped across tests/test_rollout.py, tests/test_interactive_rollout.py, tests/utils/test_cycle_timer.py, tests/policies/rtc/, tests/utils/test_stdin_input.py and tests/policies/test_pretrained_interactive_contracts.py. * chore(rollout): demote rtc log + count text to timercycle * feat(rollout): enable cycletimer sink to interactive exception * chore(rollout): raise muted logs in interactive to ERROR * chore(rollout): trim tests, comments, docstrings and docs --------- Co-authored-by: Pepijn <pepijn@huggingface.co>
* vla-jepa relative actions * vla_jepa: per-sample loss reduction for RA-BC * device autocast * refactoring processors * load reinit checkpoints onto the current device and honor strict * skip the discarded full-vocab lm_head forward * derive action and state dims in the config instead of mutating it in __init__ * only clip normalized actions under MIN_MAX * make the gripper post-steps serializable and safe outside LIBERO * separate the world-model view count from the JEPA tubelet size * wire up the config fields that were silently ignored * fix predictor attention dropout and drop its dead parameters * skip CPU autocast for dtypes it does not implement * add tests and docs for the gripper, clipping and view-count changes * keep each sample with its own camera views when merging world-model features * enable strided observations for long horizon chunks * instead of first N frames, spread them across the whole chunk * cleanup pre-commit * apply the view-merge fix to the causal world-model context path #4381's causal_world_model_context branch merges views with the same chunk+cat pattern fixed in 9622b40, so it splices features from different samples for b > 1. Extract the merge into _merge_views and use it in both places, and parametrize the regression test over the flag. * trim the long baked-in explanations to their essentials * overwrite input_features with the right state dim
feat(pi05): add optional training-time RTC Adds opt-in training-time Real-Time Chunking to pi0.5 via policy.rtc_training_max_delay (defaults to 0). When enabled, training samples a fully denoised action prefix per example, accomplishing this by supplying per-action flow timesteps (1 for prefix, 0 for the rest), and computing the flow loss only on the generated postfix. Trained checkpoints run with --inference.rtc.mode=trained; guided RTC remains the default for ordinary checkpoints.
* align molmoact2 training with original recipe * simplify molmoact2 vlm training mode * fix(molmoact2): align training configuration * fix(molmoact2): align preprocessing and normalization * fix(molmoact2): align continuous HF model path * fix(molmoact2): align continuous policy training * docs(molmoact2): document matched training recipe * test(molmoact2): cover matched continuous training * fix(molmoact2): match official LoRA initialization * fix(molmoact2): preserve continuous inference mask * fix(molmoact2): match embedding optimizer group * perf(molmoact2): specialize compiled action shapes * refactor(molmoact2): decouple shared policy helpers * docs(molmoact2): align compile and augmentation guidance * style(molmoact2): apply upstream formatters * fix(molmoact2): preserve explicit flow tensor dtype * set libero reset wait steps to 50 * refactor(molmoact2): keep one compiled training path * fix(molmoact2): load checkpoint processors through factory * fix(molmoact2): keep norm stats path initialization-only * fix(molmoact2): match official scheduler horizon * revert(libero): keep upstream reset wait default * fix(molmoact2): load LeRobot checkpoints strictly * fix(molmoact2): preserve gripper masks on checkpoint reuse * fix(molmoact2): trust saved norm tag metadata * docs(molmoact2): clarify strict topology loading * fix(molmoact2): keep norm metadata initialization-only
…eq (#4479) `save_freq` is documented as "a non-positive value disables periodic saving, keeping only the final checkpoint", and the RL learner inherits the field from TrainPipelineConfig. Its raw `optimization_step % save_freq` never honoured that: 0 raises ZeroDivisionError, and -1 saves on every step because `step % -1` is always 0.
`--peft.method` is a strict prefix of `--peft.method_type`, so argparse's `allow_abbrev` expanded it silently. draccus 0.11 sets `allow_abbrev = False`, so the flag stopped resolving when #4033 bumped the pin from 0.10.0 to 0.11.6 and all three tests now exit at argument parsing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Explain what this PR does. Feel free to tag your PR with the appropriate label(s).
Examples:
How it was tested
Explain/show how you tested your changes.
Examples:
test_somethingintests/test_stuff.py.new_featureand checked that training converges with policy X on dataset/environment Y.some_function, it now runs X times faster than previously.How to checkout & try? (for the reviewer)
Provide a simple way for the reviewer to try out your changes.
Examples:
SECTION TO REMOVE BEFORE SUBMITTING YOUR PR
Note: Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR. Try to avoid tagging more than 3 people.
Note: Before submitting this PR, please read the contributor guideline.