v1.3.4.dev0 Full Collab Integration and bug fixed#251
Merged
Conversation
…ts, and agent features (#250) * add new nworkers tests for perfs * Stop DataLoaderInterface from blocking on pause_controller during iteration pause_controller starts paused by default and is intended to be driven by the UI / training loop, not the data path. Calling _wait_if_paused() from __iter__ and __next__ meant any script that iterated the loader before an external resume would hang forever — even at num_workers=0. Workers made the failure mode look like a worker bug (leaked semaphores, freezes under load), but the same hang reproduced with no workers at all. Pause is a training-loop concern; the loader should just deliver bytes. _wait_if_paused() itself is preserved so training loops can still call it explicitly at safe points (between optimizer steps). Verified: - weightslab/tests/backend/test_data_loader_interface.py: 9/9 pass (incl. test_dataloader_interface_uses_multiple_workers, test_multiple_workers_parallelize_preprocessing) - Wider tests/backend + tests/components sweep: 108 pass, 2 unrelated pre-existing failures in test_ui_docker_bridge (cert script + Windows path test on Linux) - ws-detection example with num_workers={0,2,4} on CPU: clean runs, W=2 ~21% faster than W=0, no hangs or crashes Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Move pause_at_step trigger from DataLoaderInterface into GuardContext a4cf489 removed DataLoaderInterface._wait_if_paused's call sites to fix the num_workers>0 startup deadlock, but that was also the only place enforcing the explicit pause_at_step hyperparam. Re-add the trigger in GuardContext.__enter__ (training only), before the architecture lock so the pause blocks lock-free. pause() zeroes pause_at_step, so it fires once. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Remove dead DataLoaderInterface._wait_if_paused Its call sites were removed in a4cf489 (deadlock fix) and its pause_at_step trigger was relocated to GuardContext.__enter__ in the previous commit, so the method is now unused. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ws-detection: default num_workers=2, add num_workers benchmark harness - main.py: refactor train loop to the infinite-generator form (re-shuffles each epoch) and default both loaders to num_workers=2 (GPU sweep: ~+76% throughput vs workers=0, sweet spot; W=4 regresses). - bench.py + run_bench.sh: configurable num_workers/epochs/wall-time harness with WL_BENCH_NO_VAL for clean throughput runs; forces the model onto the target device after watch_or_edit (see device note below). - config.yaml: local run tweaks. Note: watch_or_edit(flag="model", device=...) currently drops its device= kwarg (src.py model branch returns the proxy without honoring it), so the bench applies an explicit .to(device) workaround; the framework path is still unfixed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Drop now-unused threading/pause_controller imports from dataloader_interface Both were only referenced by _wait_if_paused, removed in 41b76b8. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ws-detection: drop bench.py/run_bench.sh from the branch These were local correctness-check tooling for the num_workers sweep, not part of the example. Kept on disk locally (untracked), removed from version control. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Revert ws-detection example changes; keep this branch framework-only main.py (runnable train-loop fix) moves to a dedicated fix branch off dev; config.yaml carried machine-local tweaks. Both are kept locally but removed from this branch so it contains only the parallelism/pause framework changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix ws-detection example: restore runnable training loop An earlier merge left main.py's train loop in a non-runnable state. Restore a working loop (infinite-generator batching that re-shuffles each epoch, per-sample loss/IoU via the criterion dict) and default the loaders to num_workers=2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix oom bug on break by slice (#186) * fix oom bug on break by slice The break-by-slice handler called get_signal_history_per_sample(), which inflated the entire per-sample signal history into nested dicts and then triple-looped over it (~609 MB spike per slice query -> OOM). Separately, query_per_sample() compared sample ids as int (stored) vs str (queried), so the cheap path would have silently returned 0 rows. Query the compact per-signal arrays directly via query_per_sample, normalize the id compare to str, and derive audit_mode from the eval-marker hash. A 200-sample slice over 2.1M entries now peaks at +0.6 MB instead of +609 MB (~1000x lighter). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix BBS feature to make the main computation part on our side --------- Co-authored-by: Alexandru Rotaru <rotarualexandruandrei94@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: GuillaumePELLUET <guillaume@graybx.com> * 179 audit and export user interactions success and failure to json and csv (#183) * Implement comprehensive audit logging for all gRPC user interactions with before/after tracking - Create AuditLogger class in backend/audit_logger.py with thread-safe JSON/CSV writing - Initialize audit loggers in ExperimentService and DataService with root_log_dir - Log all 8+ gRPC handlers with detailed before/after values: * ExperimentCommand: hp_change, mode_switch, pause, resume * GetLatestLoggerData: metrics_fetch * RestoreCheckpoint: checkpoint_restore * TriggerEvaluation: evaluation_start * EditDataSample: tag_add, tag_remove, sample_discard, sample_restore * ApplyDataQuery: query_execute * GetDataSamples: data_fetch - Append-only audit_log.json and audit_log.csv files in root_log_dir - ISO 8601 timestamps with microseconds in JSON; CSV with escaped JSON details - Thread-safe file operations Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Add comprehensive unit tests and documentation for audit logging Tests: - 26 unit tests covering all AuditLogger functionality - Tests for JSON and CSV output formats - Thread-safe concurrent logging with 10+ threads - Error handling and edge cases - Real-world scenario tests (hyperparameter changes, data edits, training control) - Special characters and Unicode handling - All tests passing Documentation: - Comprehensive audit_logging.rst guide with examples - Overview of what's logged (7+ action types) - JSON and CSV format specifications - Configuration and file locations - Real-world scenarios and troubleshooting - API reference and best practices - Added to docs index for discoverability Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Reorganize documentation: separate gRPC functions and audit logging Documentation restructuring: - Created new gRPC section with subsections - docs/grpc/index.rst: Overview and architecture of gRPC communication - docs/grpc/grpc_functions.rst: Complete reference of all RPC handlers (13 methods) * ExperimentCommand (HP changes, pause/resume, mode switching) * GetLatestLoggerData (metrics and signals) * RestoreCheckpoint, TriggerEvaluation, GetEvaluationStatus, CancelEvaluation * GetDataSamples, ApplyDataQuery, EditDataSample, GetDataSplits * GetWeights, GetActivations, GetSamples * Includes: request/response types, parameters, behavior, audit logging status * Covers: error handling, performance considerations, debugging, common patterns - docs/grpc/audit_logger.rst: Comprehensive audit logging documentation * Moved from docs/audit_logging.rst with updated cross-references * Explains what gets logged, file formats (JSON/CSV), configuration * Real-world scenarios, troubleshooting, API reference, best practices - Updated docs/index.rst to reference new gRPC section Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Add configurable audit log output format via AUDIT_LOG_FORMAT environment variable - Modified AuditLogger to write only one format (json OR csv), not both - Added format parameter to AuditLogger.__init__() with environment variable support - AUDIT_LOG_FORMAT=json (default) or AUDIT_LOG_FORMAT=csv - Explicit format parameter takes precedence over environment variable - Updated all 33 tests to work with format selection: - Fixed TestAuditLoggerCSV, TestAuditLoggerErrorHandling, TestAuditLoggerThreadSafety - Added TestAuditLoggerFormat class with 4 new tests for format configuration - Updated docs/grpc/audit_logger.rst Configuration section with AUDIT_LOG_FORMAT details and precedence rules - All tests passing: 33/33 Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Add ability to disable audit logging with AUDIT_LOG_FORMAT=none - Added "none" format option to AuditLogger to disable audit logging entirely - When format="none", log_event() returns early without creating files - Added AUDIT_LOG_FORMAT=none to docs/configuration.rst environment variables - Updated docs/grpc/audit_logger.rst Configuration section with disable feature - Added 3 new tests for disable functionality (36 total tests, all passing): - test_none_format_disables_logging() - test_none_format_from_environment_variable() - test_explicit_format_none_overrides_json_default() - Precedence unchanged: explicit format > environment variable > default Use cases for disabling: - Reduce disk I/O overhead in high-performance scenarios - Disable audit history for development/debugging sessions - Focus on other logging without audit pollution Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Fix FutureWarning: Set incompatible dtype column to object before assignment When upserting data with mixed dtypes (e.g., initializing column with bool False, then assigning string/array values), pandas raises a FutureWarning about incompatible dtypes. Fix by casting both the existing column and incoming values to object dtype before assignment to prevent dtype conflicts during merge operations. This resolves: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. Tests: test_h5_dataframe_store.py passes, FutureWarning no longer raised Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Fix AttributeError: Use correct EDIT_ACCUMULATE instead of non-existent EDIT_ADD enum The SampleEditType enum only defines: - EDIT_OVERRIDE: Replace all tags - EDIT_ACCUMULATE: Add/accumulate tags - EDIT_REMOVE: Remove tags The audit logging code was trying to use the non-existent EDIT_ADD enum value. Fixed by using EDIT_ACCUMULATE for tag_add operations, which is the correct enum value for adding/accumulating tags based on _calculate_tag_column_updates docstring. Error was: AttributeError: Enum SampleEditType has no value defined for name 'EDIT_ADD' Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * fix subscribe function to allow user to compute history-based samples * remove caching * fix warning issue with h5 * add sanity check on modeling feature and fix datasampler issues * clean custom signals decorator feature in readme and doc * Refactor query_per_sample to return dict of samples instead of list of tuples Changed return format from: List of (sample_id, step, value) tuples To: Dict mapping sample_id → list of dicts with 'model_age' and 'signal_value' keys Example: {'0': [{'model_age': x, 'signal_value': y}, ...], '1': [...]} Benefits: - More structured and readable format - Keys are labeled, not positional - Easier to work with in custom signals (e.g., computing loss variance) - Matches the format expected in SignalContext.subscribed_history Both get_current_signaL_history_per_sample and query_per_sample now return the new format. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Fix registered subscribed signals * Remove data fetch operations from audit logging Audit logging should only track user actions (write operations), not read-only operations like: - GetLatestLoggerData (metrics_fetch) - GetDataSamples (data_fetch) These are passive retrieval operations, not modifications to experiment state. Changes: - Removed data_fetch and metrics_fetch from audit logging documentation - Updated audit_logger.rst to list only user action types - Changed GetDataSamples and GetLatestLoggerData to 'Audit Logged: No (read-only operation)' - Updated reproducing experiment scenario to focus on user actions only Audit logging now logs only: - Model Control: hp_change, pause, resume, mode_switch - Data Operations: tag_add, tag_remove, sample_discard, sample_restore, query_execute - Checkpoint & Evaluation: checkpoint_restore, evaluation_start Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Fix audit logger implementation: reverse chronological ordering and test fixes - Fix _flush_to_json to reverse event order within batches for strict reverse chronological ordering (newest first) - Add buffer_size=1 to all test instances to ensure events flush immediately during testing - Update test expectations for reverse chronological order (newest events appear first in JSON) - Fix timestamp assertions in training control scenario to expect decreasing order (ts1 > ts2 > ts3) - All 41 audit logger tests now pass Features verified: 1. Persistence: audit logs append when restarting experiments from existing root_log_dir 2. Reverse chronological: newest events appear first in JSON output 3. Buffering: events batch in memory before writing to disk, with configurable buffer_size Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Add audit logging for plot note operations - Log note_write action when users save or clear notes on plot points - Capture metric_name, model_age, note_text, and note_action (saved/cleared) - Update audit logger documentation to include note_write in actions list This allows compliance tracking of all user annotations and notes. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Remove buffering approach from audit logger - use immediate writes instead - Remove buffer_size parameter and _event_buffer from AuditLogger - Change to immediate writes on each log_event() call - Rename _flush_to_json/_flush_to_csv to _write_json/_write_csv for single event writes - Remove flush() and _flush_buffer() methods - Remove all buffering-related tests - Update documentation to reflect immediate write approach Benefits: - No data loss on process crash or sudden termination - All audit events are persisted immediately to disk - Simpler implementation with same persistence guarantees - Still maintains reverse chronological ordering (newest first) Tests: 38 passed (3 buffering tests removed) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Expand signal decorator and SignalContext documentation - Add comprehensive parameter reference table for @wl.signal decorator * name, subscribe_to, compute_every_n_steps, include_history, include_history_metadata * Include performance considerations and use cases - Add advanced example from weightslab_kitchen: loss coefficient of variation * Shows how to access subscribed_history for multi-step analysis * Demonstrates history entry structure (signal_value, model_age) * Real-world use case: detect training instability - Expand SignalContext documentation with detailed attribute reference * Separate sections for dynamic signals vs. static signals * Document subscribed_history structure and access patterns * Add convenience properties (image, points, is_static, is_dynamic) * Include usage patterns and code examples This makes it clearer how to write effective custom signals with full history access. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Add sanity check with iterator * Fix utests bug with AuditLogs * update hard coded signals desc * set needs btw utests and pip publish packages and test * fix code quality issues * remove data fetching from audit and slow useless part of checkpoint manager loading (rng and data iterator state) as we do not manage data state reproducibility for now --------- Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com> * 184 dataframe granularity (#187) * Upgrade database to handle multi-indexing samples_id // instance_id, and subsequent fuctions * Fix certificate generation prompts in Windows tests Remove certificate generation and validation prompts that appear when running test_ui_docker_bridge tests on Windows. The test_complete_onboarding_workflow was calling actual certificate generation code without mocking it, which would trigger Windows certificate store installation prompts. Changes: - Added proper mocking of _generate_certs_with_fallback() function - Added mocking of _run_shell_script() to prevent bootstrap script execution - Properly configured CertAuthManager mocks with check_and_apply() return value - Added from_env_or_default() mock configuration for nested calls All 40 tests in test_ui_docker_bridge.py now pass without prompting for certificate validation on Windows. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Update ws-detection to use per-instance metrics and losses Switch detection example from per-sample to per-instance metrics: - Replace PerSampleDetectionLoss with PerInstanceDetectionLoss - Replace PerSampleIoU with PerInstanceIoU - Log hierarchical loss levels: instance, sample, and batch - Enable per-instance loss tracking for multi-instance dataframe support Changes: - Import PerInstanceDetectionLoss and PerInstanceIoU from criterions - Configure losses with return_levels=True to get instance/sample/batch breakdown - Manually log per-instance and per-sample metrics via wl.log_sample_signals() - Use 'batch' level loss for backward pass to ensure proper gradient flow - Maintain per-instance IoU computation for bounding box evaluation This enables comprehensive per-annotation analysis in the UI while maintaining per-sample aggregation for backward pass compatibility. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Add both per-sample AND per-instance metrics to ws-detection Track metrics at both granularity levels: Per-Sample Metrics (aggregated): - PerSampleDetectionLoss: Bounding box, classification, DFL losses averaged per sample - PerSampleIoU: IoU averaged per sample - Auto-logged via per_sample=True - Signal names: train/bbxs_sample, train/clsf_sample, train/dfl_sample, iou/train_sample Per-Instance Metrics (per annotation): - PerInstanceDetectionLoss: Individual bbox losses for each annotation - PerInstanceIoU: IoU for each bounding box - Manually logged via wl.log_sample_signals() - Signal names: train/bbxs_instance, train/clsf_instance, train/dfl_instance, iou_instance This enables: - Aggregate per-sample analysis for model evaluation - Fine-grained per-annotation debugging - Identification of problematic detections at the instance level - Proper gradient flow through batch-level loss for training Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Auto-save per-instance signals via per_instance flag on watch_or_edit Add framework-side support for per-instance signal logging, mirroring the existing per_sample flow. Users can now wrap a per-instance loss/metric with `per_instance=True` and WeightsLab will: 1. Extract instance values from dict outputs (`{'instance','sample','batch'}` from PerInstanceDetectionLoss) or flat tensors (PerInstanceIoU). 2. Look up `batch_idx` from the second positional argument (the standard detection `batch` dict) or from kwargs, mapping each instance to its sample position. 3. Assign annotation_ids 0,1,2,... within each sample. 4. Save per-instance values to the dataframe at `(sample_id, annotation_id)` via the new `DATAFRAME_M.enqueue_instance_batch`. 5. Still log the per-sample aggregated mean for the dashboard and return the original dict to the caller so `out['batch']` works for backward. Changes: - `dataframe_manager.enqueue_instance_batch`: writes per-annotation rows using `update_values` (handles multi-index natively). - `src.save_instance_signals`: helper that maps instance values to (sample_id, annotation_id) via batch_idx and routes to the dataframe. - `wrappered_fwd`: detects `per_instance=True`, unwraps dict outputs, invokes `save_instance_signals`, and returns the original dict. - `ws-detection/main.py`: replaces manual `wl.log_sample_signals` calls with `per_instance=True` on the watch_or_edit registrations. - New unit test `test_enqueue_instance_batch_writes_per_annotation` validates the end-to-end write path. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Simplify PerInstanceDetectionLoss to return flat instance tensor Drop the dict-with-levels return type from PerInstanceDetectionLoss — only the per-instance values are needed, since PerSampleDetectionLoss already provides the per-sample gradient path for backward. PerInstanceDetectionLoss now returns a flat `(num_instances,)` tensor, ordered as in `batch['batch_idx']`. With `per_instance=True` on watch_or_edit, the framework auto-saves these values at `(sample_id, annotation_id)` in the dataframe. Changes: - `criterions.py`: remove `return_levels` param; forward returns a flat instance tensor and only that. - `main.py`: backward now uses `per_sample.mean()` from PerSampleDetectionLoss; per-instance criterions are called only for their side-effect of auto-saving annotation-level signals. - `src.py`: skip the per-sample save_signals path when `per_instance=True` (instance-length tensors don't map 1:1 to batch_ids). - New test `test_save_instance_signals_maps_batch_idx_to_annotation_ids` verifies the (sample_id, annotation_id) mapping from batch_idx. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Fix duplicate-label error when per-sample buffer flushes into multi-index df enqueue_batch produces single-level (sample_id) records, but when the global dataframe has a MultiIndex, concatenating them directly creates a hybrid index with both tuples and ints. The next flush then crashes with: ValueError: cannot reindex on an axis with duplicate labels at _apply_buffer_records_nonblocking. Root cause: _apply_buffer_records and _apply_buffer_records_nonblocking didn't bridge between the single-level buffer and the multi-index global dataframe. Fix: add _broadcast_to_multi_index which expands each single-level (sample_id) buffer record into one row per existing (sample_id, annotation_id) pair. Both apply paths now invoke it before merging, so the global dataframe stays a proper MultiIndex and per-sample signals are broadcast to every annotation of the sample. Adds regression test test_per_sample_buffer_into_multi_index_does_not_corrupt that asserts index integrity through two consecutive flushes and that per-sample values are correctly broadcast to all annotations. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Fix _normalize_arrays_for_storage on multi-index rows When the dataframe is multi-indexed, `row.name` is a `(sample_id, annotation_id)` tuple. The code passed that tuple directly to `dataset.get_index_from_sample_id` (which expects a plain sample_id string), causing every array-column normalization to fail with a KeyError. The error was caught and only logged at DEBUG level, but it flooded the log and disabled the target/prediction normalization on every flush. Extract `sample_id = row.name[0]` when `row.name` is a tuple; otherwise fall back to the original row.name. Adds regression test test_normalize_arrays_for_storage_handles_multi_index_row that injects a fake dataset and asserts the sample_id (not the tuple) is passed to `get_index_from_sample_id`. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * cleaning branch * fix usecases examples configs and python file * fix grpc and agent interface with multi-indexing * fix multi-indexing issues * fix ext files * Add cat. tag management and fix tests * Fix documentation * Fix h5 compat. with multi-indexing * Fix utests and add new ones * Fix multi-index issues with trainer gRPC functions; h5 array issues for sync batch idx; and detection tasktype bb; and finally update the documentation * Fix broken utests --------- Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com> * Ddp support wip (#185) * DDP support (WIP): SPMD primitives + 4-plane model + workers-correctness fixes Wraps the existing single-process YOLO ws-detection example for DDP via mp.spawn — no train.py edits needed past the spawn shim. Surfaces: - weightslab/components/ddp_basic_building_blocks.py — SPMD primitives: register_consistent_state, reconcile_all (bundled DOWN broadcast), register_outbox + flush_outbox (bundled UP gather), sync_step (per-step anchor + collective pause-spin). One broadcast + one gather per step. - weightslab/components/ddp_planes.py — the 4-plane model (CONFIG / CONTROL / DATAFRAME / LOGGER) + 5 dtype-keyed reducers (MAX / LATEST / UNION / RANK_0_ONLY / IGNORE) + DOWN_ONLY whitelist for cross-rank DOWN- flowing per-sample columns. - components/global_monitoring.GuardContext — guard_training_context now auto-registers the core states + invokes sync_step on first DDP entry. - data/dataframe_manager.py — shm mirror of DOWN_ONLY columns visible to DataLoader subprocess workers via fork; per-cell value-change gate so rank-N's idempotent reconcile applies don't thrash worker resets; iter invalidation triggers on real DOWN_ONLY mutations only. - backend/dataloader_interface.py — WeightsLabDataSampler composes DistributedSampler under DDP; sampler reads the shm mirror at yield time (fork-safe); DataLoaderInterface gains _invalidate_iter to drop prefetched stale batches on the trainer thread (avoids the std::terminate crash from worker shutdown on a non-owning thread). - trainer/services/experiment_service.py — RestoreCheckpoint passes force=True (data snapshot was silently skipped when hashes appeared equal) and re-pauses post-load (saved hp had is_training=True, which load_state's register_hyperparams would otherwise re-apply). - components/checkpoint_manager.py — three reset_iterator sites route through the lazy _invalidate_iter path under DDP+workers. - examples/PyTorch/ws-detection/src/main_ddp.py — spawn shim worker. - examples/PyTorch/ws-detection/src/yolo_pipeline.py — extracted YOLO pipeline (replaces the older ddp_smoke._build_pipeline / decode helpers). - examples/PyTorch/ws-detection/src/ddp_test_suite.py — 8-scenario gRPC integration suite: epoch_then_pause, discard_subset_freezes, break_by_slice, lr_batch_propagate, checkpoint_data_roundtrip, signal_coverage_all_graphs, resume_continues_curve, process_topology. - tests/test_ddp_primitives.py — trivial 3-rank gloo verification of reconcile_all (convergence + idempotency + change propagation). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add 4 gap-coverage scenarios + collective-budget instrumentation 12 scenarios green end-to-end (8 original + 4 new). The new ones cover gaps that were missing direct verification: - scenario_multi_epoch_stability — 3 epochs back-to-back, asserts (sid, model_age) entries are unique per graph (idempotent dedup at the outbox merge) + age is strictly monotonic across epochs. Catches the regression where outbox flushes would append rather than upsert per-step. - scenario_empty_shard_starvation — discards ~95% of populated samples; asserts the trainer does NOT silently hang at the next grad all-reduce when one rank's shard ends up empty. Verifies loader cycle-and-skip semantics under heavy filtering. - scenario_seed_determinism — two consecutive break_by_slice pulls of the per-sample loss history return byte-identical (sid, age, val) triples. Catches stochasticity leaks in the read path that would silently break the loss-shape descriptor downstream. - scenario_collective_budget — programmatically asserts that every training step uses EXACTLY 2 collectives (1 reconcile_all broadcast + 1 flush_outbox gather). Hard perf gate against future regressions that add a stray dist.broadcast / dist.all_reduce in a hot path. Requires a small SDK hook: WL_DDP_COLLECTIVE_LOG=<path> appends the prior step's count to a file from inside reset_collectives() — opt-in, no overhead when unset. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add scenario_curate_lifecycle — end-to-end UI curation flow Tests the realistic multi-edit workflow under DDP: epoch 1 → tag 3 suspects → discard them → epoch 2 → un-discard + tag 'verified' → epoch 3 → assert loss trajectory shows the gap. Assertions: [1] LIFECYCLE — for each suspect: pre-discard entries exist, NO entry in the (discard_age, undiscard_age] window for any of them (this is the proof that discard reached the workers' shm + sampler fast-path), AND ≥1 suspect resumes post-undiscard. [2] TAG COMPOSE — break_by_slice('verified') returns all 3 suspects (proves multi-tag stacking on the same sample). [3] PLOT METRICS — scalar plot has ≥3 epochs worth of points. Side change: Client.discard now accepts discarded=False to un-discard via the same EditDataSample RPC. This brings the suite to 13 scenarios, all green at WL_DDP_BATCH=4, WL_DDP_WORKERS=0, WL_DDP_WORLD_SIZE=2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Cleanup: drop unused primitives + document drop_last=False rationale Removes ~155 LOC of dead-code surface that accumulated during the WIP push: - weightslab/components/ddp_basic_building_blocks.py: drop aggregate_up decorator, replicate_down decorator, reconcile_down (single-state hook), plus the combine-helpers (_concat, combine_rank0, make_concat_combine) that only existed to serve aggregate_up. The outbox/flush pattern superseded ① aggregate_up for per-sample hot writes (one gather/step instead of one per call); reconcile_all replaced reconcile_down (bundled broadcast); replicate_down was never invoked. Zero external references to any of them. Net -93 LOC. - weightslab/utils/tools.py + utils/__init__.py: drop DistributedCounter (CUT-tagged in the design notes, never adopted). Net -62 LOC. - weightslab/backend/dataloader_interface.py: keep drop_last=False on the DistributedSampler, but document why. Padded yields are real training events that land in the loss trajectory as real (sid, model_age, value) encounters with distinct ages from the sample's earlier yield; the trajectory is encounter-keyed, not per-epoch-unique-keyed, so padding is honest rather than pollution. drop_last=True was considered but rejected as too trivial — it'd silently drop the trailing (world-1) samples each epoch and bias coverage downward. Verified: scenario_discard_subset_freezes PASS at WL_DDP_BATCH=4, WL_DDP_WORKERS=0, WL_DDP_WORLD_SIZE=2, WL_DDP_TEST_STEPS=20 — 156 populated samples, 5 discards held frozen across epoch 2, ~80% advance on non-discarded. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Rename ddp_planes → parallel_state, ddp_basic_building_blocks → parallel_primitives Generalizes naming away from "DDP" since the primitives don't assume the specific torch.distributed-DDP topology — they hold for FSDP / ZeRO / any SPMD setup with a rendezvous-on-collective contract. - weightslab/components/parallel_primitives.py (was ddp_basic_building_blocks) - weightslab/components/parallel_state.py (was ddp_planes) - docs/ddp_design.md (was components/ddp_design_notes.md) Updated import sites (4): global_monitoring.py, dataframe_manager.py, parallel_primitives.py (self-ref), tests/test_ddp_primitives.py. Also: scenario_lr_batch_propagate threshold loosened from `(expected + rank0_only) / 2` to `rank0_only + 1`. The old midpoint sat right at the noise band; under drop_last=False with mid-iter batch-size transitions the observed rate floats 13–14 samples/step, occasionally tripping at exactly 13.75 < 14. New threshold cleanly distinguishes "both ranks doubled" from "only rank-0 doubled" (rate ~12). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix load_state to preserve model identity on same-arch restore ledgers.register_model(new_model) replaced the registered object, orphaning any captured reference (e.g. `model = trainer.model` in a training loop, or DataLoaderInterface.self.model). Post-restore the trainer trained a stale model while pause-checks read the fresh one, so pause_at_step never fired — caught by scenario_resume_continues_curve in the DDP suite. Skip register-replace + guard updates when existing model has same keys AND shapes as the saved weights; let apply-weights load in-place. Add a regression assertion in test_06 that captures the wrapped model identity across load_state. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Rewrite docs/ddp_design.md as a concise design overview Cut 159 → 66 lines (~58% smaller, 1426 → 588 words). Drops decision-tag scratchpad ([DECIDED]/[OPEN]/[DEFER]/[CUT]), per-state placement debate, wrapper-prologue code sketches, and the open-questions section — all process residue. Keeps: - Two-space framing (train-space vs sdk-space, kernel/user analogy). - SPMD-with-one-privileged-rank constraint. - Two-kinds-of-sync framing (grad reduction = off-the-shelf; async UI = WL's job). - The loop-iteration-as-transaction insight and the train→sdk transition as the consistency boundary. - State × direction table. - DOWN broadcast / UP outbox / shm mirror mechanisms with API entry points (register_consistent_state / register_outbox + the anchor functions). - Collective budget (~2 rendezvous/step) + instrumentation env vars. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Outbox ships per-step deltas, not full snapshots local_df_writes / local_signal_triples emitted the WHOLE dataframe and the WHOLE per-sample signal history every step, gathered to rank-0 each step. The ~2-collectives/step budget bounds the COUNT of rendezvous but not their bytes, so payload scaled with N_samples x world (df) and grew unboundedly (signals) — the real scaling wall. Each rank now dumps only what changed since its last flush: changed dataframe rows (vs a process-local _LAST_SENT_DF signature cache) and signal triples past a per-(graph, exp_hash) cursor read straight off the append-only buffers. On respawn/restore the cache resets to a one-time full resend, safe because every merge is idempotent. merge_df_writes seeds rank-0's current values first (existing-first) before the per-column reducer, so a delta that omits a sample rank-0 already holds a higher value for cannot regress MAX/UNION, while LATEST still resolves to the newest delta. clear_registry resets the caches. Docs + outbox comment updated to describe the delta transport and clarify the budget governs collective count, not payload; shm section corrected to note only the bool deny-list (`discarded`) is read at __getitem__, not user_tags. Validated: scenario_signal_coverage_all_graphs PASS (per-sample 940/940 across both ranks; test_ddp_primitives 3-rank reconcile PASS). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Align DDP module docstrings with ddp_design (remove stale primitives) The parallel_primitives module docstring advertised three decorator-form primitives — AGGREGATE_UP, REPLICATE_DOWN — and a reconcile_down single-state hook, none of which exist in the code (grep finds them only in the docstring). The implemented + design-doc'd surface is two mechanisms: register_consistent_ state/reconcile_all (DOWN) and register_outbox/flush_outbox (UP), plus the shm deny-list mirror and the sync_step anchor. Rewrote the docstring to match that and dropped the ①/②/③ numbering everywhere (parallel_state plane table and the inline anchor comments now use plain DOWN/UP, matching docs/ddp_design.md). Comment/docstring-only; no logic change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Add fast unit tests for the outbox delta optimization Proves the delta change's correctness in ~30ms instead of via the multi-minute YOLO scenarios: df_writes emits only changed rows; merge_df_writes seeds rank-0's current value so a stale/lower delta can't regress a MAX column (last_seen) while a higher one still raises it and LATEST picks the newest delta; local_signal_ triples advances a per-(graph, exp_hash) cursor and resends from 0 if the buffer shrank under it (restore safety). No DDP spawn — ledger getters are monkeypatched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Document DDP sampler sharding is training-only (eval is latent-unsupported) The sampler shards EVERY loader, but the per-step anchor runs only in training context — so a sharded eval under DDP would have each rank score 1/world with no scalar-metric aggregation (undercount). No eval runs under DDP today, so this is latent; added a TODO(ddp-eval) guard at _get_dist_sampler so nobody adds a DDP eval loop without first resolving the eval sharding/aggregation policy. Comment-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Cap shm deny-list index so large/sparse uids don't blow allocation The DOWN-only shm vec is indexed directly by int(sample_id), so a single large sparse uid (e.g. an inode-based id ~1e8) allocated a ~100MB bool array. Cap the index at 1<<22 (~4M → ~4MB max per origin/col): ids at/above the cap skip the shm fast-path and fall back to the sampler's pandas deny-list check, which is the actual read site (the shm is read in the main-process sampler, not workers — docstring corrected to say so). Dense 0..N id schemes keep the fast-path. Warns once per origin when the cap is hit. Adds test_ddp_shm_cap: huge id neither allocates a giant array nor breaks the small-id fast-path; undiscard clears the cell. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Restrict shm mirror to boolean deny-list columns (skip user_tags) _propagate_to_shm ran bool(val) over every DOWN_ONLY column into the bool shm vec, so user_tags (a list column) stored a meaningless bit that nothing reads (the sampler only queries 'discarded'). Filter the mirror to genuinely boolean columns via _shm_bool_eligible: bool dtype, or an object column whose non-null cells are all bool / 0-1 scalars. user_tags still reconciles to children via the DOWN broadcast; it just no longer gets a bogus shm array. Test: a user_tags list column allocates no shm array while discarded still does. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Decouple DDP shard reshuffle from iterator reset (epoch → reshuffle_seq) _generate_indices auto-advanced the DDP epoch on every fresh iterator, so a mid-loop discard/tag — which forces an iterator reset — reshuffled the entire per-rank shard. In a curation workflow (discard-heavy by design) the shard order churned on every discard, and "epoch" counted iterator resets, not dataset passes — a meaningless abstraction here. Rename _epoch → _reshuffle_seq (it's a reshuffle generation, not a pass count) and stop auto-advancing it. The reshuffle generation now advances ONLY on a genuine pass-end reset (loader calls sampler.advance_reshuffle() on the _epoch_exhausted path); the discard-invalidation reset path leaves it untouched, so a discard re-filters the SAME permutation instead of reshuffling. Reproducibility across resets is preserved but composed correctly: the per-rank permutation is a pure fn of (ddp_seed, reshuffle_seq, rank, world), so capture/restore_iteration_state now save/restore reshuffle_seq + seed; combined with samples_yielded (offset) and the deny-list (checkpointed as a DOWN_ONLY df column) this reproduces the exact filtered stream. Warns on a seed mismatch. Side effect: also fixes the __len__-vs-iteration epoch off-by-one — neither _generate_indices nor _rank_indices_snapshot advances during iteration now, so they read the same generation. Adds test_ddp_reshuffle_seq: re-gen without advance is stable; advance reshuffles; restore reproduces; ranks partition disjoint+cover; seed mismatch warns. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Update README.md * Guard user callbacks at collective boundaries so one can't hang the group reconcile_all built its snapshot ({name: snap()}) and applied children states unguarded, and flush_outbox built its payload ({name: dump()}) unguarded. Under SPMD a callback that raises on one rank but not others crashes that rank BEFORE (snapshot/dump) or AFTER (apply) a collective, leaving every other rank blocked forever on the broadcast/gather. Wrap each snap()/apply()/dump() in try/except (merge() already was), so the collective itself is ALWAYS reached: a failed state/channel ships as None (apply/merge already tolerate None) and the rest sync normally. Matches the DDP module's existing swallowed-exception style (logger.debug("[tag] ... failed: %s", exc)). Also switch parallel_primitives' logger from getLogger("weightslab.ddp") to getLogger(__name__) to match the rest of the codebase. Adds test_ddp_collective_resilience: a "bad" state (snapshot raises on rank 0, apply raises on children) + a "bad" outbox (dump/merge raise) run two anchor rounds + a barrier without hanging, and the healthy state still syncs on all ranks. Original test_ddp_primitives still passes (no reconcile regression). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Add world>2 uneven-shard coverage to the reshuffle test n=25, world=3 exercises DistributedSampler's drop_last=False padding: each shard pads to length 9 and the union still covers the whole universe. Complements the even world=2 disjoint+cover case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Remove the redundant shm deny-list mirror The shared-memory DOWN_ONLY mirror was never load-bearing: the deny-list is enforced sampler-side (a discarded sample is never yielded), the sampler's pandas cache already refreshes on the deny-list revision bump (so a live discard is reflected within one index), and a sample already in a worker's prefetch queue is dropped by iterator invalidation tearing the workers down — NOT by shm (shm filtered at yield time, before the discard existed, so it never had power over queued samples). No test depended on it; nothing read it except the main-process sampler. Remove _propagate_to_shm / _ensure_shm_capacity / _shm_bool_eligible / is_in_down_only_shm + the shm fields and the sampler's shm fast-path (~190 LOC, plus the ctypes/multiprocessing/os imports). Replace the invalidation gate — which must stay gated on an ACTUAL value change so rank-1+ don't respawn workers every step under DDP — with a pandas before/after diff (_down_only_changed) computed before the upsert merges. Update docs (design + comments) to describe sampler-side enforcement + invalidation, dropping the inaccurate "workers fork-read the shm" story. Supersedes the shm-cap (task #3) and user_tags-shm (task #4) fixes. Drops test_ddp_shm_cap; adds test_ddp_down_only_change covering the gate, including the DDP no-respawn invariant (re-applying the same snapshot → no change). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * DDP: rebalance-on-discard sharding, drop per-flush image decode, anchor/delta cleanups - Sampler: replace shard-then-filter with a filter->pad->stripe REBALANCE so live shards are always equal length across ranks — fixes the empty-shard deadlock (scenario_empty_shard_starvation) by construction; order-preserving + deterministic (not a reshuffle). persistent_workers=True makes a discard/undiscard rebuild a cheap reset (reuse workers, drop stale prefetch) instead of a fork+reinit. drop_last=False under DDP keeps a tiny live set training so age still advances. - dataframe_manager: stop the storage-time bbox->seg get_mask conversion — it re-decoded a full image per signal flush (~13% of rank-0 wall) and silently corrupted detection prediction_raw; mask rendering stays available on-demand via get_prediction_mask. Remove the now-dead normalize/_is_array_column/_get_loader. - Anchor split DOWN(__enter__)/UP(__exit__); outbox ships per-rank deltas via a writer dirty-set; remove post-hoc active-sample masking from the model wrapper. - Test suite: compute epoch_steps from WL_DDP_BATCH (was config's mono batch=4 while the loader trained at 16 -> every "epoch" silently covered ~4 passes); add scenario_progressive_resample (shrink->grow) + per-phase timing + a WL_DDP_SELFSPY self-profiling hook. - docs/ddp_design.md: document rebalance-not-reshuffle + persistent-worker reset. Note: 2 coverage assertions (epoch_then_pause populated~=shard, progressive_resample advance%) still assume the old over-training and read the per-sample gather before it fully settles for a now-correct single epoch; recalibration deferred. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * DDP: make WL_DDP_BATCH effective, event-based pause (no sleep), resumable suite - yolo_pipeline: push WL_DDP_BATCH into the in-memory cfg, not just the DataLoader ctor. _sync_batch_size_from_ledger re-applies the ledger's batch every iteration, so without this the loader silently reverted to config's mono batch=4 and the env was dead — the suite trained at 4 while epoch_steps assumed 16 (¼-epoch coverage, which failed epoch_then_pause / progressive_resample after the epoch-math change). Now img=16 for real: full single-epoch coverage + genuine ~23% speedup (4× fewer steps, same work). config.yaml on disk untouched (mono unaffected). - parallel_primitives / global_monitoring: kill the 20ms busy-sleep pause-spin in sync_step. Rank 0 blocks on the pause_controller resume Event (new wait_for_resume); rank-1+ block inside the next reconcile_all broadcast. Neither spins (gloo socket- waits; NCCL would busy-spin — noted). Bounded timeout kept for SIGINT/SIGTERM responsiveness, not polling. - ddp_test_suite: WL_DDP_SKIP (comma-sep substrings) so a killed run resumes by skipping already-passed scenarios. Full 14-scenario suite green at batch-16 (incl. empty_shard_starvation, progressive_resample, and the event-based pause via epoch_then_pause). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * DDP: trim verbose comments (no behavior change) Condense the "novel"-length comments added during the DDP work to 1-2 tight sentences each — worst offender was dataloader_interface (rebalance/reshuffle, __iter__, __len__, persistent_workers, _reset_iterator). Key invariants kept; redundant restatements dropped. ~80 fewer comment lines. Code unchanged (reshuffle unit test still green). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * DDP: delta the DOWN reconcile + vectorize the UP merge (anchor 2x faster, no O(N)/step) Two hidden O(N)-per-step costs in the cross-rank anchor, both removed: - DOWN reconcile (rank0_df_down_state) rebuilt+pickled+broadcast {col:{sid:val}} for ALL non-null DOWN_ONLY cells every step (discarded defaults False=non-null, so all touched samples rode) — O(N). Now a DELTA: rank-0 ships only the sample-ids whose DOWN_ONLY changed since the last reconcile (drain_down_delta dirty-set, populated in upsert_df on a real DOWN diff), with one full snapshot on first reconcile / post- restore (mark_down_full_resend, hooked in _load_existing_data) so children converge. N-sweep: full build+pickle was 1.5ms@1k / 119ms@100k / 619ms@500k; delta ~0 when unchanged, O(discards) otherwise. - DOWN_ONLY trimmed to {"discarded"} — "user_tags" was never a real column (it's "tag"), tags are rank-0 UI state (tag->label override is vestigial), and tag queries gather signals UP + filter on rank-0, so nothing tag-shaped needs to reach a sampler. - UP merge (merge_df_writes): replaced per-column groupby.apply(python reducer) with one vectorized groupby.agg({col:'max'/'last'}) — _r_max/_r_latest map exactly to skipna max/last; policy_for only yields MAX/LATEST here (UNION is tags, DOWN-filtered). And _rank0_existing_seed stopped copying the WHOLE dataframe every flush — it now indexes just the delta's ~batch rows. Anchor 168 -> 153 (DOWN delta) -> 78.6 ms/step (merge), ~2x. Validated: discard_subset_freezes, progressive_resample, break_by_slice, curate_lifecycle, signal_coverage all PASS. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Add ddp_ablation.py — 3-mode WL SDK overhead harness (time/mem/IO/bytes per rank) WL_ABLATE=ul|wlimport|wl on 2 gloo ranks, WL_ABLATE_STEPS configurable. Per mode: per-section ms/step, grad bytes/step, per-rank RSS + /proc/self/io (rchar/wchar = syscall bytes incl. gloo sockets; read_bytes/write_bytes = actual disk), and the WL df RAM + H5 store sizes + H5 flush config. 256-step result: WL time tax +247ms/+17.6% (criterions+log +148ms = save_signals + NMS decode-for-logging is the biggest, anchor +89ms, loader/wrapper +28ms); RAM +108MB import-idle + ~40MB active; df/H5 tiny. I/O surfaced that rank-1 redundantly persists ~6MB to H5 (should be outbox-only — rank-0 is authoritative). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Relocate DDP integration suite + perf/ablation into tests/ PR #185 review: tests don't belong in the usecase dir. Move ddp_test_suite.py, ddp_ablation.py, aggregate_wl_ownership.py and the report driver out of examples/PyTorch/ws-detection/src into tests/integrations/ultralytics/ddp/. One god-script (run_ddp_report.sh) with modes info/scenarios/ablation/profile emits a single report: perf counters, per-scenario times, and the wl-ulmanual ablation delta. Scripts path-bootstrap back to the usecase src so yolo_pipeline / utils.* / config.yaml resolve. Added README + .gitignore. Locally-run (needs GPU + dataset) — not a CI unit test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * PR #185: all_reduce_scalar(avg) + mono/DDP usecase README - utils.tools: add all_reduce_scalar(value, reduction="sum"|"avg"); avg = sum/world since gloo has no ReduceOp.AVG. Keep all_reduce_sum_scalar as a back-compat wrapper. - examples/.../ws-detection/README.md: document the mono (main.py) and DDP (main_ddp.py) usecases, how to run, and the single-GPU gloo DDP simulation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * PR #185: exclude is_training/pause_at_step/root_log_dir from the saved HP snapshot get_HP_snapshot dumped the whole hp dict, so a restore's register_hyperparams( saved_config) resurrected experiment STATE (notably is_training=True) — the bug the post-restore force in experiment_service worked around. Strip the same state-only keys already excluded from the experiment hash, on a copy (never mutate the ledger). The post-restore pause stays, now as the intentional "user drives the next cycle", not a workaround. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix ruff F841: drop unused group_ids / pre_restore_max_plot_age / ctx Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Alexandru Rotaru <rotarualexandruandrei94@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix lint code quality issue * Fix ruff and vulture code quality issue * fix utests after merging * Code quality issues * Final code quality check * change ci code quality as important for CI * Fix dev release CI * update release note format * Improve release notes: custom template with PR links, contributor profiles, heading levels - PRs now formatted as [#N](url) title — date with author GitHub profile links - Contributors section links to github.com/{login} (from PR authors) - Commits capped at 25 most recent non-merge commits - Title: ## **Weightslab** (no version, ## level) - Sections demoted to ### level - Removed separator between title and LinkedIn/Graybx links - Dev release routes PRs from --base dev, main from --base main - Doc build gated on main-branch check (not just tag pattern) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Final release note fix * fix instance_ids bugs matching for batch target format (targets=[bb_A, bb_B], batch_idx=[batch_idA, B, ..etc] * 188 logger granularity and structure (#189) * Add to detection usecase dump history and custom signal labelling * upgrade documentation with new functions and examples * add and fix utests * refactor the logger and add instances history and queries functions, with a user wl.write_history function * add df writing for user during exp * fix code quality issues * revert merging PR #185 * Fix bug from reverted Merged Conflicts * remove hardcoded path from config * fix iou metric name * fix merged conflicts error on fct name * Remove hardcoded value from src ModelInterface * fix merged conflicts function name * Add weightslab.integrations.ultralytics SDK (#191) * Guarantee no model-internal interactions unless asked; light=True by default New `light` kwarg on `ModelInterface.__init__`, default `True`. When light=True the wrap skips `init_attributes` (the shallow `vars(model)` iteration that created class-level property forwarders), the architecture-change hook, and the `CheckpointManager` auto-load block (which would otherwise call `load_state_dict(strict=True)` on a discovered checkpoint). Also forces `compute_dependencies=False`. Result: zero traversal or mutation of the wrapped model. Retains only what's needed for consistency and metrics — device placement, ledger registration, `guard_*_context.model = self` binding, `hp_config` read, and the `get_age` / `tracking_mode` / `set_tracking_mode` methods inherited from `NetworkWithOps`. Opt out with `light=False` to enable model surgery, attribute forwarding, and checkpoint auto-load. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Ultralytics harmonization: example ladder + WL integration helper The integration ladder for blending Ultralytics YOLO idioms with WeightsLab, plus the helper module that supports the destination interface. Files (all in examples/PyTorch/ws-detection/src/): ul10_wl00_main.py pure Ultralytics anchor (zero WL). ul08_wl02_main.py + read-only WL data inspection during vanilla `model.train()`. No signal capture, no edits. ul07_wl03_main.py + per-sample signal capture (loss + IoU) via UL callbacks. Still no edits, no manual loop. ul06_wl04_main.py reserved for the edits rung (watch_or_edit on model / optimizer / hparams). Stub. ideal_main.py destination — imperative, top-level `watch_or_edit` calls only. No session, no context manager. Atexit handles the silent join so `wl.keep_serving()` is no longer needed. wl_ultralytics.py the helper module: `attach(model)` installs UL callbacks (deferred to `on_train_start`), and a dispatch around `wl.watch_or_edit` routes YOLO instances through `attach` so callers write `model = wl.watch_or_edit(model)` symmetrically with the other registrations. Atexit on first `attach` keeps the studio backend alive after training ends. Built on top of the `light=True` default from `light-mode-default`: the model wrap guarantees no model-internal interactions unless `light=False` is explicitly passed. Local-only branch — not for pushing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Trajectory checkpoint: ul05_wl05 + ul06_wl04, wl_ultralytics consolidated Consolidation: * wl_ultralytics.py absorbs utils/criterions.py + utils/data.py contents. Becomes the single canonical home for per-sample loss / IoU / detection metrics, YOLODatasetWL + collate, load_config, attach, and the wl.watch_or_edit YOLO dispatch. Env defaults also live here so they run before weightslab is imported by this module. * utils/criterions.py and utils/data.py become thin re-export shims — main.py keeps working unchanged. ideal_main.py reaches its minimal destination shape: 17 LOC, three wl.watch_or_edit calls + YOLO + serve + train. Two new convergence rungs: * ul05_wl05_main.py — one step from main.py toward ideal: keeps the DetectionTrainer subclass shell, but deletes the manual train() and do_validate() overrides. UL's natural training drives; WL listens via callbacks installed in __init__. * ul06_wl04_main.py — one step from ideal_main toward verbose (sketch, not verified): no subclass, wl.watch_or_edit(model) dispatch unfolded into explicit add_callback() lines, edits-rung opt-in visible (light=False on the model wrap), env defaults inline. ul07_wl03_main.py and ul08_wl02_main.py updated to import dataset helpers from wl_ultralytics directly. Local-only branch — not for pushing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * WL light-mode fixes + minimal YOLO integration that runs end-to-end Three bugs fixed in WL that blocked the YOLO integration trajectory: 1. backend/ledgers.py — Proxy.get(ref, default) now returns the plain default value when `ref` is not in the underlying mapping. Previously it wrapped the default in a _ValueProxy unconditionally, which broke UL's `YAML.save(args)` ("cannot represent ValueProxy") when callers used `cfg.get("epochs", 100)` on a hparams-wrapped dict. 2. backend/model_interface.py — ModelInterface.__init__ device check now normalizes via `th.device(d).type == 'cuda'`. The previous exact-string `device == 'cuda'` check silently dropped `torch.device('cuda:0')` / `torch.device('cuda')` to CPU, which moved the wrapped model off the GPU and caused later device mismatches. 3. backend/model_interface.py — `_apply` and `train` overrides on ModelInterface so `.to / .half / .float / .cuda / .cpu / .train / .eval` reach `self.model`. `self.model` is intentionally kept in `__dict__` (custom __getattr__ relies on it), so nn.Module's default submodule recursion misses it; the overrides propagate explicitly. wl_ultralytics.attach() is now minimal: * Dataset wrap, optimizer wrap. * Model wrap with `forced_model_wrapping=True` (avoids stale Proxy re-use from prior runs) — only for ledger handle + age counter. * pause_controller.resume + @wl.eval_fn. Per-sample loss / IoU / detection-metric emission is removed for now — UL's DetectionTrainer doesn't expose `trainer.preds`, so capturing per-batch predictions needs a forward hook on the underlying model. Deferred follow-up. End-to-end smoke test against TrespassColor (epoch 1, batch 4, imgsz 1024): training + validation both complete; validator reports P=0.839 R=0.747 mAP50=0.815 mAP50-95=0.394 (reasonable for a pretrained yolo11s warmup). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ul05_wl05: first step from main.py toward ideal — verified end-to-end Make ul05_wl05_main.py functional. Same overall shape as before (thin DetectionTrainer subclass, no manual loop, callbacks installed in __init__) but aligned with the working integration pattern from wl_ultralytics.attach: * `forced_model_wrapping=True` on the model wrap (avoids stale Proxy from prior runs hosting weights on the wrong device). * Per-sample loss/IoU emission dropped — UL's DetectionTrainer doesn't expose `trainer.preds`; capturing per-batch outputs needs a forward hook on the underlying model. Deferred follow-up. * Drop `workers=0` — with workers=0 the main-process dataloader iteration sees our `loader.dataset.__class__ = YOLODatasetWL` swap, but UL's default collate expects the original YOLODataset's dict output (not YOLODatasetWL's tuple). UL's default workers fork the dataset before on_train_start runs, so the swap is invisible to them — sidesteps the collate mismatch. Smoke-tested against TrespassColor (epoch 1 of 1000 partial run): training + validation complete; P=0.787 R=0.671 mAP50=0.727 mAP50-95=0.444 for a yolo11s warmup. Trajectory status: main.py (manual loop, verbose) -> ul05_wl05 (subclass shell + callbacks, UL natural training) ✅ -> ul06_wl04 (no subclass, explicit callbacks, edits opt-in) sketch -> ideal_main (clean, dispatch hides callbacks) ✅ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ul05_wl05: capture per-sample signals via forward hook + preprocess patch UL's DetectionTrainer doesn't store `trainer.preds` or `trainer.batch` — the batch is a local variable in `_do_train`'s loop body, and predictions flow through DetectionModel.forward without being stashed on the trainer. Hook two surfaces to recover what we need: * Forward hook on the raw DetectionModel: DetectionModel.forward(x) routes by input type. With a batch dict it returns (loss, loss_items); with an image tensor (recursive call inside .loss()) it returns raw preds. Both fire per training step; we keep only the prediction call. * Patch `trainer.preprocess_batch` and `validator.preprocess` to stash the device-prepared batch into shared state. UL keeps batch local in the loop, so the preprocess wrap is our only handle from callbacks. `on_train_batch_end` / `on_val_batch_end` then call the per-sample `PerSampleDetectionLoss` (bbxs / clsf / dfl) and `PerSampleIoU` channels with the captured (preds, batch). PerSampleDetectionLoss is built against the raw DetectionModel (v8DetectionLoss does `model.model[-1]` which needs the raw class, not our wrapper). Smoke test against TrespassColor: 237 train batches + full validation both complete with no callback errors (P=0.787 R=0.671 mAP50=0.727 mAP50-95=0.444 from UL's aggregated metrics). Per-sample WL signals flow through 6 channels per split (train/val × bbxs/clsf/dfl) plus miou/{split} on every batch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ul05_wl05: revert false per-sample signal capture The previous commit (2c66ea2) claimed per-sample signal emission worked end-to-end. It does NOT — though the script ran without an unhandled exception (we silently dropped errors in a try/except), the actual WL `add_scalars` call was never reached: * forward hook captured preds correctly via the monkey-patched `underlying.forward` (the register_forward_hook didn't work because DetectionModel.loss() calls self.forward(image) directly, bypassing __call__ and forward hooks). * preprocess_batch / validator.preprocess wraps captured batch. * BUT — PerSampleDetectionLoss.forward, called with our captured (preds, batch), crashes inside UL's v8DetectionLoss internals on `batch["cls"].view(-1, 1)` with "Type must be a sub-type of ndarray type". The PerSample* classes were designed for main.py's manual loop and don't slot onto UL 8.4.51's training-mode dict pred output once the per-sample slicing has happened. So nothing was actually logged to WL. Reverting to a clean shell that runs end-to-end with no per-sample emission and no false promises. Plumbing for capture (forward-patch + preprocess-patch) was removed too — it'd need to come back when we have a working per-sample computation surface. That'll likely require either: (a) decoding preds ourselves consistently with main.py's flow, or (b) hooking AFTER UL's criterion runs to read its per-anchor loss tensor and reduce per-sample (still inside the hook idiom). Per-sample work is deferred; the trajectory file is honest again. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ul05_wl05: per-sample cls + batch box/cls/dfl + aggregated val via hooks Implement option-1 MVP from the integration-hook-don't-recompute memo. No UL method is overridden; signal capture rides on: * forward_hook(criterion.bce) → per-anchor cls tensor of shape (bs, num_anchors, nc). Reduced via .sum(dim=(1,2)) gives per-sample cls. * trainer.loss_items → batch-level (box, cls, dfl) scalars already aggregated by UL; just read and emit. * validator.metrics.results_dict → aggregated val scalars (precision/recall/mAP50/ mAP50-95/fitness) read at on_val_end. `_Sink(nn.Module)` is a passt…
_slowUpdateInternals (WL-ViewRefresh) rebuilt the sample view by calling _pull_into_all_data_view_df, which pulled the full combined frame once to check .empty and then called get_collapse_annotations_to_samples_df, which pulled it AGAIN internally. On large datasets (e.g. 70k-row MNIST) that duplicated a full DataFrame copy + buffer merge + proxy conversion every refresh, tripping the >1s LockWatchdog SLOW warning. get_collapse_annotations_to_samples_df now accepts an already-materialized frame; the caller pulls once and passes it through. Output is byte-identical; get_combined_df calls per rebuild drop 2 -> 1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The classification example hardcoded every dataloader kwarg (shuffle, is_training, compute_hash, preload_*, enable_h5_persistence) inline in the wrap cell. Consolidate them into a nested config["data"]["train_loader"] / ["test_loader"] block, keyed by loader_name, and unpack each block straight into wl.watch_or_edit(..., flag="data"). The wrap cell is now declarative and adding a loader means adding a config block. Behavior is unchanged (identical kwargs). Also drops the dead max_train_samples/max_test_samples keys (the dataset no longer accepts max_samples).
Stopping the training cell sends SIGINT to the kernel's process group, which also reached the bore child process and killed the tunnel (gRPC backend, a kernel thread, survived; the tunnel did not — confirmed: port 50051 still listening, proc.poll() non-None). Launch bore with start_new_session=True so it runs in its own session/group, immune to the cell-stop SIGINT. The UI now stays connected across training stop/resume. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ar export write_history: json.dump was passed orient=orient, but orient is a pandas arg that json.dump rejects (TypeError: unexpected keyword argument 'orient') — so every JSON history dump crashed, including wl.write_history() in the loop. The payload is already a dict of per-section record lists (the documented, tested contract), so just drop the bogus kwarg. Fixes 55 failing write_history tests. write_dataframe: the orient="columns" default (compact, ~6x smaller, round-trips with pd.read_json) is intentional, but 18 tests still asserted the old list-of-records shape. Add a _records() helper that normalizes the columnar JSON back to records for row-level assertions, and update the one structural test to assert the columnar dict. Code behavior unchanged.
Walkthrough of the data-curation loop in Weights Studio: expand the grid, sort by train-loss-CE + histogram to see the easy/hard bimodal split, use the agent to tag hard_ex (>2.45) / easy_ex (<1.45) and discard the untagged middle (~60k -> ~3-4k samples), fine-tune ~6.5k steps, then evaluate on test_loader (acc ~0.86 -> ~0.95-0.99). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…parse-checkout Replace the git sparse-checkout of BDD_subset/ with a DATASET_URL Drive share link: download the zip via gdown (fuzzy=True handles the large-file scan confirmation), unzip it, and auto-detect the dataset root (the folder holding images/, so a wrapping top-level dir is fine). config["data_root"] now points at the detected DATA_ROOT. Updated the intro/data-fetch markdown accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…square transforms.Resize(size=image_size) with a single int rescales only the shorter edge and keeps aspect ratio, so non-square inputs (e.g. a user's Drive dataset) produced [128, 227] feature maps that mismatched the mask in binary_cross_entropy (target [227] vs input [128, 227]). The U-Net input is square (input_shape=(1,C,image_size,image_size)), so resize both image and mask to an explicit (image_size, image_size). Also squeeze multi-channel (RGB/RGBA) masks to a single-channel class-id map so gt matches the [H,W] prediction. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… per-instance) The per-instance path built a list of instance masks per sample and looped over them in the criterion, where a mask reduced to 1-D and broke F.binary_cross_entropy (target [128] vs input [128,128]). Switch to standard sample-wise semantic segmentation: - dataset __getitem__ returns one [H,W] class-id mask; seg_collate stacks to [B,H,W] - criterions: PerSampleCE (loss, per-pixel CrossEntropy meaned per sample) and PerSampleDice (metric, foreground Dice), each returning [B] - train/eval use the mask directly; removed _run_instance_signals, _instance_batch_idx, PerInstance*/instance signals, and the instances collate path - markdown updated to describe the sample-wise pipeline Verified: criterions return [B], Dice in [0,1], CE backprops, ignore_index respected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Aligns the ws-segmentation notebook's model/data/loss with weightslab_kitchen/.../ws-seg-wow-moment-bdd so both trainings are directly comparable: - Prefer the local bdd8k dataset (byte-identical to the reference run) over the Drive toy subset when it's present on disk, with eval-split auto-detection (train/test vs train/val layouts). - ignore_index 255 -> 0, image_size 180 -> 720, train batch_size 2 -> 8, training_steps_to_do 12000 -> 10000, eval ratio 5 -> 256 (all match the reference config.yaml). - Loss is now the reference's combined CE+Dice (CrossEntropySegLoss), ported verbatim including its class-weighting; the test criterion is unweighted to match. Added torchmetrics IoU (macro, ignore_index=0) alongside the existing per-sample Dice metric for direct comparison.
…loss PerSampleMultiClassDiceLoss already returns a dice LOSS (1 - diceScore), but the combined-loss forward was doing (1 - dice) again on top of that, which algebraically reduces to the raw dice SCORE — so a better dice score was increasing the loss instead of decreasing it. Use the dice loss directly. Same fix applied in ws-seg-wow-moment-bdd/criterions.py (separate repo).
Fixes ruff F401 lint failure in CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Collab compatibility
Dep issues like NumPy> 2 and protobuf
Set min tokens for a model at initialization. Some models need a max token set