moe_w2: skip redundant fp13/fp2 FP4-tier reads + parallel planes-cache consumption on warm boot (DS4-Flash load 5min -> ~100-120s)

[HH1162]

moe_w2: skip redundant fp13/fp2 FP4-tier reads + parallel planes-cache consumption on warm boot (DS4-Flash load 5min -> ~100-120s)

What

Two hot-path fixes in moe_w2_cubit.py for warm DeepSeek-V4-Flash boots (all 44 W2 layers already quantized in the planes cache and delta store):

1. _consume_planes_cache always re-reads the FP4 delta-tier files (fp13, fp2, ~1.25 GiB/layer) from the planes cache and re-stages them into the delta store on every boot, even when the store already persists that layer. On 44 layers that repeats ~55 GiB of useless host traffic and store writes per warm boot.

2. Planes-cache hits in build_layer_planes_nvfp4 are consumed one layer at a time by the caller, so a warm boot walks all 44 layers serially (~4s/layer), which is most of the load time even after the redundant reads are skipped.

This PR adds (a) the missing delta_present probe so layers already in the FP4 delta pack skip the fp13/fp2 read and the redundant _stage_fp4_host, and (b) a small deferred-build queue so cache-hit layers are consumed by the existing parallel builder in batches. Measured effect on a 44-layer warm boot: load time drops from ~5+ minutes to ~100-120s, with no change to serving behavior.

Root cause

Redundant FP4 staging. The delta tier (VLLM_MOE_W2_DELTA_GB / VLLM_MOE_W2_BASE_CACHE_GB machinery in moe_w2_delta.py) persists each layer's FP4 planes on first boot. The planes-cache consumption path still requests want_fp4=True and unconditionally calls _stage_fp4_host on every boot, then add_layer discards the staged data because the layer already exists. Nothing checks whether the delta pack already covers the layer.

Serialized cache consumption. The cache-hit branch of build_layer_planes_nvfp4 (and the loader-skip branch) call _consume_planes_cache inline. Each cache read + H2D + FP4 stage is ~2-4s because there is no parallel batching of the 44 independent layers.

How (single file: vllm/model_executor/layers/quantization/utils/moe_w2_cubit.py)

Delta-pack skip. In _consume_planes_cache: when an FP4 tier is configured, probe pack_has_layer("delta", layer_key, ...) first. On a delta-pack hit:

• load the planes cache with want_fp4=False (skip the fp13/fp2 files entirely), and
• skip _stage_fp4_host — the store already holds this layer's FP4 planes.

On a delta-pack miss (genuine first boot) the existing behavior is unchanged: fp13/fp2 are read and staged as before.

Parallel deferred-build queue. Two parts:

• build_layer_planes_nvfp4 — in both the loader-skip branch and the normal staged branch, a planes-cache hit is no longer consumed inline. Instead the layer is enqueued via queue_deferred_build(layer, layer_key, _layer_cutoff() + 1); a layer already owned by an active batch (key in _BATCH_ASSIGNED) is still consumed directly, which prevents recursive batch re-fire.
• queue_deferred_build / build_layer_planes_batch — the queue fires build_layer_planes_batch at VLLM_MOE_W2_BUILD_BATCH_SIZE (default 12) or when the batch covers the expected W2-layer count. The batch runs 4 workers (VLLM_MOE_W2_BUILD_WORKERS, default 4) over disjoint layer keys; the final short batch builds serially to cap staging peak. _BATCH_ASSIGNED tracks keys owned by the active batch (populated on dispatch, cleared in try/finally), and the duplicate guard covers _BATCH_PENDING / _BATCH_ASSIGNED so a layer is never enqueued twice.

Build-time and config: no new environment variables; the delta tier and the batch-size/work-count knobs are the existing ones.

Why not just delete the cache

The planes cache serves GPU-resident plane configs: planes13, sc13, planes2, sc2 must still be materialized on the GPU every boot (~1.69 GiB/layer). The delta pack is the complementary host-resident tier. The fix removes the duplicated FP4 host staging and the serialized read pattern, not the cache itself.

Verification

• DeepSeek-V4-Flash, 44 W2 layers (43 main + 1 MTP), TP1, warm planes cache + warm delta store.
• First boot: delta pack created as before (no change).
• Warm boot: fp13/fp2 no longer read; every layer still logs planes from cache for the GPU-resident planes; delta store contents unchanged; layers 2-42 complete in parallel batches.
• Load time: ~5+ minutes -> ~100-120s on the affected tree.
• py_compile clean; 44-layer batch-trigger simulation passes (all layers enqueued, _BATCH_PENDING / _BATCH_ASSIGNED drained).

Scope

• Touches only vllm/model_executor/layers/quantization/utils/moe_w2_cubit.py.
• The delta tier and planes cache themselves are untouched.