moe_w2: loader-level skip — boot DeepSeek-V4-Flash from the pack / planes cache without re-staging

[anoane]

What

Wire the existing moe_w2 loader-skip into the 2-bit Mxfp4MoEMethod path, so a DeepSeek-V4-Flash layer already served by the pack store (VLLM_MOE_W2_STORE_DIR) or the GPU-resident planes cache (VLLM_MOE_W2_PLANES_CACHE) boots straight from the cache — no per-boot re-quant of the 149 GB fp8 checkpoint, no host-RAM staging, no swap.

Why

Stock behaviour stages every expert of every layer into host RAM before building the 2-bit planes — a ~51 GiB transient that forces swap (and multi-minute boots) on a small single-GPU host, on every start. The loader-skip already exists for the nvfp4 path (build_layer_planes_nvfp4 / ModelOptNvFp4FusedMoE.create_weights); this extends the same skip to the 2-bit path so the planes cache / pack is actually used to boot.

How

Two files, mirroring the existing nvfp4 skip:

• mxfp4.py Mxfp4MoEMethod.create_weights: gate the per-expert CPU staging on not moe_w2_cubit.plan_pack_skip(layer); use the create-time build key so the pack sidecar probe and the plane build agree on the layer index.
• moe_w2_cubit.py build_layer_planes: when a layer was loader-skipped (_moe_w2_pack_skip), take the shapes from the create-time stash and serve the planes from the pack (_try_skip_requant) or the planes cache (_consume_planes_cache) instead of reading the never-staged checkpoint.

No new env vars — both paths reuse the STORE_DIR / PLANES_CACHE machinery already wired for nvfp4. On a pack miss (genuine first boot) the classic all-layers staging path is unchanged.

Patch / fingerprints

patch/vllm-moet-v0.24.0.patch regenerated from the fork branch (current base 0fd0e145 + this change). patch/FILES.txt unchanged (same files touched); patch/SOURCE.txt advanced to the fork commit carrying the change.

Note on SOURCE.txt: it points to the vllm-fork commit that carries these two edits on top of the current moet-v0.24.0 tip (0fd0e145). The change is entirely the mxfp4.py + moe_w2_cubit.py hunks in this patch — apply them to your moet-v0.24.0 branch and re-run tools/check_patch_files.py --update to stamp your own fork SHA on merge.

README

Adds one single-card block under DeepSeek-V4-Flash — 1x RTX 6000 Pro (96 GB, SM120):

• Config A (max quality): 6 GiB FP4 pool, MTP off, min_p=0.05, ~75 tok/s @ 256K, fills the card.
• Config B (throughput/concurrency): no pool, with an MTP-on-vs-off concurrency table — MTP k=2 wins for few streams (single 115 vs 89 tok/s, peaks ~1122 @ C=64), MTP off scales monotonically to ~2085 @ C=256.

Full docker + vLLM commands, every env/flag justified, a needle-in-a-haystack section (recommended 256K, absolute-max boots-only with warnings, pool-vs-retrieval tradeoff), and a link to the prebuilt cache on Hugging Face.

Prebuilt cache

The ~588 GB planes cache + FP4 packs are published so you can boot without re-quantizing: https://huggingface.co/anoane/DeepSeek-V4-Flash-vllm-moet-sm120-cache (mount planescache/ -> /planescache, moet_store/ -> /store; the model config/tokenizer still comes from deepseek-ai/DeepSeek-V4-Flash).

Testing

• git apply --check patch/vllm-moet-v0.24.0.patch on the v0.24.0 tag: clean.
• Both changed files py_compile clean; the applied result is byte-identical to the fork commit.
• Config A / B measured on 1x RTX 6000 Pro (numbers in the README).