Pr/prefetch weights by giveen · Pull Request #291 · TheTom/llama-cpp-turboquant

giveen · 2026-08-11T14:14:16Z

Overview

This is the fourth in a series of PR's to improve MoE performance

AI usage disclosure: Yes

)

* Add tagged thinking tool parser * chat : refactor and add permute helper * cont : add support for <tool_call> omission * cont : update tool delimiters * cont : add comment for qwen3-coder * cont : fix trigger pattern for <function --------- Co-authored-by: Bart de Boer <bart.deboer@gmail.com>

…l-org#26162) Incrementing `ref_count` at the beginning is important later in the `free()` method of the `ggml_backend_opencl_context` at program end. If we do not increment the `ref_count`, the result would be -1 here, and consequently, the profiling data would not be flushed and written. ( #ifdef GGML_OPENCL_PROFILING )

* convert: add option to create separate dspark GGUF * add --no-nextn * fix convert bug

The dspark- files resolve like the other speculative sidecars: the -hfd tag applies to them, a requested sidecar resolves without a full model at the tag, and an explicit -md selection disables the discovery. When no type is requested, dspark outranks dflash in the auto-selection since its sidecar carries the extra Markov head.

- Implement GGML_OP_DSV4_HC_COMB, GGML_OP_DSV4_HC_PRE, and GGML_OP_DSV4_HC_POST with SIMDgroup register and shuffle optimized kernels. - Add Metal dispatch and support plumbing and test the production Sinkhorn iteration count and embedding width. Assisted-by: Codex Co-authored-by: Thiago Padilha <thiago@padilha.cc>

* feat(silu_back): implemented silu_back op for f32 * fix(silu_back): removed redundant asserts in ggml-metal-ops.cpp function ggml_metal_op_silu_back.

* metal: implement F16 Lightning Indexer - Implement GGML_OP_LIGHTNING_INDEXER for 128-dimensional, 64-head inputs with F32 queries and weights plus F16 keys and masks. - Add tiled and tail kernels and test KV lengths around 8- and 64-element boundaries. llama-bench (--mmap 1, -fa 1, -p 512, -n 128; d=0/10k/20k/30k): Before: - pp512: 153.73 ± 0.87 t/s - tg128: 8.91 ± 0.04 t/s - pp512 @ d10000: 73.90 ± 0.39 t/s - tg128 @ d10000: 8.66 ± 0.03 t/s - pp512 @ d20000: 45.83 ± 0.18 t/s - tg128 @ d20000: 8.26 ± 0.03 t/s - pp512 @ d30000: 33.40 ± 0.21 t/s - tg128 @ d30000: 7.94 ± 0.01 t/s After: - pp512: 155.19 ± 0.91 t/s - tg128: 8.95 ± 0.04 t/s - pp512 @ d10000: 86.95 ± 0.69 t/s - tg128 @ d10000: 9.00 ± 0.05 t/s - pp512 @ d20000: 62.01 ± 0.45 t/s - tg128 @ d20000: 8.68 ± 0.04 t/s - pp512 @ d30000: 49.18 ± 0.33 t/s - tg128 @ d30000: 8.60 ± 0.02 t/s Assisted-by: Codex * metal: stage Lightning Indexer K tiles - Stage and dequantize K in F16 threadgroup memory before simdgroup matrix loads. - Zero-fill partial tiles and guard stores so all KV segments use the same numerical path. - Support F32, F16, BF16, Q4_0, Q4_1, Q5_0, Q5_1, and Q8_0 K caches. llama-bench (--mmap 1, -fa on, -p 512, -n 128; d=0/10k/20k): - pp512: 160.38 +/- 1.01 t/s - tg128: 9.08 +/- 0.03 t/s - pp512 @ d10000: 88.37 +/- 0.46 t/s - tg128 @ d10000: 9.07 +/- 0.04 t/s - pp512 @ d20000: 62.53 +/- 0.46 t/s - tg128 @ d20000: 8.84 +/- 0.03 t/s Assisted-by: Codex * dedup Lightning Indexer constants, fix flaky test * cont : fix whitespace --------- Co-authored-by: forforever73 <690105611@qq.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* llama : MTP support for DeepSeek V3.2 * model : no need to include MTP layers during DeepSeek V3.2 model type discovery --------- Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>

* mtp for qwen3nex * fix for python type-check * Fix to compute num_mtp from directly mtp layer * define opt_num_mtp_layers in _QwenMtpMixin and fix some comments * Fix for python type check * Update gguf-py/gguf/constants.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * rebase and add load_mtp flags * Update src/models/qwen3next.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Update src/models/qwen3next.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* server: (tools) add x-tool-cwd header * reuse str_to_lower from server-models

…ml-org#26508) * server: add notice for upcoming default port change 8080 --> 6631 * add link to PR * correct to 9931

@gaugarg-nv

* CUDA: Fix data-races when reusing block_reduce block_reduce currently doesn't resync after reading from SMEM, causing potential data-races when reusing SMEM for multiple reductions. One may consider simply always adding this in block_reduce, but this comes at a potential perf cost * double-buffering for single-row softmax * double-buffering for norm as well * Add comment * Add explanatory comment to block_reduce * Specify need for + do memory barrier only in multi-warp scenario * Implement review-suggestion from @gaugarg-nv

* sampling: enhance penalty handling in common_sampler_init - Set default value for penalty_last_n based on model context if not specified. - Ensure penalty_last_n and n_prev are non-negative. - Update llama_sampler_penalties structure to inherit from llama_sampler_backend and add backend input handling for penalties. - Implement backend initialization and application logic for penalties, including frequency and presence adjustments. * tests: add backend penalties sampling tests and utility functions - Introduced `accept_prompt` and `unique_prompt_tokens` functions to handle prompt acceptance and token uniqueness. - Implemented `compare_penalties_logits` to compare logits from backend and CPU samplers with penalties. - Added `test_backend_penalties_sampling` to validate backend penalties with various configurations. - Enhanced the test suite for better coverage of penalty handling in sampling. * sampling: add support for top-k penalties in backend sampling * sampling: add fix to ensure stable numerical results. Preserve masked logits as -Inf and no longer generate NaN. * sampling: enhance penalty comparison tests with masking penalties logic * add comments on padding * sampling: add comments on modifications * add the unit test to cover masked-out token as -INF * validate repeat penalty to ensure it is finite and greater than 0; add tests for invalid values * refactor: test functions to share logic and be less verbose * add test to cover case where previously penalized token is not part of candidates * remove comments * remove redundant penalty_last_n initialization and validation in common_sampler_init * add support for penalties in sampler chain with configurable positions * add validation for penalty parameters and enhance tests for non-finite values * add context parameter to common_sampler_init and set default for penalty_last_n * add llama_n_ctx parameter to common_sampler_init for improved sampler initialization * replace penalty_last_n x n_candidates comparison matrix with a vocabulary-sized count tensor * add tests for backend penalties sampling without filler entries , token_count.size() == n_active == n_max == 64 * add test for backend penalties sampling after top-p with large history window * remove as unused * add is_disabled method, tensor logits reshape, add rest review suggestions * clarify comment

…g#26474) Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>

* Move MSA logic from llama-kv-cache into llama-kv-cache-msa * cont : minor * cont : ws fix --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* add a direct size condition for `large` weights; the original dimension condition is insufficient -- q6_K lm_head for gemma-4 E2B has [1536, 262144], which is big enough to slowdown gemv_noshuffle but does not satisfy the dimension condition (ne0 >= 2048)

* ggml: use dynamic allocation for split graph inputs Replace fixed-size GGML_SCHED_MAX_SPLIT_INPUTS arrays with dynamically allocated buffers in the backend scheduler. This fixes crashes when loading wide MoE models (Gemma 4, Qwen MoE, Mixtral, DeepSeek) on multi-backend setups where graph splits exceed 30 input tensors. - split->inputs: dynamic array with grow-on-demand - sched->graph_inputs: dynamic array with grow-on-demand - graph_size calculation now uses actual input count instead of fixed constant * cont : clean-up --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* server: add get_info tool * fix --rpc in docs * server: harden get_info probe result handling Report the OS as unknown when the probe process fails to spawn or times out, so the diagnostic text from run() is never returned as an OS name. Strip the probe output on both ends, which also drops the blank line that ver prints before the version on Windows. Name the output and timeout limits, and report an unreadable working directory as unknown instead of an empty string. * server: simplify get_info result handling Drop the named limits and the working directory error branch, keeping the probe result handling to a single expression. --------- Co-authored-by: Pascal <admin@serveurperso.com>

@ngxson

…#26172) * tests: add model resolution test on synthetic repo listings Include download.cpp and arg.cpp inside a namespace with hf_cache monkey patched to serve hardcoded listings, so the resolution and the model handler assembly are tested end-to-end through the real CLI parsing, without modifying the tested code and without network access. Covers the primary, shard, mmproj, sidecar and preset resolution on layouts mimicking real vendor conventions, replays every case on permutations of the listing to assert determinism, and asserts the final wired paths for the spec type auto-selection, the -md precedence and the fallback suppression. * tests: keep model resolution checks active and let the handler resolve Replace assert with a REQUIRE macro alive in Release builds, key the fake hf_cache by repo id so the real handler init resolves every plan itself, check the exact shard sets, restrict the permutation exception to the order dependent picks, and cover dflash and eagle3. * tests: fix model resolution build on fatal warnings CI and Windows The namespaced copy of the sources leaves many static functions unused in this TU, exempt it from the unused warnings. Pre-include the windows headers so arg.cpp does not pull them inside the namespace. Declare the renamed copies of the download.h functions, verbatim from the header and renamed in sync by the macros, so missing declaration and missing prototype warnings are satisfied on every toolchain. * tests: fix winsock inclusion order for the model resolution test WIN32_LEAN_AND_MEAN and winsock2.h before windows.h, so http.h does not redefine the socket types afterwards. * tests: link cpp-httplib to the model resolution test The test compiles its own copy of download.cpp, which calls httplib directly, and the private link of llama-common does not propagate the symbols under lld-link. * common_http_client * common: finish the http client wrapper Add the virtual Head, Get and Post methods and the passthrough setters to the common_http_client skeleton, move follow_location into the constructor, expose the underlying client for the ranged pull path, and rename the missed common_http_client_init call sites. * tests: rewrite model resolution on the http client stub Replace the namespace inclusion of the sources by a plain TU: the common_http_client factory returns a stub serving hardcoded HF API responses, so the real hf_cache parsing, resolution and CLI handler run against synthetic listings in an isolated cache directory. Failures print the named case, the reordering and the actual versus expected values, the assembly cases use the full command line as context, and the empty result cases are checked once to keep the logs short. * tests: fix the model resolution on Windows and the builds without TLS Assert the exact expected paths composed like the cache does instead of suffix matching on forward slashes, set the environment portably, and serve the stub through an http endpoint so the builds rejecting the https scheme still reach it. Pause the log so the negative cases can be replayed on every reordering. * tests: make the model resolution failures self explanatory Resume the paused log before the failure report so the CI shows why the tested code bailed, and format the stub oids portably. * common: hold the http client factory behind exported functions The factory was an inline variable, and the Windows shared builds export functions but not data, so the executable and the DLL each had their own instance: the stub installed by the test was invisible to the library, which reached for the real endpoint and resolved nothing. Route the creation through functions compiled into the library and format the stub oids portably. * common: add the http client factory source missed in the previous commit * common: typedef the http client factory callback Address review from @ngxson * tests: serve the model resolution repos over the loopback Replace the client stub by a real httplib server bound to the loopback, so no C++ object crosses the module boundary anymore and the library exercises its own client and transport end to end. The debug shared build on Windows crashed inside the stubbed path. * common: add portable common_get_env and common_set_env helpers Address review from @ngxson * common: drop the http client factory left without a caller The loopback server made the stub substitution unnecessary, the client init builds the real client directly again. * common: read the model endpoint through the env helpers * nit: drop the stub leftovers from the model resolution test * common: align common_set_env and isolate the test cache per run The POSIX branch now behaves like _putenv_s, so the helper has a single contract on every platform, and common_get_env already reads an unset and an empty variable alike. The model resolution test keys its cache directory on the loopback port, where two concurrent runs on the same machine used to share one directory and the initial cleanup of either wiped the other. * tests: move the model resolution server into main * tests: support the DSpark sidecar resolution * common: revert the http client to the plain httplib client address review from @ngxson --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>

…v-cache Keep both Qwen3-TTS and Muse Glimmer projector types after merging the base. Assisted-by: Buffy

…r is met Automatic mode required two selected CUDA devices, which silently disabled the cache on single-GPU systems even when free VRAM cleared the 1 GiB slab floor. The floor exists to prevent pool thrashing; one device can satisfy it, so the count gate was wrong. - query_config advertises min_devices = 1 for automatic mode - the session-create two-device gate is removed - routing goes dormant only when no device satisfies the slab floor - automatic mode is derived from minimum_slab_bytes > 0 (the previous min_devices > 1 derivation broke once min_devices became 1) - add cache-single-gpu-auto test for the positive path - document single-device auto and the reserve knob in CUDA-MOE-CACHE.md Assisted-by: Buffy

Bring the moe-cache backend set (CUDA/Metal/Vulkan expert cache), the single-GPU auto relaxation, and the Muse Glimmer base sync onto the prefetch-weights branch. Resolved the scheduler-wiring conflicts in ggml-backend.cpp, llama-context.cpp, and llama-bench.cpp by taking the authoritative both-features-merged content from the moe-cache branch, where prefetch and moe-cache already coexist (verified: the 289-to-moe-cache delta for these files is exactly the prefetch feature, zero moe-cache divergence). Assisted-by: Buffy

The conditional worked for MSVC shared builds but disagreed with the macro on two other configurations: - MinGW defines _WIN32, so the condition took the no-explicit-extern arm while GGML_API took the visibility arm, which has no extern. That makes the line a second definition again, the exact bug the comment above it warns about. - Static builds have no GGML_SHARED, so GGML_API is plain 'extern' and the explicit extern was still emitted, leaving 'extern extern' and C2159 on MSVC static. Adding extern to the visibility branch makes GGML_API carry it on every path, so the use site needs no condition and there is nothing to keep in sync later. Expansion on all five configurations: static extern int ... shared ELF __attribute__((visibility("default"))) extern int ... shared MSVC dllexport __declspec(dllexport) extern int ... shared MSVC dllimport __declspec(dllimport) extern int ... shared MinGW __attribute__((visibility("default"))) extern int ... This also matches what PRs TheTom#289 and TheTom#291 already carry, so the two will no longer conflict.

cabelo and others added 30 commits August 1, 2026 20:53

vendor : update BoringSSL to 0.20260730.0 (ggml-org#26353)

815a2a5

vulkan: extend topk_moe fusion to support sqrt(softplus) (ggml-org#26124

a7a6d0d

)

test: fix some CI errors (ggml-org#26415)

11924d4

ggml-webgpu: add support for f16 repeat (ggml-org#26307)

7a2db1a

model : load MiMo V2 MTP tensors only if used (ggml-org#26412)

75587a0

sycl: fix classification of iGPUs (ggml-org#26105)

272700b

DeepseekV4 MTP + DSpark (ggml-org#25784)

596a579

convert: add option to create separate dspark GGUF (ggml-org#26452)

3581ba0

* convert: add option to create separate dspark GGUF * add --no-nextn * fix convert bug

opencl: limit local workgroup size for GLU operation (ggml-org#26383)

0ab9d6f

metal : add F16 support for bin ops (ggml-org#26465)

9d21b57

metal : add SILU_BACK (ggml-org#25982)

221f0f6

* feat(silu_back): implemented silu_back op for f32 * fix(silu_back): removed redundant asserts in ggml-metal-ops.cpp function ggml_metal_op_silu_back.

llama : MTP support for DeepSeek V3.2 (ggml-org#26457)

2b63e06

* llama : MTP support for DeepSeek V3.2 * model : no need to include MTP layers during DeepSeek V3.2 model type discovery --------- Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>

server: (tools) add x-tool-cwd header (ggml-org#26420)

f2b52a8

* server: (tools) add x-tool-cwd header * reuse str_to_lower from server-models

server: add notice for upcoming default port change 8080 --> 9931 (gg…

0b14b87

…ml-org#26508) * server: add notice for upcoming default port change 8080 --> 6631 * add link to PR * correct to 9931

llama : allocate indexer cache only in "full" indexer layers (ggml-or…

563dec8

…g#26474) Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>

model: M3: Move MSA into a new memory implementation (ggml-org#26338)

67d5978

* Move MSA logic from llama-kv-cache into llama-kv-cache-msa * cont : minor * cont : ws fix --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

graph : fix unused input tensors in minimax m3 graph (ggml-org#26519)

c50b34a

vocab : validate default special token ids (ggml-org#26506)

e8e06f7

github-actions Bot added examples Apple Metal Vulkan OpenCL SYCL Ascend NPU Hexagon WebGPU IBM zDNN AMD ZenDNN OpenVINO CUDA labels Aug 11, 2026

giveen added 3 commits August 11, 2026 08:45

pr/metal-vulkan-v1: resolve merge conflicts with feature/turboquant-k…

20913dd

…v-cache Keep both Qwen3-TTS and Muse Glimmer projector types after merging the base. Assisted-by: Buffy

github-actions Bot added documentation Improvements or additions to documentation server testing devops model jinja parser server/ui conversion mtmd vendor labels Aug 11, 2026

This was referenced Aug 11, 2026

Pr/cuda moe cache #288

Merged

Fix MSVC C2159: duplicate extern from GGML_API on Windows #292

Open

giveen marked this pull request as draft August 12, 2026 14:36

Navigation Menu

Uh oh!

Pr/prefetch weights - #291

Pr/prefetch weights#291
giveen wants to merge 133 commits into
TheTom:feature/turboquant-kv-cachefrom
giveen:pr/prefetch-weights

giveen commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Uh oh!

Conversation

giveen commented Aug 11, 2026

Overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants