Regression(llama-cpp backend): Models that used to load now fail with `cudaMalloc failed: out of memory`

[ackalker]

LocalAI version:
v4.5.6 (installed via Linux launcher)

Environment, CPU architecture, OS, and Version:
Linux miki-linux-ai 7.1.2-3-cachyos #1 SMP PREEMPT_DYNAMIC Mon, 29 Jun 2026 14:34:35 +0000 x86_64 GNU/Linux
CPU: Ryzen 9950X3D
RAM: 64 GB
GPU: RTX 5080

Describe the bug
When I try to use a model such as gemma-4-12b-it-qat-q4_0, or other variants in the Gemma 4 family, the model fails to load. The llama-cpp backend errors out when it tries to allocate ~ 80 GB of VRAM, vastly exceeding my GPU's capacity of 16 GB.
I can still use models such as ornith-1.0-9b without issues.

This appears to be a regression: I could use these Gemma 4 models without problems up until a week ago (on 26-06-2026 at the latest, sorry, I can't be more precise).
Apart from several updates of both LocalAI and various backends, nothing in my configuration has changed,

To Reproduce

1. Install the gemma-4-12b-it-qat-q4_0 model from the "Install Models" gallery.
2. Start a new chat with the model.

Expected behavior

• The model should load without errors and generate responses normally.
• Using an RTX 5080, max VRAM usage should be ~ 14 GB.

Logs
llama-cpp backend log for the failed load of gemma-4-12b-it-qat-q4_0 (full exported log at https://gist.github.com/ackalker/1b50b081493523e9e46a66d4e20cbc5d
):

2:04:30 PM.729stderr0.03.385.263 W llama_kv_cache_iswa: using full-size SWA cache (ref: https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
2:04:30 PM.736stderr0.03.392.016 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 81920.00 MiB on device 0: cudaMalloc failed: out of memory
2:04:30 PM.736stderr0.03.392.019 E alloc_tensor_range: failed to allocate CUDA0 buffer of size 85899345920
2:04:30 PM.740stderr0.03.395.824 E llama_init_from_model: failed to initialize the context: failed to allocate buffer for kv cache
2:04:30 PM.740stderr0.03.395.826 E common_init_result: failed to create context with model '/home/miki/.localai/models/llama-cpp/models/gemma-4-12B-it-qat-q4_0-gguf/gemma-4-12b-it-qat-q4_0.gguf'
2:04:30 PM.740stderr0.03.395.827 E common_init_from_params: failed to create context with model '/home/miki/.localai/models/llama-cpp/models/gemma-4-12B-it-qat-q4_0-gguf/gemma-4-12b-it-qat-q4_0.gguf'

llama-cpp backend log for the successful load of ornith-1.0-9b (full exported log at https://gist.github.com/ackalker/31dbef1e5a6babd6f2569e7e683c92cd)

Additional context

• An issue that may be related to this one: Multi-GPU model loading fails with CUDA OOM after Blackwell VRAM/concurrency changes #10485
  It shows similar memory allocation error.
• A friendly neighborhood AI suggested that the change to llama.cpp mentioned in the log above, which adds support for interleaved SWA cache (used by Gemma models) might be related: kv-cache : add SWA support ggml-org/llama.cpp#13194 .
  Additional explanation of the --swa-full flag: Clarification on --swa-full ggml-org/llama.cpp#24543.

[gregoryca]

same issue here. I am using a GTX 1080 and running the latest container (localai/localai:master-gpu-nvidia-cuda-12) Images that are recommended to me (qwythos-9b-claude-mythos-5-1m) won't load. I have downloaded: gemma-4-e2b-it and that runs fine, but when i run: gemma-4-e4b-it it won't run as it says

start_llama_server: starting llama server start_llama_server: waiting for model to be loaded 0.01.953.868 I system info: n_threads = 6, n_threads_batch = -1, total_threads = 12 0.01.953.871 I 0.01.953.896 I system_info: n_threads = 6 / 12 | CUDA : ARCHS = 500,610,700,750,800,860,890,900,1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 0.01.953.896 I 0.01.953.957 I srv load_model: loading model '/models/gemma-4-E4B-it-Q4_K_M.gguf' 0.02.231.699 I srv load_model: [mtmd] estimated worst-case memory usage of mmproj is 1204.56 MiB (took 277.73 ms) 0.02.231.703 I common_init_result: fitting params to device memory ... 0.02.231.704 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on) 0.02.918.064 W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 99999999, abort 0.03.405.546 W load: control-looking token: 212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden

Both models have the same size (gemma-4-e2b-it & gemma-4-e4b-it)

[mudler]

thanks for the heads up. This is on me as I was trying to improve defaults and didn't noticed in my tests, but #10611 actually allocates full context for every layer, making exceed memory on SWA models (such as gemma). Working on a fix right now, and will cut a release right after.

[gregoryca]

No worries! Thanks for your hard work! If this is a known issue I think we can close this bug.

[ackalker]

Thank you very much for looking into this, much appreciated.

[mudler]

This should be fixed on 4.6.0, if you can try it out and confirm that solves your issues that'd be appreciated, thanks!

[gregoryca]

In my case it's resolved. Both gemma models load without issues!

Thanks for the quick fix

[ackalker]

Working great here too, thanks for the fix!

[mudler]

perfect, closing then 👍 Thanks for testing it out!