Model residency in memory switch / toggle

[shpati]

The following is observed on a Mac Studio Ultra M1 128GB machine running DeepSeek-V4-Flash-Q2 via ds4-server.
The running command is:

./ds4-server --ctx 131072 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192 -m DeepSeek-V4-Flash-Vision-Exp-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8.gguf

ds4 used to unload the model weights from memory when the LLM is idle, which is handy if you use the machine for other processes. Ref commits such as 3da2da9 and earlier ones. In the latest versions the model stays loaded in memory when the LLM is idle, thus preventing other memory intensive processes to run properly.

Would it be possible to have a switch / toggle to be entered as a command line argument when running ds4-server such that the user can decide whether they want to have the model stay resident in memory or not?

[emilianbold]

This looks like a sideeffect of e2c3907

Could you try DS4_METAL_DISABLE_QUEUE_KEEPALIVE=1 to opt out and see if it still happens?

[shpati]

Thanks a lot @emilianbold, this solves the issue!

I used the following command(s) and the model unloads when idle:

export DS4_METAL_DISABLE_QUEUE_KEEPALIVE=1 && ./ds4-server --ctx 131072 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192 -m DeepSeek-V4-Flash-Vision-Exp-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8.gguf