Is your feature request related to a problem? Please describe.
When I run the model Qwen3.5-4B-Q4_K_M.gguf, I hit a NullPointerException at this line:
int modelContextLength = (int) metadata.get("qwen3.context_length");
in Qwen3Configuration.createConfiguration.
Describe the solution you'd like
Apparently, the structure of the Qwen3.5 is different to Qwen3. It would be nice if the code recognised this.
Describe alternatives you've considered
llama.cpp handles Qwen3.5-4B-Q4_K_M.gguf fine:
docker run --rm -it --gpus all -p 8080:8080 -v /home/henryp/Downloads/:/models ghcr.io/ggml-org/llama.cpp:server-cuda -m /models/Qwen3.5-4B-Q4_K_M.gguf --jinja --host 0.0.0.0 --port 8080
Additional context
I ran:
$ ./llama-tornado --gpu --verbose-init --ptx --model /home/henryp/Downloads/Qwen3.5-4B-Q4_K_M.gguf --heap-max 20g --prompt "Explain the benefits of GPU acceleration."
Detected TornadoVM backend: ptx (from /usr/local/bin/Java/Tornado/etc/tornado.backend)
WARNING: Using incubator modules: jdk.incubator.vector
Exception in thread "main" java.lang.NullPointerException: Cannot invoke "java.lang.Integer.intValue()" because the return value of "java.util.Map.get(Object)" is null
at org.beehive.gpullama3.model.loader.Qwen3ModelLoader.createConfiguration(Qwen3ModelLoader.java:49)
at org.beehive.gpullama3.model.loader.Qwen3ModelLoader.createConfiguration(Qwen3ModelLoader.java:29)
at org.beehive.gpullama3.model.loader.AbstractModelLoader.loadModel(AbstractModelLoader.java:94)
at org.beehive.gpullama3.model.ModelType$5.loadModel(ModelType.java:58)
at org.beehive.gpullama3.model.loader.ModelLoader.loadModel(ModelLoader.java:99)
at org.beehive.gpullama3.LlamaApp.main(LlamaApp.java:61)
Error: Command failed with return code 1
$ java --version
java 21.0.1 2023-10-17
Java(TM) SE Runtime Environment Oracle GraalVM 21.0.1+12.1 (build 21.0.1+12-jvmci-23.1-b19)
Java HotSpot(TM) 64-Bit Server VM Oracle GraalVM 21.0.1+12.1 (build 21.0.1+12-jvmci-23.1-b19, mixed mode, sharing)
14:45:31 henryp@thinkpad ~/Code/Java/GPU/GPULlama3.java
$ tornado --version
version=4.0.2-jdk21-dev
branch=master
commit=eab0b59
Backends installed:
- ptx
Is your feature request related to a problem? Please describe.
When I run the model
Qwen3.5-4B-Q4_K_M.gguf, I hit a NullPointerException at this line:in
Qwen3Configuration.createConfiguration.Describe the solution you'd like
Apparently, the structure of the Qwen3.5 is different to Qwen3. It would be nice if the code recognised this.
Describe alternatives you've considered
llama.cpphandlesQwen3.5-4B-Q4_K_M.gguffine:Additional context
I ran: