Skip to content

Support the Qwen3.5 architecture #143

Description

@PhillHenry

Is your feature request related to a problem? Please describe.
When I run the model Qwen3.5-4B-Q4_K_M.gguf, I hit a NullPointerException at this line:

        int modelContextLength = (int) metadata.get("qwen3.context_length");

in Qwen3Configuration.createConfiguration.

Describe the solution you'd like
Apparently, the structure of the Qwen3.5 is different to Qwen3. It would be nice if the code recognised this.

Describe alternatives you've considered
llama.cpp handles Qwen3.5-4B-Q4_K_M.gguf fine:

docker run --rm -it --gpus all -p 8080:8080 -v /home/henryp/Downloads/:/models ghcr.io/ggml-org/llama.cpp:server-cuda -m /models/Qwen3.5-4B-Q4_K_M.gguf --jinja --host 0.0.0.0 --port 8080

Additional context
I ran:

$ ./llama-tornado --gpu --verbose-init --ptx --model /home/henryp/Downloads/Qwen3.5-4B-Q4_K_M.gguf --heap-max 20g --prompt "Explain the benefits of GPU acceleration."
Detected TornadoVM backend: ptx (from /usr/local/bin/Java/Tornado/etc/tornado.backend)
WARNING: Using incubator modules: jdk.incubator.vector
Exception in thread "main" java.lang.NullPointerException: Cannot invoke "java.lang.Integer.intValue()" because the return value of "java.util.Map.get(Object)" is null
        at org.beehive.gpullama3.model.loader.Qwen3ModelLoader.createConfiguration(Qwen3ModelLoader.java:49)
        at org.beehive.gpullama3.model.loader.Qwen3ModelLoader.createConfiguration(Qwen3ModelLoader.java:29)
        at org.beehive.gpullama3.model.loader.AbstractModelLoader.loadModel(AbstractModelLoader.java:94)
        at org.beehive.gpullama3.model.ModelType$5.loadModel(ModelType.java:58)
        at org.beehive.gpullama3.model.loader.ModelLoader.loadModel(ModelLoader.java:99)
        at org.beehive.gpullama3.LlamaApp.main(LlamaApp.java:61)
Error: Command failed with return code 1
$ java --version
java 21.0.1 2023-10-17
Java(TM) SE Runtime Environment Oracle GraalVM 21.0.1+12.1 (build 21.0.1+12-jvmci-23.1-b19)
Java HotSpot(TM) 64-Bit Server VM Oracle GraalVM 21.0.1+12.1 (build 21.0.1+12-jvmci-23.1-b19, mixed mode, sharing)
14:45:31 henryp@thinkpad ~/Code/Java/GPU/GPULlama3.java
$ tornado --version
version=4.0.2-jdk21-dev
branch=master
commit=eab0b59

Backends installed: 
         - ptx

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions