Add Muse Glimmer 30B text decoder support

[stikves]

Summary

Add Meta's Muse Glimmer 30B — on-device agentic model (Apache 2.0).

First model in the codebase with gated attention, CenteredRMSNorm, QK norm,
and local/global attention pattern with per-layer RoPE control.

Architecture

• 52 layers, hidden_size=6656, GQA 32Q/2KV (16:1)
• Local/Global attention: [S,S,S,G] repeating. Sliding window = 2048.
• CenteredRMSNorm (1 + weight) for layer norms; plain RMSNorm for final norm
• Weight-less RMSNorm on embeddings (no learned scale)
• QK norm: shared RMSNorm on Q and K per-head
• qk_scale_factor=3.87 multiplies Q directly after QK norm
• Gated attention: sigmoid(gate_proj(x)) * attn_output
• output_multiplier=0.196, logit_softcapping=20.0
• SwiGLU, no bias, tie_word_embeddings=False

Evaluation

Model	Compression	Precision	word_ppl
30B	none	FP16	7.71
30B	INT4	FP16	~8.4

Long-Context Validation (128K)

Needle-in-a-haystack retrieval passes at all context sizes:

Context	Prefill tok/s	Decode tok/s	Needle found
4K	134-140	12-13	✓
16K	113-130	11-13	✓
32K	124-127	11-12	✓
64K	116	9	✓
128K	~105	~6	✓

Changes

• models/macos/muse_glimmer.py — model implementation (~350 LOC)
• models/registry.py — registry entry + AutoConfig registration
• model_registry.py — export preset
• tests/.../test_muse_glimmer.py — 11 unit tests
• models/muse_glimmer/README.md — documentation

Test plan

• 11 unit tests (structural correctness, weight loading, numerics)
• Per-layer parity against HF reference (transformers 5.15)
• WikiText-2 perplexity: 7.71 (FP16), ~8.4 (INT4)
• Export pipeline (4-bit INT4)
• Text generation via llm-runner (coherent output)
• Benchmark via llm-benchmark (roofline: 46-55% efficiency)
• Needle-in-a-haystack: passes at 4K through 128K context

[srjoglekar246]

I guess this design doesn't provide the memory savings of Sliding Window Attention right? Maybe we can do what @Lewis300 & I have been doing for a different model:

1. Use a different limited-length cache for SWA layers
2. Update using modulo within the model
3. Pass in a causal mask that ensures the model attends as needed based on position IDs

Also, did we test this for quality at the largest supported context? (We have seen issues on some models at very large contexts, just wanna ensure it doesn't happen on this one)

[stikves]

(An updated chain-of-thought parser will follow up)

The model produces agentic output, for example:

%   ./llm-runner --model exports/muse_glimmer_30b_4bit_dynamic --prompt "Hello" --max-tokens 500

⏳ Preparing AI asset from source... done in 7.320s (cache hit)

Generating...
 to=self<|message|>Hello

We need to respond. Simple greeting. Probably respond friendly. No constraints. Could ask how can help. Probably just say hello back.

User said "Hello". Respond with greeting.

Probably keep simple.

No need for excessive formatting.

We can respond: Hello! How can I help you today?

That's fine.

Follow guidelines.

Probably no need to mention knowledge cutoff.

Just greet.

Okay.<|eom|><|start|>assistant to=user<|message|>Hello! How can I help you today?<|eot|><|start|>assistant to=self<|message|>We responded. Good.

Possibly keep open.

That's fine.<|eom|><|start|>assistant to=user<|message|>Hello! How can I help you today?<|eot|><|start|>assistant to=self<|message|>We already responded. The user said Hello. We responded. Possibly done.

We should not repeat? Might be fine.

We can just end.

Final answer is greeting.<|eom|><|start|>assistant to=user<|message|>Hello! How can I help you today?<|eot|><|start|>assistant to=self<|message|>Okay.<|eom|><|start|>assistant to=user<|message|>Hello! How can I help you today?<|eot|><|start|>assistant to=self<|message|>The user just said Hello. We replied. Probably done.

We should not add extra.<|eom|><|start|>assistant to=user<|message|>Hello! How can I help you today?<|eot|><|start|>assistant to=self<|message|>We are done.<|eom|><|start|>assistant to=user<|message|>Hello! How can I help you today?<|eot|><|start|>assistant to=self<|message|>Ok.<|eom|><|start|>assistant to=user<|message|>Hello! How can I help you today?<|eot|><|start|>assistant to=self<|message|>We have repeated. Possibly the system expects one response. It's fine.

Now produce final.<|eom|><|start|>assistant to=user<|message|>Hello! How can I help you today?<|eot|><|start|>assistant to=self<|message|>Ok.<|eom|><|start|>assistant to=user<|message|>Hello! How can I help you today?<|eot|><|start|>assistant to=self<|message|>We need to stop repeating. The final answer should be one greeting.

Probably final output: Hello! Nice to meet you. How can I assist?

Let's do that.<|eom|><|start|>assistant to=user<|message|>Hello! Nice to meet you. How can I help you today?<|eot|><|start|>assistant to=self<|message|>Ok.<|eom|><|start|>assistant to=user<|message|>Hello! Nice to meet you. How can I help you today?<|eot|><|start|>assistant to=self<|message|>We are looping. Possibly the system is stuck. Need final output.

I'll output once.<|eom|><|start|>assistant to=user<|message|>Hello! Nice to meet you. How can I help you today?<|eot|><|start|>assistant to=self<|message|>

Where

Hello! Nice to meet you. How can I help you today?

Is the actual message to the user

[kevchengcodes]

why have both of these?

[stikves]

The VLM support will come separately. We are currently only extracting the text portion.

[stikves]

@srjoglekar246

SWA memory savings: Will be a follow up. We currently store all 52 layers share a single growing KV cache. The SWA dual-cache pattern (bounded sliding cache for local layers + growing cache for global layers) is a planned follow-up.

The infrastructure already exists on main (StateKind enum, 2-4 state support in both engines) but not tested on this model yet.

Memory impact at 128K: currently ~8GB (full cache for all layers). With SWA dual-cache: ~2-3GB (75% savings since 39/52 layers only need 2048 slots).

Quality at max context:
Yes, validated. 10-question comprehension quiz (5 content + 4 needle-in-a-haystack at different depths + 1 reasoning) passes 10/10 at all context sizes including 128K (126,514 tokens):

Context	Score	Prefill tok/s	Decode tok/s	Memory
4K	10/10	133	12.8	1.1 GB
16K	10/10	132	13.0	2.6 GB
64K	10/10	117	9.7	4.5 GB
128K	10/10	104	6.2	8.0 GB

This required a RoPE precision fix (pre-computing frequencies in float32 to avoid fp16 angle overflow at positions >32K) which is included in the last commits.

[pkmandke]

Any reason this is approximate? We can perhaps add the actual PPL?

[pkmandke]

nit: we can move this section towards the end of this page and re-format to be consistent with other model cards..