Skip to content

README.md

Latest commit

 

History

History
131 lines (103 loc) · 8.02 KB

README.md

File metadata and controls

131 lines (103 loc) · 8.02 KB
Raw

Cake Multimodal Benchmark Results

Date: 2026-03-18 System: RTX 3080 Laptop 16GB VRAM, i9-11900H, 32GB RAM, CUDA 13.1, Linux 6.19.6

Summary

Model Modality Reference cake Speedup VRAM (ref) VRAM (cake)
Qwen2.5-0.5B Q4_K_M Text (256 tok) ollama 421 tok/s 184 tok/s 0.44x ~600 MB 3,100 MB
Qwen2.5-0.5B F16 Text (256 tok) — 186 tok/s — — 1,700 MB
FLUX.1-dev FP8 Image (768x1024) n/a (gated) 3.5 s/step — — 13,317 MB
VibeVoice-1.5B Voice (14s audio) Python 27 ms/frame 20 ms/frame 1.35x 5,978 MB 6,565 MB

Qwen2.5-0.5B Text Generation

Prompt: "Explain the theory of general relativity in simple terms." Config: temperature=0, seed=42, max_tokens=256 | 3 runs averaged

Metric ollama (Q4_K_M) cake (Q4_K_M GGUF) cake (F16 safetensors)
Eval rate (avg) 421.2 tok/s 184.2 tok/s 185.7 tok/s
Run 1 413.4 tok/s 183.8 tok/s 183.6 tok/s
Run 2 425.1 tok/s 184.1 tok/s 187.1 tok/s
Run 3 425.1 tok/s 184.6 tok/s 186.3 tok/s
Peak VRAM ~600 MB 3,100 MB 1,700 MB
Quantization Q4_K_M (4-bit) Q4_K_M → F16 (dequant) F16 (native)

Notes:

  • Same model, same quantization: cake loads the exact same Q4_K_M GGUF file that ollama uses. Both dequantize to compute dtype at inference time.
  • ollama is 2.3x faster because it uses native quantized matmul (Q4_K×Q8_K vec_dot with SIMD/CUDA kernels that operate directly on 4-bit packed data). Cake dequantizes to F16 at load time and uses standard F16 matmul — the 4-bit compression benefit is lost for compute (only helps with disk/download size).
  • cake Q4_K_M and F16 produce identical throughput (~185 tok/s) because both run F16 matmul on GPU — the Q4_K_M weights are dequantized to F16 before any computation.
  • cake F16 uses less VRAM (1.7 GB vs 3.1 GB) because it loads F16 weights directly via mmap, while GGUF loading dequantizes all tensors to F32 on CPU first, then casts to F16 on GPU.
  • All three produce coherent, accurate explanations of general relativity.

Outputs: ollama_output.txt | cake_q4km_output.txt | cake_f16_output.txt

FLUX.1-dev FP8 Image Generation

Prompt: "A cyberpunk samurai standing on a neon-lit rooftop at night, overlooking a sprawling futuristic Tokyo cityscape with flying cars and holographic billboards, rain falling, dramatic lighting" Config: 768x1024, 20 steps, guidance=3.5

Metric cake (FP8)
Total time ~84s
T5-XXL encoding 28.0s
Transformer loading 9.0s
Denoising (20 steps) 70.0s
Per-step time 3.5s
VAE decode 4.0s
Peak VRAM 13,317 MB

Per-step breakdown (GPU-synced timing)

Component Blocks Notes
Double stream blocks 19 MMDiT: joint img+txt attention, separate MLPs
Single stream blocks 38 Combined QKV + MLP in one linear
F8→F16 on-the-fly dequant 228/step ~36GB bandwidth per step
BF16↔F16 dtype conversions 456/step BF16 outer, F16 inner (fastest on Ampere)
Compute per step 43 TFLOPs ~20% GPU utilization (batch=1)

Optimization investigation

Approach Result Finding
F16 outer compute (eliminate BF16↔F16 conversions) Black image F16 overflow in attention scores
Direct F8→BF16 dequant (skip F16 intermediate) 4.0s/step (+14%) BF16 matmul slower than F16 on Ampere
broadcast_matmul instead of 2D reshape 4.1s/step (+17%) cuBLAS batched GEMM less efficient
Pre-computed timestep/dt tensors ~0 change GPU-bound, not launch-bound
BF16 outer + F16 inner (original) 3.5s/step Already optimal for this hardware

Notes:

  • Reference (diffusers) benchmark skipped — black-forest-labs/FLUX.1-dev is a gated repo requiring HuggingFace authentication. Cake uses the publicly available evilsocket/flux1-dev FP8 checkpoint.
  • FLUX.1-dev is compute-bound (unlike VibeVoice which was kernel-launch-bound). The bottleneck is 43 TFLOPs/step across 57 transformer blocks with on-the-fly F8→F16 dequantization.
  • FP8 weights kept on GPU (~8.6GB), dequantized to F16 on-the-fly per layer. Caching as F16 would need ~17GB — too much for 16GB GPU.
  • BF16 outer compute with F16 inner matmul is the optimal configuration: BF16 prevents overflow, F16 Tensor Cores are fastest on Ampere.
  • Further speedup requires fusing F8 dequant directly into matmul (custom CUTLASS kernel) or INT8 quantization with native cuBLAS support.
  • Fits on a 16GB GPU with room for activations — ComfyUI-comparable memory footprint.

Output: cake_output.png

VibeVoice-1.5B Voice Synthesis

Text: "Generating long-form, multi-speaker conversational audio like podcasts poses significant challenges for traditional text-to-speech systems, particularly in scalability, speaker consistency, and natural turn-taking." Voice: en-Carter_man.wav | Config: 10 diffusion steps, CFG=1.3

Metric Python (BF16, SDPA) cake (BF16, flash attn)
Total time 8.9s 9.0s
Audio duration 15.6s 13.7s
Speech frames 325 103
Per-frame time 27 ms 20 ms
Peak VRAM 5,978 MB 6,565 MB
Real-time factor 0.57x (1.75x RT) 0.66x (1.52x RT)

Per-frame breakdown (steady state)

Component Python (est.) cake Technique
Neg LM forward ~2ms 2.5ms Pre-computed embedding, skip for non-diffusion tokens
Diffusion (10 steps) ~10ms 4.0ms Pre-computed timestep embeddings, cached silu(cond), fused adaln_modulate kernel
VAE decode ~4ms 7.2ms Fused rms_norm_channel, depthwise_conv1d_bias_ctx, add_scaled kernels
Semantic encode ~4ms 6.3ms Same fused kernels + streaming cache
Connectors ~1ms 0.2ms —
Total overhead ~27ms 20ms 6 custom CUDA kernels

Optimization history

Version Per-frame Speedup vs Python
Initial (manual depthwise conv) 34ms 0.79x
+ streaming VAE caches, pre-alloc, skip neg LM 32ms 0.84x
+ fused depthwise_conv1d_bias kernel (14→1 launches) 28ms 0.96x
+ fused rms_norm_channel + add_scaled (eliminate transposes) 23ms 1.17x
+ fused depthwise_conv1d_bias_ctx (eliminate cat alloc) 21ms 1.29x
+ pre-computed t_emb/cond, cached silu, fused adaln_modulate 20ms 1.35x

Notes:

  • cake is now 35% faster than the Python reference per frame.
  • 6 custom CUDA fused kernels eliminate ~800 kernel launches per frame.
  • Streaming VAE caches provide correct inter-frame Conv1d context (matching Python's VibeVoiceTokenizerStreamingCache).
  • Python generates 3x more speech frames (325 vs 103) for similar audio duration — uses streaming VAE decode which generates shorter audio per frame.

Outputs: python_output.wav | cake_output.wav

Key Takeaways

  1. Text generation — cake supports GGUF Q4_K_M models (same files ollama uses). At 184 tok/s vs ollama's 421 tok/s, the gap is from dequantize-at-load vs native quantized matmul. To close the gap, cake would need native Q4_K vec_dot kernels (quantized matmul without dequantization).
  2. Image generation works on consumer hardware — FLUX.1-dev FP8 at 3.5s/step (768x1024) on a 16GB laptop GPU. Compute-bound at 43 TFLOPs/step — optimal BF16/F16 mixed-precision pipeline already saturates Tensor Cores.
  3. Voice synthesis is 35% faster than Python — 20ms/frame vs Python's 27ms, achieved through 6 custom CUDA fused kernels that eliminate ~800 kernel launches per frame. Generates 13.7s of audio in 9.0s (~1.5x real-time).