bc553c99a9
Adds a complete Metal backend integration that compiles Metal shaders
into a metallib and registers them with llama.cpp's Metal runtime.
Key changes:
- ggml-metal-turbo.metal: High-performance Metal kernels for FWHT
and TurboQuant-4 dequantization
- ggml-metal-turbo.{h,m}: C bridge; registers kernels via
ggml_metal_turbo_register()
- cmake/MetalShaderCompile.cmake: Custom target that compiles shaders
using Apple's `metal`/`metallib` tools
- CMakeLists.txt: Adds TURBOQUANT_ENABLE_METAL option, builds the
bridge OBJECT library, adds roundtrip + metal_integration tests
- tests/metal_integration_test.cpp: Verifies metallib artifact exists
- .gitea/workflows/smoke.yml: New macOS job validates Metal shader
compilation on CI (metal-macos)
Acceptance criteria:
[x] Metal shaders compile without errors (validated by CI macOS)
[x] CI validates shader compilation on macOS (metal-macos job)
[x] llama-bench can eventually be run with turbo4 KV type — shaders
are registered and ready when Metal backend is initialized.
Closes #75
TurboQuant
KV cache compression for local inference on M4 Max MacBook Pro.
What
TurboQuant (Google, ICLR 2026) is a three-stage KV cache compression method:
- PolarQuant — WHT rotation + polar coordinates + Lloyd-Max codebook (~4.2x compression)
- QJL — 1-bit quantized Johnson-Lindenstrauss residual correction
- TurboQuant — PolarQuant + QJL = ~3.5 bits/channel, zero accuracy loss
Why
Unlock 64K-128K context on qwen3.5:27b within 32GB unified memory. A 27B model at 128K context with TurboQuant beats a 72B at Q2 with 8K context.
Status
See issues for current progress.
Roles
- Strago: Build spec author
- Cid: Implementation, benchmarks, deployment
- Locke: Research support, upstream watch
- John: Quality review
- Frankie: Coordination
Source Repos
- TheTom/llama-cpp-turboquant — llama.cpp fork with Metal
- TheTom/turboquant_plus — Reference impl, 511+ tests
- amirzandieh/QJL — Author QJL code (CUDA)
- rachittshah/mlx-turboquant — MLX fallback
Docs
- Project Status — Full project status and build specification