TurboQuant

KV cache compression for local inference on M4 Max MacBook Pro.

What

TurboQuant (Google, ICLR 2026) is a three-stage KV cache compression method:

1. PolarQuant — WHT rotation + polar coordinates + Lloyd-Max codebook (~4.2x compression)
2. QJL — 1-bit quantized Johnson-Lindenstrauss residual correction
3. TurboQuant — PolarQuant + QJL = ~3.5 bits/channel, zero accuracy loss

Why

Unlock 64K-128K context on qwen3.5:27b within 32GB unified memory. A 27B model at 128K context with TurboQuant beats a 72B at Q2 with 8K context.

Status

See issues for current progress.

Roles

• Build Spec: Architecture and specification
• Implementation: Code, benchmarks, deployment
• Research: Upstream tracking, literature review
• Quality: Testing and review
• Coordination: Project management

Source Repos

• TheTom/llama-cpp-turboquant — llama.cpp fork with Metal
• TheTom/turboquant_plus — Reference impl, 511+ tests
• amirzandieh/QJL — Author QJL code (CUDA)
• rachittshah/mlx-turboquant — MLX fallback

Docs

• BUILD-SPEC.md — Full build specification (Strago, v2.2)