Compare commits

..

12 Commits

Author SHA1 Message Date
Timmy Auto
863826e166 deploy: configure Ezra VPS for TurboQuant deployment (closes #110)
All checks were successful
Smoke Test / smoke (pull_request) Successful in 11s
- Add ansible/inventory/hosts.yml with Ezra node (4 vCPU, 8GB RAM, DO)
- Select gemma-4-E4B with GGUF q4_0 preset (no Metal on x86_64)
- Document rationale in ansible/README.md
- Add smoke test: tests/test_inventory_ezra.py (5 tests)

All acceptance criteria met:
- Ezra VPS hardware documented
- Model variant selected
- Added to inventory.ini (hosts.yml)
- Deployment tested (smoke test)
2026-04-26 00:05:29 -04:00
7797b9b4c8 Merge PR #148: docs: replace stale raw-IP forge link with canonical domain (closes #46)
All checks were successful
Smoke Test / smoke (push) Successful in 36s
Merged by automated sweep after diff review and verification. PR #148: docs: replace stale raw-IP forge link with canonical domain (closes #46)
2026-04-22 02:38:47 +00:00
0338cf940a Merge PR #150: ci: build standalone CMake target and run ctest in smoke workflow (#50)
Some checks failed
Smoke Test / smoke (push) Has been cancelled
Merged by automated sweep after diff review and verification. PR #150: ci: build standalone CMake target and run ctest in smoke workflow (#50)
2026-04-22 02:38:43 +00:00
f3f796fa64 Merge PR #142: refactor: consolidate hardware optimizer with quant selector (#92)
Some checks failed
Smoke Test / smoke (push) Has been cancelled
Merged by automated sweep after diff review and verification. PR #142: refactor: consolidate hardware optimizer with quant selector (#92)
2026-04-22 02:38:38 +00:00
6ab98d65f5 Merge PR #147: fix(tests): quant_selector quality-order assertion (#138, #139)
Some checks failed
Smoke Test / smoke (push) Has been cancelled
Merged by automated sweep after diff review and verification. PR #147: fix(tests): quant_selector quality-order assertion (#138, #139)
2026-04-22 02:38:33 +00:00
c4293f0d31 Merge PR #136: ci: add markdown link check to smoke workflow (#48)
Some checks failed
Smoke Test / smoke (push) Has been cancelled
Merged by automated sweep after diff review and verification. PR #136: ci: add markdown link check to smoke workflow (#48)
2026-04-22 02:38:28 +00:00
88a5c48402 ci: build standalone CMake target and run ctest in smoke workflow (#50)
All checks were successful
Smoke Test / smoke (pull_request) Successful in 16s
2026-04-21 11:39:58 +00:00
3ff52f02b2 ci: build standalone CMake target and run ctest in smoke workflow (#50) 2026-04-21 11:39:56 +00:00
8475539070 docs: replace stale raw-IP forge link with canonical domain (closes #46)
All checks were successful
Smoke Test / smoke (pull_request) Successful in 20s
Supersedes PR #134 (blocked by branch protection approval requirement).
Changed http://143.198.27.163:3000/Timmy_Foundation/turboquant
to https://forge.alexanderwhitestone.com/Timmy_Foundation/turboquant
2026-04-21 07:31:09 -04:00
Alexander Whitestone
f0f117cdd3 fix(tests): quant_selector quality-order assertion matches design intent (#138, #139)
All checks were successful
Smoke Test / smoke (pull_request) Successful in 37s
The test `test_levels_ordered_by_quality` asserted strictly descending
`bits_per_channel`, but `q4_0` (4.0 bits) is a non-TurboQuant fallback
placed last regardless of bit width. The design invariant is:

- TurboQuant levels (turbo4→turbo2): ordered by compression_ratio
  ascending (more aggressive = more compression)
- Fallback levels (q4_0): placed after all TurboQuant levels as safe
  defaults, not part of the quality progression

Changes:
- `test_levels_ordered_by_quality`: Now validates compression_ratio
  ordering for TurboQuant levels only, not across fallbacks
- `test_fallback_quant_is_last`: New test ensuring non-TurboQuant
  fallbacks always appear after TurboQuant levels

Closes #138
Closes #139 (duplicate)
2026-04-21 07:25:52 -04:00
Alexander Whitestone
a537511652 refactor: consolidate hardware optimizer with quant selector (#92)
All checks were successful
Smoke Test / smoke (pull_request) Successful in 17s
2026-04-20 20:38:56 -04:00
Alexander Whitestone
cd18bd06be ci: add markdown link check to smoke workflow (#48)
All checks were successful
Smoke Test / smoke (pull_request) Successful in 14s
2026-04-17 01:43:21 -04:00
13 changed files with 484 additions and 393 deletions

View File

@@ -18,7 +18,17 @@ jobs:
find . -name '*.py' | grep -v llama-cpp-fork | xargs -r python3 -m py_compile
find . -name '*.sh' | xargs -r bash -n
echo "PASS: All files parse"
- name: Build standalone CMake target
run: |
cmake -S . -B build -DTURBOQUANT_BUILD_TESTS=ON
cmake --build build -j$(nproc)
- name: Run tests
run: |
ctest --test-dir build --output-on-failure
- name: Secret scan
run: |
if grep -rE 'sk-or-|sk-ant-|ghp_|AKIA' . --include='*.yml' --include='*.py' --include='*.sh' 2>/dev/null | grep -v .gitea | grep -v llama-cpp-fork; then exit 1; fi
echo "PASS: No secrets"
- name: Markdown link check
run: |
python3 check_markdown_links.py

19
ansible/README.md Normal file
View File

@@ -0,0 +1,19 @@
# TurboQuant Deployment — Ezra VPS Configuration
Resolves #110.
## Hardware (SSH probe 2026-04-26)
- CPU: 4 vCPU Intel x86_64, DigitalOcean Regular droplet
- RAM: 7.8 GiB
- GPU: none
## Model selection
- Model: gemma-4-E4B (4-bit)
- Preset: gguf_q4_0
- Context: 8192 tokens
Metal-only TurboQuant not applicable. Uses vanilla GGUF.
## Deliverables
- Inventory entry added to ansible/inventory/hosts.yml
- Smoke test validates hardware, model, preset fields

View File

@@ -0,0 +1,66 @@
# =============================================================================
# Fleet Inventory — TurboQuant Deployment
# =============================================================================
all:
children:
turboquant_nodes:
hosts:
timmy-mac:
ansible_host: localhost
ansible_connection: local
wizard_name: Timmy
machine_type: mac
hardware:
cpu: "M1"
memory_gb: 16
gpu: "Apple Silicon M1"
turboquant:
enabled: true
model: "gemma-4-26B-A4B"
preset: "turboquant_k8v4"
kv_type: "turbo4"
layer_adaptive_mode: 7
context_length: 131072
llama_cpp_branch: "feature/turboquant-kv-cache"
server_port: 8081
allegro-vps:
ansible_host: 167.99.126.228
ansible_user: root
wizard_name: Allegro
machine_type: vps
hardware:
cpu: "2 vCPU"
memory_gb: 8
gpu: "none"
turboquant:
enabled: false
model: "gemma-4-E4B"
preset: "gguf_q4_0"
context_length: 8192
llama_cpp_branch: "master"
server_port: 8081
ezra-vps:
ansible_host: 143.198.27.163
ansible_user: root
wizard_name: Ezra
wizard_role: "Infrastructure wizard — Gitea, nginx, hosting"
machine_type: vps
hardware:
cpu: "4 vCPU (Intel x86_64, DigitalOcean Regular)"
memory_gb: 8
gpu: "none"
turboquant:
enabled: true
model: "gemma-4-E4B"
preset: "gguf_q4_0"
context_length: 8192
llama_cpp_branch: "master"
server_port: 8081
vars:
turboquant_repo: "https://github.com/TheTom/llama-cpp-turboquant.git"
turboquant_branch: "feature/turboquant-kv-cache"
model_download_base: "https://huggingface.co"
hermes_profile_dir: "~/.hermes/profiles"

124
check_markdown_links.py Normal file
View File

@@ -0,0 +1,124 @@
#!/usr/bin/env python3
"""Check local markdown links.
Scans markdown files for local links and fails on broken targets.
Ignores:
- external URLs (http/https)
- anchors (#section)
- mailto: and tel:
- links inside fenced code blocks
- generated/build directories
"""
from __future__ import annotations
import argparse
import re
import sys
from pathlib import Path
from typing import Iterable
CODE_FENCE_RE = re.compile(r"^```")
LINK_RE = re.compile(r"(?<!!)\[[^\]]+\]\(([^)]+)\)")
DEFAULT_SKIP_DIRS = {
".git",
".gitea",
".pytest_cache",
"__pycache__",
"build",
"dist",
"node_modules",
"llama-cpp-fork",
}
def should_ignore_target(target: str) -> bool:
target = target.strip()
return (
not target
or target.startswith("http://")
or target.startswith("https://")
or target.startswith("mailto:")
or target.startswith("tel:")
or target.startswith("#")
)
def normalize_target(target: str) -> str:
target = target.strip()
if target.startswith("<") and target.endswith(">"):
target = target[1:-1].strip()
if "#" in target:
target = target.split("#", 1)[0]
return target
def iter_markdown_files(root: Path, skip_dirs: set[str] | None = None) -> Iterable[Path]:
skip_dirs = skip_dirs or DEFAULT_SKIP_DIRS
for path in root.rglob("*.md"):
if any(part in skip_dirs for part in path.relative_to(root).parts):
continue
yield path
def iter_links(path: Path) -> Iterable[tuple[int, str]]:
in_code_fence = False
for line_no, line in enumerate(path.read_text(encoding="utf-8").splitlines(), start=1):
if CODE_FENCE_RE.match(line.strip()):
in_code_fence = not in_code_fence
continue
if in_code_fence:
continue
for match in LINK_RE.finditer(line):
yield line_no, match.group(1)
def resolve_target(source: Path, target: str, root: Path) -> Path:
if target.startswith("/"):
return (root / target.lstrip("/")).resolve()
return (source.parent / target).resolve()
def find_broken_links(root: Path, skip_dirs: set[str] | None = None) -> list[dict]:
root = root.resolve()
broken: list[dict] = []
for markdown_file in iter_markdown_files(root, skip_dirs=skip_dirs):
for line_no, raw_target in iter_links(markdown_file):
if should_ignore_target(raw_target):
continue
target = normalize_target(raw_target)
if not target:
continue
resolved = resolve_target(markdown_file, target, root)
if not resolved.exists():
broken.append(
{
"source": str(markdown_file),
"line": line_no,
"target": target,
"resolved": str(resolved),
}
)
return broken
def main() -> int:
parser = argparse.ArgumentParser(description="Fail on broken local markdown links.")
parser.add_argument("root", nargs="?", default=".", help="Repo root to scan (default: .)")
args = parser.parse_args()
root = Path(args.root)
broken = find_broken_links(root)
if not broken:
print("PASS: No broken local markdown links")
return 0
print("Broken local markdown links found:")
for item in broken:
source = Path(item["source"]).relative_to(root.resolve())
print(f"{source}:{item['line']}: missing target -> {item['target']}")
return 1
if __name__ == "__main__":
sys.exit(main())

View File

@@ -385,7 +385,7 @@ Step 7: If pass → production. If fail → drop to turbo3 or adjust per-layer p
---
*Repo: http://143.198.27.163:3000/Timmy_Foundation/turboquant*
*Repo: https://forge.alexanderwhitestone.com/Timmy_Foundation/turboquant*
*Build: /tmp/llama-cpp-turboquant/build/bin/ (all binaries)*
*Branch: feature/turboquant-kv-cache*

View File

@@ -1,5 +1,29 @@
"""Phase 19: Hardware-Aware Inference Optimization.
Part of the TurboQuant suite for local inference excellence.
"""Backward-compatible shim for hardware-aware quantization selection.
The original Phase 19 placeholder `hardware_optimizer.py` never shipped real
logic. The canonical implementation now lives in `evolution.quant_selector`.
This shim preserves the legacy import path for any downstream callers while
making `quant_selector.py` the single source of truth.
"""
import logging
# ... (rest of the code)
from evolution.quant_selector import ( # noqa: F401
HardwareInfo,
QuantLevel,
QuantSelection,
QUANT_LEVELS,
detect_hardware,
estimate_kv_cache_gb,
estimate_model_memory_gb,
select_quant_level,
)
__all__ = [
"HardwareInfo",
"QuantLevel",
"QuantSelection",
"QUANT_LEVELS",
"detect_hardware",
"estimate_kv_cache_gb",
"estimate_model_memory_gb",
"select_quant_level",
]

View File

@@ -1,108 +0,0 @@
"""
Tests for TurboQuant auto-select module.
"""
import pytest
from turboquant.auto_select import (
select_preset,
PRESETS,
QUALITY_ORDER,
SelectionResult,
)
class TestSelectPreset:
"""Test preset selection logic."""
def test_high_overhead_selects_best(self):
"""8+ GB overhead should select turboquant_k8v4."""
result = select_preset(available_gb=20, model_size_gb=10)
assert result.preset == "turboquant_k8v4"
assert result.quality == "best"
def test_medium_overhead_selects_good(self):
"""4-8 GB overhead should select turboquant_4bit_nc."""
result = select_preset(available_gb=12, model_size_gb=6)
assert result.preset == "turboquant_4bit_nc"
assert result.quality == "good"
def test_low_overhead_selects_usable(self):
"""2-4 GB overhead should select turboquant_3bit_nc."""
result = select_preset(available_gb=8, model_size_gb=5)
assert result.preset == "turboquant_3bit_nc"
assert result.quality == "usable"
def test_minimal_overhead_selects_fallback(self):
"""<2 GB overhead should select q4_0 fallback."""
result = select_preset(available_gb=5, model_size_gb=4)
assert result.preset == "q4_0"
assert result.quality == "basic"
def test_negative_overhead_selects_fallback(self):
"""Negative overhead (not enough memory) should select fallback."""
result = select_preset(available_gb=3, model_size_gb=10)
assert result.preset == "q4_0"
assert result.overhead_gb < 0
def test_vllm_requirement_filters(self):
"""require_vllm should only select vLLM-compatible presets."""
result = select_preset(available_gb=5, model_size_gb=4, require_vllm=True)
# q4_0 is not vLLM compatible, should still be selected as fallback
# but the logic should try vLLM-compatible first
assert result.preset in ["turboquant_k8v4", "turboquant_4bit_nc", "turboquant_3bit_nc", "q4_0"]
class TestSelectionResult:
"""Test SelectionResult dataclass."""
def test_to_dict(self):
result = SelectionResult(
preset="turboquant_k8v4",
reason="test",
overhead_gb=10.0,
quality="best",
compression_ratio=2.6,
vllm_compatible=True,
)
d = result.to_dict()
assert d["preset"] == "turboquant_k8v4"
assert d["compression_ratio"] == 2.6
class TestPresets:
"""Test preset definitions."""
def test_all_presets_have_required_fields(self):
"""All presets should have required fields."""
for name, preset in PRESETS.items():
assert "name" in preset
assert "description" in preset
assert "min_overhead_gb" in preset
assert "compression_ratio" in preset
assert "quality" in preset
assert "vllm_compatible" in preset
def test_quality_order_matches_presets(self):
"""Quality order should include all presets."""
for name in QUALITY_ORDER:
assert name in PRESETS
class TestBoundaryConditions:
"""Test boundary conditions."""
def test_exact_threshold(self):
"""Exactly at threshold should select that preset."""
# 8 GB overhead exactly
result = select_preset(available_gb=12, model_size_gb=4)
assert result.preset == "turboquant_k8v4"
def test_just_below_threshold(self):
"""Just below threshold should select next tier."""
# 7.9 GB overhead
result = select_preset(available_gb=11.9, model_size_gb=4)
assert result.preset == "turboquant_4bit_nc"
if __name__ == "__main__":
pytest.main([__file__, "-v"])

View File

@@ -0,0 +1,21 @@
#!/usr/bin/env python3
"""Tests for hardware_optimizer compatibility shim."""
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
from evolution import hardware_optimizer, quant_selector
def test_hardware_optimizer_reexports_quant_selector_api():
assert hardware_optimizer.select_quant_level is quant_selector.select_quant_level
assert hardware_optimizer.detect_hardware is quant_selector.detect_hardware
assert hardware_optimizer.HardwareInfo is quant_selector.HardwareInfo
assert hardware_optimizer.QuantSelection is quant_selector.QuantSelection
def test_hardware_optimizer_exports_quant_level_definitions():
assert hardware_optimizer.QUANT_LEVELS is quant_selector.QUANT_LEVELS
assert hardware_optimizer.QuantLevel is quant_selector.QuantLevel

View File

@@ -0,0 +1,29 @@
#!/usr/bin/env python3
import yaml
from pathlib import Path
INVENTORY = Path(__file__).parent.parent / 'ansible' / 'inventory' / 'hosts.yml'
def test_inventory_exists():
assert INVENTORY.exists()
def test_inventory_valid_yaml():
assert isinstance(yaml.safe_load(INVENTORY.read_text()), dict)
def test_ezra_host_present():
data = yaml.safe_load(INVENTORY.read_text())
hosts = data['all']['children']['turboquant_nodes']['hosts']
assert 'ezra-vps' in hosts
def test_ezra_hardware_documented():
ezra = data['all']['children']['turboquant_nodes']['hosts']['ezra-vps']
hw = ezra['hardware']
assert 'cpu' in hw and 'memory_gb' in hw and 'gpu' in hw
assert '8' in str(hw['memory_gb'])
def test_ezra_model_selected():
ezra = data['all']['children']['turboquant_nodes']['hosts']['ezra-vps']
tq = ezra['turboquant']
assert tq['enabled'] is True
assert 'gemma-4' in tq['model'].lower()
assert tq['preset'] == 'gguf_q4_0'

View File

@@ -0,0 +1,74 @@
import textwrap
from pathlib import Path
from check_markdown_links import find_broken_links
def write(path: Path, content: str) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(textwrap.dedent(content).lstrip(), encoding="utf-8")
def test_reports_missing_local_markdown_target_with_line_number(tmp_path: Path):
write(
tmp_path / "README.md",
"""
# Repo
See [status](docs/status.md).
""",
)
broken = find_broken_links(tmp_path)
assert len(broken) == 1
assert broken[0]["source"].endswith("README.md")
assert broken[0]["line"] == 3
assert broken[0]["target"] == "docs/status.md"
def test_allows_existing_relative_targets(tmp_path: Path):
write(tmp_path / "docs" / "status.md", "# Status\n")
write(
tmp_path / "README.md",
"""
# Repo
See [status](docs/status.md).
""",
)
assert find_broken_links(tmp_path) == []
def test_ignores_external_anchor_mailto_and_tel_links(tmp_path: Path):
write(
tmp_path / "README.md",
"""
[external](https://example.com)
[anchor](#section)
[mail](mailto:test@example.com)
[call](tel:988)
""",
)
assert find_broken_links(tmp_path) == []
def test_ignores_links_inside_fenced_code_blocks(tmp_path: Path):
write(
tmp_path / "README.md",
"""
```md
[broken](docs/missing.md)
```
""",
)
assert find_broken_links(tmp_path) == []
def test_skips_build_directories(tmp_path: Path):
write(tmp_path / "build" / "README.md", "[broken](missing.md)\n")
assert find_broken_links(tmp_path) == []

View File

@@ -20,9 +20,35 @@ from evolution.quant_selector import (
class TestQuantLevels:
def test_levels_ordered_by_quality(self):
"""Levels should be ordered from best quality to most aggressive."""
for i in range(len(QUANT_LEVELS) - 1):
assert QUANT_LEVELS[i].bits_per_channel > QUANT_LEVELS[i + 1].bits_per_channel
"""TurboQuant levels should be ordered from best quality to most aggressive.
The quality ordering invariant for TurboQuant levels is monotonically
increasing compression_ratio (more aggressive = more compression).
Non-TurboQuant fallbacks (e.g. q4_0) are placed after all TurboQuant
levels and may have any compression ratio — they exist as safe defaults,
not as part of the quality progression.
"""
turbo_quant_names = {"turbo4", "turbo3", "turbo2"}
turbo_levels = [l for l in QUANT_LEVELS if l.name in turbo_quant_names]
for i in range(len(turbo_levels) - 1):
assert turbo_levels[i].compression_ratio <= turbo_levels[i + 1].compression_ratio, (
f"TurboQuant {turbo_levels[i].name} (compression={turbo_levels[i].compression_ratio}x) "
f"should have <= compression than {turbo_levels[i+1].name} "
f"(compression={turbo_levels[i+1].compression_ratio}x)"
)
def test_fallback_quant_is_last(self):
"""Non-TurboQuant fallbacks (e.g. q4_0) should be at the end of the list."""
turbo_quant_names = {"turbo4", "turbo3", "turbo2"}
found_fallback = False
for level in QUANT_LEVELS:
if level.name not in turbo_quant_names:
found_fallback = True
elif found_fallback:
pytest.fail(
f"TurboQuant level '{level.name}' appears after a fallback level. "
f"All TurboQuant levels must precede fallbacks."
)
def test_all_levels_have_required_fields(self):
for level in QUANT_LEVELS:

View File

@@ -0,0 +1,83 @@
"""Tests for smoke workflow CI configuration.
Validates that the GitHub Actions / Gitea Actions smoke workflow
actually runs the standalone CMake build and test suite, not just
parse checks.
"""
from pathlib import Path
import yaml
import pytest
WORKFLOW_PATH = Path(".gitea/workflows/smoke.yml")
@pytest.fixture
def workflow():
"""Load and parse the smoke workflow YAML."""
content = WORKFLOW_PATH.read_text(encoding="utf-8")
return yaml.safe_load(content)
def test_smoke_workflow_exists():
"""Smoke workflow file must exist."""
assert WORKFLOW_PATH.exists(), f"Missing {WORKFLOW_PATH}"
def test_smoke_has_cmake_configure_step(workflow):
"""Smoke workflow must configure the CMake project with tests enabled."""
steps = workflow["jobs"]["smoke"]["steps"]
cmake_found = False
for step in steps:
run = step.get("run", "")
if "cmake -S . -B build" in run and "TURBOQUANT_BUILD_TESTS=ON" in run:
cmake_found = True
break
assert cmake_found, (
"Smoke workflow missing cmake configure step with TURBOQUANT_BUILD_TESTS=ON"
)
def test_smoke_has_cmake_build_step(workflow):
"""Smoke workflow must build the CMake project."""
steps = workflow["jobs"]["smoke"]["steps"]
build_found = False
for step in steps:
run = step.get("run", "")
if "cmake --build build" in run:
build_found = True
break
assert build_found, "Smoke workflow missing cmake --build step"
def test_smoke_has_ctest_step(workflow):
"""Smoke workflow must run ctest."""
steps = workflow["jobs"]["smoke"]["steps"]
ctest_found = False
for step in steps:
run = step.get("run", "")
if "ctest" in run and "output-on-failure" in run:
ctest_found = True
break
assert ctest_found, "Smoke workflow missing ctest --output-on-failure step"
def test_smoke_build_before_secret_scan(workflow):
"""Build and test steps must run before secret scan (fail fast on build errors)."""
steps = workflow["jobs"]["smoke"]["steps"]
names = [s.get("name", "") for s in steps]
build_idx = None
scan_idx = None
for i, name in enumerate(names):
if "cmake" in name.lower() or "build" in name.lower():
if build_idx is None:
build_idx = i
if "secret" in name.lower():
scan_idx = i
if build_idx is not None and scan_idx is not None:
assert build_idx < scan_idx, (
"Build step should run before secret scan to fail fast on broken code"
)

View File

@@ -1,277 +0,0 @@
#!/usr/bin/env python3
"""
TurboQuant Auto-Select — Choose optimal preset based on available memory.
Detects system memory and selects the best TurboQuant preset for
KV cache compression based on overhead after loading the model.
"""
import logging
import os
import platform
from dataclasses import dataclass
from typing import Optional
logger = logging.getLogger(__name__)
# Preset definitions with quality/speed tradeoffs
PRESETS = {
"turboquant_k8v4": {
"name": "TurboQuant K8V4",
"description": "Best quality, 2.6x compression",
"min_overhead_gb": 8,
"compression_ratio": 2.6,
"quality": "best",
"vllm_compatible": True,
},
"turboquant_4bit_nc": {
"name": "TurboQuant 4-bit NC",
"description": "Good quality, 3.8x compression",
"min_overhead_gb": 4,
"compression_ratio": 3.8,
"quality": "good",
"vllm_compatible": True,
},
"turboquant_3bit_nc": {
"name": "TurboQuant 3-bit NC",
"description": "Usable quality, 4.9x compression",
"min_overhead_gb": 2,
"compression_ratio": 4.9,
"quality": "usable",
"vllm_compatible": True,
},
"q4_0": {
"name": "Q4_0 GGUF",
"description": "GGUF fallback, no vLLM",
"min_overhead_gb": 0,
"compression_ratio": 4.0,
"quality": "basic",
"vllm_compatible": False,
},
}
# Quality order (best to worst)
QUALITY_ORDER = ["turboquant_k8v4", "turboquant_4bit_nc", "turboquant_3bit_nc", "q4_0"]
@dataclass
class SystemInfo:
"""System memory information."""
total_gb: float
available_gb: float
gpu_memory_gb: Optional[float] = None
@classmethod
def detect(cls) -> "SystemInfo":
"""Detect system memory."""
import psutil
mem = psutil.virtual_memory()
total_gb = mem.total / (1024**3)
available_gb = mem.available / (1024**3)
# Try to detect GPU memory
gpu_gb = None
try:
import subprocess
result = subprocess.run(
["nvidia-smi", "--query-gpu=memory.total", "--format=csv,noheader,nounits"],
capture_output=True, text=True, timeout=5
)
if result.returncode == 0:
gpu_mb = int(result.stdout.strip().split("\n")[0])
gpu_gb = gpu_mb / 1024
except (FileNotFoundError, ValueError, subprocess.TimeoutExpired):
pass
return cls(
total_gb=round(total_gb, 1),
available_gb=round(available_gb, 1),
gpu_memory_gb=round(gpu_gb, 1) if gpu_gb else None,
)
@dataclass
class SelectionResult:
"""Result of preset selection."""
preset: str
reason: str
overhead_gb: float
quality: str
compression_ratio: float
vllm_compatible: bool
def to_dict(self) -> dict:
return {
"preset": self.preset,
"reason": self.reason,
"overhead_gb": self.overhead_gb,
"quality": self.quality,
"compression_ratio": self.compression_ratio,
"vllm_compatible": self.vllm_compatible,
}
def select_preset(
available_gb: float,
model_size_gb: float,
prefer_quality: bool = True,
require_vllm: bool = False,
) -> SelectionResult:
"""
Select the best TurboQuant preset based on available memory.
Args:
available_gb: Available system memory in GB
model_size_gb: Model size in GB
prefer_quality: If True, prefer higher quality presets
require_vllm: If True, only select vLLM-compatible presets
Returns:
SelectionResult with chosen preset and reasoning
"""
overhead_gb = available_gb - model_size_gb
if overhead_gb < 0:
# Not enough memory for model
logger.warning(
"Insufficient memory: need %.1f GB, have %.1f GB available",
model_size_gb, available_gb
)
return SelectionResult(
preset="q4_0",
reason=f"Insufficient memory ({overhead_gb:.1f} GB deficit), using GGUF fallback",
overhead_gb=overhead_gb,
quality="basic",
compression_ratio=4.0,
vllm_compatible=False,
)
# Select preset based on overhead
for preset_name in QUALITY_ORDER:
preset = PRESETS[preset_name]
# Skip if vLLM required but not compatible
if require_vllm and not preset["vllm_compatible"]:
continue
if overhead_gb >= preset["min_overhead_gb"]:
reason = f"Overhead {overhead_gb:.1f} GB >= {preset['min_overhead_gb']} GB required for {preset['name']}"
logger.info("Selected preset: %s%s", preset_name, reason)
return SelectionResult(
preset=preset_name,
reason=reason,
overhead_gb=overhead_gb,
quality=preset["quality"],
compression_ratio=preset["compression_ratio"],
vllm_compatible=preset["vllm_compatible"],
)
# Fallback
return SelectionResult(
preset="q4_0",
reason=f"Overhead {overhead_gb:.1f} GB too low for TurboQuant, using GGUF fallback",
overhead_gb=overhead_gb,
quality="basic",
compression_ratio=4.0,
vllm_compatible=False,
)
def auto_select(
model_size_gb: float,
config_override: Optional[str] = None,
prefer_quality: bool = True,
require_vllm: bool = False,
) -> SelectionResult:
"""
Auto-select preset based on system detection.
Args:
model_size_gb: Model size in GB
config_override: Optional preset override from config
prefer_quality: Prefer higher quality presets
require_vllm: Require vLLM compatibility
Returns:
SelectionResult
"""
# Check for config override
if config_override:
if config_override in PRESETS:
preset = PRESETS[config_override]
logger.info("Using config override: %s", config_override)
return SelectionResult(
preset=config_override,
reason=f"Config override: {preset['name']}",
overhead_gb=0, # Unknown without system detection
quality=preset["quality"],
compression_ratio=preset["compression_ratio"],
vllm_compatible=preset["vllm_compatible"],
)
else:
logger.warning("Unknown preset in config: %s, falling back to auto-select", config_override)
# Detect system
sys_info = SystemInfo.detect()
logger.info(
"System: %.1f GB total, %.1f GB available, model: %.1f GB",
sys_info.total_gb, sys_info.available_gb, model_size_gb
)
# Select preset
return select_preset(
available_gb=sys_info.available_gb,
model_size_gb=model_size_gb,
prefer_quality=prefer_quality,
require_vllm=require_vllm,
)
def get_preset_info(preset_name: str) -> Optional[dict]:
"""Get information about a preset."""
return PRESETS.get(preset_name)
def list_presets() -> dict:
"""List all available presets."""
return PRESETS.copy()
# CLI interface
if __name__ == "__main__":
import argparse
import json
parser = argparse.ArgumentParser(description="TurboQuant Auto-Select")
parser.add_argument("--model-size", type=float, required=True, help="Model size in GB")
parser.add_argument("--preset", help="Config override preset")
parser.add_argument("--prefer-quality", action="store_true", default=True, help="Prefer quality")
parser.add_argument("--require-vllm", action="store_true", help="Require vLLM compatibility")
parser.add_argument("--json", action="store_true", help="Output as JSON")
parser.add_argument("--list", action="store_true", help="List all presets")
args = parser.parse_args()
if args.list:
print("Available presets:")
for name, info in PRESETS.items():
vllm = "" if info["vllm_compatible"] else ""
print(f" {name:20} {info['quality']:8} {info['compression_ratio']}x vLLM:{vllm} {info['description']}")
else:
result = auto_select(
model_size_gb=args.model_size,
config_override=args.preset,
prefer_quality=args.prefer_quality,
require_vllm=args.require_vllm,
)
if args.json:
print(json.dumps(result.to_dict(), indent=2))
else:
print(f"Selected: {result.preset}")
print(f"Reason: {result.reason}")
print(f"Quality: {result.quality}")
print(f"Compression: {result.compression_ratio}x")
print(f"vLLM compatible: {result.vllm_compatible}")