---
name: llama-server-local-agent-tuning
description: Optimal llama-server configuration for local LLM agent tool-calling. Hard-won settings from trial and error getting Hermes models to execute tool calls reliably.
tags: [llama.cpp, local-llm, tool-calling, inference, optimization]
---

# llama-server Local Agent Tuning

## When to Use
When setting up llama-server to power a local AI agent that needs reliable tool calling (Hermes models, function-calling models).

## Critical Flags

```bash
llama-server \
  -m /path/to/model.gguf \
  --port 8081 \
  --jinja \        # CRITICAL: converts Hermes <tool_call> XML to OpenAI structured format
  -ngl 99 \        # GPU layers (use 0 for CPU-only)
  -c 8192 \        # Context window — 8192 is the sweet spot for agent work
  -np 1            # CRITICAL: single slot gets full context
```

## Key Findings (Trial and Error)

### --jinja flag is mandatory
Without it, Hermes models output tool calls as `<tool_call>` XML tags in raw text. The OpenAI-compatible API endpoint returns these as plain text content, not structured `tool_calls`. The `--jinja` flag makes llama-server process the chat template properly so tool calls appear as structured API responses.

### -np 1 (single parallel slot) is critical
Default is `-np 4`, which splits context evenly: 8192/4 = 2048 tokens per slot. That's too small for system prompt + tool definitions + conversation. Symptoms: `n_decoded: 0` — model never starts generating because the prompt doesn't fit in a slot. Fix: `-np 1` gives full context to one request at a time.

### Context window sizing
- 65536 (model default): too slow, huge KV cache allocation, unnecessary for agent tasks
- 4096: too small for system prompt + tools + multi-turn
- 8192: sweet spot — fits full tool definitions + 3-5 turn conversation
- If using multiple parallel slots, multiply: `-np 2 -c 16384` gives 8192 per slot

### Response time expectations (M3 Max, 14B Q4_K_M)
- Simple tool call (read file): 7-20 seconds
- Complex reasoning: 20-40 seconds
- First request after startup: 30-60 seconds (cold KV cache)
- Subsequent requests: 2-3x faster (warm cache)

## Pitfalls

1. **Multiple slots silently fail**: No error message. Model just never responds. Check `/slots` endpoint to see `n_decoded: 0`.
2. **Hermes-3 8B at ~/.models/hermes3-8b.gguf**: faster, weaker. Good for overnight loops and simple tasks.
3. **Hermes-4 14B**: better quality but 2x slower. Worth it for complex reasoning.
4. **Health check**: `curl localhost:8081/health` — also check `/slots` for actual slot utilization.
5. **Draft model for speculative decoding**: `-md /path/to/small.gguf --draft-max 8` — can give 2-3x speedup.

## Verification

```bash
# Check server is up
curl -s localhost:8081/health | python3 -m json.tool

# Check slot status (is it actually processing?)
curl -s localhost:8081/slots | python3 -m json.tool

# Quick tool-call test
curl -s localhost:8081/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"local","messages":[{"role":"user","content":"What is 2+2?"}]}'
```
