Add stuck initiatives audit report
This commit is contained in:
Ezra
@@ -0,0 +1,160 @@
|
||||
---
|
||||
name: local-timmy-overnight-loop
|
||||
description: Deploy an unattended overnight loop that runs grounded tasks against local llama-server via Hermes, logging every result with timing. Produces rich capability data for morning review.
|
||||
version: 1.0.0
|
||||
author: Ezra
|
||||
license: MIT
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [local-model, llama.cpp, overnight, data-generation, sovereignty, timmy]
|
||||
related_skills: [local-llama-tool-calling-debug, wizard-house-remote-triage]
|
||||
---
|
||||
|
||||
# Local Timmy Overnight Loop
|
||||
|
||||
## When to Use
|
||||
|
||||
- Local Timmy needs to generate capability data overnight
|
||||
- You want to measure tool-call success rates, response times, and failure modes
|
||||
- The model is too slow for interactive use but can produce useful data unattended
|
||||
- Issue #93 (proof test) needs empirical evidence from many runs
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- llama-server running with `--jinja` flag (required for tool calls)
|
||||
- Hermes agent installed at `~/.hermes/hermes-agent/`
|
||||
- Timmy workspace at `~/.timmy/`
|
||||
- Model path known (e.g., `/Users/apayne/models/hermes4-14b/NousResearch_Hermes-4-14B-Q4_K_M.gguf`)
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
### Strip the system prompt
|
||||
The default Timmy system prompt is ~12K tokens (SOUL.md + skills list + memory). On a 14B Q4 model, this causes multi-minute prompt processing. The overnight loop uses a minimal prompt (~100 tokens):
|
||||
|
||||
```
|
||||
You are Timmy. You run locally on llama.cpp.
|
||||
You MUST use the tools provided. Do not narrate tool calls as text.
|
||||
When asked to read a file, call the read_file tool.
|
||||
When asked to write a file, call the write_file tool.
|
||||
When asked to search, call the search_files tool.
|
||||
Be brief. Do the task. Report what you found.
|
||||
```
|
||||
|
||||
### Skip context files and memory
|
||||
Pass `skip_context_files=True` and `skip_memory=True` to AIAgent to prevent injecting AGENTS.md, project context, skills, and memory into the prompt.
|
||||
|
||||
### Restrict toolsets
|
||||
Each task specifies only the toolsets it needs (usually just `file`). Fewer tool schemas = less context = faster processing.
|
||||
|
||||
### Reduce context length and use single slot
|
||||
Start llama-server with `-c 8192 -np 1` instead of `-c 65536`. The `-np 1` is critical — without it, llama-server defaults to 4 parallel slots, splitting 8192 into 2048 per slot. That's not enough for tool schemas + prompt, and the server silently hangs with `n_decoded: 0`. Single slot gives the full context to the loop's requests.
|
||||
|
||||
### Use the venv python
|
||||
macOS system python is 3.9 which lacks `X | None` syntax. Always use `~/.hermes/hermes-agent/venv/bin/python3`.
|
||||
|
||||
## Script Location
|
||||
|
||||
Deploy to: `~/.timmy/scripts/timmy_overnight_loop.py`
|
||||
Results in: `~/.timmy/overnight-loop/`
|
||||
|
||||
## Output Format
|
||||
|
||||
- `overnight_run_YYYYMMDD_HHMMSS.jsonl` — one JSON line per task with full result
|
||||
- `overnight_summary_YYYYMMDD_HHMMSS.md` — rolling human-readable summary
|
||||
|
||||
Each JSONL entry contains:
|
||||
```json
|
||||
{
|
||||
"task_id": "read-soul",
|
||||
"run": 1,
|
||||
"started_at": "...",
|
||||
"finished_at": "...",
|
||||
"elapsed_seconds": 45.2,
|
||||
"status": "pass|empty|error",
|
||||
"response": "...",
|
||||
"session_id": "...",
|
||||
"provider": "custom",
|
||||
"base_url": "http://localhost:8081/v1",
|
||||
"model": "hermes4:14b",
|
||||
"prompt": "...",
|
||||
"error": null
|
||||
}
|
||||
```
|
||||
|
||||
## Task Design
|
||||
|
||||
Good overnight tasks are:
|
||||
1. **Single tool call** — read one file, search one pattern
|
||||
2. **Verifiable** — expected output is known (file exists, content is deterministic)
|
||||
3. **Varied** — mix of read_file, write_file, search_files
|
||||
4. **Grounded** — require actual file operations, not knowledge recall
|
||||
5. **Short prompt** — under 100 words
|
||||
|
||||
Example tasks:
|
||||
- "Read ~/.timmy/SOUL.md. Quote the first sentence of the Prime Directive."
|
||||
- "Search ~/.hermes/bin/ for the string 'chatgpt.com'. Report which files."
|
||||
- "Write a file to ~/.timmy/overnight-loop/timmy_wrote_this.md with content: ..."
|
||||
- "Read ~/.hermes/config.yaml. What model is configured as default?"
|
||||
|
||||
## Starting the Loop
|
||||
|
||||
```bash
|
||||
cd ~/.hermes/hermes-agent
|
||||
nohup venv/bin/python3 ~/.timmy/scripts/timmy_overnight_loop.py \
|
||||
> ~/.timmy/overnight-loop/loop_stdout.log 2>&1 &
|
||||
echo "PID: $!"
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
```bash
|
||||
# Check if running
|
||||
pgrep -f timmy_overnight_loop
|
||||
|
||||
# Live progress
|
||||
tail -f ~/.timmy/overnight-loop/loop_stdout.log
|
||||
|
||||
# Latest summary
|
||||
cat ~/.timmy/overnight-loop/overnight_summary_*.md | tail -30
|
||||
|
||||
# Count completed tasks
|
||||
wc -l ~/.timmy/overnight-loop/overnight_run_*.jsonl
|
||||
```
|
||||
|
||||
## Stopping
|
||||
|
||||
```bash
|
||||
pkill -f timmy_overnight_loop
|
||||
```
|
||||
|
||||
## Morning Analysis
|
||||
|
||||
Key metrics to extract:
|
||||
1. **Tool call success rate** — did the model actually use tools?
|
||||
2. **Average response time** — baseline for performance tuning
|
||||
3. **Error patterns** — which tasks fail and why?
|
||||
4. **Pass/empty ratio** — empty responses mean the model responded but didn't use tools
|
||||
5. **Time-series trend** — does performance degrade over cycles?
|
||||
|
||||
```bash
|
||||
# Quick stats
|
||||
python3 -c "
|
||||
import json
|
||||
results = [json.loads(l) for l in open('overnight_run_*.jsonl')]
|
||||
passes = sum(1 for r in results if r['status'] == 'pass')
|
||||
total = len(results)
|
||||
avg = sum(r.get('elapsed_seconds',0) for r in results) / max(total,1)
|
||||
print(f'Pass: {passes}/{total} ({100*passes//max(total,1)}%)')
|
||||
print(f'Avg time: {avg:.1f}s')
|
||||
print(f'Errors: {sum(1 for r in results if r[\"status\"]==\"error\")}')
|
||||
"
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
1. **Kill stale hermes processes first.** Old stuck sessions compete for llama-server slots. Run `pkill -f "hermes chat"` before starting the loop. Also kill legacy loops: `pkill -f gemini-loop; pkill -f ops-dashboard; pkill -f timmy-status`.
|
||||
2. **Also kill legacy loops.** gemini-loop.sh, ops-dashboard.sh, timmy-status.sh may be running. They waste resources.
|
||||
3. **Check llama-server health before starting.** `curl -s http://localhost:8081/health` — if it's processing a stale request, restart it.
|
||||
4. **The loop sleeps 30s between cycles.** This prevents hammering the model. Adjust if needed.
|
||||
5. **Gemini fallback may silently activate.** If `fallback_model` in config.yaml points to Gemini, slow/failed local requests may route to cloud. Check config before running.
|
||||
6. **Security guards block remote process kills.** If running remotely via SSH, `pkill` commands on the Mac may need user approval. Have Alexander run kill commands directly.
|
||||
Reference in New Issue
Block a user