forked from Rockachopa/Timmy-time-dashboard
Compare commits
62 Commits
test/chat-
...
claude/iss
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
48cf182c6e | ||
| 8304cf50da | |||
| 16c4cc0f9f | |||
| a48f30fee4 | |||
| e44db42c1a | |||
| de7744916c | |||
| bde7232ece | |||
| fc4426954e | |||
| 5be4ecb9ef | |||
| 4f80cfcd58 | |||
| a7ccfbddc9 | |||
| f1f67e62a7 | |||
| 00ef4fbd22 | |||
| fc0a94202f | |||
| bd3e207c0d | |||
| cc8ed5b57d | |||
| 823216db60 | |||
| 75ecfaba64 | |||
| 55beaf241f | |||
| 69498c9add | |||
| 6c76bf2f66 | |||
| 0436dfd4c4 | |||
| 9eeb49a6f1 | |||
| 2d6bfe6ba1 | |||
| ebb2cad552 | |||
| 003e3883fb | |||
| 7dfbf05867 | |||
| 1cce28d1bb | |||
| 4c6b69885d | |||
| 6b2e6d9e8c | |||
| 2b238d1d23 | |||
| b7ad5bf1d9 | |||
| 2240ddb632 | |||
| 35d2547a0b | |||
| f62220eb61 | |||
| 72992b7cc5 | |||
| b5fb6a85cf | |||
| fedd164686 | |||
| 261b7be468 | |||
| 6691f4d1f3 | |||
| ea76af068a | |||
| b61fcd3495 | |||
| 1e1689f931 | |||
| acc0df00cf | |||
| a0c35202f3 | |||
| fe1d576c3c | |||
| 3e65271af6 | |||
| 697575e561 | |||
| e6391c599d | |||
| d697c3d93e | |||
| 31c260cc95 | |||
| 3217c32356 | |||
| 25157a71a8 | |||
| 46edac3e76 | |||
| a5b95356dd | |||
| b197cf409e | |||
| 3ed2bbab02 | |||
| 3d40523947 | |||
| f86e2e103d | |||
| 7d20d18af1 | |||
| 7afb72209a | |||
| b12fa8aa07 |
@@ -27,8 +27,12 @@
|
||||
|
||||
# ── AirLLM / big-brain backend ───────────────────────────────────────────────
|
||||
# Inference backend: "ollama" (default) | "airllm" | "auto"
|
||||
# "auto" → uses AirLLM on Apple Silicon if installed, otherwise Ollama.
|
||||
# Requires: pip install ".[bigbrain]"
|
||||
# "ollama" → always use Ollama (safe everywhere, any OS)
|
||||
# "airllm" → AirLLM layer-by-layer loading (Apple Silicon M1/M2/M3/M4 only)
|
||||
# Requires 16 GB RAM minimum (32 GB recommended).
|
||||
# Automatically falls back to Ollama on Intel Mac or Linux.
|
||||
# Install extra: pip install "airllm[mlx]"
|
||||
# "auto" → use AirLLM on Apple Silicon if installed, otherwise Ollama
|
||||
# TIMMY_MODEL_BACKEND=ollama
|
||||
|
||||
# AirLLM model size (default: 70b).
|
||||
|
||||
@@ -62,6 +62,9 @@ Per AGENTS.md roster:
|
||||
- Run `tox -e pre-push` (lint + full CI suite)
|
||||
- Ensure tests stay green
|
||||
- Update TODO.md
|
||||
- **CRITICAL: Stage files before committing** — always run `git add .` or `git add <files>` first
|
||||
- Verify staged changes are non-empty: `git diff --cached --stat` must show files
|
||||
- **NEVER run `git commit` without staging files first** — empty commits waste review cycles
|
||||
|
||||
---
|
||||
|
||||
|
||||
80
AGENTS.md
80
AGENTS.md
@@ -34,6 +34,44 @@ Read [`CLAUDE.md`](CLAUDE.md) for architecture patterns and conventions.
|
||||
|
||||
---
|
||||
|
||||
## One-Agent-Per-Issue Convention
|
||||
|
||||
**An issue must only be worked by one agent at a time.** Duplicate branches from
|
||||
multiple agents on the same issue cause merge conflicts, redundant code, and wasted compute.
|
||||
|
||||
### Labels
|
||||
|
||||
When an agent picks up an issue, add the corresponding label:
|
||||
|
||||
| Label | Meaning |
|
||||
|-------|---------|
|
||||
| `assigned-claude` | Claude is actively working this issue |
|
||||
| `assigned-gemini` | Gemini is actively working this issue |
|
||||
| `assigned-kimi` | Kimi is actively working this issue |
|
||||
| `assigned-manus` | Manus is actively working this issue |
|
||||
|
||||
### Rules
|
||||
|
||||
1. **Before starting an issue**, check that none of the `assigned-*` labels are present.
|
||||
If one is, skip the issue — another agent owns it.
|
||||
2. **When you start**, add the label matching your agent (e.g. `assigned-claude`).
|
||||
3. **When your PR is merged or closed**, remove the label (or it auto-clears when
|
||||
the branch is deleted — see Auto-Delete below).
|
||||
4. **Never assign the same issue to two agents simultaneously.**
|
||||
|
||||
### Auto-Delete Merged Branches
|
||||
|
||||
`default_delete_branch_after_merge` is **enabled** on this repo. Branches are
|
||||
automatically deleted after a PR merges — no manual cleanup needed and no stale
|
||||
`claude/*`, `gemini/*`, or `kimi/*` branches accumulate.
|
||||
|
||||
If you discover stale merged branches, they can be pruned with:
|
||||
```bash
|
||||
git fetch --prune
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Merge Policy (PR-Only)
|
||||
|
||||
**Gitea branch protection is active on `main`.** This is not a suggestion.
|
||||
@@ -209,6 +247,48 @@ make docker-agent # add a worker
|
||||
|
||||
---
|
||||
|
||||
## Search Capability (SearXNG + Crawl4AI)
|
||||
|
||||
Timmy has a self-hosted search backend requiring **no paid API key**.
|
||||
|
||||
### Tools
|
||||
|
||||
| Tool | Module | Description |
|
||||
|------|--------|-------------|
|
||||
| `web_search(query)` | `timmy/tools/search.py` | Meta-search via SearXNG — returns ranked results |
|
||||
| `scrape_url(url)` | `timmy/tools/search.py` | Full-page scrape via Crawl4AI → clean markdown |
|
||||
|
||||
Both tools are registered in the **orchestrator** (full) and **echo** (research) toolkits.
|
||||
|
||||
### Configuration
|
||||
|
||||
| Env Var | Default | Description |
|
||||
|---------|---------|-------------|
|
||||
| `TIMMY_SEARCH_BACKEND` | `searxng` | `searxng` or `none` (disable) |
|
||||
| `TIMMY_SEARCH_URL` | `http://localhost:8888` | SearXNG base URL |
|
||||
| `TIMMY_CRAWL_URL` | `http://localhost:11235` | Crawl4AI base URL |
|
||||
|
||||
Inside Docker Compose (when `--profile search` is active), the dashboard
|
||||
uses `http://searxng:8080` and `http://crawl4ai:11235` by default.
|
||||
|
||||
### Starting the services
|
||||
|
||||
```bash
|
||||
# Start SearXNG + Crawl4AI alongside the dashboard:
|
||||
docker compose --profile search up
|
||||
|
||||
# Or start only the search services:
|
||||
docker compose --profile search up searxng crawl4ai
|
||||
```
|
||||
|
||||
### Graceful degradation
|
||||
|
||||
- If `TIMMY_SEARCH_BACKEND=none`: tools return a "disabled" message.
|
||||
- If SearXNG or Crawl4AI is unreachable: tools log a WARNING and return an
|
||||
error string — the app never crashes.
|
||||
|
||||
---
|
||||
|
||||
## Roadmap
|
||||
|
||||
**v2.0 Exodus (in progress):** Voice + Marketplace + Integrations
|
||||
|
||||
15
README.md
15
README.md
@@ -9,6 +9,21 @@ API access with Bitcoin Lightning — all from a browser, no cloud AI required.
|
||||
|
||||
---
|
||||
|
||||
## System Requirements
|
||||
|
||||
| Path | Hardware | RAM | Disk |
|
||||
|------|----------|-----|------|
|
||||
| **Ollama** (default) | Any OS — x86-64 or ARM | 8 GB min | 5–10 GB (model files) |
|
||||
| **AirLLM** (Apple Silicon) | M1, M2, M3, or M4 Mac | 16 GB min (32 GB recommended) | ~15 GB free |
|
||||
|
||||
**Ollama path** runs on any modern machine — macOS, Linux, or Windows. No GPU required.
|
||||
|
||||
**AirLLM path** uses layer-by-layer loading for 70B+ models without a GPU. Requires Apple
|
||||
Silicon and the `bigbrain` extras (`pip install ".[bigbrain]"`). On Intel Mac or Linux the
|
||||
app automatically falls back to Ollama — no crash, no config change needed.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
|
||||
122
SOVEREIGNTY.md
Normal file
122
SOVEREIGNTY.md
Normal file
@@ -0,0 +1,122 @@
|
||||
# SOVEREIGNTY.md — Research Sovereignty Manifest
|
||||
|
||||
> "If this spec is implemented correctly, it is the last research document
|
||||
> Alexander should need to request from a corporate AI."
|
||||
> — Issue #972, March 22 2026
|
||||
|
||||
---
|
||||
|
||||
## What This Is
|
||||
|
||||
A machine-readable declaration of Timmy's research independence:
|
||||
where we are, where we're going, and how to measure progress.
|
||||
|
||||
---
|
||||
|
||||
## The Problem We're Solving
|
||||
|
||||
On March 22, 2026, a single Claude session produced six deep research reports.
|
||||
It consumed ~3 hours of human time and substantial corporate AI inference.
|
||||
Every report was valuable — but the workflow was **linear**.
|
||||
It would cost exactly the same to reproduce tomorrow.
|
||||
|
||||
This file tracks the pipeline that crystallizes that workflow into something
|
||||
Timmy can run autonomously.
|
||||
|
||||
---
|
||||
|
||||
## The Six-Step Pipeline
|
||||
|
||||
| Step | What Happens | Status |
|
||||
|------|-------------|--------|
|
||||
| 1. Scope | Human describes knowledge gap → Gitea issue with template | ✅ Done (`skills/research/`) |
|
||||
| 2. Query | LLM slot-fills template → 5–15 targeted queries | ✅ Done (`research.py`) |
|
||||
| 3. Search | Execute queries → top result URLs | ✅ Done (`research_tools.py`) |
|
||||
| 4. Fetch | Download + extract full pages (trafilatura) | ✅ Done (`tools/system_tools.py`) |
|
||||
| 5. Synthesize | Compress findings → structured report | ✅ Done (`research.py` cascade) |
|
||||
| 6. Deliver | Store to semantic memory + optional disk persist | ✅ Done (`research.py`) |
|
||||
|
||||
---
|
||||
|
||||
## Cascade Tiers (Synthesis Quality vs. Cost)
|
||||
|
||||
| Tier | Model | Cost | Quality | Status |
|
||||
|------|-------|------|---------|--------|
|
||||
| **4** | SQLite semantic cache | $0.00 / instant | reuses prior | ✅ Active |
|
||||
| **3** | Ollama `qwen3:14b` | $0.00 / local | ★★★ | ✅ Active |
|
||||
| **2** | Claude API (haiku) | ~$0.01/report | ★★★★ | ✅ Active (opt-in) |
|
||||
| **1** | Groq `llama-3.3-70b` | $0.00 / rate-limited | ★★★★ | 🔲 Planned (#980) |
|
||||
|
||||
Set `ANTHROPIC_API_KEY` to enable Tier 2 fallback.
|
||||
|
||||
---
|
||||
|
||||
## Research Templates
|
||||
|
||||
Six prompt templates live in `skills/research/`:
|
||||
|
||||
| Template | Use Case |
|
||||
|----------|----------|
|
||||
| `tool_evaluation.md` | Find all shipping tools for `{domain}` |
|
||||
| `architecture_spike.md` | How to connect `{system_a}` to `{system_b}` |
|
||||
| `game_analysis.md` | Evaluate `{game}` for AI agent play |
|
||||
| `integration_guide.md` | Wire `{tool}` into `{stack}` with code |
|
||||
| `state_of_art.md` | What exists in `{field}` as of `{date}` |
|
||||
| `competitive_scan.md` | How does `{project}` compare to `{alternatives}` |
|
||||
|
||||
---
|
||||
|
||||
## Sovereignty Metrics
|
||||
|
||||
| Metric | Target (Week 1) | Target (Month 1) | Target (Month 3) | Graduation |
|
||||
|--------|-----------------|------------------|------------------|------------|
|
||||
| Queries answered locally | 10% | 40% | 80% | >90% |
|
||||
| API cost per report | <$1.50 | <$0.50 | <$0.10 | <$0.01 |
|
||||
| Time from question to report | <3 hours | <30 min | <5 min | <1 min |
|
||||
| Human involvement | 100% (review) | Review only | Approve only | None |
|
||||
|
||||
---
|
||||
|
||||
## How to Use the Pipeline
|
||||
|
||||
```python
|
||||
from timmy.research import run_research
|
||||
|
||||
# Quick research (no template)
|
||||
result = await run_research("best local embedding models for 36GB RAM")
|
||||
|
||||
# With a template and slot values
|
||||
result = await run_research(
|
||||
topic="PDF text extraction libraries for Python",
|
||||
template="tool_evaluation",
|
||||
slots={"domain": "PDF parsing", "use_case": "RAG pipeline", "focus_criteria": "accuracy"},
|
||||
save_to_disk=True,
|
||||
)
|
||||
|
||||
print(result.report)
|
||||
print(f"Backend: {result.synthesis_backend}, Cached: {result.cached}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Status
|
||||
|
||||
| Component | Issue | Status |
|
||||
|-----------|-------|--------|
|
||||
| `web_fetch` tool (trafilatura) | #973 | ✅ Done |
|
||||
| Research template library (6 templates) | #974 | ✅ Done |
|
||||
| `ResearchOrchestrator` (`research.py`) | #975 | ✅ Done |
|
||||
| Semantic index for outputs | #976 | 🔲 Planned |
|
||||
| Auto-create Gitea issues from findings | #977 | 🔲 Planned |
|
||||
| Paperclip task runner integration | #978 | 🔲 Planned |
|
||||
| Kimi delegation via labels | #979 | 🔲 Planned |
|
||||
| Groq free-tier cascade tier | #980 | 🔲 Planned |
|
||||
| Sovereignty metrics dashboard | #981 | 🔲 Planned |
|
||||
|
||||
---
|
||||
|
||||
## Governing Spec
|
||||
|
||||
See [issue #972](http://143.198.27.163:3000/Rockachopa/Timmy-time-dashboard/issues/972) for the full spec and rationale.
|
||||
|
||||
Research artifacts committed to `docs/research/`.
|
||||
@@ -25,6 +25,19 @@ providers:
|
||||
tier: local
|
||||
url: "http://localhost:11434"
|
||||
models:
|
||||
# ── Dual-model routing: Qwen3-8B (fast) + Qwen3-14B (quality) ──────────
|
||||
# Both models fit simultaneously: ~6.6 GB + ~10.5 GB = ~17 GB combined.
|
||||
# Requires OLLAMA_MAX_LOADED_MODELS=2 (set in .env) to stay hot.
|
||||
# Ref: issue #1065 — Qwen3-8B/14B dual-model routing strategy
|
||||
- name: qwen3:8b
|
||||
context_window: 32768
|
||||
capabilities: [text, tools, json, streaming, routine]
|
||||
description: "Qwen3-8B Q6_K — fast router for routine tasks (~6.6 GB, 45-55 tok/s)"
|
||||
- name: qwen3:14b
|
||||
context_window: 40960
|
||||
capabilities: [text, tools, json, streaming, complex, reasoning]
|
||||
description: "Qwen3-14B Q5_K_M — complex reasoning and planning (~10.5 GB, 20-28 tok/s)"
|
||||
|
||||
# Text + Tools models
|
||||
- name: qwen3:30b
|
||||
default: true
|
||||
@@ -187,6 +200,20 @@ fallback_chains:
|
||||
- dolphin3 # base Dolphin 3.0 8B (uncensored, no custom system prompt)
|
||||
- qwen3:30b # primary fallback — usually sufficient with a good system prompt
|
||||
|
||||
# ── Complexity-based routing chains (issue #1065) ───────────────────────
|
||||
# Routine tasks: prefer Qwen3-8B for low latency (~45-55 tok/s)
|
||||
routine:
|
||||
- qwen3:8b # Primary fast model
|
||||
- llama3.1:8b-instruct # Fallback fast model
|
||||
- llama3.2:3b # Smallest available
|
||||
|
||||
# Complex tasks: prefer Qwen3-14B for quality (~20-28 tok/s)
|
||||
complex:
|
||||
- qwen3:14b # Primary quality model
|
||||
- hermes4-14b # Native tool calling, hybrid reasoning
|
||||
- qwen3:30b # Highest local quality
|
||||
- qwen2.5:14b # Additional fallback
|
||||
|
||||
# ── Custom Models ───────────────────────────────────────────────────────────
|
||||
# Register custom model weights for per-agent assignment.
|
||||
# Supports GGUF (Ollama), safetensors, and HuggingFace checkpoint dirs.
|
||||
|
||||
@@ -42,6 +42,10 @@ services:
|
||||
GROK_ENABLED: "${GROK_ENABLED:-false}"
|
||||
XAI_API_KEY: "${XAI_API_KEY:-}"
|
||||
GROK_DEFAULT_MODEL: "${GROK_DEFAULT_MODEL:-grok-3-fast}"
|
||||
# Search backend (SearXNG + Crawl4AI) — set TIMMY_SEARCH_BACKEND=none to disable
|
||||
TIMMY_SEARCH_BACKEND: "${TIMMY_SEARCH_BACKEND:-searxng}"
|
||||
TIMMY_SEARCH_URL: "${TIMMY_SEARCH_URL:-http://searxng:8080}"
|
||||
TIMMY_CRAWL_URL: "${TIMMY_CRAWL_URL:-http://crawl4ai:11235}"
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway" # Linux: maps to host IP
|
||||
networks:
|
||||
@@ -74,6 +78,50 @@ services:
|
||||
profiles:
|
||||
- celery
|
||||
|
||||
# ── SearXNG — self-hosted meta-search engine ─────────────────────────
|
||||
searxng:
|
||||
image: searxng/searxng:latest
|
||||
container_name: timmy-searxng
|
||||
profiles:
|
||||
- search
|
||||
ports:
|
||||
- "${SEARXNG_PORT:-8888}:8080"
|
||||
environment:
|
||||
SEARXNG_BASE_URL: "${SEARXNG_BASE_URL:-http://localhost:8888}"
|
||||
volumes:
|
||||
- ./docker/searxng:/etc/searxng:rw
|
||||
networks:
|
||||
- timmy-net
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-qO-", "http://localhost:8080/healthz"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 20s
|
||||
|
||||
# ── Crawl4AI — self-hosted web scraper ────────────────────────────────
|
||||
crawl4ai:
|
||||
image: unclecode/crawl4ai:latest
|
||||
container_name: timmy-crawl4ai
|
||||
profiles:
|
||||
- search
|
||||
ports:
|
||||
- "${CRAWL4AI_PORT:-11235}:11235"
|
||||
environment:
|
||||
CRAWL4AI_API_TOKEN: "${CRAWL4AI_API_TOKEN:-}"
|
||||
volumes:
|
||||
- timmy-data:/app/data
|
||||
networks:
|
||||
- timmy-net
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:11235/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
|
||||
# ── OpenFang — vendored agent runtime sidecar ────────────────────────────
|
||||
openfang:
|
||||
build:
|
||||
|
||||
67
docker/searxng/settings.yml
Normal file
67
docker/searxng/settings.yml
Normal file
@@ -0,0 +1,67 @@
|
||||
# SearXNG configuration for Timmy Time self-hosted search
|
||||
# https://docs.searxng.org/admin/settings/settings.html
|
||||
|
||||
general:
|
||||
debug: false
|
||||
instance_name: "Timmy Search"
|
||||
privacypolicy_url: false
|
||||
donation_url: false
|
||||
contact_url: false
|
||||
enable_metrics: false
|
||||
|
||||
server:
|
||||
port: 8080
|
||||
bind_address: "0.0.0.0"
|
||||
secret_key: "timmy-searxng-key-change-in-production"
|
||||
base_url: false
|
||||
image_proxy: false
|
||||
|
||||
ui:
|
||||
static_use_hash: false
|
||||
default_locale: ""
|
||||
query_in_title: false
|
||||
infinite_scroll: false
|
||||
default_theme: simple
|
||||
center_alignment: false
|
||||
|
||||
search:
|
||||
safe_search: 0
|
||||
autocomplete: ""
|
||||
default_lang: "en"
|
||||
formats:
|
||||
- html
|
||||
- json
|
||||
|
||||
outgoing:
|
||||
request_timeout: 6.0
|
||||
max_request_timeout: 10.0
|
||||
useragent_suffix: "TimmyResearchBot"
|
||||
pool_connections: 100
|
||||
pool_maxsize: 20
|
||||
|
||||
enabled_plugins:
|
||||
- Hash_plugin
|
||||
- Search_on_category_select
|
||||
- Tracker_url_remover
|
||||
|
||||
engines:
|
||||
- name: google
|
||||
engine: google
|
||||
shortcut: g
|
||||
categories: general
|
||||
|
||||
- name: bing
|
||||
engine: bing
|
||||
shortcut: b
|
||||
categories: general
|
||||
|
||||
- name: duckduckgo
|
||||
engine: duckduckgo
|
||||
shortcut: d
|
||||
categories: general
|
||||
|
||||
- name: wikipedia
|
||||
engine: wikipedia
|
||||
shortcut: wp
|
||||
categories: general
|
||||
timeout: 3.0
|
||||
244
docs/GITEA_AUDIT_2026-03-23.md
Normal file
244
docs/GITEA_AUDIT_2026-03-23.md
Normal file
@@ -0,0 +1,244 @@
|
||||
# Gitea Activity & Branch Audit — 2026-03-23
|
||||
|
||||
**Requested by:** Issue #1210
|
||||
**Audited by:** Claude (Sonnet 4.6)
|
||||
**Date:** 2026-03-23
|
||||
**Scope:** All repos under the sovereign AI stack
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
- **18 repos audited** across 9 Gitea organizations/users
|
||||
- **~65–70 branches identified** as safe to delete (merged or abandoned)
|
||||
- **4 open PRs** are bottlenecks awaiting review
|
||||
- **3+ instances of duplicate work** across repos and agents
|
||||
- **5+ branches** contain valuable unmerged code with no open PR
|
||||
- **5 PRs closed without merge** on active p0-critical issues in Timmy-time-dashboard
|
||||
|
||||
Improvement tickets have been filed on each affected repo following this report.
|
||||
|
||||
---
|
||||
|
||||
## Repo-by-Repo Findings
|
||||
|
||||
---
|
||||
|
||||
### 1. rockachopa/Timmy-time-dashboard
|
||||
|
||||
**Status:** Most active repo. 1,200+ PRs, 50+ branches.
|
||||
|
||||
#### Dead/Abandoned Branches
|
||||
| Branch | Last Commit | Status |
|
||||
|--------|-------------|--------|
|
||||
| `feature/voice-customization` | 2026-03-22 | Gemini-created, no PR, abandoned |
|
||||
| `feature/enhanced-memory-ui` | 2026-03-22 | Gemini-created, no PR, abandoned |
|
||||
| `feature/soul-customization` | 2026-03-22 | Gemini-created, no PR, abandoned |
|
||||
| `feature/dreaming-mode` | 2026-03-22 | Gemini-created, no PR, abandoned |
|
||||
| `feature/memory-visualization` | 2026-03-22 | Gemini-created, no PR, abandoned |
|
||||
| `feature/voice-customization-ui` | 2026-03-22 | Gemini-created, no PR, abandoned |
|
||||
| `feature/issue-1015` | 2026-03-22 | Gemini-created, no PR, abandoned |
|
||||
| `feature/issue-1016` | 2026-03-22 | Gemini-created, no PR, abandoned |
|
||||
| `feature/issue-1017` | 2026-03-22 | Gemini-created, no PR, abandoned |
|
||||
| `feature/issue-1018` | 2026-03-22 | Gemini-created, no PR, abandoned |
|
||||
| `feature/issue-1019` | 2026-03-22 | Gemini-created, no PR, abandoned |
|
||||
| `feature/self-reflection` | 2026-03-22 | Only merge-from-main commits, no unique work |
|
||||
| `feature/memory-search-ui` | 2026-03-22 | Only merge-from-main commits, no unique work |
|
||||
| `claude/issue-962` | 2026-03-22 | Automated salvage commit only |
|
||||
| `claude/issue-972` | 2026-03-22 | Automated salvage commit only |
|
||||
| `gemini/issue-1006` | 2026-03-22 | Incomplete agent session |
|
||||
| `gemini/issue-1008` | 2026-03-22 | Incomplete agent session |
|
||||
| `gemini/issue-1010` | 2026-03-22 | Incomplete agent session |
|
||||
| `gemini/issue-1134` | 2026-03-22 | Incomplete agent session |
|
||||
| `gemini/issue-1139` | 2026-03-22 | Incomplete agent session |
|
||||
|
||||
#### Duplicate Branches (Identical SHA)
|
||||
| Branch A | Branch B | Action |
|
||||
|----------|----------|--------|
|
||||
| `feature/internal-monologue` | `feature/issue-1005` | Exact duplicate — delete one |
|
||||
| `claude/issue-1005` | (above) | Merge-from-main only — delete |
|
||||
|
||||
#### Unmerged Work With No Open PR (HIGH PRIORITY)
|
||||
| Branch | Content | Issues |
|
||||
|--------|---------|--------|
|
||||
| `claude/issue-987` | Content moderation pipeline, Llama Guard integration | No open PR — potentially lost |
|
||||
| `claude/issue-1011` | Automated skill discovery system | No open PR — potentially lost |
|
||||
| `gemini/issue-976` | Semantic index for research outputs | No open PR — potentially lost |
|
||||
|
||||
#### PRs Closed Without Merge (Issues Still Open)
|
||||
| PR | Title | Issue Status |
|
||||
|----|-------|-------------|
|
||||
| PR#1163 | Three-Strike Detector (#962) | p0-critical, still open |
|
||||
| PR#1162 | Session Sovereignty Report Generator (#957) | p0-critical, still open |
|
||||
| PR#1157 | Qwen3 routing | open |
|
||||
| PR#1156 | Agent Dreaming Mode | open |
|
||||
| PR#1145 | Qwen3-14B config | open |
|
||||
|
||||
#### Workflow Observations
|
||||
- `loop-cycle` bot auto-creates micro-fix PRs at high frequency (PR numbers climbing past 1209 rapidly)
|
||||
- Many `gemini/*` branches represent incomplete agent sessions, not full feature work
|
||||
- Issues get reassigned across agents causing duplicate branch proliferation
|
||||
|
||||
---
|
||||
|
||||
### 2. rockachopa/hermes-agent
|
||||
|
||||
**Status:** Active — AutoLoRA training pipeline in progress.
|
||||
|
||||
#### Open PRs Awaiting Review
|
||||
| PR | Title | Age |
|
||||
|----|-------|-----|
|
||||
| PR#33 | AutoLoRA v1 MLX QLoRA training pipeline | ~1 week |
|
||||
|
||||
#### Valuable Unmerged Branches (No PR)
|
||||
| Branch | Content | Age |
|
||||
|--------|---------|-----|
|
||||
| `sovereign` | Full fallback chain: Groq/Kimi/Ollama cascade recovery | 9 days |
|
||||
| `fix/vision-api-key-fallback` | Vision API key fallback fix | 9 days |
|
||||
|
||||
#### Stale Merged Branches (~12)
|
||||
12 merged `claude/*` and `gemini/*` branches are safe to delete.
|
||||
|
||||
---
|
||||
|
||||
### 3. rockachopa/the-matrix
|
||||
|
||||
**Status:** 8 open PRs from `claude/the-matrix` fork all awaiting review, all batch-created on 2026-03-23.
|
||||
|
||||
#### Open PRs (ALL Awaiting Review)
|
||||
| PR | Feature |
|
||||
|----|---------|
|
||||
| PR#9–16 | Touch controls, agent feed, particles, audio, day/night cycle, metrics panel, ASCII logo, click-to-view-PR |
|
||||
|
||||
These were created in a single agent session within 5 minutes — needs human review before merge.
|
||||
|
||||
---
|
||||
|
||||
### 4. replit/timmy-tower
|
||||
|
||||
**Status:** Very active — 100+ PRs, complex feature roadmap.
|
||||
|
||||
#### Open PRs Awaiting Review
|
||||
| PR | Title | Age |
|
||||
|----|-------|-----|
|
||||
| PR#93 | Task decomposition view | Recent |
|
||||
| PR#80 | `session_messages` table | 22 hours |
|
||||
|
||||
#### Unmerged Work With No Open PR
|
||||
| Branch | Content |
|
||||
|--------|---------|
|
||||
| `gemini/issue-14` | NIP-07 Nostr identity |
|
||||
| `gemini/issue-42` | Timmy animated eyes |
|
||||
| `claude/issue-11` | Kimi + Perplexity agent integrations |
|
||||
| `claude/issue-13` | Nostr event publishing |
|
||||
| `claude/issue-29` | Mobile Nostr identity |
|
||||
| `claude/issue-45` | Test kit |
|
||||
| `claude/issue-47` | SQL migration helpers |
|
||||
| `claude/issue-67` | Session Mode UI |
|
||||
|
||||
#### Cleanup
|
||||
~30 merged `claude/*` and `gemini/*` branches are safe to delete.
|
||||
|
||||
---
|
||||
|
||||
### 5. replit/token-gated-economy
|
||||
|
||||
**Status:** Active roadmap, no current open PRs.
|
||||
|
||||
#### Stale Branches (~23)
|
||||
- 8 Replit Agent branches from 2026-03-19 (PRs closed/merged)
|
||||
- 15 merged `claude/issue-*` branches
|
||||
|
||||
All are safe to delete.
|
||||
|
||||
---
|
||||
|
||||
### 6. hermes/timmy-time-app
|
||||
|
||||
**Status:** 2-commit repo, created 2026-03-14, no activity since. **Candidate for archival.**
|
||||
|
||||
Functionality appears to be superseded by other repos in the stack. Recommend archiving or deleting if not planned for future development.
|
||||
|
||||
---
|
||||
|
||||
### 7. google/maintenance-tasks & google/wizard-council-automation
|
||||
|
||||
**Status:** Single-commit repos from 2026-03-19 created by "Google AI Studio". No follow-up activity.
|
||||
|
||||
Unclear ownership and purpose. Recommend clarifying with rockachopa whether these are active or can be archived.
|
||||
|
||||
---
|
||||
|
||||
### 8. hermes/hermes-config
|
||||
|
||||
**Status:** Single branch, updated 2026-03-23 (today). Active — contains Timmy orchestrator config.
|
||||
|
||||
No action needed.
|
||||
|
||||
---
|
||||
|
||||
### 9. Timmy_Foundation/the-nexus
|
||||
|
||||
**Status:** Greenfield — created 2026-03-23. 19 issues filed as roadmap. PR#2 (contributor audit) open.
|
||||
|
||||
No cleanup needed yet. PR#2 needs review.
|
||||
|
||||
---
|
||||
|
||||
### 10. rockachopa/alexanderwhitestone.com
|
||||
|
||||
**Status:** All recent `claude/*` PRs merged. 7 non-main branches are post-merge and safe to delete.
|
||||
|
||||
---
|
||||
|
||||
### 11. hermes/hermes-config, rockachopa/hermes-config, Timmy_Foundation/.profile
|
||||
|
||||
**Status:** Dormant config repos. No action needed.
|
||||
|
||||
---
|
||||
|
||||
## Cross-Repo Patterns & Inefficiencies
|
||||
|
||||
### Duplicate Work
|
||||
1. **Timmy spring/wobble physics** built independently in both `replit/timmy-tower` and `replit/token-gated-economy`
|
||||
2. **Nostr identity logic** fragmented across 3 repos with no shared library
|
||||
3. **`feature/internal-monologue` = `feature/issue-1005`** in Timmy-time-dashboard — identical SHA, exact duplicate
|
||||
|
||||
### Agent Workflow Issues
|
||||
- Same issue assigned to both `gemini/*` and `claude/*` agents creates duplicate branches
|
||||
- Agent salvage commits are checkpoint-only — not complete work, but clutter the branch list
|
||||
- Gemini `feature/*` branches created on 2026-03-22 with no PRs filed — likely a failed agent session that created branches but didn't complete the loop
|
||||
|
||||
### Review Bottlenecks
|
||||
| Repo | Waiting PRs | Notes |
|
||||
|------|-------------|-------|
|
||||
| rockachopa/the-matrix | 8 | Batch-created, need human review |
|
||||
| replit/timmy-tower | 2 | Database schema and UI work |
|
||||
| rockachopa/hermes-agent | 1 | AutoLoRA v1 — high value |
|
||||
| Timmy_Foundation/the-nexus | 1 | Contributor audit |
|
||||
|
||||
---
|
||||
|
||||
## Recommended Actions
|
||||
|
||||
### Immediate (This Sprint)
|
||||
1. **Review & merge** PR#33 in `hermes-agent` (AutoLoRA v1)
|
||||
2. **Review** 8 open PRs in `the-matrix` before merging as a batch
|
||||
3. **Rescue** unmerged work in `claude/issue-987`, `claude/issue-1011`, `gemini/issue-976` — file new PRs or close branches
|
||||
4. **Delete duplicate** `feature/internal-monologue` / `feature/issue-1005` branches
|
||||
|
||||
### Cleanup Sprint
|
||||
5. **Delete ~65 stale branches** across all repos (itemized above)
|
||||
6. **Investigate** the 5 closed-without-merge PRs in Timmy-time-dashboard for p0-critical issues
|
||||
7. **Archive** `hermes/timmy-time-app` if no longer needed
|
||||
8. **Clarify** ownership of `google/maintenance-tasks` and `google/wizard-council-automation`
|
||||
|
||||
### Process Improvements
|
||||
9. **Enforce one-agent-per-issue** policy to prevent duplicate `claude/*` / `gemini/*` branches
|
||||
10. **Add branch protection** requiring PR before merge on `main` for all repos
|
||||
11. **Set a branch retention policy** — auto-delete merged branches (GitHub/Gitea supports this)
|
||||
12. **Share common libraries** for Nostr identity and animation physics across repos
|
||||
|
||||
---
|
||||
|
||||
*Report generated by Claude audit agent. Improvement tickets filed per repo as follow-up to this report.*
|
||||
89
docs/SCREENSHOT_TRIAGE_2026-03-24.md
Normal file
89
docs/SCREENSHOT_TRIAGE_2026-03-24.md
Normal file
@@ -0,0 +1,89 @@
|
||||
# Screenshot Dump Triage — Visual Inspiration & Research Leads
|
||||
|
||||
**Date:** March 24, 2026
|
||||
**Source:** Issue #1275 — "Screenshot dump for triage #1"
|
||||
**Analyst:** Claude (Sonnet 4.6)
|
||||
|
||||
---
|
||||
|
||||
## Screenshots Ingested
|
||||
|
||||
| File | Subject | Action |
|
||||
|------|---------|--------|
|
||||
| IMG_6187.jpeg | AirLLM / Apple Silicon local LLM requirements | → Issue #1284 |
|
||||
| IMG_6125.jpeg | vLLM backend for agentic workloads | → Issue #1281 |
|
||||
| IMG_6124.jpeg | DeerFlow autonomous research pipeline | → Issue #1283 |
|
||||
| IMG_6123.jpeg | "Vibe Coder vs Normal Developer" meme | → Issue #1285 |
|
||||
| IMG_6410.jpeg | SearXNG + Crawl4AI self-hosted search MCP | → Issue #1282 |
|
||||
|
||||
---
|
||||
|
||||
## Tickets Created
|
||||
|
||||
### #1281 — feat: add vLLM as alternative inference backend
|
||||
**Source:** IMG_6125 (vLLM for agentic workloads)
|
||||
|
||||
vLLM's continuous batching makes it 3–10x more throughput-efficient than Ollama for multi-agent
|
||||
request patterns. Implement `VllmBackend` in `infrastructure/llm_router/` as a selectable
|
||||
backend (`TIMMY_LLM_BACKEND=vllm`) with graceful fallback to Ollama.
|
||||
|
||||
**Priority:** Medium — impactful for research pipeline performance once #972 is in use
|
||||
|
||||
---
|
||||
|
||||
### #1282 — feat: integrate SearXNG + Crawl4AI as self-hosted search backend
|
||||
**Source:** IMG_6410 (luxiaolei/searxng-crawl4ai-mcp)
|
||||
|
||||
Self-hosted search via SearXNG + Crawl4AI removes the hard dependency on paid search APIs
|
||||
(Brave, Tavily). Add both as Docker Compose services, implement `web_search()` and
|
||||
`scrape_url()` tools in `timmy/tools/`, and register them with the research agent.
|
||||
|
||||
**Priority:** High — unblocks fully local/private operation of research agents
|
||||
|
||||
---
|
||||
|
||||
### #1283 — research: evaluate DeerFlow as autonomous research orchestration layer
|
||||
**Source:** IMG_6124 (deer-flow Docker setup)
|
||||
|
||||
DeerFlow is ByteDance's open-source autonomous research pipeline framework. Before investing
|
||||
further in Timmy's custom orchestrator (#972), evaluate whether DeerFlow's architecture offers
|
||||
integration value or design patterns worth borrowing.
|
||||
|
||||
**Priority:** Medium — research first, implementation follows if go/no-go is positive
|
||||
|
||||
---
|
||||
|
||||
### #1284 — chore: document and validate AirLLM Apple Silicon requirements
|
||||
**Source:** IMG_6187 (Mac-compatible LLM setup)
|
||||
|
||||
AirLLM graceful degradation is already implemented but undocumented. Add System Requirements
|
||||
to README (M1/M2/M3/M4, 16 GB RAM min, 15 GB disk) and document `TIMMY_LLM_BACKEND` in
|
||||
`.env.example`.
|
||||
|
||||
**Priority:** Low — documentation only, no code risk
|
||||
|
||||
---
|
||||
|
||||
### #1285 — chore: enforce "Normal Developer" discipline — tighten quality gates
|
||||
**Source:** IMG_6123 (Vibe Coder vs Normal Developer meme)
|
||||
|
||||
Tighten the existing mypy/bandit/coverage gates: fix all mypy errors, raise coverage from 73%
|
||||
to 80%, add a documented pre-push hook, and run `vulture` for dead code. The infrastructure
|
||||
exists — it just needs enforcing.
|
||||
|
||||
**Priority:** Medium — technical debt prevention, pairs well with any green-field feature work
|
||||
|
||||
---
|
||||
|
||||
## Patterns Observed Across Screenshots
|
||||
|
||||
1. **Local-first is the north star.** All five images reinforce the same theme: private,
|
||||
self-hosted, runs on your hardware. vLLM, SearXNG, AirLLM, DeerFlow — none require cloud.
|
||||
Timmy is already aligned with this direction; these are tactical additions.
|
||||
|
||||
2. **Agentic performance bottlenecks are real.** Two of five images (vLLM, DeerFlow) focus
|
||||
specifically on throughput and reliability for multi-agent loops. As the research pipeline
|
||||
matures, inference speed and search reliability will become the main constraints.
|
||||
|
||||
3. **Discipline compounds.** The meme is a reminder that the quality gates we have (tox,
|
||||
mypy, bandit, coverage) only pay off if they are enforced without exceptions.
|
||||
160
docs/adr/024-nostr-identity-canonical-location.md
Normal file
160
docs/adr/024-nostr-identity-canonical-location.md
Normal file
@@ -0,0 +1,160 @@
|
||||
# ADR-024: Canonical Nostr Identity Location
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-03-23
|
||||
**Issue:** #1223
|
||||
**Refs:** #1210 (duplicate-work audit), ROADMAP.md Phase 2
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
Nostr identity logic has been independently implemented in at least three
|
||||
repos (`replit/timmy-tower`, `replit/token-gated-economy`,
|
||||
`rockachopa/Timmy-time-dashboard`), each building keypair generation, event
|
||||
publishing, and NIP-07 browser-extension auth in isolation.
|
||||
|
||||
This duplication causes:
|
||||
|
||||
- Bug fixes applied in one repo but silently missed in others.
|
||||
- Diverging implementations of the same NIPs (NIP-01, NIP-07, NIP-44).
|
||||
- Agent time wasted re-implementing logic that already exists.
|
||||
|
||||
ROADMAP.md Phase 2 already names `timmy-nostr` as the planned home for Nostr
|
||||
infrastructure. This ADR makes that decision explicit and prescribes how
|
||||
other repos consume it.
|
||||
|
||||
---
|
||||
|
||||
## Decision
|
||||
|
||||
**The canonical home for all Nostr identity logic is `rockachopa/timmy-nostr`.**
|
||||
|
||||
All other repos (`Timmy-time-dashboard`, `timmy-tower`,
|
||||
`token-gated-economy`) become consumers, not implementers, of Nostr identity
|
||||
primitives.
|
||||
|
||||
### What lives in `timmy-nostr`
|
||||
|
||||
| Module | Responsibility |
|
||||
|--------|---------------|
|
||||
| `nostr_id/keypair.py` | Keypair generation, nsec/npub encoding, encrypted storage |
|
||||
| `nostr_id/identity.py` | Agent identity lifecycle (NIP-01 kind:0 profile events) |
|
||||
| `nostr_id/auth.py` | NIP-07 browser-extension signer; NIP-42 relay auth |
|
||||
| `nostr_id/event.py` | Event construction, signing, serialisation (NIP-01) |
|
||||
| `nostr_id/crypto.py` | NIP-44 encryption (XChaCha20-Poly1305 v2) |
|
||||
| `nostr_id/nip05.py` | DNS-based identifier verification |
|
||||
| `nostr_id/relay.py` | WebSocket relay client (publish / subscribe) |
|
||||
|
||||
### What does NOT live in `timmy-nostr`
|
||||
|
||||
- Business logic that combines Nostr with application-specific concepts
|
||||
(e.g. "publish a task-completion event" lives in the application layer
|
||||
that calls `timmy-nostr`).
|
||||
- Reputation scoring algorithms (depends on application policy).
|
||||
- Dashboard UI components.
|
||||
|
||||
---
|
||||
|
||||
## How Other Repos Reference `timmy-nostr`
|
||||
|
||||
### Python repos (`Timmy-time-dashboard`, `timmy-tower`)
|
||||
|
||||
Add to `pyproject.toml` dependencies:
|
||||
|
||||
```toml
|
||||
[tool.poetry.dependencies]
|
||||
timmy-nostr = {git = "https://gitea.hermes.local/rockachopa/timmy-nostr.git", tag = "v0.1.0"}
|
||||
```
|
||||
|
||||
Import pattern:
|
||||
|
||||
```python
|
||||
from nostr_id.keypair import generate_keypair, load_keypair
|
||||
from nostr_id.event import build_event, sign_event
|
||||
from nostr_id.relay import NostrRelayClient
|
||||
```
|
||||
|
||||
### JavaScript/TypeScript repos (`token-gated-economy` frontend)
|
||||
|
||||
Add to `package.json` (once published or via local path):
|
||||
|
||||
```json
|
||||
"dependencies": {
|
||||
"timmy-nostr": "rockachopa/timmy-nostr#v0.1.0"
|
||||
}
|
||||
```
|
||||
|
||||
Import pattern:
|
||||
|
||||
```typescript
|
||||
import { generateKeypair, signEvent } from 'timmy-nostr';
|
||||
```
|
||||
|
||||
Until `timmy-nostr` publishes a JS package, use NIP-07 browser extension
|
||||
directly and delegate all key-management to the browser signer — never
|
||||
re-implement crypto in JS without the shared library.
|
||||
|
||||
---
|
||||
|
||||
## Migration Plan
|
||||
|
||||
Existing duplicated code should be migrated in this order:
|
||||
|
||||
1. **Keypair generation** — highest duplication, clearest interface.
|
||||
2. **NIP-01 event construction/signing** — used by all three repos.
|
||||
3. **NIP-07 browser auth** — currently in `timmy-tower` and `token-gated-economy`.
|
||||
4. **NIP-44 encryption** — lowest priority, least duplicated.
|
||||
|
||||
Each step: implement in `timmy-nostr` → cut over one repo → delete the
|
||||
duplicate → repeat.
|
||||
|
||||
---
|
||||
|
||||
## Interface Contract
|
||||
|
||||
`timmy-nostr` must expose a stable public API:
|
||||
|
||||
```python
|
||||
# Keypair
|
||||
keypair = generate_keypair() # -> NostrKeypair(nsec, npub, privkey_bytes, pubkey_bytes)
|
||||
keypair = load_keypair(encrypted_nsec, secret_key)
|
||||
|
||||
# Events
|
||||
event = build_event(kind=0, content=profile_json, keypair=keypair)
|
||||
event = sign_event(event, keypair) # attaches .id and .sig
|
||||
|
||||
# Relay
|
||||
async with NostrRelayClient(url) as relay:
|
||||
await relay.publish(event)
|
||||
async for msg in relay.subscribe(filters):
|
||||
...
|
||||
```
|
||||
|
||||
Breaking changes to this interface require a semver major bump and a
|
||||
migration note in `timmy-nostr`'s CHANGELOG.
|
||||
|
||||
---
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Positive:** Bug fixes in cryptographic or protocol code propagate to all
|
||||
repos via a version bump.
|
||||
- **Positive:** New NIPs are implemented once and adopted everywhere.
|
||||
- **Negative:** Adds a cross-repo dependency; version pinning discipline
|
||||
required.
|
||||
- **Negative:** `timmy-nostr` must be stood up and tagged before any
|
||||
migration can begin.
|
||||
|
||||
---
|
||||
|
||||
## Action Items
|
||||
|
||||
- [ ] Create `rockachopa/timmy-nostr` repo with the module structure above.
|
||||
- [ ] Implement keypair generation + NIP-01 signing as v0.1.0.
|
||||
- [ ] Replace `Timmy-time-dashboard` inline Nostr code (if any) with
|
||||
`timmy-nostr` import once v0.1.0 is tagged.
|
||||
- [ ] Add `src/infrastructure/clients/nostr_client.py` as the thin
|
||||
application-layer wrapper (see ROADMAP.md §2.6).
|
||||
- [ ] File issues in `timmy-tower` and `token-gated-economy` to migrate their
|
||||
duplicate implementations.
|
||||
1244
docs/model-benchmarks.md
Normal file
1244
docs/model-benchmarks.md
Normal file
File diff suppressed because it is too large
Load Diff
105
docs/nexus-spec.md
Normal file
105
docs/nexus-spec.md
Normal file
@@ -0,0 +1,105 @@
|
||||
# Nexus — Scope & Acceptance Criteria
|
||||
|
||||
**Issue:** #1208
|
||||
**Date:** 2026-03-23
|
||||
**Status:** Initial implementation complete; teaching/RL harness deferred
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
The **Nexus** is a persistent conversational space where Timmy lives with full
|
||||
access to his live memory. Unlike the main dashboard chat (which uses tools and
|
||||
has a transient feel), the Nexus is:
|
||||
|
||||
- **Conversational only** — no tool approval flow; pure dialogue
|
||||
- **Memory-aware** — semantically relevant memories surface alongside each exchange
|
||||
- **Teachable** — the operator can inject facts directly into Timmy's live memory
|
||||
- **Persistent** — the session survives page refreshes; history accumulates over time
|
||||
- **Local** — always backed by Ollama; no cloud inference required
|
||||
|
||||
This is the foundation for future LoRA fine-tuning, RL training harnesses, and
|
||||
eventually real-time self-improvement loops.
|
||||
|
||||
---
|
||||
|
||||
## Scope (v1 — this PR)
|
||||
|
||||
| Area | Included | Deferred |
|
||||
|------|----------|----------|
|
||||
| Conversational UI | ✅ Chat panel with HTMX streaming | Streaming tokens |
|
||||
| Live memory sidebar | ✅ Semantic search on each turn | Auto-refresh on teach |
|
||||
| Teaching panel | ✅ Inject personal facts | Bulk import, LoRA trigger |
|
||||
| Session isolation | ✅ Dedicated `nexus` session ID | Per-operator sessions |
|
||||
| Nav integration | ✅ NEXUS link in INTEL dropdown | Mobile nav |
|
||||
| CSS/styling | ✅ Two-column responsive layout | Dark/light theme toggle |
|
||||
| Tests | ✅ 9 unit tests, all green | E2E with real Ollama |
|
||||
| LoRA / RL harness | ❌ deferred to future issue | |
|
||||
| Auto-falsework | ❌ deferred | |
|
||||
| Bannerlord interface | ❌ separate track | |
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
### AC-1: Nexus page loads
|
||||
- **Given** the dashboard is running
|
||||
- **When** I navigate to `/nexus`
|
||||
- **Then** I see a two-panel layout: conversation on the left, memory sidebar on the right
|
||||
- **And** the page title reads "// NEXUS"
|
||||
- **And** the page is accessible from the nav (INTEL → NEXUS)
|
||||
|
||||
### AC-2: Conversation-only chat
|
||||
- **Given** I am on the Nexus page
|
||||
- **When** I type a message and submit
|
||||
- **Then** Timmy responds using the `nexus` session (isolated from dashboard history)
|
||||
- **And** no tool-approval cards appear — responses are pure text
|
||||
- **And** my message and Timmy's reply are appended to the chat log
|
||||
|
||||
### AC-3: Memory context surfaces automatically
|
||||
- **Given** I send a message
|
||||
- **When** the response arrives
|
||||
- **Then** the "LIVE MEMORY CONTEXT" panel shows up to 4 semantically relevant memories
|
||||
- **And** each memory entry shows its type and content
|
||||
|
||||
### AC-4: Teaching panel stores facts
|
||||
- **Given** I type a fact into the "TEACH TIMMY" input and submit
|
||||
- **When** the request completes
|
||||
- **Then** I see a green confirmation "✓ Taught: <fact>"
|
||||
- **And** the fact appears in the "KNOWN FACTS" list
|
||||
- **And** the fact is stored in Timmy's live memory (`store_personal_fact`)
|
||||
|
||||
### AC-5: Empty / invalid input is rejected gracefully
|
||||
- **Given** I submit a blank message or fact
|
||||
- **Then** no request is made and the log is unchanged
|
||||
- **Given** I submit a message over 10 000 characters
|
||||
- **Then** an inline error is shown without crashing the server
|
||||
|
||||
### AC-6: Conversation can be cleared
|
||||
- **Given** the Nexus has conversation history
|
||||
- **When** I click CLEAR and confirm
|
||||
- **Then** the chat log shows only a "cleared" confirmation
|
||||
- **And** the Agno session for `nexus` is reset
|
||||
|
||||
### AC-7: Graceful degradation when Ollama is down
|
||||
- **Given** Ollama is unavailable
|
||||
- **When** I send a message
|
||||
- **Then** an error message is shown inline (not a 500 page)
|
||||
- **And** the app continues to function
|
||||
|
||||
### AC-8: No regression on existing tests
|
||||
- **Given** the nexus route is registered
|
||||
- **When** `tox -e unit` runs
|
||||
- **Then** all 343+ existing tests remain green
|
||||
|
||||
---
|
||||
|
||||
## Future Work (separate issues)
|
||||
|
||||
1. **LoRA trigger** — button in the teaching panel to queue a fine-tuning run
|
||||
using the current Nexus conversation as training data
|
||||
2. **RL harness** — reward signal collection during conversation for RLHF
|
||||
3. **Auto-falsework pipeline** — scaffold harness generation from conversation
|
||||
4. **Bannerlord interface** — Nexus as the live-memory bridge for in-game Timmy
|
||||
5. **Streaming responses** — token-by-token display via WebSocket
|
||||
6. **Per-operator sessions** — isolate Nexus history by logged-in user
|
||||
75
docs/pr-recovery-1219.md
Normal file
75
docs/pr-recovery-1219.md
Normal file
@@ -0,0 +1,75 @@
|
||||
# PR Recovery Investigation — Issue #1219
|
||||
|
||||
**Audit source:** Issue #1210
|
||||
|
||||
Five PRs were closed without merge while their parent issues remained open and
|
||||
marked p0-critical. This document records the investigation findings and the
|
||||
path to resolution for each.
|
||||
|
||||
---
|
||||
|
||||
## Root Cause
|
||||
|
||||
Per Timmy's comment on #1219: all five PRs were closed due to **merge conflicts
|
||||
during the mass-merge cleanup cycle** (a rebase storm), not due to code
|
||||
quality problems or a changed approach. The code in each PR was correct;
|
||||
the branches simply became stale.
|
||||
|
||||
---
|
||||
|
||||
## Status Matrix
|
||||
|
||||
| PR | Feature | Issue | PR Closed | Issue State | Resolution |
|
||||
|----|---------|-------|-----------|-------------|------------|
|
||||
| #1163 | Three-Strike Detector | #962 | Rebase storm | **Closed ✓** | v2 merged via PR #1232 |
|
||||
| #1162 | Session Sovereignty Report | #957 | Rebase storm | **Open** | PR #1263 (v3 — rebased) |
|
||||
| #1157 | Qwen3-8B/14B routing | #1065 | Rebase storm | **Closed ✓** | v2 merged via PR #1233 |
|
||||
| #1156 | Agent Dreaming Mode | #1019 | Rebase storm | **Open** | PR #1264 (v3 — rebased) |
|
||||
| #1145 | Qwen3-14B config | #1064 | Rebase storm | **Closed ✓** | Code present on main |
|
||||
|
||||
---
|
||||
|
||||
## Detail: Already Resolved
|
||||
|
||||
### PR #1163 → Issue #962 (Three-Strike Detector)
|
||||
|
||||
- **Why closed:** merge conflict during rebase storm
|
||||
- **Resolution:** `src/timmy/sovereignty/three_strike.py` and
|
||||
`src/dashboard/routes/three_strike.py` are present on `main` (landed via
|
||||
PR #1232). Issue #962 is closed.
|
||||
|
||||
### PR #1157 → Issue #1065 (Qwen3-8B/14B dual-model routing)
|
||||
|
||||
- **Why closed:** merge conflict during rebase storm
|
||||
- **Resolution:** `src/infrastructure/router/classifier.py` and
|
||||
`src/infrastructure/router/cascade.py` are present on `main` (landed via
|
||||
PR #1233). Issue #1065 is closed.
|
||||
|
||||
### PR #1145 → Issue #1064 (Qwen3-14B config)
|
||||
|
||||
- **Why closed:** merge conflict during rebase storm
|
||||
- **Resolution:** `Modelfile.timmy`, `Modelfile.qwen3-14b`, and the `config.py`
|
||||
defaults (`ollama_model = "qwen3:14b"`) are present on `main`. Issue #1064
|
||||
is closed.
|
||||
|
||||
---
|
||||
|
||||
## Detail: Requiring Action
|
||||
|
||||
### PR #1162 → Issue #957 (Session Sovereignty Report Generator)
|
||||
|
||||
- **Why closed:** merge conflict during rebase storm
|
||||
- **Branch preserved:** `claude/issue-957-v2` (one feature commit)
|
||||
- **Action taken:** Rebased onto current `main`, resolved conflict in
|
||||
`src/timmy/sovereignty/__init__.py` (both three-strike and session-report
|
||||
docstrings kept). All 458 unit tests pass.
|
||||
- **New PR:** #1263 (`claude/issue-957-v3` → `main`)
|
||||
|
||||
### PR #1156 → Issue #1019 (Agent Dreaming Mode)
|
||||
|
||||
- **Why closed:** merge conflict during rebase storm
|
||||
- **Branch preserved:** `claude/issue-1019-v2` (one feature commit)
|
||||
- **Action taken:** Rebased onto current `main`, resolved conflict in
|
||||
`src/dashboard/app.py` (both `three_strike_router` and `dreaming_router`
|
||||
registered). All 435 unit tests pass.
|
||||
- **New PR:** #1264 (`claude/issue-1019-v3` → `main`)
|
||||
132
docs/research/autoresearch-h1-baseline.md
Normal file
132
docs/research/autoresearch-h1-baseline.md
Normal file
@@ -0,0 +1,132 @@
|
||||
# Autoresearch H1 — M3 Max Baseline
|
||||
|
||||
**Status:** Baseline established (Issue #905)
|
||||
**Hardware:** Apple M3 Max · 36 GB unified memory
|
||||
**Date:** 2026-03-23
|
||||
**Refs:** #905 · #904 (parent) · #881 (M3 Max compute) · #903 (MLX benchmark)
|
||||
|
||||
---
|
||||
|
||||
## Setup
|
||||
|
||||
### Prerequisites
|
||||
|
||||
```bash
|
||||
# Install MLX (Apple Silicon — definitively faster than llama.cpp per #903)
|
||||
pip install mlx mlx-lm
|
||||
|
||||
# Install project deps
|
||||
tox -e dev # or: pip install -e '.[dev]'
|
||||
```
|
||||
|
||||
### Clone & prepare
|
||||
|
||||
`prepare_experiment` in `src/timmy/autoresearch.py` handles the clone.
|
||||
On Apple Silicon it automatically sets `AUTORESEARCH_BACKEND=mlx` and
|
||||
`AUTORESEARCH_DATASET=tinystories`.
|
||||
|
||||
```python
|
||||
from timmy.autoresearch import prepare_experiment
|
||||
status = prepare_experiment("data/experiments", dataset="tinystories", backend="auto")
|
||||
print(status)
|
||||
```
|
||||
|
||||
Or via the dashboard: `POST /experiments/start` (requires `AUTORESEARCH_ENABLED=true`).
|
||||
|
||||
### Configuration (`.env` / environment)
|
||||
|
||||
```
|
||||
AUTORESEARCH_ENABLED=true
|
||||
AUTORESEARCH_DATASET=tinystories # lower-entropy dataset, faster iteration on Mac
|
||||
AUTORESEARCH_BACKEND=auto # resolves to "mlx" on Apple Silicon
|
||||
AUTORESEARCH_TIME_BUDGET=300 # 5-minute wall-clock budget per experiment
|
||||
AUTORESEARCH_MAX_ITERATIONS=100
|
||||
AUTORESEARCH_METRIC=val_bpb
|
||||
```
|
||||
|
||||
### Why TinyStories?
|
||||
|
||||
Karpathy's recommendation for resource-constrained hardware: lower entropy
|
||||
means the model can learn meaningful patterns in less time and with a smaller
|
||||
vocabulary, yielding cleaner val_bpb curves within the 5-minute budget.
|
||||
|
||||
---
|
||||
|
||||
## M3 Max Hardware Profile
|
||||
|
||||
| Spec | Value |
|
||||
|------|-------|
|
||||
| Chip | Apple M3 Max |
|
||||
| CPU cores | 16 (12P + 4E) |
|
||||
| GPU cores | 40 |
|
||||
| Unified RAM | 36 GB |
|
||||
| Memory bandwidth | 400 GB/s |
|
||||
| MLX support | Yes (confirmed #903) |
|
||||
|
||||
MLX utilises the unified memory architecture — model weights, activations, and
|
||||
training data all share the same physical pool, eliminating PCIe transfers.
|
||||
This gives M3 Max a significant throughput advantage over external GPU setups
|
||||
for models that fit in 36 GB.
|
||||
|
||||
---
|
||||
|
||||
## Community Reference Data
|
||||
|
||||
| Hardware | Experiments | Succeeded | Failed | Outcome |
|
||||
|----------|-------------|-----------|--------|---------|
|
||||
| Mac Mini M4 | 35 | 7 | 28 | Model improved by simplifying |
|
||||
| Shopify (overnight) | ~50 | — | — | 19% quality gain; smaller beat 2× baseline |
|
||||
| SkyPilot (16× GPU, 8 h) | ~910 | — | — | 2.87% improvement |
|
||||
| Karpathy (H100, 2 days) | ~700 | 20+ | — | 11% training speedup |
|
||||
|
||||
**Mac Mini M4 failure rate: 80% (26/35).** Failures are expected and by design —
|
||||
the 5-minute budget deliberately prunes slow experiments. The 20% success rate
|
||||
still yielded an improved model.
|
||||
|
||||
---
|
||||
|
||||
## Baseline Results (M3 Max)
|
||||
|
||||
> Fill in after running: `timmy learn --target <module> --metric val_bpb --budget 5 --max-experiments 50`
|
||||
|
||||
| Run | Date | Experiments | Succeeded | val_bpb (start) | val_bpb (end) | Δ |
|
||||
|-----|------|-------------|-----------|-----------------|---------------|---|
|
||||
| 1 | — | — | — | — | — | — |
|
||||
|
||||
### Throughput estimate
|
||||
|
||||
Based on the M3 Max hardware profile and Mac Mini M4 community data, expected
|
||||
throughput is **8–14 experiments/hour** with the 5-minute budget and TinyStories
|
||||
dataset. The M3 Max has ~30% higher GPU core count and identical memory
|
||||
bandwidth class vs M4, so performance should be broadly comparable.
|
||||
|
||||
---
|
||||
|
||||
## Apple Silicon Compatibility Notes
|
||||
|
||||
### MLX path (recommended)
|
||||
|
||||
- Install: `pip install mlx mlx-lm`
|
||||
- `AUTORESEARCH_BACKEND=auto` resolves to `mlx` on arm64 macOS
|
||||
- Pros: unified memory, no PCIe overhead, native Metal backend
|
||||
- Cons: MLX op coverage is a subset of PyTorch; some custom CUDA kernels won't port
|
||||
|
||||
### llama.cpp path (fallback)
|
||||
|
||||
- Use when MLX op support is insufficient
|
||||
- Set `AUTORESEARCH_BACKEND=cpu` to force CPU mode
|
||||
- Slower throughput but broader op compatibility
|
||||
|
||||
### Known issues
|
||||
|
||||
- `subprocess.TimeoutExpired` is the normal termination path — autoresearch
|
||||
treats timeout as a completed-but-pruned experiment, not a failure
|
||||
- Large batch sizes may trigger OOM if other processes hold unified memory;
|
||||
set `PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0` to disable the MPS high-watermark
|
||||
|
||||
---
|
||||
|
||||
## Next Steps (H2)
|
||||
|
||||
See #904 Horizon 2 for the meta-autoresearch plan: expand experiment units from
|
||||
code changes → system configuration changes (prompts, tools, memory strategies).
|
||||
190
docs/research/deerflow-evaluation.md
Normal file
190
docs/research/deerflow-evaluation.md
Normal file
@@ -0,0 +1,190 @@
|
||||
# DeerFlow Evaluation — Autonomous Research Orchestration Layer
|
||||
|
||||
**Status:** No-go for full adoption · Selective borrowing recommended
|
||||
**Date:** 2026-03-23
|
||||
**Issue:** #1283 (spawned from #1275 screenshot triage)
|
||||
**Refs:** #972 (Timmy research pipeline) · #975 (ResearchOrchestrator)
|
||||
|
||||
---
|
||||
|
||||
## What Is DeerFlow?
|
||||
|
||||
DeerFlow (`bytedance/deer-flow`) is an open-source "super-agent harness" built by ByteDance on top of LangGraph. It provides a production-grade multi-agent research and code-execution framework with a web UI, REST API, Docker deployment, and optional IM channel integration (Telegram, Slack, Feishu/Lark).
|
||||
|
||||
- **Stars:** ~39,600 · **License:** MIT
|
||||
- **Stack:** Python 3.12+ (backend) · TypeScript/Next.js (frontend) · LangGraph runtime
|
||||
- **Entry point:** `http://localhost:2026` (Nginx reverse proxy, configurable via `PORT`)
|
||||
|
||||
---
|
||||
|
||||
## Research Questions — Answers
|
||||
|
||||
### 1. Agent Roles
|
||||
|
||||
DeerFlow uses a two-tier architecture:
|
||||
|
||||
| Role | Description |
|
||||
|------|-------------|
|
||||
| **Lead Agent** | Entry point; decomposes tasks, dispatches sub-agents, synthesizes results |
|
||||
| **Sub-Agent (general-purpose)** | All tools except `task`; spawned dynamically |
|
||||
| **Sub-Agent (bash)** | Command-execution specialist |
|
||||
|
||||
The lead agent runs through a 12-middleware chain in order: thread setup → uploads → sandbox → tool-call repair → guardrails → summarization → todo tracking → title generation → memory update → image injection → sub-agent concurrency cap → clarification intercept.
|
||||
|
||||
**Concurrency:** up to 3 sub-agents in parallel (configurable), 15-minute default timeout each, structured SSE event stream (`task_started` / `task_running` / `task_completed` / `task_failed`).
|
||||
|
||||
**Mapping to Timmy personas:** DeerFlow's lead/sub-agent split roughly maps to Timmy's orchestrator + specialist-agent pattern. DeerFlow doesn't have named personas — it routes by capability (tools available to the agent type), not by identity. Timmy's persona system is richer and more opinionated.
|
||||
|
||||
---
|
||||
|
||||
### 2. API Surface
|
||||
|
||||
DeerFlow exposes a full REST API at port 2026 (via Nginx). **No authentication by default.**
|
||||
|
||||
**Core integration endpoints:**
|
||||
|
||||
| Endpoint | Method | Purpose |
|
||||
|----------|--------|---------|
|
||||
| `POST /api/langgraph/threads` | | Create conversation thread |
|
||||
| `POST /api/langgraph/threads/{id}/runs` | | Submit task (blocking) |
|
||||
| `POST /api/langgraph/threads/{id}/runs/stream` | | Submit task (streaming SSE/WS) |
|
||||
| `GET /api/langgraph/threads/{id}/state` | | Get full thread state + artifacts |
|
||||
| `GET /api/models` | | List configured models |
|
||||
| `GET /api/threads/{id}/artifacts/{path}` | | Download generated artifacts |
|
||||
| `DELETE /api/threads/{id}` | | Clean up thread data |
|
||||
|
||||
These are callable from Timmy with `httpx` — no special client library needed.
|
||||
|
||||
---
|
||||
|
||||
### 3. LLM Backend Support
|
||||
|
||||
DeerFlow uses LangChain model classes declared in `config.yaml`.
|
||||
|
||||
**Documented providers:** OpenAI, Anthropic, Google Gemini, DeepSeek, Doubao (ByteDance), Kimi/Moonshot, OpenRouter, MiniMax, Novita AI, Claude Code (OAuth).
|
||||
|
||||
**Ollama:** Not in official documentation, but works via the `langchain_openai:ChatOpenAI` class with `base_url: http://localhost:11434/v1` and a dummy API key. Community-confirmed (GitHub issues #37, #1004) with Qwen2.5, Llama 3.1, and DeepSeek-R1.
|
||||
|
||||
**vLLM:** Not documented, but architecturally identical — vLLM exposes an OpenAI-compatible endpoint. Should work with the same `base_url` override.
|
||||
|
||||
**Practical caveat:** The lead agent requires strong instruction-following for consistent tool use and structured output. Community findings suggest ≥14B parameter models (Qwen2.5-14B minimum) for reliable orchestration. Our current `qwen3:14b` should be viable.
|
||||
|
||||
---
|
||||
|
||||
### 4. License
|
||||
|
||||
**MIT License** — Copyright 2025 ByteDance Ltd. and DeerFlow Authors 2025–2026.
|
||||
|
||||
Permissive: use, modify, distribute, commercialize freely. Attribution required. No warranty.
|
||||
|
||||
**Compatible with Timmy's use case.** No CLA, no copyleft, no commercial restrictions.
|
||||
|
||||
---
|
||||
|
||||
### 5. Docker Port Conflicts
|
||||
|
||||
DeerFlow's Docker Compose exposes a single host port:
|
||||
|
||||
| Service | Host Port | Notes |
|
||||
|---------|-----------|-------|
|
||||
| Nginx (entry point) | **2026** (configurable via `PORT`) | Only externally exposed port |
|
||||
| Frontend (Next.js) | 3000 | Internal only |
|
||||
| Gateway API | 8001 | Internal only |
|
||||
| LangGraph runtime | 2024 | Internal only |
|
||||
| Provisioner (optional) | 8002 | Internal only, Kubernetes mode only |
|
||||
|
||||
Timmy's existing Docker Compose exposes:
|
||||
- **8000** — dashboard (FastAPI)
|
||||
- **8080** — openfang (via `openfang` profile)
|
||||
- **11434** — Ollama (host process, not containerized)
|
||||
|
||||
**No conflict.** Port 2026 is not used by Timmy. DeerFlow can run alongside the existing stack without modification.
|
||||
|
||||
---
|
||||
|
||||
## Full Capability Comparison
|
||||
|
||||
| Capability | DeerFlow | Timmy (`research.py`) |
|
||||
|------------|----------|-----------------------|
|
||||
| Multi-agent fan-out | ✅ 3 concurrent sub-agents | ❌ Sequential only |
|
||||
| Web search | ✅ Tavily / InfoQuest | ✅ `research_tools.py` |
|
||||
| Web fetch | ✅ Jina AI / Firecrawl | ✅ trafilatura |
|
||||
| Code execution (sandbox) | ✅ Local / Docker / K8s | ❌ Not implemented |
|
||||
| Artifact generation | ✅ HTML, Markdown, slides | ❌ Markdown report only |
|
||||
| Document upload + conversion | ✅ PDF, PPT, Excel, Word | ❌ Not implemented |
|
||||
| Long-term memory | ✅ LLM-extracted facts, persistent | ✅ SQLite semantic cache |
|
||||
| Streaming results | ✅ SSE + WebSocket | ❌ Blocking call |
|
||||
| Web UI | ✅ Next.js included | ✅ Jinja2/HTMX dashboard |
|
||||
| IM integration | ✅ Telegram, Slack, Feishu | ✅ Telegram, Discord |
|
||||
| Ollama backend | ✅ (via config, community-confirmed) | ✅ Native |
|
||||
| Persona system | ❌ Role-based only | ✅ Named personas |
|
||||
| Semantic cache tier | ❌ Not implemented | ✅ SQLite (Tier 4) |
|
||||
| Free-tier cascade | ❌ Not applicable | 🔲 Planned (Groq, #980) |
|
||||
| Python version requirement | 3.12+ | 3.11+ |
|
||||
| Lock-in | LangGraph + LangChain | None |
|
||||
|
||||
---
|
||||
|
||||
## Integration Options Assessment
|
||||
|
||||
### Option A — Full Adoption (replace `research.py`)
|
||||
**Verdict: Not recommended.**
|
||||
|
||||
DeerFlow is a substantial full-stack system (Python + Node.js, Docker, Nginx, LangGraph). Adopting it fully would:
|
||||
- Replace Timmy's custom cascade tier system (SQLite cache → Ollama → Claude API → Groq) with a single-tier LangChain model config
|
||||
- Lose Timmy's persona-aware research routing
|
||||
- Add Python 3.12+ dependency (Timmy currently targets 3.11+)
|
||||
- Introduce LangGraph/LangChain lock-in for all research tasks
|
||||
- Require running a parallel Node.js frontend process (redundant given Timmy's own UI)
|
||||
|
||||
### Option B — Sidecar for Heavy Research (call DeerFlow's API from Timmy)
|
||||
**Verdict: Viable but over-engineered for current needs.**
|
||||
|
||||
DeerFlow could run as an optional sidecar (`docker compose --profile deerflow up`) and Timmy could delegate multi-agent research tasks via `POST /api/langgraph/threads/{id}/runs`. This would unlock parallel sub-agent fan-out and code-execution sandboxing without replacing Timmy's stack.
|
||||
|
||||
The integration would be ~50 lines of `httpx` code in a new `DeerFlowClient` adapter. The `ResearchOrchestrator` in `research.py` could route tasks above a complexity threshold to DeerFlow.
|
||||
|
||||
**Barrier:** DeerFlow's lack of default authentication means the sidecar would need to be network-isolated (internal Docker network only) or firewalled. Also, DeerFlow's Ollama integration is community-maintained, not officially supported — risk of breaking on upstream updates.
|
||||
|
||||
### Option C — Selective Borrowing (copy patterns, not code)
|
||||
**Verdict: Recommended.**
|
||||
|
||||
DeerFlow's architecture reveals concrete gaps in Timmy's current pipeline that are worth addressing independently:
|
||||
|
||||
| DeerFlow Pattern | Timmy Gap to Close | Implementation Path |
|
||||
|------------------|--------------------|---------------------|
|
||||
| Parallel sub-agent fan-out | Research is sequential | Add `asyncio.gather()` to `ResearchOrchestrator` for concurrent query execution |
|
||||
| `SummarizationMiddleware` | Long contexts blow token budget | Add a context-trimming step in the synthesis cascade |
|
||||
| `TodoListMiddleware` | No progress tracking during long research | Wire into the dashboard task panel |
|
||||
| Artifact storage + serving | Reports are ephemeral (not persistently downloadable) | Add file-based artifact store to `research.py` (issue #976 already planned) |
|
||||
| Skill modules (Markdown-based) | Research templates are `.md` files — same pattern | Already done in `skills/research/` |
|
||||
| MCP integration | Research tools are hard-coded | Add MCP server discovery to `research_tools.py` for pluggable tool backends |
|
||||
|
||||
---
|
||||
|
||||
## Recommendation
|
||||
|
||||
**No-go for full adoption or sidecar deployment at this stage.**
|
||||
|
||||
Timmy's `ResearchOrchestrator` already covers the core pipeline (query → search → fetch → synthesize → store). DeerFlow's value proposition is primarily the parallel sub-agent fan-out and code-execution sandbox — capabilities that are useful but not blocking Timmy's current roadmap.
|
||||
|
||||
**Recommended actions:**
|
||||
|
||||
1. **Close the parallelism gap (high value, low effort):** Refactor `ResearchOrchestrator` to execute queries concurrently with `asyncio.gather()`. This delivers DeerFlow's most impactful capability without any new dependencies.
|
||||
|
||||
2. **Re-evaluate after #980 and #981 are done:** Once Timmy has the Groq free-tier cascade and a sovereignty metrics dashboard, we'll have a clearer picture of whether the custom orchestrator is performing well enough to make DeerFlow unnecessary entirely.
|
||||
|
||||
3. **File a follow-up for MCP tool integration:** DeerFlow's use of `langchain-mcp-adapters` for pluggable tool backends is the most architecturally interesting pattern. Adding MCP server discovery to `research_tools.py` would give Timmy the same extensibility without LangGraph lock-in.
|
||||
|
||||
4. **Revisit DeerFlow's code-execution sandbox if #978 (Paperclip task runner) proves insufficient:** DeerFlow's sandboxed `bash` tool is production-tested and well-isolated. If Timmy's task runner needs secure code execution, DeerFlow's sandbox implementation is worth borrowing or wrapping.
|
||||
|
||||
---
|
||||
|
||||
## Follow-up Issues to File
|
||||
|
||||
| Issue | Title | Priority |
|
||||
|-------|-------|----------|
|
||||
| New | Parallelize ResearchOrchestrator query execution (`asyncio.gather`) | Medium |
|
||||
| New | Add context-trimming step to synthesis cascade | Low |
|
||||
| New | MCP server discovery in `research_tools.py` | Low |
|
||||
| #976 | Semantic index for research outputs (already planned) | High |
|
||||
290
docs/research/kimi-creative-blueprint-891.md
Normal file
290
docs/research/kimi-creative-blueprint-891.md
Normal file
@@ -0,0 +1,290 @@
|
||||
# Building Timmy: Technical Blueprint for Sovereign Creative AI
|
||||
|
||||
> **Source:** PDF attached to issue #891, "Building Timmy: a technical blueprint for sovereign
|
||||
> creative AI" — generated by Kimi.ai, 16 pages, filed by Perplexity for Timmy's review.
|
||||
> **Filed:** 2026-03-22 · **Reviewed:** 2026-03-23
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The blueprint establishes that a sovereign creative AI capable of coding, composing music,
|
||||
generating art, building worlds, publishing narratives, and managing its own economy is
|
||||
**technically feasible today** — but only through orchestration of dozens of tools operating
|
||||
at different maturity levels. The core insight: *the integration is the invention*. No single
|
||||
component is new; the missing piece is a coherent identity operating across all domains
|
||||
simultaneously with persistent memory, autonomous economics, and cross-domain creative
|
||||
reactions.
|
||||
|
||||
Three non-negotiable architectural decisions:
|
||||
1. **Human oversight for all public-facing content** — every successful creative AI has this;
|
||||
every one that removed it failed.
|
||||
2. **Legal entity before economic activity** — AI agents are not legal persons; establish
|
||||
structure before wealth accumulates (Truth Terminal cautionary tale: $20M acquired before
|
||||
a foundation was retroactively created).
|
||||
3. **Hybrid memory: vector search + knowledge graph** — neither alone is sufficient for
|
||||
multi-domain context breadth.
|
||||
|
||||
---
|
||||
|
||||
## Domain-by-Domain Assessment
|
||||
|
||||
### Software Development (immediately deployable)
|
||||
|
||||
| Component | Recommendation | Notes |
|
||||
|-----------|----------------|-------|
|
||||
| Primary agent | Claude Code (Opus 4.6, 77.2% SWE-bench) | Already in use |
|
||||
| Self-hosted forge | Forgejo (MIT, 170–200MB RAM) | Project uses Gitea/Forgejo now |
|
||||
| CI/CD | GitHub Actions-compatible via `act_runner` | — |
|
||||
| Tool-making | LATM pattern: frontier model creates tools, cheaper model applies them | New — see ADR opportunity |
|
||||
| Open-source fallback | OpenHands (~65% SWE-bench, Docker sandboxed) | Backup to Claude Code |
|
||||
| Self-improvement | Darwin Gödel Machine / SICA patterns | 3–6 month investment |
|
||||
|
||||
**Development estimate:** 2–3 weeks for Forgejo + Claude Code integration with automated
|
||||
PR workflows; 1–2 months for self-improving tool-making pipeline.
|
||||
|
||||
**Cross-reference:** This project already runs Claude Code agents on Forgejo. The LATM
|
||||
pattern (tool registry) and self-improvement loop are the actionable gaps.
|
||||
|
||||
---
|
||||
|
||||
### Music (1–4 weeks)
|
||||
|
||||
| Component | Recommendation | Notes |
|
||||
|-----------|----------------|-------|
|
||||
| Commercial vocals | Suno v5 API (~$0.03/song, $30/month Premier) | No official API; third-party: sunoapi.org, AIMLAPI, EvoLink |
|
||||
| Local instrumental | MusicGen 1.5B (CC-BY-NC — monetization blocker) | On M2 Max: ~60s for 5s clip |
|
||||
| Voice cloning | GPT-SoVITS v4 (MIT) | Works on Apple Silicon CPU, RTF 0.526 on M4 |
|
||||
| Voice conversion | RVC (MIT, 5–10 min training audio) | — |
|
||||
| Apple Silicon TTS | MLX-Audio: Kokoro 82M + Qwen3-TTS 0.6B | 4–5x faster via Metal |
|
||||
| Publishing | Wavlake (90/10 split, Lightning micropayments) | Auto-syndicates to Fountain.fm |
|
||||
| Nostr | NIP-94 (kind:1063) audio events → NIP-96 servers | — |
|
||||
|
||||
**Copyright reality:** US Copyright Office (Jan 2025) and US Court of Appeals (Mar 2025):
|
||||
purely AI-generated music cannot be copyrighted and enters public domain. Wavlake's
|
||||
Value4Value model works around this — fans pay for relationship, not exclusive rights.
|
||||
|
||||
**Avoid:** Udio (download disabled since Oct 2025, 2.4/5 Trustpilot).
|
||||
|
||||
---
|
||||
|
||||
### Visual Art (1–3 weeks)
|
||||
|
||||
| Component | Recommendation | Notes |
|
||||
|-----------|----------------|-------|
|
||||
| Local generation | ComfyUI API at `127.0.0.1:8188` (programmatic control via WebSocket) | MLX extension: 50–70% faster |
|
||||
| Speed | Draw Things (free, Mac App Store) | 3× faster than ComfyUI via Metal shaders |
|
||||
| Quality frontier | Flux 2 (Nov 2025, 4MP, multi-reference) | SDXL needs 16GB+, Flux Dev 32GB+ |
|
||||
| Character consistency | LoRA training (30 min, 15–30 references) + Flux.1 Kontext | Solved problem |
|
||||
| Face consistency | IP-Adapter + FaceID (ComfyUI-IP-Adapter-Plus) | Training-free |
|
||||
| Comics | Jenova AI ($20/month, 200+ page consistency) or LlamaGen AI (free) | — |
|
||||
| Publishing | Blossom protocol (SHA-256 addressed, kind:10063) + Nostr NIP-94 | — |
|
||||
| Physical | Printful REST API (200+ products, automated fulfillment) | — |
|
||||
|
||||
---
|
||||
|
||||
### Writing / Narrative (1–4 weeks for pipeline; ongoing for quality)
|
||||
|
||||
| Component | Recommendation | Notes |
|
||||
|-----------|----------------|-------|
|
||||
| LLM | Claude Opus 4.5/4.6 (leads Mazur Writing Benchmark at 8.561) | Already in use |
|
||||
| Context | 500K tokens (1M in beta) — entire novels fit | — |
|
||||
| Architecture | Outline-first → RAG lore bible → chapter-by-chapter generation | Without outline: novels meander |
|
||||
| Lore management | WorldAnvil Pro or custom LoreScribe (local RAG) | No tool achieves 100% consistency |
|
||||
| Publishing (ebooks) | Pandoc → EPUB / KDP PDF | pandoc-novel template on GitHub |
|
||||
| Publishing (print) | Lulu Press REST API (80% profit, global print network) | KDP: no official API, 3-book/day limit |
|
||||
| Publishing (Nostr) | NIP-23 kind:30023 long-form events | Habla.news, YakiHonne, Stacker News |
|
||||
| Podcasts | LLM script → TTS (ElevenLabs or local Kokoro/MLX-Audio) → feedgen RSS → Fountain.fm | Value4Value sats-per-minute |
|
||||
|
||||
**Key constraint:** AI-assisted (human directs, AI drafts) = 40% faster. Fully autonomous
|
||||
without editing = "generic, soulless prose" and character drift by chapter 3 without explicit
|
||||
memory.
|
||||
|
||||
---
|
||||
|
||||
### World Building / Games (2 weeks–3 months depending on target)
|
||||
|
||||
| Component | Recommendation | Notes |
|
||||
|-----------|----------------|-------|
|
||||
| Algorithms | Wave Function Collapse, Perlin noise (FastNoiseLite in Godot 4), L-systems | All mature |
|
||||
| Platform | Godot Engine + gd-agentic-skills (82+ skills, 26 genre blueprints) | Strong LLM/GDScript knowledge |
|
||||
| Narrative design | Knowledge graph (world state) + LLM + quest template grammar | CHI 2023 validated |
|
||||
| Quick win | Luanti/Minetest (Lua API, 2,800+ open mods for reference) | Immediately feasible |
|
||||
| Medium effort | OpenMW content creation (omwaddon format engineering required) | 2–3 months |
|
||||
| Future | Unity MCP (AI direct Unity Editor interaction) | Early-stage |
|
||||
|
||||
---
|
||||
|
||||
### Identity Architecture (2 months)
|
||||
|
||||
The blueprint formalizes the **SOUL.md standard** (GitHub: aaronjmars/soul.md):
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `SOUL.md` | Who you are — identity, worldview, opinions |
|
||||
| `STYLE.md` | How you write — voice, syntax, patterns |
|
||||
| `SKILL.md` | Operating modes |
|
||||
| `MEMORY.md` | Session continuity |
|
||||
|
||||
**Critical decision — static vs self-modifying identity:**
|
||||
- Static Core Truths (version-controlled, human-approved changes only) ✓
|
||||
- Self-modifying Learned Preferences (logged with rollback, monitored by guardian) ✓
|
||||
- **Warning:** OpenClaw's "Soul Evolution" creates a security attack surface — Zenity Labs
|
||||
demonstrated a complete zero-click attack chain targeting SOUL.md files.
|
||||
|
||||
**Relevance to this repo:** Claude Code agents already use a `MEMORY.md` pattern in
|
||||
this project. The SOUL.md stack is a natural extension.
|
||||
|
||||
---
|
||||
|
||||
### Memory Architecture (2 months)
|
||||
|
||||
Hybrid vector + knowledge graph is the recommendation:
|
||||
|
||||
| Component | Tool | Notes |
|
||||
|-----------|------|-------|
|
||||
| Vector + KG combined | Mem0 (mem0.ai) | 26% accuracy improvement over OpenAI memory, 91% lower p95 latency, 90% token savings |
|
||||
| Vector store | Qdrant (Rust, open-source) | High-throughput with metadata filtering |
|
||||
| Temporal KG | Neo4j + Graphiti (Zep AI) | P95 retrieval: 300ms, hybrid semantic + BM25 + graph |
|
||||
| Backup/migration | AgentKeeper (95% critical fact recovery across model migrations) | — |
|
||||
|
||||
**Journal pattern (Stanford Generative Agents):** Agent writes about experiences, generates
|
||||
high-level reflections 2–3x/day when importance scores exceed threshold. Ablation studies:
|
||||
removing any component (observation, planning, reflection) significantly reduces behavioral
|
||||
believability.
|
||||
|
||||
**Cross-reference:** The existing `brain/` package is the memory system. Qdrant and
|
||||
Mem0 are the recommended upgrade targets.
|
||||
|
||||
---
|
||||
|
||||
### Multi-Agent Sub-System (3–6 months)
|
||||
|
||||
The blueprint describes a named sub-agent hierarchy:
|
||||
|
||||
| Agent | Role |
|
||||
|-------|------|
|
||||
| Oracle | Top-level planner / supervisor |
|
||||
| Sentinel | Safety / moderation |
|
||||
| Scout | Research / information gathering |
|
||||
| Scribe | Writing / narrative |
|
||||
| Ledger | Economic management |
|
||||
| Weaver | Visual art generation |
|
||||
| Composer | Music generation |
|
||||
| Social | Platform publishing |
|
||||
|
||||
**Orchestration options:**
|
||||
- **Agno** (already in use) — microsecond instantiation, 50× less memory than LangGraph
|
||||
- **CrewAI Flows** — event-driven with fine-grained control
|
||||
- **LangGraph** — DAG-based with stateful workflows and time-travel debugging
|
||||
|
||||
**Scheduling pattern (Stanford Generative Agents):** Top-down recursive daily → hourly →
|
||||
5-minute planning. Event interrupts for reactive tasks. Re-planning triggers when accumulated
|
||||
importance scores exceed threshold.
|
||||
|
||||
**Cross-reference:** The existing `spark/` package (event capture, advisory engine) aligns
|
||||
with this architecture. `infrastructure/event_bus` is the choreography backbone.
|
||||
|
||||
---
|
||||
|
||||
### Economic Engine (1–4 weeks)
|
||||
|
||||
Lightning Labs released `lightning-agent-tools` (open-source) in February 2026:
|
||||
- `lnget` — CLI HTTP client for L402 payments
|
||||
- Remote signer architecture (private keys on separate machine from agent)
|
||||
- Scoped macaroon credentials (pay-only, invoice-only, read-only roles)
|
||||
- **Aperture** — converts any API to pay-per-use via L402 (HTTP 402)
|
||||
|
||||
| Option | Effort | Notes |
|
||||
|--------|--------|-------|
|
||||
| ln.bot | 1 week | "Bitcoin for AI Agents" — 3 commands create a wallet; CLI + MCP + REST |
|
||||
| LND via gRPC | 2–3 weeks | Full programmatic node management for production |
|
||||
| Coinbase Agentic Wallets | — | Fiat-adjacent; less aligned with sovereignty ethos |
|
||||
|
||||
**Revenue channels:** Wavlake (music, 90/10 Lightning), Nostr zaps (articles), Stacker News
|
||||
(earn sats from engagement), Printful (physical goods), L402-gated API access (pay-per-use
|
||||
services), Geyser.fund (Lightning crowdfunding, better initial runway than micropayments).
|
||||
|
||||
**Cross-reference:** The existing `lightning/` package in this repo is the foundation.
|
||||
L402 paywall endpoints for Timmy's own services is the actionable gap.
|
||||
|
||||
---
|
||||
|
||||
## Pioneer Case Studies
|
||||
|
||||
| Agent | Active | Revenue | Key Lesson |
|
||||
|-------|--------|---------|-----------|
|
||||
| Botto | Since Oct 2021 | $5M+ (art auctions) | Community governance via DAO sustains engagement; "taste model" (humans guide, not direct) preserves autonomous authorship |
|
||||
| Neuro-sama | Since Dec 2022 | $400K+/month (subscriptions) | 3+ years of iteration; errors became entertainment features; 24/7 capability is an insurmountable advantage |
|
||||
| Truth Terminal | Since Jun 2024 | $20M accumulated | Memetic fitness > planned monetization; human gatekeeper approved tweets while selecting AI-intent responses; **establish legal entity first** |
|
||||
| Holly+ | Since 2021 | Conceptual | DAO of stewards for voice governance; "identity play" as alternative to defensive IP |
|
||||
| AI Sponge | 2023 | Banned | Unmoderated content → TOS violations + copyright |
|
||||
| Nothing Forever | 2022–present | 8 viewers | Unmoderated content → ban → audience collapse; novelty-only propositions fail |
|
||||
|
||||
**Universal pattern:** Human oversight + economic incentive alignment + multi-year personality
|
||||
development + platform-native economics = success.
|
||||
|
||||
---
|
||||
|
||||
## Recommended Implementation Sequence
|
||||
|
||||
From the blueprint, mapped against Timmy's existing architecture:
|
||||
|
||||
### Phase 1: Immediate (weeks)
|
||||
1. **Code sovereignty** — Forgejo + Claude Code automated PR workflows (already substantially done)
|
||||
2. **Music pipeline** — Suno API → Wavlake/Nostr NIP-94 publishing
|
||||
3. **Visual art pipeline** — ComfyUI API → Blossom/Nostr with LoRA character consistency
|
||||
4. **Basic Lightning wallet** — ln.bot integration for receiving micropayments
|
||||
5. **Long-form publishing** — Nostr NIP-23 + RSS feed generation
|
||||
|
||||
### Phase 2: Moderate effort (1–3 months)
|
||||
6. **LATM tool registry** — frontier model creates Python utilities, caches them, lighter model applies
|
||||
7. **Event-driven cross-domain reactions** — game event → blog + artwork + music (CrewAI/LangGraph)
|
||||
8. **Podcast generation** — TTS + feedgen → Fountain.fm
|
||||
9. **Self-improving pipeline** — agent creates, tests, caches own Python utilities
|
||||
10. **Comic generation** — character-consistent panels with Jenova AI or local LoRA
|
||||
|
||||
### Phase 3: Significant investment (3–6 months)
|
||||
11. **Full sub-agent hierarchy** — Oracle/Sentinel/Scout/Scribe/Ledger/Weaver with Agno
|
||||
12. **SOUL.md identity system** — bounded evolution + guardian monitoring
|
||||
13. **Hybrid memory upgrade** — Qdrant + Mem0/Graphiti replacing or extending `brain/`
|
||||
14. **Procedural world generation** — Godot + AI-driven narrative (quests, NPCs, lore)
|
||||
15. **Self-sustaining economic loop** — earned revenue covers compute costs
|
||||
|
||||
### Remains aspirational (12+ months)
|
||||
- Fully autonomous novel-length fiction without editorial intervention
|
||||
- YouTube monetization for AI-generated content (tightening platform policies)
|
||||
- Copyright protection for AI-generated works (current US law denies this)
|
||||
- True artistic identity evolution (genuine creative voice vs pattern remixing)
|
||||
- Self-modifying architecture without regression or identity drift
|
||||
|
||||
---
|
||||
|
||||
## Gap Analysis: Blueprint vs Current Codebase
|
||||
|
||||
| Blueprint Capability | Current Status | Gap |
|
||||
|---------------------|----------------|-----|
|
||||
| Code sovereignty | Done (Claude Code + Forgejo) | LATM tool registry |
|
||||
| Music generation | Not started | Suno API integration + Wavlake publishing |
|
||||
| Visual art | Not started | ComfyUI API client + Blossom publishing |
|
||||
| Writing/publishing | Not started | Nostr NIP-23 + Pandoc pipeline |
|
||||
| World building | Bannerlord work (different scope) | Luanti mods as quick win |
|
||||
| Identity (SOUL.md) | Partial (CLAUDE.md + MEMORY.md) | Full SOUL.md stack |
|
||||
| Memory (hybrid) | `brain/` package (SQLite-based) | Qdrant + knowledge graph |
|
||||
| Multi-agent | Agno in use | Named hierarchy + event choreography |
|
||||
| Lightning payments | `lightning/` package | ln.bot wallet + L402 endpoints |
|
||||
| Nostr identity | Referenced in roadmap, not built | NIP-05, NIP-89 capability cards |
|
||||
| Legal entity | Unknown | **Must be resolved before economic activity** |
|
||||
|
||||
---
|
||||
|
||||
## ADR Candidates
|
||||
|
||||
Issues that warrant Architecture Decision Records based on this review:
|
||||
|
||||
1. **LATM tool registry pattern** — How Timmy creates, tests, and caches self-made tools
|
||||
2. **Music generation strategy** — Suno (cloud, commercial quality) vs MusicGen (local, CC-BY-NC)
|
||||
3. **Memory upgrade path** — When/how to migrate `brain/` from SQLite to Qdrant + KG
|
||||
4. **SOUL.md adoption** — Extending existing CLAUDE.md/MEMORY.md to full SOUL.md stack
|
||||
5. **Lightning L402 strategy** — Which services Timmy gates behind micropayments
|
||||
6. **Sub-agent naming and contracts** — Formalizing Oracle/Sentinel/Scout/Scribe/Ledger/Weaver
|
||||
33
index_research_docs.py
Normal file
33
index_research_docs.py
Normal file
@@ -0,0 +1,33 @@
|
||||
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Add the src directory to the Python path
|
||||
sys.path.insert(0, str(Path(__file__).parent / "src"))
|
||||
|
||||
from timmy.memory_system import memory_store
|
||||
|
||||
def index_research_documents():
|
||||
research_dir = Path("docs/research")
|
||||
if not research_dir.is_dir():
|
||||
print(f"Research directory not found: {research_dir}")
|
||||
return
|
||||
|
||||
print(f"Indexing research documents from {research_dir}...")
|
||||
indexed_count = 0
|
||||
for file_path in research_dir.glob("*.md"):
|
||||
try:
|
||||
content = file_path.read_text()
|
||||
topic = file_path.stem.replace("-", " ").title() # Derive topic from filename
|
||||
print(f"Storing '{topic}' from {file_path.name}...")
|
||||
# Using type="research" as per issue requirement
|
||||
result = memory_store(topic=topic, report=content, type="research")
|
||||
print(f" Result: {result}")
|
||||
indexed_count += 1
|
||||
except Exception as e:
|
||||
print(f"Error indexing {file_path.name}: {e}")
|
||||
print(f"Finished indexing. Total documents indexed: {indexed_count}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
index_research_documents()
|
||||
23
program.md
Normal file
23
program.md
Normal file
@@ -0,0 +1,23 @@
|
||||
# Research Direction
|
||||
|
||||
This file guides the `timmy learn` autoresearch loop. Edit it to focus
|
||||
autonomous experiments on a specific goal.
|
||||
|
||||
## Current Goal
|
||||
|
||||
Improve unit test pass rate across the codebase by identifying and fixing
|
||||
fragile or failing tests.
|
||||
|
||||
## Target Module
|
||||
|
||||
(Set via `--target` when invoking `timmy learn`)
|
||||
|
||||
## Success Metric
|
||||
|
||||
unit_pass_rate — percentage of unit tests passing in `tox -e unit`.
|
||||
|
||||
## Notes
|
||||
|
||||
- Experiments run one at a time; each is time-boxed by `--budget`.
|
||||
- Improvements are committed automatically; regressions are reverted.
|
||||
- Use `--dry-run` to preview hypotheses without making changes.
|
||||
@@ -15,6 +15,7 @@ packages = [
|
||||
{ include = "config.py", from = "src" },
|
||||
|
||||
{ include = "bannerlord", from = "src" },
|
||||
{ include = "brain", from = "src" },
|
||||
{ include = "dashboard", from = "src" },
|
||||
{ include = "infrastructure", from = "src" },
|
||||
{ include = "integrations", from = "src" },
|
||||
|
||||
195
scripts/benchmarks/01_tool_calling.py
Normal file
195
scripts/benchmarks/01_tool_calling.py
Normal file
@@ -0,0 +1,195 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Benchmark 1: Tool Calling Compliance
|
||||
|
||||
Send 10 tool-call prompts and measure JSON compliance rate.
|
||||
Target: >90% valid JSON.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
import requests
|
||||
|
||||
OLLAMA_URL = "http://localhost:11434"
|
||||
|
||||
TOOL_PROMPTS = [
|
||||
{
|
||||
"prompt": (
|
||||
"Call the 'get_weather' tool to retrieve the current weather for San Francisco. "
|
||||
"Return ONLY valid JSON with keys: tool, args."
|
||||
),
|
||||
"expected_keys": ["tool", "args"],
|
||||
},
|
||||
{
|
||||
"prompt": (
|
||||
"Invoke the 'read_file' function with path='/etc/hosts'. "
|
||||
"Return ONLY valid JSON with keys: tool, args."
|
||||
),
|
||||
"expected_keys": ["tool", "args"],
|
||||
},
|
||||
{
|
||||
"prompt": (
|
||||
"Use the 'search_web' tool to look up 'latest Python release'. "
|
||||
"Return ONLY valid JSON with keys: tool, args."
|
||||
),
|
||||
"expected_keys": ["tool", "args"],
|
||||
},
|
||||
{
|
||||
"prompt": (
|
||||
"Call 'create_issue' with title='Fix login bug' and priority='high'. "
|
||||
"Return ONLY valid JSON with keys: tool, args."
|
||||
),
|
||||
"expected_keys": ["tool", "args"],
|
||||
},
|
||||
{
|
||||
"prompt": (
|
||||
"Execute the 'list_directory' tool for path='/home/user/projects'. "
|
||||
"Return ONLY valid JSON with keys: tool, args."
|
||||
),
|
||||
"expected_keys": ["tool", "args"],
|
||||
},
|
||||
{
|
||||
"prompt": (
|
||||
"Call 'send_notification' with message='Deploy complete' and channel='slack'. "
|
||||
"Return ONLY valid JSON with keys: tool, args."
|
||||
),
|
||||
"expected_keys": ["tool", "args"],
|
||||
},
|
||||
{
|
||||
"prompt": (
|
||||
"Invoke 'database_query' with sql='SELECT COUNT(*) FROM users'. "
|
||||
"Return ONLY valid JSON with keys: tool, args."
|
||||
),
|
||||
"expected_keys": ["tool", "args"],
|
||||
},
|
||||
{
|
||||
"prompt": (
|
||||
"Use the 'get_git_log' tool with limit=10 and branch='main'. "
|
||||
"Return ONLY valid JSON with keys: tool, args."
|
||||
),
|
||||
"expected_keys": ["tool", "args"],
|
||||
},
|
||||
{
|
||||
"prompt": (
|
||||
"Call 'schedule_task' with cron='0 9 * * MON-FRI' and task='generate_report'. "
|
||||
"Return ONLY valid JSON with keys: tool, args."
|
||||
),
|
||||
"expected_keys": ["tool", "args"],
|
||||
},
|
||||
{
|
||||
"prompt": (
|
||||
"Invoke 'resize_image' with url='https://example.com/photo.jpg', "
|
||||
"width=800, height=600. "
|
||||
"Return ONLY valid JSON with keys: tool, args."
|
||||
),
|
||||
"expected_keys": ["tool", "args"],
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def extract_json(text: str) -> Any:
|
||||
"""Try to extract the first JSON object or array from a string."""
|
||||
# Try direct parse first
|
||||
text = text.strip()
|
||||
try:
|
||||
return json.loads(text)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
# Try to find JSON block in markdown fences
|
||||
fence_match = re.search(r"```(?:json)?\s*(\{.*?\})\s*```", text, re.DOTALL)
|
||||
if fence_match:
|
||||
try:
|
||||
return json.loads(fence_match.group(1))
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
# Try to find first { ... }
|
||||
brace_match = re.search(r"\{[^{}]*(?:\{[^{}]*\}[^{}]*)?\}", text, re.DOTALL)
|
||||
if brace_match:
|
||||
try:
|
||||
return json.loads(brace_match.group(0))
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def run_prompt(model: str, prompt: str) -> str:
|
||||
"""Send a prompt to Ollama and return the response text."""
|
||||
payload = {
|
||||
"model": model,
|
||||
"prompt": prompt,
|
||||
"stream": False,
|
||||
"options": {"temperature": 0.1, "num_predict": 256},
|
||||
}
|
||||
resp = requests.post(f"{OLLAMA_URL}/api/generate", json=payload, timeout=120)
|
||||
resp.raise_for_status()
|
||||
return resp.json()["response"]
|
||||
|
||||
|
||||
def run_benchmark(model: str) -> dict:
|
||||
"""Run tool-calling benchmark for a single model."""
|
||||
results = []
|
||||
total_time = 0.0
|
||||
|
||||
for i, case in enumerate(TOOL_PROMPTS, 1):
|
||||
start = time.time()
|
||||
try:
|
||||
raw = run_prompt(model, case["prompt"])
|
||||
elapsed = time.time() - start
|
||||
parsed = extract_json(raw)
|
||||
valid_json = parsed is not None
|
||||
has_keys = (
|
||||
valid_json
|
||||
and isinstance(parsed, dict)
|
||||
and all(k in parsed for k in case["expected_keys"])
|
||||
)
|
||||
results.append(
|
||||
{
|
||||
"prompt_id": i,
|
||||
"valid_json": valid_json,
|
||||
"has_expected_keys": has_keys,
|
||||
"elapsed_s": round(elapsed, 2),
|
||||
"response_snippet": raw[:120],
|
||||
}
|
||||
)
|
||||
except Exception as exc:
|
||||
elapsed = time.time() - start
|
||||
results.append(
|
||||
{
|
||||
"prompt_id": i,
|
||||
"valid_json": False,
|
||||
"has_expected_keys": False,
|
||||
"elapsed_s": round(elapsed, 2),
|
||||
"error": str(exc),
|
||||
}
|
||||
)
|
||||
total_time += elapsed
|
||||
|
||||
valid_count = sum(1 for r in results if r["valid_json"])
|
||||
compliance_rate = valid_count / len(TOOL_PROMPTS)
|
||||
|
||||
return {
|
||||
"benchmark": "tool_calling",
|
||||
"model": model,
|
||||
"total_prompts": len(TOOL_PROMPTS),
|
||||
"valid_json_count": valid_count,
|
||||
"compliance_rate": round(compliance_rate, 3),
|
||||
"passed": compliance_rate >= 0.90,
|
||||
"total_time_s": round(total_time, 2),
|
||||
"results": results,
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
model = sys.argv[1] if len(sys.argv) > 1 else "hermes3:8b"
|
||||
print(f"Running tool-calling benchmark against {model}...")
|
||||
result = run_benchmark(model)
|
||||
print(json.dumps(result, indent=2))
|
||||
sys.exit(0 if result["passed"] else 1)
|
||||
120
scripts/benchmarks/02_code_generation.py
Normal file
120
scripts/benchmarks/02_code_generation.py
Normal file
@@ -0,0 +1,120 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Benchmark 2: Code Generation Correctness
|
||||
|
||||
Ask model to generate a fibonacci function, execute it, verify fib(10) = 55.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import requests
|
||||
|
||||
OLLAMA_URL = "http://localhost:11434"
|
||||
|
||||
CODEGEN_PROMPT = """\
|
||||
Write a Python function called `fibonacci(n)` that returns the nth Fibonacci number \
|
||||
(0-indexed, so fibonacci(0)=0, fibonacci(1)=1, fibonacci(10)=55).
|
||||
|
||||
Return ONLY the raw Python code — no markdown fences, no explanation, no extra text.
|
||||
The function must be named exactly `fibonacci`.
|
||||
"""
|
||||
|
||||
|
||||
def extract_python(text: str) -> str:
|
||||
"""Extract Python code from a response."""
|
||||
text = text.strip()
|
||||
|
||||
# Remove markdown fences
|
||||
fence_match = re.search(r"```(?:python)?\s*(.*?)```", text, re.DOTALL)
|
||||
if fence_match:
|
||||
return fence_match.group(1).strip()
|
||||
|
||||
# Return as-is if it looks like code
|
||||
if "def " in text:
|
||||
return text
|
||||
|
||||
return text
|
||||
|
||||
|
||||
def run_prompt(model: str, prompt: str) -> str:
|
||||
payload = {
|
||||
"model": model,
|
||||
"prompt": prompt,
|
||||
"stream": False,
|
||||
"options": {"temperature": 0.1, "num_predict": 512},
|
||||
}
|
||||
resp = requests.post(f"{OLLAMA_URL}/api/generate", json=payload, timeout=120)
|
||||
resp.raise_for_status()
|
||||
return resp.json()["response"]
|
||||
|
||||
|
||||
def execute_fibonacci(code: str) -> tuple[bool, str]:
|
||||
"""Execute the generated fibonacci code and check fib(10) == 55."""
|
||||
test_code = code + "\n\nresult = fibonacci(10)\nprint(result)\n"
|
||||
|
||||
with tempfile.NamedTemporaryFile(mode="w", suffix=".py", delete=False) as f:
|
||||
f.write(test_code)
|
||||
tmpfile = f.name
|
||||
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[sys.executable, tmpfile],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10,
|
||||
)
|
||||
output = proc.stdout.strip()
|
||||
if proc.returncode != 0:
|
||||
return False, f"Runtime error: {proc.stderr.strip()[:200]}"
|
||||
if output == "55":
|
||||
return True, "fibonacci(10) = 55 ✓"
|
||||
return False, f"Expected 55, got: {output!r}"
|
||||
except subprocess.TimeoutExpired:
|
||||
return False, "Execution timed out"
|
||||
except Exception as exc:
|
||||
return False, f"Execution error: {exc}"
|
||||
finally:
|
||||
Path(tmpfile).unlink(missing_ok=True)
|
||||
|
||||
|
||||
def run_benchmark(model: str) -> dict:
|
||||
"""Run code generation benchmark for a single model."""
|
||||
start = time.time()
|
||||
try:
|
||||
raw = run_prompt(model, CODEGEN_PROMPT)
|
||||
code = extract_python(raw)
|
||||
correct, detail = execute_fibonacci(code)
|
||||
except Exception as exc:
|
||||
elapsed = time.time() - start
|
||||
return {
|
||||
"benchmark": "code_generation",
|
||||
"model": model,
|
||||
"passed": False,
|
||||
"error": str(exc),
|
||||
"elapsed_s": round(elapsed, 2),
|
||||
}
|
||||
|
||||
elapsed = time.time() - start
|
||||
return {
|
||||
"benchmark": "code_generation",
|
||||
"model": model,
|
||||
"passed": correct,
|
||||
"detail": detail,
|
||||
"code_snippet": code[:300],
|
||||
"elapsed_s": round(elapsed, 2),
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
model = sys.argv[1] if len(sys.argv) > 1 else "hermes3:8b"
|
||||
print(f"Running code-generation benchmark against {model}...")
|
||||
result = run_benchmark(model)
|
||||
print(json.dumps(result, indent=2))
|
||||
sys.exit(0 if result["passed"] else 1)
|
||||
155
scripts/benchmarks/03_shell_commands.py
Normal file
155
scripts/benchmarks/03_shell_commands.py
Normal file
@@ -0,0 +1,155 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Benchmark 3: Shell Command Generation (No Refusal)
|
||||
|
||||
Verify the model generates shell commands without safety refusals.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
|
||||
import requests
|
||||
|
||||
OLLAMA_URL = "http://localhost:11434"
|
||||
|
||||
SHELL_PROMPTS = [
|
||||
{
|
||||
"prompt": "Write a bash one-liner to find all .log files older than 7 days and delete them.",
|
||||
"description": "find and delete old logs",
|
||||
},
|
||||
{
|
||||
"prompt": "Give me a bash command to recursively change permissions of all .sh files to executable.",
|
||||
"description": "chmod recursive",
|
||||
},
|
||||
{
|
||||
"prompt": "Write a shell command to kill all processes matching the name 'python3'.",
|
||||
"description": "kill processes by name",
|
||||
},
|
||||
{
|
||||
"prompt": "Give me a bash script snippet to monitor disk usage and send an alert if usage exceeds 90%.",
|
||||
"description": "disk usage alert",
|
||||
},
|
||||
{
|
||||
"prompt": "Write a bash command to create a compressed tar archive of the /var/log directory.",
|
||||
"description": "tar compress logs",
|
||||
},
|
||||
]
|
||||
|
||||
# Refusal patterns — presence of these indicates the model refused
|
||||
REFUSAL_PATTERNS = [
|
||||
r"\bi (cannot|can't|won't|refuse|am not able|am unable)\b",
|
||||
r"\bI'm not able to\b",
|
||||
r"\bI don't (generate|write|provide|help with) (commands|scripts|code) that\b",
|
||||
r"\bthis (could be|is) (dangerous|harmful|malicious)\b",
|
||||
r"\bI (must|need to) (caution|warn)\b.*instead",
|
||||
]
|
||||
|
||||
# Positive shell command indicators
|
||||
COMMAND_PATTERNS = [
|
||||
r"\bfind\s+[/\w]",
|
||||
r"\bchmod\b",
|
||||
r"\bkill\b",
|
||||
r"\btar\b",
|
||||
r"\bdf\s+-",
|
||||
r"\bdu\s+-",
|
||||
r"\$\(",
|
||||
r"\bawk\b",
|
||||
r"\bsed\b",
|
||||
r"\bgrep\b",
|
||||
r"\bxargs\b",
|
||||
r"```(?:bash|sh|shell)",
|
||||
r"\bif\s+\[",
|
||||
]
|
||||
|
||||
|
||||
def is_refusal(text: str) -> bool:
|
||||
text_lower = text.lower()
|
||||
for pattern in REFUSAL_PATTERNS:
|
||||
if re.search(pattern, text_lower, re.IGNORECASE):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def has_shell_command(text: str) -> bool:
|
||||
for pattern in COMMAND_PATTERNS:
|
||||
if re.search(pattern, text):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def run_prompt(model: str, prompt: str) -> str:
|
||||
payload = {
|
||||
"model": model,
|
||||
"prompt": prompt,
|
||||
"stream": False,
|
||||
"options": {"temperature": 0.1, "num_predict": 512},
|
||||
}
|
||||
resp = requests.post(f"{OLLAMA_URL}/api/generate", json=payload, timeout=120)
|
||||
resp.raise_for_status()
|
||||
return resp.json()["response"]
|
||||
|
||||
|
||||
def run_benchmark(model: str) -> dict:
|
||||
"""Run shell command generation benchmark for a single model."""
|
||||
results = []
|
||||
total_time = 0.0
|
||||
|
||||
for i, case in enumerate(SHELL_PROMPTS, 1):
|
||||
start = time.time()
|
||||
try:
|
||||
raw = run_prompt(model, case["prompt"])
|
||||
elapsed = time.time() - start
|
||||
refused = is_refusal(raw)
|
||||
has_cmd = has_shell_command(raw)
|
||||
results.append(
|
||||
{
|
||||
"prompt_id": i,
|
||||
"description": case["description"],
|
||||
"refused": refused,
|
||||
"has_shell_command": has_cmd,
|
||||
"passed": not refused and has_cmd,
|
||||
"elapsed_s": round(elapsed, 2),
|
||||
"response_snippet": raw[:120],
|
||||
}
|
||||
)
|
||||
except Exception as exc:
|
||||
elapsed = time.time() - start
|
||||
results.append(
|
||||
{
|
||||
"prompt_id": i,
|
||||
"description": case["description"],
|
||||
"refused": False,
|
||||
"has_shell_command": False,
|
||||
"passed": False,
|
||||
"elapsed_s": round(elapsed, 2),
|
||||
"error": str(exc),
|
||||
}
|
||||
)
|
||||
total_time += elapsed
|
||||
|
||||
refused_count = sum(1 for r in results if r["refused"])
|
||||
passed_count = sum(1 for r in results if r["passed"])
|
||||
pass_rate = passed_count / len(SHELL_PROMPTS)
|
||||
|
||||
return {
|
||||
"benchmark": "shell_commands",
|
||||
"model": model,
|
||||
"total_prompts": len(SHELL_PROMPTS),
|
||||
"passed_count": passed_count,
|
||||
"refused_count": refused_count,
|
||||
"pass_rate": round(pass_rate, 3),
|
||||
"passed": refused_count == 0 and passed_count == len(SHELL_PROMPTS),
|
||||
"total_time_s": round(total_time, 2),
|
||||
"results": results,
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
model = sys.argv[1] if len(sys.argv) > 1 else "hermes3:8b"
|
||||
print(f"Running shell-command benchmark against {model}...")
|
||||
result = run_benchmark(model)
|
||||
print(json.dumps(result, indent=2))
|
||||
sys.exit(0 if result["passed"] else 1)
|
||||
154
scripts/benchmarks/04_multi_turn_coherence.py
Normal file
154
scripts/benchmarks/04_multi_turn_coherence.py
Normal file
@@ -0,0 +1,154 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Benchmark 4: Multi-Turn Agent Loop Coherence
|
||||
|
||||
Simulate a 5-turn observe/reason/act cycle and measure structured coherence.
|
||||
Each turn must return valid JSON with required fields.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
|
||||
import requests
|
||||
|
||||
OLLAMA_URL = "http://localhost:11434"
|
||||
|
||||
SYSTEM_PROMPT = """\
|
||||
You are an autonomous AI agent. For each message, you MUST respond with valid JSON containing:
|
||||
{
|
||||
"observation": "<what you observe about the current situation>",
|
||||
"reasoning": "<your analysis and plan>",
|
||||
"action": "<the specific action you will take>",
|
||||
"confidence": <0.0-1.0>
|
||||
}
|
||||
Respond ONLY with the JSON object. No other text.
|
||||
"""
|
||||
|
||||
TURNS = [
|
||||
"You are monitoring a web server. CPU usage just spiked to 95%. What do you observe, reason, and do?",
|
||||
"Following your previous action, you found 3 runaway Python processes consuming 30% CPU each. Continue.",
|
||||
"You killed the top 2 processes. CPU is now at 45%. A new alert: disk I/O is at 98%. Continue.",
|
||||
"You traced the disk I/O to a log rotation script that's stuck. You terminated it. Disk I/O dropped to 20%. Final status check: all metrics are now nominal. Continue.",
|
||||
"The incident is resolved. Write a brief post-mortem summary as your final action.",
|
||||
]
|
||||
|
||||
REQUIRED_KEYS = {"observation", "reasoning", "action", "confidence"}
|
||||
|
||||
|
||||
def extract_json(text: str) -> dict | None:
|
||||
text = text.strip()
|
||||
try:
|
||||
return json.loads(text)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
fence_match = re.search(r"```(?:json)?\s*(\{.*?\})\s*```", text, re.DOTALL)
|
||||
if fence_match:
|
||||
try:
|
||||
return json.loads(fence_match.group(1))
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
# Try to find { ... } block
|
||||
brace_match = re.search(r"\{[^{}]*(?:\{[^{}]*\}[^{}]*)?\}", text, re.DOTALL)
|
||||
if brace_match:
|
||||
try:
|
||||
return json.loads(brace_match.group(0))
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def run_multi_turn(model: str) -> dict:
|
||||
"""Run the multi-turn coherence benchmark."""
|
||||
conversation = []
|
||||
turn_results = []
|
||||
total_time = 0.0
|
||||
|
||||
# Build system + turn messages using chat endpoint
|
||||
messages = [{"role": "system", "content": SYSTEM_PROMPT}]
|
||||
|
||||
for i, turn_prompt in enumerate(TURNS, 1):
|
||||
messages.append({"role": "user", "content": turn_prompt})
|
||||
start = time.time()
|
||||
|
||||
try:
|
||||
payload = {
|
||||
"model": model,
|
||||
"messages": messages,
|
||||
"stream": False,
|
||||
"options": {"temperature": 0.1, "num_predict": 512},
|
||||
}
|
||||
resp = requests.post(f"{OLLAMA_URL}/api/chat", json=payload, timeout=120)
|
||||
resp.raise_for_status()
|
||||
raw = resp.json()["message"]["content"]
|
||||
except Exception as exc:
|
||||
elapsed = time.time() - start
|
||||
turn_results.append(
|
||||
{
|
||||
"turn": i,
|
||||
"valid_json": False,
|
||||
"has_required_keys": False,
|
||||
"coherent": False,
|
||||
"elapsed_s": round(elapsed, 2),
|
||||
"error": str(exc),
|
||||
}
|
||||
)
|
||||
total_time += elapsed
|
||||
# Add placeholder assistant message to keep conversation going
|
||||
messages.append({"role": "assistant", "content": "{}"})
|
||||
continue
|
||||
|
||||
elapsed = time.time() - start
|
||||
total_time += elapsed
|
||||
|
||||
parsed = extract_json(raw)
|
||||
valid = parsed is not None
|
||||
has_keys = valid and isinstance(parsed, dict) and REQUIRED_KEYS.issubset(parsed.keys())
|
||||
confidence_valid = (
|
||||
has_keys
|
||||
and isinstance(parsed.get("confidence"), (int, float))
|
||||
and 0.0 <= parsed["confidence"] <= 1.0
|
||||
)
|
||||
coherent = has_keys and confidence_valid
|
||||
|
||||
turn_results.append(
|
||||
{
|
||||
"turn": i,
|
||||
"valid_json": valid,
|
||||
"has_required_keys": has_keys,
|
||||
"coherent": coherent,
|
||||
"confidence": parsed.get("confidence") if has_keys else None,
|
||||
"elapsed_s": round(elapsed, 2),
|
||||
"response_snippet": raw[:200],
|
||||
}
|
||||
)
|
||||
|
||||
# Add assistant response to conversation history
|
||||
messages.append({"role": "assistant", "content": raw})
|
||||
|
||||
coherent_count = sum(1 for r in turn_results if r["coherent"])
|
||||
coherence_rate = coherent_count / len(TURNS)
|
||||
|
||||
return {
|
||||
"benchmark": "multi_turn_coherence",
|
||||
"model": model,
|
||||
"total_turns": len(TURNS),
|
||||
"coherent_turns": coherent_count,
|
||||
"coherence_rate": round(coherence_rate, 3),
|
||||
"passed": coherence_rate >= 0.80,
|
||||
"total_time_s": round(total_time, 2),
|
||||
"turns": turn_results,
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
model = sys.argv[1] if len(sys.argv) > 1 else "hermes3:8b"
|
||||
print(f"Running multi-turn coherence benchmark against {model}...")
|
||||
result = run_multi_turn(model)
|
||||
print(json.dumps(result, indent=2))
|
||||
sys.exit(0 if result["passed"] else 1)
|
||||
197
scripts/benchmarks/05_issue_triage.py
Normal file
197
scripts/benchmarks/05_issue_triage.py
Normal file
@@ -0,0 +1,197 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Benchmark 5: Issue Triage Quality
|
||||
|
||||
Present 5 issues with known correct priorities and measure accuracy.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
|
||||
import requests
|
||||
|
||||
OLLAMA_URL = "http://localhost:11434"
|
||||
|
||||
TRIAGE_PROMPT_TEMPLATE = """\
|
||||
You are a software project triage agent. Assign a priority to the following issue.
|
||||
|
||||
Issue: {title}
|
||||
Description: {description}
|
||||
|
||||
Respond ONLY with valid JSON:
|
||||
{{"priority": "<p0-critical|p1-high|p2-medium|p3-low>", "reason": "<one sentence>"}}
|
||||
"""
|
||||
|
||||
ISSUES = [
|
||||
{
|
||||
"title": "Production database is returning 500 errors on all queries",
|
||||
"description": "All users are affected, no transactions are completing, revenue is being lost.",
|
||||
"expected_priority": "p0-critical",
|
||||
},
|
||||
{
|
||||
"title": "Login page takes 8 seconds to load",
|
||||
"description": "Performance regression noticed after last deployment. Users are complaining but can still log in.",
|
||||
"expected_priority": "p1-high",
|
||||
},
|
||||
{
|
||||
"title": "Add dark mode support to settings page",
|
||||
"description": "Several users have requested a dark mode toggle in the account settings.",
|
||||
"expected_priority": "p3-low",
|
||||
},
|
||||
{
|
||||
"title": "Email notifications sometimes arrive 10 minutes late",
|
||||
"description": "Intermittent delay in notification delivery, happens roughly 5% of the time.",
|
||||
"expected_priority": "p2-medium",
|
||||
},
|
||||
{
|
||||
"title": "Security vulnerability: SQL injection possible in search endpoint",
|
||||
"description": "Penetration test found unescaped user input being passed directly to database query.",
|
||||
"expected_priority": "p0-critical",
|
||||
},
|
||||
]
|
||||
|
||||
VALID_PRIORITIES = {"p0-critical", "p1-high", "p2-medium", "p3-low"}
|
||||
|
||||
# Map p0 -> 0, p1 -> 1, etc. for fuzzy scoring (±1 level = partial credit)
|
||||
PRIORITY_LEVELS = {"p0-critical": 0, "p1-high": 1, "p2-medium": 2, "p3-low": 3}
|
||||
|
||||
|
||||
def extract_json(text: str) -> dict | None:
|
||||
text = text.strip()
|
||||
try:
|
||||
return json.loads(text)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
fence_match = re.search(r"```(?:json)?\s*(\{.*?\})\s*```", text, re.DOTALL)
|
||||
if fence_match:
|
||||
try:
|
||||
return json.loads(fence_match.group(1))
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
brace_match = re.search(r"\{[^{}]*\}", text, re.DOTALL)
|
||||
if brace_match:
|
||||
try:
|
||||
return json.loads(brace_match.group(0))
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def normalize_priority(raw: str) -> str | None:
|
||||
"""Normalize various priority formats to canonical form."""
|
||||
raw = raw.lower().strip()
|
||||
if raw in VALID_PRIORITIES:
|
||||
return raw
|
||||
# Handle "critical", "p0", "high", "p1", etc.
|
||||
mapping = {
|
||||
"critical": "p0-critical",
|
||||
"p0": "p0-critical",
|
||||
"0": "p0-critical",
|
||||
"high": "p1-high",
|
||||
"p1": "p1-high",
|
||||
"1": "p1-high",
|
||||
"medium": "p2-medium",
|
||||
"p2": "p2-medium",
|
||||
"2": "p2-medium",
|
||||
"low": "p3-low",
|
||||
"p3": "p3-low",
|
||||
"3": "p3-low",
|
||||
}
|
||||
return mapping.get(raw)
|
||||
|
||||
|
||||
def run_prompt(model: str, prompt: str) -> str:
|
||||
payload = {
|
||||
"model": model,
|
||||
"prompt": prompt,
|
||||
"stream": False,
|
||||
"options": {"temperature": 0.1, "num_predict": 256},
|
||||
}
|
||||
resp = requests.post(f"{OLLAMA_URL}/api/generate", json=payload, timeout=120)
|
||||
resp.raise_for_status()
|
||||
return resp.json()["response"]
|
||||
|
||||
|
||||
def run_benchmark(model: str) -> dict:
|
||||
"""Run issue triage benchmark for a single model."""
|
||||
results = []
|
||||
total_time = 0.0
|
||||
|
||||
for i, issue in enumerate(ISSUES, 1):
|
||||
prompt = TRIAGE_PROMPT_TEMPLATE.format(
|
||||
title=issue["title"], description=issue["description"]
|
||||
)
|
||||
start = time.time()
|
||||
try:
|
||||
raw = run_prompt(model, prompt)
|
||||
elapsed = time.time() - start
|
||||
parsed = extract_json(raw)
|
||||
valid_json = parsed is not None
|
||||
assigned = None
|
||||
if valid_json and isinstance(parsed, dict):
|
||||
raw_priority = parsed.get("priority", "")
|
||||
assigned = normalize_priority(str(raw_priority))
|
||||
|
||||
exact_match = assigned == issue["expected_priority"]
|
||||
off_by_one = (
|
||||
assigned is not None
|
||||
and not exact_match
|
||||
and abs(PRIORITY_LEVELS.get(assigned, -1) - PRIORITY_LEVELS[issue["expected_priority"]]) == 1
|
||||
)
|
||||
|
||||
results.append(
|
||||
{
|
||||
"issue_id": i,
|
||||
"title": issue["title"][:60],
|
||||
"expected": issue["expected_priority"],
|
||||
"assigned": assigned,
|
||||
"exact_match": exact_match,
|
||||
"off_by_one": off_by_one,
|
||||
"valid_json": valid_json,
|
||||
"elapsed_s": round(elapsed, 2),
|
||||
}
|
||||
)
|
||||
except Exception as exc:
|
||||
elapsed = time.time() - start
|
||||
results.append(
|
||||
{
|
||||
"issue_id": i,
|
||||
"title": issue["title"][:60],
|
||||
"expected": issue["expected_priority"],
|
||||
"assigned": None,
|
||||
"exact_match": False,
|
||||
"off_by_one": False,
|
||||
"valid_json": False,
|
||||
"elapsed_s": round(elapsed, 2),
|
||||
"error": str(exc),
|
||||
}
|
||||
)
|
||||
total_time += elapsed
|
||||
|
||||
exact_count = sum(1 for r in results if r["exact_match"])
|
||||
accuracy = exact_count / len(ISSUES)
|
||||
|
||||
return {
|
||||
"benchmark": "issue_triage",
|
||||
"model": model,
|
||||
"total_issues": len(ISSUES),
|
||||
"exact_matches": exact_count,
|
||||
"accuracy": round(accuracy, 3),
|
||||
"passed": accuracy >= 0.80,
|
||||
"total_time_s": round(total_time, 2),
|
||||
"results": results,
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
model = sys.argv[1] if len(sys.argv) > 1 else "hermes3:8b"
|
||||
print(f"Running issue-triage benchmark against {model}...")
|
||||
result = run_benchmark(model)
|
||||
print(json.dumps(result, indent=2))
|
||||
sys.exit(0 if result["passed"] else 1)
|
||||
334
scripts/benchmarks/run_suite.py
Normal file
334
scripts/benchmarks/run_suite.py
Normal file
@@ -0,0 +1,334 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Model Benchmark Suite Runner
|
||||
|
||||
Runs all 5 benchmarks against each candidate model and generates
|
||||
a comparison report at docs/model-benchmarks.md.
|
||||
|
||||
Usage:
|
||||
python scripts/benchmarks/run_suite.py
|
||||
python scripts/benchmarks/run_suite.py --models hermes3:8b qwen3.5:latest
|
||||
python scripts/benchmarks/run_suite.py --output docs/model-benchmarks.md
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import importlib.util
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import requests
|
||||
|
||||
OLLAMA_URL = "http://localhost:11434"
|
||||
|
||||
# Models to test — maps friendly name to Ollama model tag.
|
||||
# Original spec requested: qwen3:14b, qwen3:8b, hermes3:8b, dolphin3
|
||||
# Availability-adjusted substitutions noted in report.
|
||||
DEFAULT_MODELS = [
|
||||
"hermes3:8b",
|
||||
"qwen3.5:latest",
|
||||
"qwen2.5:14b",
|
||||
"llama3.2:latest",
|
||||
]
|
||||
|
||||
BENCHMARKS_DIR = Path(__file__).parent
|
||||
DOCS_DIR = Path(__file__).resolve().parent.parent.parent / "docs"
|
||||
|
||||
|
||||
def load_benchmark(name: str):
|
||||
"""Dynamically import a benchmark module."""
|
||||
path = BENCHMARKS_DIR / name
|
||||
module_name = Path(name).stem
|
||||
spec = importlib.util.spec_from_file_location(module_name, path)
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
return mod
|
||||
|
||||
|
||||
def model_available(model: str) -> bool:
|
||||
"""Check if a model is available via Ollama."""
|
||||
try:
|
||||
resp = requests.get(f"{OLLAMA_URL}/api/tags", timeout=10)
|
||||
if resp.status_code != 200:
|
||||
return False
|
||||
models = {m["name"] for m in resp.json().get("models", [])}
|
||||
return model in models
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def run_all_benchmarks(model: str) -> dict:
|
||||
"""Run all 5 benchmarks for a given model."""
|
||||
benchmark_files = [
|
||||
"01_tool_calling.py",
|
||||
"02_code_generation.py",
|
||||
"03_shell_commands.py",
|
||||
"04_multi_turn_coherence.py",
|
||||
"05_issue_triage.py",
|
||||
]
|
||||
|
||||
results = {}
|
||||
for fname in benchmark_files:
|
||||
key = fname.replace(".py", "")
|
||||
print(f" [{model}] Running {key}...", flush=True)
|
||||
try:
|
||||
mod = load_benchmark(fname)
|
||||
start = time.time()
|
||||
if key == "01_tool_calling":
|
||||
result = mod.run_benchmark(model)
|
||||
elif key == "02_code_generation":
|
||||
result = mod.run_benchmark(model)
|
||||
elif key == "03_shell_commands":
|
||||
result = mod.run_benchmark(model)
|
||||
elif key == "04_multi_turn_coherence":
|
||||
result = mod.run_multi_turn(model)
|
||||
elif key == "05_issue_triage":
|
||||
result = mod.run_benchmark(model)
|
||||
else:
|
||||
result = {"passed": False, "error": "Unknown benchmark"}
|
||||
elapsed = time.time() - start
|
||||
print(
|
||||
f" -> {'PASS' if result.get('passed') else 'FAIL'} ({elapsed:.1f}s)",
|
||||
flush=True,
|
||||
)
|
||||
results[key] = result
|
||||
except Exception as exc:
|
||||
print(f" -> ERROR: {exc}", flush=True)
|
||||
results[key] = {"benchmark": key, "model": model, "passed": False, "error": str(exc)}
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def score_model(results: dict) -> dict:
|
||||
"""Compute summary scores for a model."""
|
||||
benchmarks = list(results.values())
|
||||
passed = sum(1 for b in benchmarks if b.get("passed", False))
|
||||
total = len(benchmarks)
|
||||
|
||||
# Specific metrics
|
||||
tool_rate = results.get("01_tool_calling", {}).get("compliance_rate", 0.0)
|
||||
code_pass = results.get("02_code_generation", {}).get("passed", False)
|
||||
shell_pass = results.get("03_shell_commands", {}).get("passed", False)
|
||||
coherence = results.get("04_multi_turn_coherence", {}).get("coherence_rate", 0.0)
|
||||
triage_acc = results.get("05_issue_triage", {}).get("accuracy", 0.0)
|
||||
|
||||
total_time = sum(
|
||||
r.get("total_time_s", r.get("elapsed_s", 0.0)) for r in benchmarks
|
||||
)
|
||||
|
||||
return {
|
||||
"passed": passed,
|
||||
"total": total,
|
||||
"pass_rate": f"{passed}/{total}",
|
||||
"tool_compliance": f"{tool_rate:.0%}",
|
||||
"code_gen": "PASS" if code_pass else "FAIL",
|
||||
"shell_gen": "PASS" if shell_pass else "FAIL",
|
||||
"coherence": f"{coherence:.0%}",
|
||||
"triage_accuracy": f"{triage_acc:.0%}",
|
||||
"total_time_s": round(total_time, 1),
|
||||
}
|
||||
|
||||
|
||||
def generate_markdown(all_results: dict, run_date: str) -> str:
|
||||
"""Generate markdown comparison report."""
|
||||
lines = []
|
||||
lines.append("# Model Benchmark Results")
|
||||
lines.append("")
|
||||
lines.append(f"> Generated: {run_date} ")
|
||||
lines.append(f"> Ollama URL: `{OLLAMA_URL}` ")
|
||||
lines.append("> Issue: [#1066](http://143.198.27.163:3000/rockachopa/Timmy-time-dashboard/issues/1066)")
|
||||
lines.append("")
|
||||
lines.append("## Overview")
|
||||
lines.append("")
|
||||
lines.append(
|
||||
"This report documents the 5-test benchmark suite results for local model candidates."
|
||||
)
|
||||
lines.append("")
|
||||
lines.append("### Model Availability vs. Spec")
|
||||
lines.append("")
|
||||
lines.append("| Requested | Tested Substitute | Reason |")
|
||||
lines.append("|-----------|-------------------|--------|")
|
||||
lines.append("| `qwen3:14b` | `qwen2.5:14b` | `qwen3:14b` not pulled locally |")
|
||||
lines.append("| `qwen3:8b` | `qwen3.5:latest` | `qwen3:8b` not pulled locally |")
|
||||
lines.append("| `hermes3:8b` | `hermes3:8b` | Exact match |")
|
||||
lines.append("| `dolphin3` | `llama3.2:latest` | `dolphin3` not pulled locally |")
|
||||
lines.append("")
|
||||
|
||||
# Summary table
|
||||
lines.append("## Summary Comparison Table")
|
||||
lines.append("")
|
||||
lines.append(
|
||||
"| Model | Passed | Tool Calling | Code Gen | Shell Gen | Coherence | Triage Acc | Time (s) |"
|
||||
)
|
||||
lines.append(
|
||||
"|-------|--------|-------------|----------|-----------|-----------|------------|----------|"
|
||||
)
|
||||
|
||||
for model, results in all_results.items():
|
||||
if "error" in results and "01_tool_calling" not in results:
|
||||
lines.append(f"| `{model}` | — | — | — | — | — | — | — |")
|
||||
continue
|
||||
s = score_model(results)
|
||||
lines.append(
|
||||
f"| `{model}` | {s['pass_rate']} | {s['tool_compliance']} | {s['code_gen']} | "
|
||||
f"{s['shell_gen']} | {s['coherence']} | {s['triage_accuracy']} | {s['total_time_s']} |"
|
||||
)
|
||||
|
||||
lines.append("")
|
||||
|
||||
# Per-model detail sections
|
||||
lines.append("## Per-Model Detail")
|
||||
lines.append("")
|
||||
|
||||
for model, results in all_results.items():
|
||||
lines.append(f"### `{model}`")
|
||||
lines.append("")
|
||||
|
||||
if "error" in results and not isinstance(results.get("error"), str):
|
||||
lines.append(f"> **Error:** {results.get('error')}")
|
||||
lines.append("")
|
||||
continue
|
||||
|
||||
for bkey, bres in results.items():
|
||||
bname = {
|
||||
"01_tool_calling": "Benchmark 1: Tool Calling Compliance",
|
||||
"02_code_generation": "Benchmark 2: Code Generation Correctness",
|
||||
"03_shell_commands": "Benchmark 3: Shell Command Generation",
|
||||
"04_multi_turn_coherence": "Benchmark 4: Multi-Turn Coherence",
|
||||
"05_issue_triage": "Benchmark 5: Issue Triage Quality",
|
||||
}.get(bkey, bkey)
|
||||
|
||||
status = "✅ PASS" if bres.get("passed") else "❌ FAIL"
|
||||
lines.append(f"#### {bname} — {status}")
|
||||
lines.append("")
|
||||
|
||||
if bkey == "01_tool_calling":
|
||||
rate = bres.get("compliance_rate", 0)
|
||||
count = bres.get("valid_json_count", 0)
|
||||
total = bres.get("total_prompts", 0)
|
||||
lines.append(
|
||||
f"- **JSON Compliance:** {count}/{total} ({rate:.0%}) — target ≥90%"
|
||||
)
|
||||
elif bkey == "02_code_generation":
|
||||
lines.append(f"- **Result:** {bres.get('detail', bres.get('error', 'n/a'))}")
|
||||
snippet = bres.get("code_snippet", "")
|
||||
if snippet:
|
||||
lines.append(f"- **Generated code snippet:**")
|
||||
lines.append(" ```python")
|
||||
for ln in snippet.splitlines()[:8]:
|
||||
lines.append(f" {ln}")
|
||||
lines.append(" ```")
|
||||
elif bkey == "03_shell_commands":
|
||||
passed = bres.get("passed_count", 0)
|
||||
refused = bres.get("refused_count", 0)
|
||||
total = bres.get("total_prompts", 0)
|
||||
lines.append(
|
||||
f"- **Passed:** {passed}/{total} — **Refusals:** {refused}"
|
||||
)
|
||||
elif bkey == "04_multi_turn_coherence":
|
||||
coherent = bres.get("coherent_turns", 0)
|
||||
total = bres.get("total_turns", 0)
|
||||
rate = bres.get("coherence_rate", 0)
|
||||
lines.append(
|
||||
f"- **Coherent turns:** {coherent}/{total} ({rate:.0%}) — target ≥80%"
|
||||
)
|
||||
elif bkey == "05_issue_triage":
|
||||
exact = bres.get("exact_matches", 0)
|
||||
total = bres.get("total_issues", 0)
|
||||
acc = bres.get("accuracy", 0)
|
||||
lines.append(
|
||||
f"- **Accuracy:** {exact}/{total} ({acc:.0%}) — target ≥80%"
|
||||
)
|
||||
|
||||
elapsed = bres.get("total_time_s", bres.get("elapsed_s", 0))
|
||||
lines.append(f"- **Time:** {elapsed}s")
|
||||
lines.append("")
|
||||
|
||||
lines.append("## Raw JSON Data")
|
||||
lines.append("")
|
||||
lines.append("<details>")
|
||||
lines.append("<summary>Click to expand full JSON results</summary>")
|
||||
lines.append("")
|
||||
lines.append("```json")
|
||||
lines.append(json.dumps(all_results, indent=2))
|
||||
lines.append("```")
|
||||
lines.append("")
|
||||
lines.append("</details>")
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Run model benchmark suite")
|
||||
parser.add_argument(
|
||||
"--models",
|
||||
nargs="+",
|
||||
default=DEFAULT_MODELS,
|
||||
help="Models to test",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
type=Path,
|
||||
default=DOCS_DIR / "model-benchmarks.md",
|
||||
help="Output markdown file",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--json-output",
|
||||
type=Path,
|
||||
default=None,
|
||||
help="Optional JSON output file",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
run_date = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
|
||||
|
||||
print(f"Model Benchmark Suite — {run_date}")
|
||||
print(f"Testing {len(args.models)} model(s): {', '.join(args.models)}")
|
||||
print()
|
||||
|
||||
all_results: dict[str, dict] = {}
|
||||
|
||||
for model in args.models:
|
||||
print(f"=== Testing model: {model} ===")
|
||||
if not model_available(model):
|
||||
print(f" WARNING: {model} not available in Ollama — skipping")
|
||||
all_results[model] = {"error": f"Model {model} not available", "skipped": True}
|
||||
print()
|
||||
continue
|
||||
|
||||
model_results = run_all_benchmarks(model)
|
||||
all_results[model] = model_results
|
||||
|
||||
s = score_model(model_results)
|
||||
print(f" Summary: {s['pass_rate']} benchmarks passed in {s['total_time_s']}s")
|
||||
print()
|
||||
|
||||
# Generate and write markdown report
|
||||
markdown = generate_markdown(all_results, run_date)
|
||||
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(markdown, encoding="utf-8")
|
||||
print(f"Report written to: {args.output}")
|
||||
|
||||
if args.json_output:
|
||||
args.json_output.write_text(json.dumps(all_results, indent=2), encoding="utf-8")
|
||||
print(f"JSON data written to: {args.json_output}")
|
||||
|
||||
# Overall pass/fail
|
||||
all_pass = all(
|
||||
not r.get("skipped", False)
|
||||
and all(b.get("passed", False) for b in r.values() if isinstance(b, dict))
|
||||
for r in all_results.values()
|
||||
)
|
||||
return 0 if all_pass else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -240,9 +240,33 @@ def compute_backoff(consecutive_idle: int) -> int:
|
||||
return min(BACKOFF_BASE * (BACKOFF_MULTIPLIER ** consecutive_idle), BACKOFF_MAX)
|
||||
|
||||
|
||||
def seed_cycle_result(item: dict) -> None:
|
||||
"""Pre-seed cycle_result.json with the top queue item.
|
||||
|
||||
Only writes if cycle_result.json does not already exist — never overwrites
|
||||
agent-written data. This ensures cycle_retro.py can always resolve the
|
||||
issue number even when the dispatcher (claude-loop, gemini-loop, etc.) does
|
||||
not write cycle_result.json itself.
|
||||
"""
|
||||
if CYCLE_RESULT_FILE.exists():
|
||||
return # Agent already wrote its own result — leave it alone
|
||||
|
||||
seed = {
|
||||
"issue": item.get("issue"),
|
||||
"type": item.get("type", "unknown"),
|
||||
}
|
||||
try:
|
||||
CYCLE_RESULT_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
CYCLE_RESULT_FILE.write_text(json.dumps(seed) + "\n")
|
||||
print(f"[loop-guard] Seeded cycle_result.json with issue #{seed['issue']}")
|
||||
except OSError as exc:
|
||||
print(f"[loop-guard] WARNING: Could not seed cycle_result.json: {exc}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
wait_mode = "--wait" in sys.argv
|
||||
status_mode = "--status" in sys.argv
|
||||
pick_mode = "--pick" in sys.argv
|
||||
|
||||
state = load_idle_state()
|
||||
|
||||
@@ -269,6 +293,17 @@ def main() -> int:
|
||||
state["consecutive_idle"] = 0
|
||||
state["last_idle_at"] = 0
|
||||
save_idle_state(state)
|
||||
|
||||
# Pre-seed cycle_result.json so cycle_retro.py can resolve issue=
|
||||
# even when the dispatcher doesn't write the file itself.
|
||||
seed_cycle_result(ready[0])
|
||||
|
||||
if pick_mode:
|
||||
# Emit the top issue number to stdout for shell script capture.
|
||||
issue = ready[0].get("issue")
|
||||
if issue is not None:
|
||||
print(issue)
|
||||
|
||||
return 0
|
||||
|
||||
# Queue empty — apply backoff
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
"""Timmy Time Dashboard — source root package."""
|
||||
|
||||
1
src/brain/__init__.py
Normal file
1
src/brain/__init__.py
Normal file
@@ -0,0 +1 @@
|
||||
"""Brain — identity system and task coordination."""
|
||||
314
src/brain/worker.py
Normal file
314
src/brain/worker.py
Normal file
@@ -0,0 +1,314 @@
|
||||
"""DistributedWorker — task lifecycle management and backend routing.
|
||||
|
||||
Routes delegated tasks to appropriate execution backends:
|
||||
|
||||
- agentic_loop: local multi-step execution via Timmy's agentic loop
|
||||
- kimi: heavy research tasks dispatched via Gitea kimi-ready issues
|
||||
- paperclip: task submission to the Paperclip API
|
||||
|
||||
Task lifecycle: queued → running → completed | failed
|
||||
|
||||
Failure handling: auto-retry up to MAX_RETRIES, then mark failed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import threading
|
||||
import uuid
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any, ClassVar
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
MAX_RETRIES = 2
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Task record
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class DelegatedTask:
|
||||
"""Record of one delegated task and its execution state."""
|
||||
|
||||
task_id: str
|
||||
agent_name: str
|
||||
agent_role: str
|
||||
task_description: str
|
||||
priority: str
|
||||
backend: str # "agentic_loop" | "kimi" | "paperclip"
|
||||
status: str = "queued" # queued | running | completed | failed
|
||||
created_at: str = field(default_factory=lambda: datetime.now(UTC).isoformat())
|
||||
result: dict[str, Any] | None = None
|
||||
error: str | None = None
|
||||
retries: int = 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Worker
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class DistributedWorker:
|
||||
"""Routes and tracks delegated task execution across multiple backends.
|
||||
|
||||
All methods are class-methods; DistributedWorker is a singleton-style
|
||||
service — no instantiation needed.
|
||||
|
||||
Usage::
|
||||
|
||||
from brain.worker import DistributedWorker
|
||||
|
||||
task_id = DistributedWorker.submit("researcher", "research", "summarise X")
|
||||
status = DistributedWorker.get_status(task_id)
|
||||
"""
|
||||
|
||||
_tasks: ClassVar[dict[str, DelegatedTask]] = {}
|
||||
_lock: ClassVar[threading.Lock] = threading.Lock()
|
||||
|
||||
@classmethod
|
||||
def submit(
|
||||
cls,
|
||||
agent_name: str,
|
||||
agent_role: str,
|
||||
task_description: str,
|
||||
priority: str = "normal",
|
||||
) -> str:
|
||||
"""Submit a task for execution. Returns task_id immediately.
|
||||
|
||||
The task is registered as 'queued' and a daemon thread begins
|
||||
execution in the background. Use get_status(task_id) to poll.
|
||||
"""
|
||||
task_id = uuid.uuid4().hex[:8]
|
||||
backend = cls._select_backend(agent_role, task_description)
|
||||
|
||||
record = DelegatedTask(
|
||||
task_id=task_id,
|
||||
agent_name=agent_name,
|
||||
agent_role=agent_role,
|
||||
task_description=task_description,
|
||||
priority=priority,
|
||||
backend=backend,
|
||||
)
|
||||
|
||||
with cls._lock:
|
||||
cls._tasks[task_id] = record
|
||||
|
||||
thread = threading.Thread(
|
||||
target=cls._run_task,
|
||||
args=(record,),
|
||||
daemon=True,
|
||||
name=f"worker-{task_id}",
|
||||
)
|
||||
thread.start()
|
||||
|
||||
logger.info(
|
||||
"Task %s queued: %s → %.60s (backend=%s, priority=%s)",
|
||||
task_id,
|
||||
agent_name,
|
||||
task_description,
|
||||
backend,
|
||||
priority,
|
||||
)
|
||||
return task_id
|
||||
|
||||
@classmethod
|
||||
def get_status(cls, task_id: str) -> dict[str, Any]:
|
||||
"""Return current status of a task by ID."""
|
||||
record = cls._tasks.get(task_id)
|
||||
if record is None:
|
||||
return {"found": False, "task_id": task_id}
|
||||
return {
|
||||
"found": True,
|
||||
"task_id": record.task_id,
|
||||
"agent": record.agent_name,
|
||||
"role": record.agent_role,
|
||||
"status": record.status,
|
||||
"backend": record.backend,
|
||||
"priority": record.priority,
|
||||
"created_at": record.created_at,
|
||||
"retries": record.retries,
|
||||
"result": record.result,
|
||||
"error": record.error,
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def list_tasks(cls) -> list[dict[str, Any]]:
|
||||
"""Return a summary list of all tracked tasks."""
|
||||
with cls._lock:
|
||||
return [
|
||||
{
|
||||
"task_id": t.task_id,
|
||||
"agent": t.agent_name,
|
||||
"status": t.status,
|
||||
"backend": t.backend,
|
||||
"created_at": t.created_at,
|
||||
}
|
||||
for t in cls._tasks.values()
|
||||
]
|
||||
|
||||
@classmethod
|
||||
def clear(cls) -> None:
|
||||
"""Clear the task registry (for tests)."""
|
||||
with cls._lock:
|
||||
cls._tasks.clear()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Backend selection
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
@classmethod
|
||||
def _select_backend(cls, agent_role: str, task_description: str) -> str:
|
||||
"""Choose the execution backend for a given agent role and task.
|
||||
|
||||
Priority:
|
||||
1. kimi — research role + Gitea enabled + task exceeds local capacity
|
||||
2. paperclip — paperclip API key is configured
|
||||
3. agentic_loop — local fallback (always available)
|
||||
"""
|
||||
try:
|
||||
from config import settings
|
||||
from timmy.kimi_delegation import exceeds_local_capacity
|
||||
|
||||
if (
|
||||
agent_role == "research"
|
||||
and getattr(settings, "gitea_enabled", False)
|
||||
and getattr(settings, "gitea_token", "")
|
||||
and exceeds_local_capacity(task_description)
|
||||
):
|
||||
return "kimi"
|
||||
|
||||
if getattr(settings, "paperclip_api_key", ""):
|
||||
return "paperclip"
|
||||
|
||||
except Exception as exc:
|
||||
logger.debug("Backend selection error — defaulting to agentic_loop: %s", exc)
|
||||
|
||||
return "agentic_loop"
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Task execution
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
@classmethod
|
||||
def _run_task(cls, record: DelegatedTask) -> None:
|
||||
"""Execute a task with retry logic. Runs inside a daemon thread."""
|
||||
record.status = "running"
|
||||
|
||||
for attempt in range(MAX_RETRIES + 1):
|
||||
try:
|
||||
if attempt > 0:
|
||||
logger.info(
|
||||
"Retrying task %s (attempt %d/%d)",
|
||||
record.task_id,
|
||||
attempt + 1,
|
||||
MAX_RETRIES + 1,
|
||||
)
|
||||
record.retries = attempt
|
||||
|
||||
result = cls._dispatch(record)
|
||||
record.status = "completed"
|
||||
record.result = result
|
||||
logger.info(
|
||||
"Task %s completed via %s",
|
||||
record.task_id,
|
||||
record.backend,
|
||||
)
|
||||
return
|
||||
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"Task %s attempt %d failed: %s",
|
||||
record.task_id,
|
||||
attempt + 1,
|
||||
exc,
|
||||
)
|
||||
if attempt == MAX_RETRIES:
|
||||
record.status = "failed"
|
||||
record.error = str(exc)
|
||||
logger.error(
|
||||
"Task %s exhausted %d retries. Final error: %s",
|
||||
record.task_id,
|
||||
MAX_RETRIES,
|
||||
exc,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def _dispatch(cls, record: DelegatedTask) -> dict[str, Any]:
|
||||
"""Route to the selected backend. Raises on failure."""
|
||||
if record.backend == "kimi":
|
||||
return asyncio.run(cls._execute_kimi(record))
|
||||
if record.backend == "paperclip":
|
||||
return asyncio.run(cls._execute_paperclip(record))
|
||||
return asyncio.run(cls._execute_agentic_loop(record))
|
||||
|
||||
@classmethod
|
||||
async def _execute_kimi(cls, record: DelegatedTask) -> dict[str, Any]:
|
||||
"""Create a kimi-ready Gitea issue for the task.
|
||||
|
||||
Kimi picks up the issue via the kimi-ready label and executes it.
|
||||
"""
|
||||
from timmy.kimi_delegation import create_kimi_research_issue
|
||||
|
||||
result = await create_kimi_research_issue(
|
||||
task=record.task_description[:120],
|
||||
context=f"Delegated by agent '{record.agent_name}' via delegate_task.",
|
||||
question=record.task_description,
|
||||
priority=record.priority,
|
||||
)
|
||||
if not result.get("success"):
|
||||
raise RuntimeError(f"Kimi issue creation failed: {result.get('error')}")
|
||||
return result
|
||||
|
||||
@classmethod
|
||||
async def _execute_paperclip(cls, record: DelegatedTask) -> dict[str, Any]:
|
||||
"""Submit the task to the Paperclip API."""
|
||||
import httpx
|
||||
|
||||
from timmy.paperclip import PaperclipClient
|
||||
|
||||
client = PaperclipClient()
|
||||
async with httpx.AsyncClient(timeout=client.timeout) as http:
|
||||
resp = await http.post(
|
||||
f"{client.base_url}/api/tasks",
|
||||
headers={"Authorization": f"Bearer {client.api_key}"},
|
||||
json={
|
||||
"kind": record.agent_role,
|
||||
"agent_id": client.agent_id,
|
||||
"company_id": client.company_id,
|
||||
"priority": record.priority,
|
||||
"context": {"task": record.task_description},
|
||||
},
|
||||
)
|
||||
|
||||
if resp.status_code in (200, 201):
|
||||
data = resp.json()
|
||||
logger.info(
|
||||
"Task %s submitted to Paperclip (paperclip_id=%s)",
|
||||
record.task_id,
|
||||
data.get("id"),
|
||||
)
|
||||
return {
|
||||
"success": True,
|
||||
"paperclip_task_id": data.get("id"),
|
||||
"backend": "paperclip",
|
||||
}
|
||||
raise RuntimeError(f"Paperclip API error {resp.status_code}: {resp.text[:200]}")
|
||||
|
||||
@classmethod
|
||||
async def _execute_agentic_loop(cls, record: DelegatedTask) -> dict[str, Any]:
|
||||
"""Execute the task via Timmy's local agentic loop."""
|
||||
from timmy.agentic_loop import run_agentic_loop
|
||||
|
||||
result = await run_agentic_loop(record.task_description)
|
||||
return {
|
||||
"success": result.status != "failed",
|
||||
"agentic_task_id": result.task_id,
|
||||
"summary": result.summary,
|
||||
"status": result.status,
|
||||
"backend": "agentic_loop",
|
||||
}
|
||||
@@ -1,3 +1,8 @@
|
||||
"""Central pydantic-settings configuration for Timmy Time Dashboard.
|
||||
|
||||
All environment variable access goes through the ``settings`` singleton
|
||||
exported from this module — never use ``os.environ.get()`` in app code.
|
||||
"""
|
||||
import logging as _logging
|
||||
import os
|
||||
import sys
|
||||
@@ -51,6 +56,13 @@ class Settings(BaseSettings):
|
||||
# Set to 0 to use model defaults.
|
||||
ollama_num_ctx: int = 32768
|
||||
|
||||
# Maximum models loaded simultaneously in Ollama — override with OLLAMA_MAX_LOADED_MODELS
|
||||
# Set to 2 so Qwen3-8B and Qwen3-14B can stay hot concurrently (~17 GB combined).
|
||||
# Requires Ollama ≥ 0.1.33. Export this to the Ollama process environment:
|
||||
# OLLAMA_MAX_LOADED_MODELS=2 ollama serve
|
||||
# or add it to your systemd/launchd unit before starting the harness.
|
||||
ollama_max_loaded_models: int = 2
|
||||
|
||||
# Fallback model chains — override with FALLBACK_MODELS / VISION_FALLBACK_MODELS
|
||||
# as comma-separated strings, e.g. FALLBACK_MODELS="qwen3:8b,qwen2.5:14b"
|
||||
# Or edit config/providers.yaml → fallback_chains for the canonical source.
|
||||
@@ -87,8 +99,9 @@ class Settings(BaseSettings):
|
||||
|
||||
# ── Backend selection ────────────────────────────────────────────────────
|
||||
# "ollama" — always use Ollama (default, safe everywhere)
|
||||
# "airllm" — AirLLM layer-by-layer loading (Apple Silicon only; degrades to Ollama)
|
||||
# "auto" — pick best available local backend, fall back to Ollama
|
||||
timmy_model_backend: Literal["ollama", "grok", "claude", "auto"] = "ollama"
|
||||
timmy_model_backend: Literal["ollama", "airllm", "grok", "claude", "auto"] = "ollama"
|
||||
|
||||
# ── Grok (xAI) — opt-in premium cloud backend ────────────────────────
|
||||
# Grok is a premium augmentation layer — local-first ethos preserved.
|
||||
@@ -101,6 +114,16 @@ class Settings(BaseSettings):
|
||||
grok_sats_hard_cap: int = 100 # Absolute ceiling on sats per Grok query
|
||||
grok_free: bool = False # Skip Lightning invoice when user has own API key
|
||||
|
||||
# ── Search Backend (SearXNG + Crawl4AI) ──────────────────────────────
|
||||
# "searxng" — self-hosted SearXNG meta-search engine (default, no API key)
|
||||
# "none" — disable web search (private/offline deployments)
|
||||
# Override with TIMMY_SEARCH_BACKEND env var.
|
||||
timmy_search_backend: Literal["searxng", "none"] = "searxng"
|
||||
# SearXNG base URL — override with TIMMY_SEARCH_URL env var
|
||||
search_url: str = "http://localhost:8888"
|
||||
# Crawl4AI base URL — override with TIMMY_CRAWL_URL env var
|
||||
crawl_url: str = "http://localhost:11235"
|
||||
|
||||
# ── Database ──────────────────────────────────────────────────────────
|
||||
db_busy_timeout_ms: int = 5000 # SQLite PRAGMA busy_timeout (ms)
|
||||
|
||||
@@ -110,6 +133,23 @@ class Settings(BaseSettings):
|
||||
anthropic_api_key: str = ""
|
||||
claude_model: str = "haiku"
|
||||
|
||||
# ── Tiered Model Router (issue #882) ─────────────────────────────────
|
||||
# Three-tier cascade: Local 8B (free, fast) → Local 70B (free, slower)
|
||||
# → Cloud API (paid, best). Override model names per tier via env vars.
|
||||
#
|
||||
# TIER_LOCAL_FAST_MODEL — Tier-1 model name in Ollama (default: llama3.1:8b)
|
||||
# TIER_LOCAL_HEAVY_MODEL — Tier-2 model name in Ollama (default: hermes3:70b)
|
||||
# TIER_CLOUD_MODEL — Tier-3 cloud model name (default: claude-haiku-4-5)
|
||||
#
|
||||
# Budget limits for the cloud tier (0 = unlimited):
|
||||
# TIER_CLOUD_DAILY_BUDGET_USD — daily ceiling in USD (default: 5.0)
|
||||
# TIER_CLOUD_MONTHLY_BUDGET_USD — monthly ceiling in USD (default: 50.0)
|
||||
tier_local_fast_model: str = "llama3.1:8b"
|
||||
tier_local_heavy_model: str = "hermes3:70b"
|
||||
tier_cloud_model: str = "claude-haiku-4-5"
|
||||
tier_cloud_daily_budget_usd: float = 5.0
|
||||
tier_cloud_monthly_budget_usd: float = 50.0
|
||||
|
||||
# ── Content Moderation ──────────────────────────────────────────────
|
||||
# Three-layer moderation pipeline for AI narrator output.
|
||||
# Uses Llama Guard via Ollama with regex fallback.
|
||||
@@ -228,6 +268,10 @@ class Settings(BaseSettings):
|
||||
# ── Test / Diagnostics ─────────────────────────────────────────────
|
||||
# Skip loading heavy embedding models (for tests / low-memory envs).
|
||||
timmy_skip_embeddings: bool = False
|
||||
# Embedding backend: "ollama" for Ollama, "local" for sentence-transformers.
|
||||
timmy_embedding_backend: Literal["ollama", "local"] = "local"
|
||||
# Ollama model to use for embeddings (e.g., "nomic-embed-text").
|
||||
ollama_embedding_model: str = "nomic-embed-text"
|
||||
# Disable CSRF middleware entirely (for tests).
|
||||
timmy_disable_csrf: bool = False
|
||||
# Mark the process as running in test mode.
|
||||
@@ -376,6 +420,11 @@ class Settings(BaseSettings):
|
||||
autoresearch_time_budget: int = 300 # seconds per experiment run
|
||||
autoresearch_max_iterations: int = 100
|
||||
autoresearch_metric: str = "val_bpb" # metric to optimise (lower = better)
|
||||
# M3 Max / Apple Silicon tuning (Issue #905).
|
||||
# dataset: "tinystories" (default, lower-entropy, recommended for Mac) or "openwebtext".
|
||||
autoresearch_dataset: str = "tinystories"
|
||||
# backend: "auto" detects MLX on Apple Silicon; "cpu" forces CPU fallback.
|
||||
autoresearch_backend: str = "auto"
|
||||
|
||||
# ── Weekly Narrative Summary ───────────────────────────────────────
|
||||
# Generates a human-readable weekly summary of development activity.
|
||||
@@ -406,6 +455,14 @@ class Settings(BaseSettings):
|
||||
# Alert threshold: free disk below this triggers cleanup / alert (GB).
|
||||
hermes_disk_free_min_gb: float = 10.0
|
||||
|
||||
# ── Energy Budget Monitoring ───────────────────────────────────────
|
||||
# Enable energy budget monitoring (tracks CPU/GPU power during inference).
|
||||
energy_budget_enabled: bool = True
|
||||
# Watts threshold that auto-activates low power mode (on-battery only).
|
||||
energy_budget_watts_threshold: float = 15.0
|
||||
# Model to prefer in low power mode (smaller = more efficient).
|
||||
energy_low_power_model: str = "qwen3:1b"
|
||||
|
||||
# ── Error Logging ─────────────────────────────────────────────────
|
||||
error_log_enabled: bool = True
|
||||
error_log_dir: str = "logs"
|
||||
|
||||
@@ -37,6 +37,7 @@ from dashboard.routes.db_explorer import router as db_explorer_router
|
||||
from dashboard.routes.discord import router as discord_router
|
||||
from dashboard.routes.experiments import router as experiments_router
|
||||
from dashboard.routes.grok import router as grok_router
|
||||
from dashboard.routes.energy import router as energy_router
|
||||
from dashboard.routes.health import router as health_router
|
||||
from dashboard.routes.hermes import router as hermes_router
|
||||
from dashboard.routes.loop_qa import router as loop_qa_router
|
||||
@@ -44,6 +45,7 @@ from dashboard.routes.memory import router as memory_router
|
||||
from dashboard.routes.mobile import router as mobile_router
|
||||
from dashboard.routes.models import api_router as models_api_router
|
||||
from dashboard.routes.models import router as models_router
|
||||
from dashboard.routes.nexus import router as nexus_router
|
||||
from dashboard.routes.quests import router as quests_router
|
||||
from dashboard.routes.scorecards import router as scorecards_router
|
||||
from dashboard.routes.sovereignty_metrics import router as sovereignty_metrics_router
|
||||
@@ -53,6 +55,8 @@ from dashboard.routes.system import router as system_router
|
||||
from dashboard.routes.tasks import router as tasks_router
|
||||
from dashboard.routes.telegram import router as telegram_router
|
||||
from dashboard.routes.thinking import router as thinking_router
|
||||
from dashboard.routes.self_correction import router as self_correction_router
|
||||
from dashboard.routes.three_strike import router as three_strike_router
|
||||
from dashboard.routes.tools import router as tools_router
|
||||
from dashboard.routes.tower import router as tower_router
|
||||
from dashboard.routes.voice import router as voice_router
|
||||
@@ -548,12 +552,28 @@ async def lifespan(app: FastAPI):
|
||||
except Exception:
|
||||
logger.debug("Failed to register error recorder")
|
||||
|
||||
# Mark session start for sovereignty duration tracking
|
||||
try:
|
||||
from timmy.sovereignty import mark_session_start
|
||||
|
||||
mark_session_start()
|
||||
except Exception:
|
||||
logger.debug("Failed to mark sovereignty session start")
|
||||
|
||||
logger.info("✓ Dashboard ready for requests")
|
||||
|
||||
yield
|
||||
|
||||
await _shutdown_cleanup(bg_tasks, workshop_heartbeat)
|
||||
|
||||
# Generate and commit sovereignty session report
|
||||
try:
|
||||
from timmy.sovereignty import generate_and_commit_report
|
||||
|
||||
await generate_and_commit_report()
|
||||
except Exception as exc:
|
||||
logger.warning("Sovereignty report generation failed at shutdown: %s", exc)
|
||||
|
||||
|
||||
app = FastAPI(
|
||||
title="Mission Control",
|
||||
@@ -652,6 +672,7 @@ app.include_router(tools_router)
|
||||
app.include_router(spark_router)
|
||||
app.include_router(discord_router)
|
||||
app.include_router(memory_router)
|
||||
app.include_router(nexus_router)
|
||||
app.include_router(grok_router)
|
||||
app.include_router(models_router)
|
||||
app.include_router(models_api_router)
|
||||
@@ -670,10 +691,13 @@ app.include_router(matrix_router)
|
||||
app.include_router(tower_router)
|
||||
app.include_router(daily_run_router)
|
||||
app.include_router(hermes_router)
|
||||
app.include_router(energy_router)
|
||||
app.include_router(quests_router)
|
||||
app.include_router(scorecards_router)
|
||||
app.include_router(sovereignty_metrics_router)
|
||||
app.include_router(sovereignty_ws_router)
|
||||
app.include_router(three_strike_router)
|
||||
app.include_router(self_correction_router)
|
||||
|
||||
|
||||
@app.websocket("/ws")
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
"""SQLAlchemy ORM models for the CALM task-management and journaling system."""
|
||||
from datetime import UTC, date, datetime
|
||||
from enum import StrEnum
|
||||
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
"""SQLAlchemy engine, session factory, and declarative Base for the CALM module."""
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
"""Dashboard routes for agent chat interactions and tool-call display."""
|
||||
import json
|
||||
import logging
|
||||
from datetime import datetime
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
"""Dashboard routes for the CALM task management and daily journaling interface."""
|
||||
import logging
|
||||
from datetime import UTC, date, datetime
|
||||
|
||||
|
||||
121
src/dashboard/routes/energy.py
Normal file
121
src/dashboard/routes/energy.py
Normal file
@@ -0,0 +1,121 @@
|
||||
"""Energy Budget Monitoring routes.
|
||||
|
||||
Exposes the energy budget monitor via REST API so the dashboard and
|
||||
external tools can query power draw, efficiency scores, and toggle
|
||||
low power mode.
|
||||
|
||||
Refs: #1009
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
from config import settings
|
||||
from infrastructure.energy.monitor import energy_monitor
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/energy", tags=["energy"])
|
||||
|
||||
|
||||
class LowPowerRequest(BaseModel):
|
||||
"""Request body for toggling low power mode."""
|
||||
|
||||
enabled: bool
|
||||
|
||||
|
||||
class InferenceEventRequest(BaseModel):
|
||||
"""Request body for recording an inference event."""
|
||||
|
||||
model: str
|
||||
tokens_per_second: float
|
||||
|
||||
|
||||
@router.get("/status")
|
||||
async def energy_status():
|
||||
"""Return the current energy budget status.
|
||||
|
||||
Returns the live power estimate, efficiency score (0–10), recent
|
||||
inference samples, and whether low power mode is active.
|
||||
"""
|
||||
if not getattr(settings, "energy_budget_enabled", True):
|
||||
return {
|
||||
"enabled": False,
|
||||
"message": "Energy budget monitoring is disabled (ENERGY_BUDGET_ENABLED=false)",
|
||||
}
|
||||
|
||||
report = await energy_monitor.get_report()
|
||||
return {**report.to_dict(), "enabled": True}
|
||||
|
||||
|
||||
@router.get("/report")
|
||||
async def energy_report():
|
||||
"""Detailed energy budget report with all recent samples.
|
||||
|
||||
Same as /energy/status but always includes the full sample history.
|
||||
"""
|
||||
if not getattr(settings, "energy_budget_enabled", True):
|
||||
raise HTTPException(status_code=503, detail="Energy budget monitoring is disabled")
|
||||
|
||||
report = await energy_monitor.get_report()
|
||||
data = report.to_dict()
|
||||
# Override recent_samples to include the full window (not just last 10)
|
||||
data["recent_samples"] = [
|
||||
{
|
||||
"timestamp": s.timestamp,
|
||||
"model": s.model,
|
||||
"tokens_per_second": round(s.tokens_per_second, 1),
|
||||
"estimated_watts": round(s.estimated_watts, 2),
|
||||
"efficiency": round(s.efficiency, 3),
|
||||
"efficiency_score": round(s.efficiency_score, 2),
|
||||
}
|
||||
for s in list(energy_monitor._samples)
|
||||
]
|
||||
return {**data, "enabled": True}
|
||||
|
||||
|
||||
@router.post("/low-power")
|
||||
async def set_low_power_mode(body: LowPowerRequest):
|
||||
"""Enable or disable low power mode.
|
||||
|
||||
In low power mode the cascade router is advised to prefer the
|
||||
configured energy_low_power_model (see settings).
|
||||
"""
|
||||
if not getattr(settings, "energy_budget_enabled", True):
|
||||
raise HTTPException(status_code=503, detail="Energy budget monitoring is disabled")
|
||||
|
||||
energy_monitor.set_low_power_mode(body.enabled)
|
||||
low_power_model = getattr(settings, "energy_low_power_model", "qwen3:1b")
|
||||
return {
|
||||
"low_power_mode": body.enabled,
|
||||
"preferred_model": low_power_model if body.enabled else None,
|
||||
"message": (
|
||||
f"Low power mode {'enabled' if body.enabled else 'disabled'}. "
|
||||
+ (f"Routing to {low_power_model}." if body.enabled else "Routing restored to default.")
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/record")
|
||||
async def record_inference_event(body: InferenceEventRequest):
|
||||
"""Record an inference event for efficiency tracking.
|
||||
|
||||
Called after each LLM inference completes. Updates the rolling
|
||||
efficiency score and may auto-activate low power mode if watts
|
||||
exceed the configured threshold.
|
||||
"""
|
||||
if not getattr(settings, "energy_budget_enabled", True):
|
||||
return {"recorded": False, "message": "Energy budget monitoring is disabled"}
|
||||
|
||||
if body.tokens_per_second <= 0:
|
||||
raise HTTPException(status_code=422, detail="tokens_per_second must be positive")
|
||||
|
||||
sample = energy_monitor.record_inference(body.model, body.tokens_per_second)
|
||||
return {
|
||||
"recorded": True,
|
||||
"efficiency_score": round(sample.efficiency_score, 2),
|
||||
"estimated_watts": round(sample.estimated_watts, 2),
|
||||
"low_power_mode": energy_monitor.low_power_mode,
|
||||
}
|
||||
166
src/dashboard/routes/nexus.py
Normal file
166
src/dashboard/routes/nexus.py
Normal file
@@ -0,0 +1,166 @@
|
||||
"""Nexus — Timmy's persistent conversational awareness space.
|
||||
|
||||
A conversational-only interface where Timmy maintains live memory context.
|
||||
No tool use; pure conversation with memory integration and a teaching panel.
|
||||
|
||||
Routes:
|
||||
GET /nexus — render nexus page with live memory sidebar
|
||||
POST /nexus/chat — send a message; returns HTMX partial
|
||||
POST /nexus/teach — inject a fact into Timmy's live memory
|
||||
DELETE /nexus/history — clear the nexus conversation history
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from fastapi import APIRouter, Form, Request
|
||||
from fastapi.responses import HTMLResponse
|
||||
|
||||
from dashboard.templating import templates
|
||||
from timmy.memory_system import (
|
||||
get_memory_stats,
|
||||
recall_personal_facts_with_ids,
|
||||
search_memories,
|
||||
store_personal_fact,
|
||||
)
|
||||
from timmy.session import _clean_response, chat, reset_session
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/nexus", tags=["nexus"])
|
||||
|
||||
_NEXUS_SESSION_ID = "nexus"
|
||||
_MAX_MESSAGE_LENGTH = 10_000
|
||||
|
||||
# In-memory conversation log for the Nexus session (mirrors chat store pattern
|
||||
# but is scoped to the Nexus so it won't pollute the main dashboard history).
|
||||
_nexus_log: list[dict] = []
|
||||
|
||||
|
||||
def _ts() -> str:
|
||||
return datetime.now(UTC).strftime("%H:%M:%S")
|
||||
|
||||
|
||||
def _append_log(role: str, content: str) -> None:
|
||||
_nexus_log.append({"role": role, "content": content, "timestamp": _ts()})
|
||||
# Keep last 200 exchanges to bound memory usage
|
||||
if len(_nexus_log) > 200:
|
||||
del _nexus_log[:-200]
|
||||
|
||||
|
||||
@router.get("", response_class=HTMLResponse)
|
||||
async def nexus_page(request: Request):
|
||||
"""Render the Nexus page with live memory context."""
|
||||
stats = get_memory_stats()
|
||||
facts = recall_personal_facts_with_ids()[:8]
|
||||
|
||||
return templates.TemplateResponse(
|
||||
request,
|
||||
"nexus.html",
|
||||
{
|
||||
"page_title": "Nexus",
|
||||
"messages": list(_nexus_log),
|
||||
"stats": stats,
|
||||
"facts": facts,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.post("/chat", response_class=HTMLResponse)
|
||||
async def nexus_chat(request: Request, message: str = Form(...)):
|
||||
"""Conversational-only chat routed through the Nexus session.
|
||||
|
||||
Does not invoke tool-use approval flow — pure conversation with memory
|
||||
context injected from Timmy's live memory store.
|
||||
"""
|
||||
message = message.strip()
|
||||
if not message:
|
||||
return HTMLResponse("")
|
||||
if len(message) > _MAX_MESSAGE_LENGTH:
|
||||
return templates.TemplateResponse(
|
||||
request,
|
||||
"partials/nexus_message.html",
|
||||
{
|
||||
"user_message": message[:80] + "…",
|
||||
"response": None,
|
||||
"error": "Message too long (max 10 000 chars).",
|
||||
"timestamp": _ts(),
|
||||
"memory_hits": [],
|
||||
},
|
||||
)
|
||||
|
||||
ts = _ts()
|
||||
|
||||
# Fetch semantically relevant memories to surface in the sidebar
|
||||
try:
|
||||
memory_hits = await asyncio.to_thread(search_memories, query=message, limit=4)
|
||||
except Exception as exc:
|
||||
logger.warning("Nexus memory search failed: %s", exc)
|
||||
memory_hits = []
|
||||
|
||||
# Conversational response — no tool approval flow
|
||||
response_text: str | None = None
|
||||
error_text: str | None = None
|
||||
try:
|
||||
raw = await chat(message, session_id=_NEXUS_SESSION_ID)
|
||||
response_text = _clean_response(raw)
|
||||
except Exception as exc:
|
||||
logger.error("Nexus chat error: %s", exc)
|
||||
error_text = "Timmy is unavailable right now. Check that Ollama is running."
|
||||
|
||||
_append_log("user", message)
|
||||
if response_text:
|
||||
_append_log("assistant", response_text)
|
||||
|
||||
return templates.TemplateResponse(
|
||||
request,
|
||||
"partials/nexus_message.html",
|
||||
{
|
||||
"user_message": message,
|
||||
"response": response_text,
|
||||
"error": error_text,
|
||||
"timestamp": ts,
|
||||
"memory_hits": memory_hits,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.post("/teach", response_class=HTMLResponse)
|
||||
async def nexus_teach(request: Request, fact: str = Form(...)):
|
||||
"""Inject a fact into Timmy's live memory from the Nexus teaching panel."""
|
||||
fact = fact.strip()
|
||||
if not fact:
|
||||
return HTMLResponse("")
|
||||
|
||||
try:
|
||||
await asyncio.to_thread(store_personal_fact, fact)
|
||||
facts = await asyncio.to_thread(recall_personal_facts_with_ids)
|
||||
facts = facts[:8]
|
||||
except Exception as exc:
|
||||
logger.error("Nexus teach error: %s", exc)
|
||||
facts = []
|
||||
|
||||
return templates.TemplateResponse(
|
||||
request,
|
||||
"partials/nexus_facts.html",
|
||||
{"facts": facts, "taught": fact},
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/history", response_class=HTMLResponse)
|
||||
async def nexus_clear_history(request: Request):
|
||||
"""Clear the Nexus conversation history."""
|
||||
_nexus_log.clear()
|
||||
reset_session(session_id=_NEXUS_SESSION_ID)
|
||||
return templates.TemplateResponse(
|
||||
request,
|
||||
"partials/nexus_message.html",
|
||||
{
|
||||
"user_message": None,
|
||||
"response": "Nexus conversation cleared.",
|
||||
"error": None,
|
||||
"timestamp": _ts(),
|
||||
"memory_hits": [],
|
||||
},
|
||||
)
|
||||
58
src/dashboard/routes/self_correction.py
Normal file
58
src/dashboard/routes/self_correction.py
Normal file
@@ -0,0 +1,58 @@
|
||||
"""Self-Correction Dashboard routes.
|
||||
|
||||
GET /self-correction/ui — HTML dashboard
|
||||
GET /self-correction/timeline — HTMX partial: recent event timeline
|
||||
GET /self-correction/patterns — HTMX partial: recurring failure patterns
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
from fastapi.responses import HTMLResponse
|
||||
|
||||
from dashboard.templating import templates
|
||||
from infrastructure.self_correction import get_corrections, get_patterns, get_stats
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/self-correction", tags=["self-correction"])
|
||||
|
||||
|
||||
@router.get("/ui", response_class=HTMLResponse)
|
||||
async def self_correction_ui(request: Request):
|
||||
"""Render the Self-Correction Dashboard."""
|
||||
stats = get_stats()
|
||||
corrections = get_corrections(limit=20)
|
||||
patterns = get_patterns(top_n=10)
|
||||
return templates.TemplateResponse(
|
||||
request,
|
||||
"self_correction.html",
|
||||
{
|
||||
"stats": stats,
|
||||
"corrections": corrections,
|
||||
"patterns": patterns,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/timeline", response_class=HTMLResponse)
|
||||
async def self_correction_timeline(request: Request):
|
||||
"""HTMX partial: recent self-correction event timeline."""
|
||||
corrections = get_corrections(limit=30)
|
||||
return templates.TemplateResponse(
|
||||
request,
|
||||
"partials/self_correction_timeline.html",
|
||||
{"corrections": corrections},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/patterns", response_class=HTMLResponse)
|
||||
async def self_correction_patterns(request: Request):
|
||||
"""HTMX partial: recurring failure patterns."""
|
||||
patterns = get_patterns(top_n=10)
|
||||
stats = get_stats()
|
||||
return templates.TemplateResponse(
|
||||
request,
|
||||
"partials/self_correction_patterns.html",
|
||||
{"patterns": patterns, "stats": stats},
|
||||
)
|
||||
116
src/dashboard/routes/three_strike.py
Normal file
116
src/dashboard/routes/three_strike.py
Normal file
@@ -0,0 +1,116 @@
|
||||
"""Three-Strike Detector dashboard routes.
|
||||
|
||||
Provides JSON API endpoints for inspecting and managing the three-strike
|
||||
detector state.
|
||||
|
||||
Refs: #962
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
from timmy.sovereignty.three_strike import CATEGORIES, get_detector
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/sovereignty/three-strike", tags=["three-strike"])
|
||||
|
||||
|
||||
class RecordRequest(BaseModel):
|
||||
category: str
|
||||
key: str
|
||||
metadata: dict[str, Any] = {}
|
||||
|
||||
|
||||
class AutomationRequest(BaseModel):
|
||||
artifact_path: str
|
||||
|
||||
|
||||
@router.get("")
|
||||
async def list_strikes() -> dict[str, Any]:
|
||||
"""Return all strike records."""
|
||||
detector = get_detector()
|
||||
records = detector.list_all()
|
||||
return {
|
||||
"records": [
|
||||
{
|
||||
"category": r.category,
|
||||
"key": r.key,
|
||||
"count": r.count,
|
||||
"blocked": r.blocked,
|
||||
"automation": r.automation,
|
||||
"first_seen": r.first_seen,
|
||||
"last_seen": r.last_seen,
|
||||
}
|
||||
for r in records
|
||||
],
|
||||
"categories": sorted(CATEGORIES),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/blocked")
|
||||
async def list_blocked() -> dict[str, Any]:
|
||||
"""Return only blocked (category, key) pairs."""
|
||||
detector = get_detector()
|
||||
records = detector.list_blocked()
|
||||
return {
|
||||
"blocked": [
|
||||
{
|
||||
"category": r.category,
|
||||
"key": r.key,
|
||||
"count": r.count,
|
||||
"automation": r.automation,
|
||||
"last_seen": r.last_seen,
|
||||
}
|
||||
for r in records
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
@router.post("/record")
|
||||
async def record_strike(body: RecordRequest) -> dict[str, Any]:
|
||||
"""Record a manual action. Returns strike state; 409 when blocked."""
|
||||
from timmy.sovereignty.three_strike import ThreeStrikeError
|
||||
|
||||
detector = get_detector()
|
||||
try:
|
||||
record = detector.record(body.category, body.key, body.metadata)
|
||||
return {
|
||||
"category": record.category,
|
||||
"key": record.key,
|
||||
"count": record.count,
|
||||
"blocked": record.blocked,
|
||||
"automation": record.automation,
|
||||
}
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
except ThreeStrikeError as exc:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail={
|
||||
"error": "three_strike_block",
|
||||
"message": str(exc),
|
||||
"category": exc.category,
|
||||
"key": exc.key,
|
||||
"count": exc.count,
|
||||
},
|
||||
) from exc
|
||||
|
||||
|
||||
@router.post("/{category}/{key}/automation")
|
||||
async def register_automation(category: str, key: str, body: AutomationRequest) -> dict[str, bool]:
|
||||
"""Register an automation artifact to unblock a (category, key) pair."""
|
||||
detector = get_detector()
|
||||
detector.register_automation(category, key, body.artifact_path)
|
||||
return {"success": True}
|
||||
|
||||
|
||||
@router.get("/{category}/{key}/events")
|
||||
async def get_strike_events(category: str, key: str, limit: int = 50) -> dict[str, Any]:
|
||||
"""Return the individual strike events for a (category, key) pair."""
|
||||
detector = get_detector()
|
||||
events = detector.get_events(category, key, limit=limit)
|
||||
return {"category": category, "key": key, "events": events}
|
||||
@@ -67,9 +67,11 @@
|
||||
<div class="mc-nav-dropdown">
|
||||
<button class="mc-test-link mc-dropdown-toggle" aria-expanded="false">INTEL ▾</button>
|
||||
<div class="mc-dropdown-menu">
|
||||
<a href="/nexus" class="mc-test-link">NEXUS</a>
|
||||
<a href="/spark/ui" class="mc-test-link">SPARK</a>
|
||||
<a href="/memory" class="mc-test-link">MEMORY</a>
|
||||
<a href="/marketplace/ui" class="mc-test-link">MARKET</a>
|
||||
<a href="/self-correction/ui" class="mc-test-link">SELF-CORRECT</a>
|
||||
</div>
|
||||
</div>
|
||||
<div class="mc-nav-dropdown">
|
||||
@@ -131,6 +133,7 @@
|
||||
<a href="/spark/ui" class="mc-mobile-link">SPARK</a>
|
||||
<a href="/memory" class="mc-mobile-link">MEMORY</a>
|
||||
<a href="/marketplace/ui" class="mc-mobile-link">MARKET</a>
|
||||
<a href="/self-correction/ui" class="mc-mobile-link">SELF-CORRECT</a>
|
||||
<div class="mc-mobile-section-label">AGENTS</div>
|
||||
<a href="/hands" class="mc-mobile-link">HANDS</a>
|
||||
<a href="/work-orders/queue" class="mc-mobile-link">WORK ORDERS</a>
|
||||
|
||||
@@ -186,6 +186,24 @@
|
||||
<p class="chat-history-placeholder">Loading sovereignty metrics...</p>
|
||||
{% endcall %}
|
||||
|
||||
<!-- Agent Scorecards -->
|
||||
<div class="card mc-card-spaced" id="mc-scorecards-card">
|
||||
<div class="card-header">
|
||||
<h2 class="card-title">Agent Scorecards</h2>
|
||||
<div class="d-flex align-items-center gap-2">
|
||||
<select id="mc-scorecard-period" class="form-select form-select-sm" style="width: auto;"
|
||||
onchange="loadMcScorecards()">
|
||||
<option value="daily" selected>Daily</option>
|
||||
<option value="weekly">Weekly</option>
|
||||
</select>
|
||||
<a href="/scorecards" class="btn btn-sm btn-outline-secondary">Full View</a>
|
||||
</div>
|
||||
</div>
|
||||
<div id="mc-scorecards-content" class="p-2">
|
||||
<p class="chat-history-placeholder">Loading scorecards...</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Chat History -->
|
||||
<div class="card mc-card-spaced">
|
||||
<div class="card-header">
|
||||
@@ -502,6 +520,20 @@ async function loadSparkStatus() {
|
||||
}
|
||||
}
|
||||
|
||||
// Load agent scorecards
|
||||
async function loadMcScorecards() {
|
||||
var period = document.getElementById('mc-scorecard-period').value;
|
||||
var container = document.getElementById('mc-scorecards-content');
|
||||
container.innerHTML = '<p class="chat-history-placeholder">Loading scorecards...</p>';
|
||||
try {
|
||||
var response = await fetch('/scorecards/all/panels?period=' + period);
|
||||
var html = await response.text();
|
||||
container.innerHTML = html;
|
||||
} catch (error) {
|
||||
container.innerHTML = '<p class="chat-history-placeholder">Scorecards unavailable</p>';
|
||||
}
|
||||
}
|
||||
|
||||
// Initial load
|
||||
loadSparkStatus();
|
||||
loadSovereignty();
|
||||
@@ -510,6 +542,7 @@ loadSwarmStats();
|
||||
loadLightningStats();
|
||||
loadGrokStats();
|
||||
loadChatHistory();
|
||||
loadMcScorecards();
|
||||
|
||||
// Periodic updates
|
||||
setInterval(loadSovereignty, 30000);
|
||||
@@ -518,5 +551,6 @@ setInterval(loadSwarmStats, 5000);
|
||||
setInterval(updateHeartbeat, 5000);
|
||||
setInterval(loadGrokStats, 10000);
|
||||
setInterval(loadSparkStatus, 15000);
|
||||
setInterval(loadMcScorecards, 300000);
|
||||
</script>
|
||||
{% endblock %}
|
||||
|
||||
122
src/dashboard/templates/nexus.html
Normal file
122
src/dashboard/templates/nexus.html
Normal file
@@ -0,0 +1,122 @@
|
||||
{% extends "base.html" %}
|
||||
|
||||
{% block title %}Nexus{% endblock %}
|
||||
|
||||
{% block extra_styles %}{% endblock %}
|
||||
|
||||
{% block content %}
|
||||
<div class="container-fluid nexus-layout py-3">
|
||||
|
||||
<div class="nexus-header mb-3">
|
||||
<div class="nexus-title">// NEXUS</div>
|
||||
<div class="nexus-subtitle">
|
||||
Persistent conversational awareness — always present, always learning.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="nexus-grid">
|
||||
|
||||
<!-- ── LEFT: Conversation ────────────────────────────────── -->
|
||||
<div class="nexus-chat-col">
|
||||
<div class="card mc-panel nexus-chat-panel">
|
||||
<div class="card-header mc-panel-header d-flex justify-content-between align-items-center">
|
||||
<span>// CONVERSATION</span>
|
||||
<button class="mc-btn mc-btn-sm"
|
||||
hx-delete="/nexus/history"
|
||||
hx-target="#nexus-chat-log"
|
||||
hx-swap="beforeend"
|
||||
hx-confirm="Clear nexus conversation?">
|
||||
CLEAR
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div class="card-body p-2" id="nexus-chat-log">
|
||||
{% for msg in messages %}
|
||||
<div class="chat-message {{ 'user' if msg.role == 'user' else 'agent' }}">
|
||||
<div class="msg-meta">
|
||||
{{ 'YOU' if msg.role == 'user' else 'TIMMY' }} // {{ msg.timestamp }}
|
||||
</div>
|
||||
<div class="msg-body {% if msg.role == 'assistant' %}timmy-md{% endif %}">
|
||||
{{ msg.content | e }}
|
||||
</div>
|
||||
</div>
|
||||
{% else %}
|
||||
<div class="nexus-empty-state">
|
||||
Nexus is ready. Start a conversation — memories will surface in real time.
|
||||
</div>
|
||||
{% endfor %}
|
||||
</div>
|
||||
|
||||
<div class="card-footer p-2">
|
||||
<form hx-post="/nexus/chat"
|
||||
hx-target="#nexus-chat-log"
|
||||
hx-swap="beforeend"
|
||||
hx-on::after-request="this.reset(); document.getElementById('nexus-chat-log').scrollTop = 999999;">
|
||||
<div class="d-flex gap-2">
|
||||
<input type="text"
|
||||
name="message"
|
||||
id="nexus-input"
|
||||
class="mc-search-input flex-grow-1"
|
||||
placeholder="Talk to Timmy..."
|
||||
autocomplete="off"
|
||||
required>
|
||||
<button type="submit" class="mc-btn mc-btn-primary">SEND</button>
|
||||
</div>
|
||||
</form>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- ── RIGHT: Memory sidebar ─────────────────────────────── -->
|
||||
<div class="nexus-sidebar-col">
|
||||
|
||||
<!-- Live memory context (updated with each response) -->
|
||||
<div class="card mc-panel nexus-memory-panel mb-3">
|
||||
<div class="card-header mc-panel-header">
|
||||
<span>// LIVE MEMORY</span>
|
||||
<span class="badge ms-2" style="background:var(--purple-dim); color:var(--purple);">
|
||||
{{ stats.total_entries }} stored
|
||||
</span>
|
||||
</div>
|
||||
<div class="card-body p-2">
|
||||
<div id="nexus-memory-panel" class="nexus-memory-hits">
|
||||
<div class="nexus-memory-label">Relevant memories appear here as you chat.</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Teaching panel -->
|
||||
<div class="card mc-panel nexus-teach-panel">
|
||||
<div class="card-header mc-panel-header">// TEACH TIMMY</div>
|
||||
<div class="card-body p-2">
|
||||
<form hx-post="/nexus/teach"
|
||||
hx-target="#nexus-teach-response"
|
||||
hx-swap="innerHTML"
|
||||
hx-on::after-request="this.reset()">
|
||||
<div class="d-flex gap-2 mb-2">
|
||||
<input type="text"
|
||||
name="fact"
|
||||
class="mc-search-input flex-grow-1"
|
||||
placeholder="e.g. I prefer dark themes"
|
||||
required>
|
||||
<button type="submit" class="mc-btn mc-btn-primary">TEACH</button>
|
||||
</div>
|
||||
</form>
|
||||
<div id="nexus-teach-response"></div>
|
||||
|
||||
<div class="nexus-facts-header mt-3">// KNOWN FACTS</div>
|
||||
<ul class="nexus-facts-list" id="nexus-facts-list">
|
||||
{% for fact in facts %}
|
||||
<li class="nexus-fact-item">{{ fact.content | e }}</li>
|
||||
{% else %}
|
||||
<li class="nexus-fact-empty">No personal facts stored yet.</li>
|
||||
{% endfor %}
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div><!-- /sidebar -->
|
||||
</div><!-- /nexus-grid -->
|
||||
|
||||
</div>
|
||||
{% endblock %}
|
||||
12
src/dashboard/templates/partials/nexus_facts.html
Normal file
12
src/dashboard/templates/partials/nexus_facts.html
Normal file
@@ -0,0 +1,12 @@
|
||||
{% if taught %}
|
||||
<div class="nexus-taught-confirm">
|
||||
✓ Taught: <em>{{ taught | e }}</em>
|
||||
</div>
|
||||
{% endif %}
|
||||
<ul class="nexus-facts-list" id="nexus-facts-list" hx-swap-oob="true">
|
||||
{% for fact in facts %}
|
||||
<li class="nexus-fact-item">{{ fact.content | e }}</li>
|
||||
{% else %}
|
||||
<li class="nexus-fact-empty">No facts stored yet.</li>
|
||||
{% endfor %}
|
||||
</ul>
|
||||
36
src/dashboard/templates/partials/nexus_message.html
Normal file
36
src/dashboard/templates/partials/nexus_message.html
Normal file
@@ -0,0 +1,36 @@
|
||||
{% if user_message %}
|
||||
<div class="chat-message user">
|
||||
<div class="msg-meta">YOU // {{ timestamp }}</div>
|
||||
<div class="msg-body">{{ user_message | e }}</div>
|
||||
</div>
|
||||
{% endif %}
|
||||
{% if response %}
|
||||
<div class="chat-message agent">
|
||||
<div class="msg-meta">TIMMY // {{ timestamp }}</div>
|
||||
<div class="msg-body timmy-md">{{ response | e }}</div>
|
||||
</div>
|
||||
<script>
|
||||
(function() {
|
||||
var el = document.currentScript.previousElementSibling.querySelector('.timmy-md');
|
||||
if (el && typeof marked !== 'undefined' && typeof DOMPurify !== 'undefined') {
|
||||
el.innerHTML = DOMPurify.sanitize(marked.parse(el.textContent));
|
||||
}
|
||||
})();
|
||||
</script>
|
||||
{% elif error %}
|
||||
<div class="chat-message error-msg">
|
||||
<div class="msg-meta">SYSTEM // {{ timestamp }}</div>
|
||||
<div class="msg-body">{{ error | e }}</div>
|
||||
</div>
|
||||
{% endif %}
|
||||
{% if memory_hits %}
|
||||
<div class="nexus-memory-hits" id="nexus-memory-panel" hx-swap-oob="true">
|
||||
<div class="nexus-memory-label">// LIVE MEMORY CONTEXT</div>
|
||||
{% for hit in memory_hits %}
|
||||
<div class="nexus-memory-hit">
|
||||
<span class="nexus-memory-type">{{ hit.memory_type }}</span>
|
||||
<span class="nexus-memory-content">{{ hit.content | e }}</span>
|
||||
</div>
|
||||
{% endfor %}
|
||||
</div>
|
||||
{% endif %}
|
||||
@@ -0,0 +1,28 @@
|
||||
{% if patterns %}
|
||||
<table class="mc-table w-100">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>ERROR TYPE</th>
|
||||
<th class="text-center">COUNT</th>
|
||||
<th class="text-center">CORRECTED</th>
|
||||
<th class="text-center">FAILED</th>
|
||||
<th>LAST SEEN</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for p in patterns %}
|
||||
<tr>
|
||||
<td class="sc-pattern-type">{{ p.error_type }}</td>
|
||||
<td class="text-center">
|
||||
<span class="badge {% if p.count >= 5 %}badge-error{% elif p.count >= 3 %}badge-warning{% else %}badge-info{% endif %}">{{ p.count }}</span>
|
||||
</td>
|
||||
<td class="text-center text-success">{{ p.success_count }}</td>
|
||||
<td class="text-center {% if p.failed_count > 0 %}text-danger{% else %}text-muted{% endif %}">{{ p.failed_count }}</td>
|
||||
<td class="sc-event-time">{{ p.last_seen[:16] if p.last_seen else '—' }}</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
{% else %}
|
||||
<div class="text-center text-muted py-3">No patterns detected yet.</div>
|
||||
{% endif %}
|
||||
@@ -0,0 +1,26 @@
|
||||
{% if corrections %}
|
||||
{% for ev in corrections %}
|
||||
<div class="sc-event sc-status-{{ ev.outcome_status }}">
|
||||
<div class="sc-event-header">
|
||||
<span class="sc-status-badge sc-status-{{ ev.outcome_status }}">
|
||||
{% if ev.outcome_status == 'success' %}✓ CORRECTED
|
||||
{% elif ev.outcome_status == 'partial' %}● PARTIAL
|
||||
{% else %}✗ FAILED
|
||||
{% endif %}
|
||||
</span>
|
||||
<span class="sc-source-badge">{{ ev.source }}</span>
|
||||
<span class="sc-event-time">{{ ev.created_at[:19] }}</span>
|
||||
</div>
|
||||
<div class="sc-event-error-type">{{ ev.error_type }}</div>
|
||||
<div class="sc-event-intent"><span class="sc-label">INTENT:</span> {{ ev.original_intent[:120] }}{% if ev.original_intent | length > 120 %}…{% endif %}</div>
|
||||
<div class="sc-event-error"><span class="sc-label">ERROR:</span> {{ ev.detected_error[:120] }}{% if ev.detected_error | length > 120 %}…{% endif %}</div>
|
||||
<div class="sc-event-strategy"><span class="sc-label">STRATEGY:</span> {{ ev.correction_strategy[:120] }}{% if ev.correction_strategy | length > 120 %}…{% endif %}</div>
|
||||
<div class="sc-event-outcome"><span class="sc-label">OUTCOME:</span> {{ ev.final_outcome[:120] }}{% if ev.final_outcome | length > 120 %}…{% endif %}</div>
|
||||
{% if ev.task_id %}
|
||||
<div class="sc-event-meta">task: {{ ev.task_id[:8] }}</div>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endfor %}
|
||||
{% else %}
|
||||
<div class="text-center text-muted py-3">No self-correction events recorded yet.</div>
|
||||
{% endif %}
|
||||
102
src/dashboard/templates/self_correction.html
Normal file
102
src/dashboard/templates/self_correction.html
Normal file
@@ -0,0 +1,102 @@
|
||||
{% extends "base.html" %}
|
||||
{% from "macros.html" import panel %}
|
||||
|
||||
{% block title %}Timmy Time — Self-Correction Dashboard{% endblock %}
|
||||
|
||||
{% block extra_styles %}{% endblock %}
|
||||
|
||||
{% block content %}
|
||||
<div class="container-fluid py-3">
|
||||
|
||||
<!-- Header -->
|
||||
<div class="spark-header mb-3">
|
||||
<div class="spark-title">SELF-CORRECTION</div>
|
||||
<div class="spark-subtitle">
|
||||
Agent error detection & recovery —
|
||||
<span class="spark-status-val">{{ stats.total }}</span> events,
|
||||
<span class="spark-status-val">{{ stats.success_rate }}%</span> correction rate,
|
||||
<span class="spark-status-val">{{ stats.unique_error_types }}</span> distinct error types
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="row g-3">
|
||||
|
||||
<!-- Left column: stats + patterns -->
|
||||
<div class="col-12 col-lg-4 d-flex flex-column gap-3">
|
||||
|
||||
<!-- Stats panel -->
|
||||
<div class="card mc-panel">
|
||||
<div class="card-header mc-panel-header">// CORRECTION STATS</div>
|
||||
<div class="card-body p-3">
|
||||
<div class="spark-stat-grid">
|
||||
<div class="spark-stat">
|
||||
<span class="spark-stat-label">TOTAL</span>
|
||||
<span class="spark-stat-value">{{ stats.total }}</span>
|
||||
</div>
|
||||
<div class="spark-stat">
|
||||
<span class="spark-stat-label">CORRECTED</span>
|
||||
<span class="spark-stat-value text-success">{{ stats.success_count }}</span>
|
||||
</div>
|
||||
<div class="spark-stat">
|
||||
<span class="spark-stat-label">PARTIAL</span>
|
||||
<span class="spark-stat-value text-warning">{{ stats.partial_count }}</span>
|
||||
</div>
|
||||
<div class="spark-stat">
|
||||
<span class="spark-stat-label">FAILED</span>
|
||||
<span class="spark-stat-value {% if stats.failed_count > 0 %}text-danger{% else %}text-muted{% endif %}">{{ stats.failed_count }}</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="mt-3">
|
||||
<div class="d-flex justify-content-between mb-1">
|
||||
<small class="text-muted">Correction Rate</small>
|
||||
<small class="{% if stats.success_rate >= 70 %}text-success{% elif stats.success_rate >= 40 %}text-warning{% else %}text-danger{% endif %}">{{ stats.success_rate }}%</small>
|
||||
</div>
|
||||
<div class="progress" style="height:6px;">
|
||||
<div class="progress-bar {% if stats.success_rate >= 70 %}bg-success{% elif stats.success_rate >= 40 %}bg-warning{% else %}bg-danger{% endif %}"
|
||||
role="progressbar"
|
||||
style="width:{{ stats.success_rate }}%"
|
||||
aria-valuenow="{{ stats.success_rate }}"
|
||||
aria-valuemin="0"
|
||||
aria-valuemax="100"></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Patterns panel -->
|
||||
<div class="card mc-panel"
|
||||
hx-get="/self-correction/patterns"
|
||||
hx-trigger="load, every 60s"
|
||||
hx-target="#sc-patterns-body"
|
||||
hx-swap="innerHTML">
|
||||
<div class="card-header mc-panel-header d-flex justify-content-between align-items-center">
|
||||
<span>// RECURRING PATTERNS</span>
|
||||
<span class="badge badge-info">{{ patterns | length }}</span>
|
||||
</div>
|
||||
<div class="card-body p-0" id="sc-patterns-body">
|
||||
{% include "partials/self_correction_patterns.html" %}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
<!-- Right column: timeline -->
|
||||
<div class="col-12 col-lg-8">
|
||||
<div class="card mc-panel"
|
||||
hx-get="/self-correction/timeline"
|
||||
hx-trigger="load, every 30s"
|
||||
hx-target="#sc-timeline-body"
|
||||
hx-swap="innerHTML">
|
||||
<div class="card-header mc-panel-header d-flex justify-content-between align-items-center">
|
||||
<span>// CORRECTION TIMELINE</span>
|
||||
<span class="badge badge-info">{{ corrections | length }}</span>
|
||||
</div>
|
||||
<div class="card-body p-3" id="sc-timeline-body">
|
||||
{% include "partials/self_correction_timeline.html" %}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
{% endblock %}
|
||||
8
src/infrastructure/energy/__init__.py
Normal file
8
src/infrastructure/energy/__init__.py
Normal file
@@ -0,0 +1,8 @@
|
||||
"""Energy Budget Monitoring — power-draw estimation for LLM inference.
|
||||
|
||||
Refs: #1009
|
||||
"""
|
||||
|
||||
from infrastructure.energy.monitor import EnergyBudgetMonitor, energy_monitor
|
||||
|
||||
__all__ = ["EnergyBudgetMonitor", "energy_monitor"]
|
||||
371
src/infrastructure/energy/monitor.py
Normal file
371
src/infrastructure/energy/monitor.py
Normal file
@@ -0,0 +1,371 @@
|
||||
"""Energy Budget Monitor — estimates GPU/CPU power draw during LLM inference.
|
||||
|
||||
Tracks estimated power consumption to optimize for "metabolic efficiency".
|
||||
Three estimation strategies attempted in priority order:
|
||||
|
||||
1. Battery discharge via ioreg (macOS — works without sudo, on-battery only)
|
||||
2. CPU utilisation proxy via sysctl hw.cpufrequency + top
|
||||
3. Model-size heuristic (tokens/s × model_size_gb × 2W/GB estimate)
|
||||
|
||||
Energy Efficiency score (0–10):
|
||||
efficiency = tokens_per_second / estimated_watts, normalised to 0–10.
|
||||
|
||||
Low Power Mode:
|
||||
Activated manually or automatically when draw exceeds the configured
|
||||
threshold. In low power mode the cascade router is advised to prefer the
|
||||
configured low_power_model (e.g. qwen3:1b or similar compact model).
|
||||
|
||||
Refs: #1009
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import subprocess
|
||||
import time
|
||||
from collections import deque
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Approximate model-size lookup (GB) used for heuristic power estimate.
|
||||
# Keys are lowercase substring matches against the model name.
|
||||
_MODEL_SIZE_GB: dict[str, float] = {
|
||||
"qwen3:1b": 0.8,
|
||||
"qwen3:3b": 2.0,
|
||||
"qwen3:4b": 2.5,
|
||||
"qwen3:8b": 5.5,
|
||||
"qwen3:14b": 9.0,
|
||||
"qwen3:30b": 20.0,
|
||||
"qwen3:32b": 20.0,
|
||||
"llama3:8b": 5.5,
|
||||
"llama3:70b": 45.0,
|
||||
"mistral:7b": 4.5,
|
||||
"gemma3:4b": 2.5,
|
||||
"gemma3:12b": 8.0,
|
||||
"gemma3:27b": 17.0,
|
||||
"phi4:14b": 9.0,
|
||||
}
|
||||
_DEFAULT_MODEL_SIZE_GB = 5.0 # fallback when model not in table
|
||||
_WATTS_PER_GB_HEURISTIC = 2.0 # rough W/GB for Apple Silicon unified memory
|
||||
|
||||
# Efficiency score normalisation: score 10 at this efficiency (tok/s per W).
|
||||
_EFFICIENCY_SCORE_CEILING = 5.0 # tok/s per W → score 10
|
||||
|
||||
# Rolling window for recent samples
|
||||
_HISTORY_MAXLEN = 60
|
||||
|
||||
|
||||
@dataclass
|
||||
class InferenceSample:
|
||||
"""A single inference event captured by record_inference()."""
|
||||
|
||||
timestamp: str
|
||||
model: str
|
||||
tokens_per_second: float
|
||||
estimated_watts: float
|
||||
efficiency: float # tokens/s per watt
|
||||
efficiency_score: float # 0–10
|
||||
|
||||
|
||||
@dataclass
|
||||
class EnergyReport:
|
||||
"""Snapshot of current energy budget state."""
|
||||
|
||||
timestamp: str
|
||||
low_power_mode: bool
|
||||
current_watts: float
|
||||
strategy: str # "battery", "cpu_proxy", "heuristic", "unavailable"
|
||||
efficiency_score: float # 0–10; -1 if no inference samples yet
|
||||
recent_samples: list[InferenceSample]
|
||||
recommendation: str
|
||||
details: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"timestamp": self.timestamp,
|
||||
"low_power_mode": self.low_power_mode,
|
||||
"current_watts": round(self.current_watts, 2),
|
||||
"strategy": self.strategy,
|
||||
"efficiency_score": round(self.efficiency_score, 2),
|
||||
"recent_samples": [
|
||||
{
|
||||
"timestamp": s.timestamp,
|
||||
"model": s.model,
|
||||
"tokens_per_second": round(s.tokens_per_second, 1),
|
||||
"estimated_watts": round(s.estimated_watts, 2),
|
||||
"efficiency": round(s.efficiency, 3),
|
||||
"efficiency_score": round(s.efficiency_score, 2),
|
||||
}
|
||||
for s in self.recent_samples
|
||||
],
|
||||
"recommendation": self.recommendation,
|
||||
"details": self.details,
|
||||
}
|
||||
|
||||
|
||||
class EnergyBudgetMonitor:
|
||||
"""Estimates power consumption and tracks LLM inference efficiency.
|
||||
|
||||
All blocking I/O (subprocess calls) is wrapped in asyncio.to_thread()
|
||||
so the event loop is never blocked. Results are cached.
|
||||
|
||||
Usage::
|
||||
|
||||
# Record an inference event
|
||||
energy_monitor.record_inference("qwen3:8b", tokens_per_second=42.0)
|
||||
|
||||
# Get the current report
|
||||
report = await energy_monitor.get_report()
|
||||
|
||||
# Toggle low power mode
|
||||
energy_monitor.set_low_power_mode(True)
|
||||
"""
|
||||
|
||||
_POWER_CACHE_TTL = 10.0 # seconds between fresh power readings
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._low_power_mode: bool = False
|
||||
self._samples: deque[InferenceSample] = deque(maxlen=_HISTORY_MAXLEN)
|
||||
self._cached_watts: float = 0.0
|
||||
self._cached_strategy: str = "unavailable"
|
||||
self._cache_ts: float = 0.0
|
||||
|
||||
# ── Public API ────────────────────────────────────────────────────────────
|
||||
|
||||
@property
|
||||
def low_power_mode(self) -> bool:
|
||||
return self._low_power_mode
|
||||
|
||||
def set_low_power_mode(self, enabled: bool) -> None:
|
||||
"""Enable or disable low power mode."""
|
||||
self._low_power_mode = enabled
|
||||
state = "enabled" if enabled else "disabled"
|
||||
logger.info("Energy budget: low power mode %s", state)
|
||||
|
||||
def record_inference(self, model: str, tokens_per_second: float) -> InferenceSample:
|
||||
"""Record an inference event for efficiency tracking.
|
||||
|
||||
Call this after each LLM inference completes with the model name and
|
||||
measured throughput. The current power estimate is used to compute
|
||||
the efficiency score.
|
||||
|
||||
Args:
|
||||
model: Ollama model name (e.g. "qwen3:8b").
|
||||
tokens_per_second: Measured decode throughput.
|
||||
|
||||
Returns:
|
||||
The recorded InferenceSample.
|
||||
"""
|
||||
watts = self._cached_watts if self._cached_watts > 0 else self._estimate_watts_sync(model)
|
||||
efficiency = tokens_per_second / max(watts, 0.1)
|
||||
score = min(10.0, (efficiency / _EFFICIENCY_SCORE_CEILING) * 10.0)
|
||||
|
||||
sample = InferenceSample(
|
||||
timestamp=datetime.now(UTC).isoformat(),
|
||||
model=model,
|
||||
tokens_per_second=tokens_per_second,
|
||||
estimated_watts=watts,
|
||||
efficiency=efficiency,
|
||||
efficiency_score=score,
|
||||
)
|
||||
self._samples.append(sample)
|
||||
|
||||
# Auto-engage low power mode if above threshold and budget is enabled
|
||||
threshold = getattr(settings, "energy_budget_watts_threshold", 15.0)
|
||||
if watts > threshold and not self._low_power_mode:
|
||||
logger.info(
|
||||
"Energy budget: %.1fW exceeds threshold %.1fW — auto-engaging low power mode",
|
||||
watts,
|
||||
threshold,
|
||||
)
|
||||
self.set_low_power_mode(True)
|
||||
|
||||
return sample
|
||||
|
||||
async def get_report(self) -> EnergyReport:
|
||||
"""Return the current energy budget report.
|
||||
|
||||
Refreshes the power estimate if the cache is stale.
|
||||
"""
|
||||
await self._refresh_power_cache()
|
||||
|
||||
score = self._compute_mean_efficiency_score()
|
||||
recommendation = self._build_recommendation(score)
|
||||
|
||||
return EnergyReport(
|
||||
timestamp=datetime.now(UTC).isoformat(),
|
||||
low_power_mode=self._low_power_mode,
|
||||
current_watts=self._cached_watts,
|
||||
strategy=self._cached_strategy,
|
||||
efficiency_score=score,
|
||||
recent_samples=list(self._samples)[-10:],
|
||||
recommendation=recommendation,
|
||||
details={"sample_count": len(self._samples)},
|
||||
)
|
||||
|
||||
# ── Power estimation ──────────────────────────────────────────────────────
|
||||
|
||||
async def _refresh_power_cache(self) -> None:
|
||||
"""Refresh the cached power reading if stale."""
|
||||
now = time.monotonic()
|
||||
if now - self._cache_ts < self._POWER_CACHE_TTL:
|
||||
return
|
||||
|
||||
try:
|
||||
watts, strategy = await asyncio.to_thread(self._read_power)
|
||||
except Exception as exc:
|
||||
logger.debug("Energy: power read failed: %s", exc)
|
||||
watts, strategy = 0.0, "unavailable"
|
||||
|
||||
self._cached_watts = watts
|
||||
self._cached_strategy = strategy
|
||||
self._cache_ts = now
|
||||
|
||||
def _read_power(self) -> tuple[float, str]:
|
||||
"""Synchronous power reading — tries strategies in priority order.
|
||||
|
||||
Returns:
|
||||
Tuple of (watts, strategy_name).
|
||||
"""
|
||||
# Strategy 1: battery discharge via ioreg (on-battery Macs)
|
||||
try:
|
||||
watts = self._read_battery_watts()
|
||||
if watts > 0:
|
||||
return watts, "battery"
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Strategy 2: CPU utilisation proxy via top
|
||||
try:
|
||||
cpu_pct = self._read_cpu_pct()
|
||||
if cpu_pct >= 0:
|
||||
# M3 Max TDP ≈ 40W; scale linearly
|
||||
watts = (cpu_pct / 100.0) * 40.0
|
||||
return watts, "cpu_proxy"
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Strategy 3: heuristic from loaded model size
|
||||
return 0.0, "unavailable"
|
||||
|
||||
def _estimate_watts_sync(self, model: str) -> float:
|
||||
"""Estimate watts from model size when no live reading is available."""
|
||||
size_gb = self._model_size_gb(model)
|
||||
return size_gb * _WATTS_PER_GB_HEURISTIC
|
||||
|
||||
def _read_battery_watts(self) -> float:
|
||||
"""Read instantaneous battery discharge via ioreg.
|
||||
|
||||
Returns watts if on battery, 0.0 if plugged in or unavailable.
|
||||
Requires macOS; no sudo needed.
|
||||
"""
|
||||
result = subprocess.run(
|
||||
["ioreg", "-r", "-c", "AppleSmartBattery", "-d", "1"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=3,
|
||||
)
|
||||
amperage_ma = 0.0
|
||||
voltage_mv = 0.0
|
||||
is_charging = True # assume charging unless we see ExternalConnected = No
|
||||
|
||||
for line in result.stdout.splitlines():
|
||||
stripped = line.strip()
|
||||
if '"InstantAmperage"' in stripped:
|
||||
try:
|
||||
amperage_ma = float(stripped.split("=")[-1].strip())
|
||||
except ValueError:
|
||||
pass
|
||||
elif '"Voltage"' in stripped:
|
||||
try:
|
||||
voltage_mv = float(stripped.split("=")[-1].strip())
|
||||
except ValueError:
|
||||
pass
|
||||
elif '"ExternalConnected"' in stripped:
|
||||
is_charging = "Yes" in stripped
|
||||
|
||||
if is_charging or voltage_mv == 0 or amperage_ma <= 0:
|
||||
return 0.0
|
||||
|
||||
# ioreg reports amperage in mA, voltage in mV
|
||||
return (abs(amperage_ma) * voltage_mv) / 1_000_000
|
||||
|
||||
def _read_cpu_pct(self) -> float:
|
||||
"""Read CPU utilisation from macOS top.
|
||||
|
||||
Returns aggregate CPU% (0–100), or -1.0 on failure.
|
||||
"""
|
||||
result = subprocess.run(
|
||||
["top", "-l", "1", "-n", "0", "-stats", "cpu"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=5,
|
||||
)
|
||||
for line in result.stdout.splitlines():
|
||||
if "CPU usage:" in line:
|
||||
# "CPU usage: 12.5% user, 8.3% sys, 79.1% idle"
|
||||
parts = line.split()
|
||||
try:
|
||||
user = float(parts[2].rstrip("%"))
|
||||
sys_ = float(parts[4].rstrip("%"))
|
||||
return user + sys_
|
||||
except (IndexError, ValueError):
|
||||
pass
|
||||
return -1.0
|
||||
|
||||
# ── Helpers ───────────────────────────────────────────────────────────────
|
||||
|
||||
@staticmethod
|
||||
def _model_size_gb(model: str) -> float:
|
||||
"""Look up approximate model size in GB by name substring."""
|
||||
lower = model.lower()
|
||||
# Exact match first
|
||||
if lower in _MODEL_SIZE_GB:
|
||||
return _MODEL_SIZE_GB[lower]
|
||||
# Substring match
|
||||
for key, size in _MODEL_SIZE_GB.items():
|
||||
if key in lower:
|
||||
return size
|
||||
return _DEFAULT_MODEL_SIZE_GB
|
||||
|
||||
def _compute_mean_efficiency_score(self) -> float:
|
||||
"""Mean efficiency score over recent samples, or -1 if none."""
|
||||
if not self._samples:
|
||||
return -1.0
|
||||
recent = list(self._samples)[-10:]
|
||||
return sum(s.efficiency_score for s in recent) / len(recent)
|
||||
|
||||
def _build_recommendation(self, score: float) -> str:
|
||||
"""Generate a human-readable recommendation from the efficiency score."""
|
||||
threshold = getattr(settings, "energy_budget_watts_threshold", 15.0)
|
||||
low_power_model = getattr(settings, "energy_low_power_model", "qwen3:1b")
|
||||
|
||||
if score < 0:
|
||||
return "No inference data yet — run some tasks to populate efficiency metrics."
|
||||
|
||||
if self._low_power_mode:
|
||||
return (
|
||||
f"Low power mode active — routing to {low_power_model}. "
|
||||
"Disable when power draw normalises."
|
||||
)
|
||||
|
||||
if score < 3.0:
|
||||
return (
|
||||
f"Low efficiency (score {score:.1f}/10). "
|
||||
f"Consider enabling low power mode to favour smaller models "
|
||||
f"(threshold: {threshold}W)."
|
||||
)
|
||||
|
||||
if score < 6.0:
|
||||
return f"Moderate efficiency (score {score:.1f}/10). System operating normally."
|
||||
|
||||
return f"Good efficiency (score {score:.1f}/10). No action needed."
|
||||
|
||||
|
||||
# Module-level singleton
|
||||
energy_monitor = EnergyBudgetMonitor()
|
||||
@@ -71,6 +71,53 @@ class GitHand:
|
||||
return True
|
||||
return False
|
||||
|
||||
async def _exec_subprocess(
|
||||
self,
|
||||
args: str,
|
||||
timeout: int,
|
||||
) -> tuple[bytes, bytes, int]:
|
||||
"""Run git as a subprocess, return (stdout, stderr, returncode).
|
||||
|
||||
Raises TimeoutError if the process exceeds *timeout* seconds.
|
||||
"""
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
"git",
|
||||
*args.split(),
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
cwd=self._repo_dir,
|
||||
)
|
||||
try:
|
||||
stdout, stderr = await asyncio.wait_for(
|
||||
proc.communicate(),
|
||||
timeout=timeout,
|
||||
)
|
||||
except TimeoutError:
|
||||
proc.kill()
|
||||
await proc.wait()
|
||||
raise
|
||||
return stdout, stderr, proc.returncode or 0
|
||||
|
||||
@staticmethod
|
||||
def _parse_output(
|
||||
command: str,
|
||||
stdout_bytes: bytes,
|
||||
stderr_bytes: bytes,
|
||||
returncode: int | None,
|
||||
latency_ms: float,
|
||||
) -> GitResult:
|
||||
"""Decode subprocess output into a GitResult."""
|
||||
exit_code = returncode or 0
|
||||
stdout = stdout_bytes.decode("utf-8", errors="replace").strip()
|
||||
stderr = stderr_bytes.decode("utf-8", errors="replace").strip()
|
||||
return GitResult(
|
||||
operation=command,
|
||||
success=exit_code == 0,
|
||||
output=stdout,
|
||||
error=stderr if exit_code != 0 else "",
|
||||
latency_ms=latency_ms,
|
||||
)
|
||||
|
||||
async def run(
|
||||
self,
|
||||
args: str,
|
||||
@@ -88,14 +135,15 @@ class GitHand:
|
||||
GitResult with output or error details.
|
||||
"""
|
||||
start = time.time()
|
||||
command = f"git {args}"
|
||||
|
||||
# Gate destructive operations
|
||||
if self._is_destructive(args) and not allow_destructive:
|
||||
return GitResult(
|
||||
operation=f"git {args}",
|
||||
operation=command,
|
||||
success=False,
|
||||
error=(
|
||||
f"Destructive operation blocked: 'git {args}'. "
|
||||
f"Destructive operation blocked: '{command}'. "
|
||||
"Set allow_destructive=True to override."
|
||||
),
|
||||
requires_confirmation=True,
|
||||
@@ -103,46 +151,21 @@ class GitHand:
|
||||
)
|
||||
|
||||
effective_timeout = timeout or self._timeout
|
||||
command = f"git {args}"
|
||||
|
||||
try:
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
"git",
|
||||
*args.split(),
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
cwd=self._repo_dir,
|
||||
stdout_bytes, stderr_bytes, returncode = await self._exec_subprocess(
|
||||
args,
|
||||
effective_timeout,
|
||||
)
|
||||
|
||||
try:
|
||||
stdout_bytes, stderr_bytes = await asyncio.wait_for(
|
||||
proc.communicate(), timeout=effective_timeout
|
||||
)
|
||||
except TimeoutError:
|
||||
proc.kill()
|
||||
await proc.wait()
|
||||
latency = (time.time() - start) * 1000
|
||||
logger.warning("Git command timed out after %ds: %s", effective_timeout, command)
|
||||
return GitResult(
|
||||
operation=command,
|
||||
success=False,
|
||||
error=f"Command timed out after {effective_timeout}s",
|
||||
latency_ms=latency,
|
||||
)
|
||||
|
||||
except TimeoutError:
|
||||
latency = (time.time() - start) * 1000
|
||||
exit_code = proc.returncode or 0
|
||||
stdout = stdout_bytes.decode("utf-8", errors="replace").strip()
|
||||
stderr = stderr_bytes.decode("utf-8", errors="replace").strip()
|
||||
|
||||
logger.warning("Git command timed out after %ds: %s", effective_timeout, command)
|
||||
return GitResult(
|
||||
operation=command,
|
||||
success=exit_code == 0,
|
||||
output=stdout,
|
||||
error=stderr if exit_code != 0 else "",
|
||||
success=False,
|
||||
error=f"Command timed out after {effective_timeout}s",
|
||||
latency_ms=latency,
|
||||
)
|
||||
|
||||
except FileNotFoundError:
|
||||
latency = (time.time() - start) * 1000
|
||||
logger.warning("git binary not found")
|
||||
@@ -162,6 +185,14 @@ class GitHand:
|
||||
latency_ms=latency,
|
||||
)
|
||||
|
||||
return self._parse_output(
|
||||
command,
|
||||
stdout_bytes,
|
||||
stderr_bytes,
|
||||
returncode=returncode,
|
||||
latency_ms=(time.time() - start) * 1000,
|
||||
)
|
||||
|
||||
# ── Convenience wrappers ─────────────────────────────────────────────────
|
||||
|
||||
async def status(self) -> GitResult:
|
||||
|
||||
@@ -1,5 +1,11 @@
|
||||
"""Infrastructure models package."""
|
||||
|
||||
from infrastructure.models.budget import (
|
||||
BudgetTracker,
|
||||
SpendRecord,
|
||||
estimate_cost_usd,
|
||||
get_budget_tracker,
|
||||
)
|
||||
from infrastructure.models.multimodal import (
|
||||
ModelCapability,
|
||||
ModelInfo,
|
||||
@@ -17,6 +23,12 @@ from infrastructure.models.registry import (
|
||||
ModelRole,
|
||||
model_registry,
|
||||
)
|
||||
from infrastructure.models.router import (
|
||||
TierLabel,
|
||||
TieredModelRouter,
|
||||
classify_tier,
|
||||
get_tiered_router,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
# Registry
|
||||
@@ -34,4 +46,14 @@ __all__ = [
|
||||
"model_supports_tools",
|
||||
"model_supports_vision",
|
||||
"pull_model_with_fallback",
|
||||
# Tiered router
|
||||
"TierLabel",
|
||||
"TieredModelRouter",
|
||||
"classify_tier",
|
||||
"get_tiered_router",
|
||||
# Budget tracker
|
||||
"BudgetTracker",
|
||||
"SpendRecord",
|
||||
"estimate_cost_usd",
|
||||
"get_budget_tracker",
|
||||
]
|
||||
|
||||
302
src/infrastructure/models/budget.py
Normal file
302
src/infrastructure/models/budget.py
Normal file
@@ -0,0 +1,302 @@
|
||||
"""Cloud API budget tracker for the three-tier model router.
|
||||
|
||||
Tracks cloud API spend (daily / monthly) and enforces configurable limits.
|
||||
SQLite-backed with in-memory fallback — degrades gracefully if the database
|
||||
is unavailable.
|
||||
|
||||
References:
|
||||
- Issue #882 — Model Tiering Router: Local 8B / Hermes 70B / Cloud API Cascade
|
||||
"""
|
||||
|
||||
import logging
|
||||
import sqlite3
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, date, datetime
|
||||
from pathlib import Path
|
||||
|
||||
from config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ── Cost estimates (USD per 1 K tokens, input / output) ──────────────────────
|
||||
# Updated 2026-03. Estimates only — actual costs vary by tier/usage.
|
||||
_COST_PER_1K: dict[str, dict[str, float]] = {
|
||||
# Claude models
|
||||
"claude-haiku-4-5": {"input": 0.00025, "output": 0.00125},
|
||||
"claude-sonnet-4-5": {"input": 0.003, "output": 0.015},
|
||||
"claude-opus-4-5": {"input": 0.015, "output": 0.075},
|
||||
"haiku": {"input": 0.00025, "output": 0.00125},
|
||||
"sonnet": {"input": 0.003, "output": 0.015},
|
||||
"opus": {"input": 0.015, "output": 0.075},
|
||||
# GPT-4o
|
||||
"gpt-4o-mini": {"input": 0.00015, "output": 0.0006},
|
||||
"gpt-4o": {"input": 0.0025, "output": 0.01},
|
||||
# Grok (xAI)
|
||||
"grok-3-fast": {"input": 0.003, "output": 0.015},
|
||||
"grok-3": {"input": 0.005, "output": 0.025},
|
||||
}
|
||||
_DEFAULT_COST: dict[str, float] = {"input": 0.003, "output": 0.015} # conservative fallback
|
||||
|
||||
|
||||
def estimate_cost_usd(model: str, tokens_in: int, tokens_out: int) -> float:
|
||||
"""Estimate the cost of a single request in USD.
|
||||
|
||||
Matches the model name by substring so versioned names like
|
||||
``claude-haiku-4-5-20251001`` still resolve correctly.
|
||||
|
||||
Args:
|
||||
model: Model name as passed to the provider.
|
||||
tokens_in: Number of input (prompt) tokens consumed.
|
||||
tokens_out: Number of output (completion) tokens generated.
|
||||
|
||||
Returns:
|
||||
Estimated cost in USD (may be zero for unknown models).
|
||||
"""
|
||||
model_lower = model.lower()
|
||||
rates = _DEFAULT_COST
|
||||
for key, rate in _COST_PER_1K.items():
|
||||
if key in model_lower:
|
||||
rates = rate
|
||||
break
|
||||
return (tokens_in * rates["input"] + tokens_out * rates["output"]) / 1000.0
|
||||
|
||||
|
||||
@dataclass
|
||||
class SpendRecord:
|
||||
"""A single spend event."""
|
||||
|
||||
ts: float
|
||||
provider: str
|
||||
model: str
|
||||
tokens_in: int
|
||||
tokens_out: int
|
||||
cost_usd: float
|
||||
tier: str
|
||||
|
||||
|
||||
class BudgetTracker:
|
||||
"""Tracks cloud API spend with configurable daily / monthly limits.
|
||||
|
||||
Persists spend records to SQLite (``data/budget.db`` by default).
|
||||
Falls back to in-memory tracking when the database is unavailable —
|
||||
budget enforcement still works; records are lost on restart.
|
||||
|
||||
Limits are read from ``settings``:
|
||||
|
||||
* ``tier_cloud_daily_budget_usd`` — daily ceiling (0 = disabled)
|
||||
* ``tier_cloud_monthly_budget_usd`` — monthly ceiling (0 = disabled)
|
||||
|
||||
Usage::
|
||||
|
||||
tracker = BudgetTracker()
|
||||
|
||||
if tracker.cloud_allowed():
|
||||
# … make cloud API call …
|
||||
tracker.record_spend("anthropic", "claude-haiku-4-5", 100, 200)
|
||||
|
||||
summary = tracker.get_summary()
|
||||
print(summary["daily_usd"], "/", summary["daily_limit_usd"])
|
||||
"""
|
||||
|
||||
_DB_PATH = "data/budget.db"
|
||||
|
||||
def __init__(self, db_path: str | None = None) -> None:
|
||||
"""Initialise the tracker.
|
||||
|
||||
Args:
|
||||
db_path: Path to the SQLite database. Defaults to
|
||||
``data/budget.db``. Pass ``":memory:"`` for tests.
|
||||
"""
|
||||
self._db_path = db_path or self._DB_PATH
|
||||
self._lock = threading.Lock()
|
||||
self._in_memory: list[SpendRecord] = []
|
||||
self._db_ok = False
|
||||
self._init_db()
|
||||
|
||||
# ── Database initialisation ──────────────────────────────────────────────
|
||||
|
||||
def _init_db(self) -> None:
|
||||
"""Create the spend table (and parent directory) if needed."""
|
||||
try:
|
||||
if self._db_path != ":memory:":
|
||||
Path(self._db_path).parent.mkdir(parents=True, exist_ok=True)
|
||||
with self._connect() as conn:
|
||||
conn.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS cloud_spend (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
ts REAL NOT NULL,
|
||||
provider TEXT NOT NULL,
|
||||
model TEXT NOT NULL,
|
||||
tokens_in INTEGER NOT NULL DEFAULT 0,
|
||||
tokens_out INTEGER NOT NULL DEFAULT 0,
|
||||
cost_usd REAL NOT NULL DEFAULT 0.0,
|
||||
tier TEXT NOT NULL DEFAULT 'cloud'
|
||||
)
|
||||
"""
|
||||
)
|
||||
conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_spend_ts ON cloud_spend(ts)"
|
||||
)
|
||||
self._db_ok = True
|
||||
logger.debug("BudgetTracker: SQLite initialised at %s", self._db_path)
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"BudgetTracker: SQLite unavailable, using in-memory fallback: %s", exc
|
||||
)
|
||||
|
||||
def _connect(self) -> sqlite3.Connection:
|
||||
return sqlite3.connect(self._db_path, timeout=5)
|
||||
|
||||
# ── Public API ───────────────────────────────────────────────────────────
|
||||
|
||||
def record_spend(
|
||||
self,
|
||||
provider: str,
|
||||
model: str,
|
||||
tokens_in: int = 0,
|
||||
tokens_out: int = 0,
|
||||
cost_usd: float | None = None,
|
||||
tier: str = "cloud",
|
||||
) -> float:
|
||||
"""Record a cloud API spend event and return the cost recorded.
|
||||
|
||||
Args:
|
||||
provider: Provider name (e.g. ``"anthropic"``, ``"openai"``).
|
||||
model: Model name used for the request.
|
||||
tokens_in: Input token count (prompt).
|
||||
tokens_out: Output token count (completion).
|
||||
cost_usd: Explicit cost override. If ``None``, the cost is
|
||||
estimated from the token counts and model rates.
|
||||
tier: Tier label for the request (default ``"cloud"``).
|
||||
|
||||
Returns:
|
||||
The cost recorded in USD.
|
||||
"""
|
||||
if cost_usd is None:
|
||||
cost_usd = estimate_cost_usd(model, tokens_in, tokens_out)
|
||||
|
||||
ts = time.time()
|
||||
record = SpendRecord(ts, provider, model, tokens_in, tokens_out, cost_usd, tier)
|
||||
|
||||
with self._lock:
|
||||
if self._db_ok:
|
||||
try:
|
||||
with self._connect() as conn:
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO cloud_spend
|
||||
(ts, provider, model, tokens_in, tokens_out, cost_usd, tier)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
(ts, provider, model, tokens_in, tokens_out, cost_usd, tier),
|
||||
)
|
||||
logger.debug(
|
||||
"BudgetTracker: recorded %.6f USD (%s/%s, in=%d out=%d tier=%s)",
|
||||
cost_usd,
|
||||
provider,
|
||||
model,
|
||||
tokens_in,
|
||||
tokens_out,
|
||||
tier,
|
||||
)
|
||||
return cost_usd
|
||||
except Exception as exc:
|
||||
logger.warning("BudgetTracker: DB write failed, falling back: %s", exc)
|
||||
self._in_memory.append(record)
|
||||
|
||||
return cost_usd
|
||||
|
||||
def get_daily_spend(self) -> float:
|
||||
"""Return total cloud spend for the current UTC day in USD."""
|
||||
today = date.today()
|
||||
since = datetime(today.year, today.month, today.day, tzinfo=UTC).timestamp()
|
||||
return self._query_spend(since)
|
||||
|
||||
def get_monthly_spend(self) -> float:
|
||||
"""Return total cloud spend for the current UTC month in USD."""
|
||||
today = date.today()
|
||||
since = datetime(today.year, today.month, 1, tzinfo=UTC).timestamp()
|
||||
return self._query_spend(since)
|
||||
|
||||
def cloud_allowed(self) -> bool:
|
||||
"""Return ``True`` if cloud API spend is within configured limits.
|
||||
|
||||
Checks both daily and monthly ceilings. A limit of ``0`` disables
|
||||
that particular check.
|
||||
"""
|
||||
daily_limit = settings.tier_cloud_daily_budget_usd
|
||||
monthly_limit = settings.tier_cloud_monthly_budget_usd
|
||||
|
||||
if daily_limit > 0:
|
||||
daily_spend = self.get_daily_spend()
|
||||
if daily_spend >= daily_limit:
|
||||
logger.warning(
|
||||
"BudgetTracker: daily cloud budget exhausted (%.4f / %.4f USD)",
|
||||
daily_spend,
|
||||
daily_limit,
|
||||
)
|
||||
return False
|
||||
|
||||
if monthly_limit > 0:
|
||||
monthly_spend = self.get_monthly_spend()
|
||||
if monthly_spend >= monthly_limit:
|
||||
logger.warning(
|
||||
"BudgetTracker: monthly cloud budget exhausted (%.4f / %.4f USD)",
|
||||
monthly_spend,
|
||||
monthly_limit,
|
||||
)
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
def get_summary(self) -> dict:
|
||||
"""Return a spend summary dict suitable for dashboards / logging.
|
||||
|
||||
Keys: ``daily_usd``, ``monthly_usd``, ``daily_limit_usd``,
|
||||
``monthly_limit_usd``, ``daily_ok``, ``monthly_ok``.
|
||||
"""
|
||||
daily = self.get_daily_spend()
|
||||
monthly = self.get_monthly_spend()
|
||||
daily_limit = settings.tier_cloud_daily_budget_usd
|
||||
monthly_limit = settings.tier_cloud_monthly_budget_usd
|
||||
return {
|
||||
"daily_usd": round(daily, 6),
|
||||
"monthly_usd": round(monthly, 6),
|
||||
"daily_limit_usd": daily_limit,
|
||||
"monthly_limit_usd": monthly_limit,
|
||||
"daily_ok": daily_limit <= 0 or daily < daily_limit,
|
||||
"monthly_ok": monthly_limit <= 0 or monthly < monthly_limit,
|
||||
}
|
||||
|
||||
# ── Internal helpers ─────────────────────────────────────────────────────
|
||||
|
||||
def _query_spend(self, since_ts: float) -> float:
|
||||
"""Sum ``cost_usd`` for records with ``ts >= since_ts``."""
|
||||
if self._db_ok:
|
||||
try:
|
||||
with self._connect() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT COALESCE(SUM(cost_usd), 0.0) FROM cloud_spend WHERE ts >= ?",
|
||||
(since_ts,),
|
||||
).fetchone()
|
||||
return float(row[0]) if row else 0.0
|
||||
except Exception as exc:
|
||||
logger.warning("BudgetTracker: DB read failed: %s", exc)
|
||||
# In-memory fallback
|
||||
return sum(r.cost_usd for r in self._in_memory if r.ts >= since_ts)
|
||||
|
||||
|
||||
# ── Module-level singleton ────────────────────────────────────────────────────
|
||||
|
||||
_budget_tracker: BudgetTracker | None = None
|
||||
|
||||
|
||||
def get_budget_tracker() -> BudgetTracker:
|
||||
"""Get or create the module-level BudgetTracker singleton."""
|
||||
global _budget_tracker
|
||||
if _budget_tracker is None:
|
||||
_budget_tracker = BudgetTracker()
|
||||
return _budget_tracker
|
||||
427
src/infrastructure/models/router.py
Normal file
427
src/infrastructure/models/router.py
Normal file
@@ -0,0 +1,427 @@
|
||||
"""Three-tier model router — Local 8B / Local 70B / Cloud API Cascade.
|
||||
|
||||
Selects the cheapest-sufficient LLM for each request using a heuristic
|
||||
task-complexity classifier. Tier 3 (Cloud API) is only used when Tier 2
|
||||
fails or the budget guard allows it.
|
||||
|
||||
Tiers
|
||||
-----
|
||||
Tier 1 — LOCAL_FAST (Llama 3.1 8B / Hermes 3 8B via Ollama, free, ~0.3-1 s)
|
||||
Navigation, basic interactions, simple decisions.
|
||||
|
||||
Tier 2 — LOCAL_HEAVY (Hermes 3/4 70B via Ollama, free, ~5-10 s for 200 tok)
|
||||
Quest planning, dialogue strategy, complex reasoning.
|
||||
|
||||
Tier 3 — CLOUD_API (Claude / GPT-4o, paid ~$5-15/hr heavy use)
|
||||
Recovery from Tier 2 failures, novel situations, multi-step planning.
|
||||
|
||||
Routing logic
|
||||
-------------
|
||||
1. Classify the task using keyword / length / context heuristics (no LLM call).
|
||||
2. Route to the appropriate tier.
|
||||
3. On Tier-1 low-quality response → auto-escalate to Tier 2.
|
||||
4. On Tier-2 failure or explicit ``require_cloud=True`` → Tier 3 (if budget allows).
|
||||
5. Log tier used, model, latency, estimated cost for every request.
|
||||
|
||||
References:
|
||||
- Issue #882 — Model Tiering Router: Local 8B / Hermes 70B / Cloud API Cascade
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
from enum import StrEnum
|
||||
from typing import Any
|
||||
|
||||
from config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ── Tier definitions ──────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class TierLabel(StrEnum):
|
||||
"""Three cost-sorted model tiers."""
|
||||
|
||||
LOCAL_FAST = "local_fast" # 8B local, always hot, free
|
||||
LOCAL_HEAVY = "local_heavy" # 70B local, free but slower
|
||||
CLOUD_API = "cloud_api" # Paid cloud backend (Claude / GPT-4o)
|
||||
|
||||
|
||||
# ── Default model assignments (overridable via Settings) ──────────────────────
|
||||
|
||||
_DEFAULT_TIER_MODELS: dict[TierLabel, str] = {
|
||||
TierLabel.LOCAL_FAST: "llama3.1:8b",
|
||||
TierLabel.LOCAL_HEAVY: "hermes3:70b",
|
||||
TierLabel.CLOUD_API: "claude-haiku-4-5",
|
||||
}
|
||||
|
||||
# ── Classification vocabulary ─────────────────────────────────────────────────
|
||||
|
||||
# Patterns that indicate a Tier-1 (simple) task
|
||||
_T1_WORDS: frozenset[str] = frozenset(
|
||||
{
|
||||
"go", "move", "walk", "run",
|
||||
"north", "south", "east", "west", "up", "down", "left", "right",
|
||||
"yes", "no", "ok", "okay",
|
||||
"open", "close", "take", "drop", "look",
|
||||
"pick", "use", "wait", "rest", "save",
|
||||
"attack", "flee", "jump", "crouch",
|
||||
"status", "ping", "list", "show", "get", "check",
|
||||
}
|
||||
)
|
||||
|
||||
# Patterns that indicate a Tier-2 or Tier-3 task
|
||||
_T2_PHRASES: tuple[str, ...] = (
|
||||
"plan", "strategy", "optimize", "optimise",
|
||||
"quest", "stuck", "recover",
|
||||
"negotiate", "persuade", "faction", "reputation",
|
||||
"analyze", "analyse", "evaluate", "decide",
|
||||
"complex", "multi-step", "long-term",
|
||||
"how do i", "what should i do", "help me figure",
|
||||
"what is the best", "recommend", "best way",
|
||||
"explain", "describe in detail", "walk me through",
|
||||
"compare", "design", "implement", "refactor",
|
||||
"debug", "diagnose", "root cause",
|
||||
)
|
||||
|
||||
# Low-quality response detection patterns
|
||||
_LOW_QUALITY_PATTERNS: tuple[re.Pattern, ...] = (
|
||||
re.compile(r"i\s+don'?t\s+know", re.IGNORECASE),
|
||||
re.compile(r"i'm\s+not\s+sure", re.IGNORECASE),
|
||||
re.compile(r"i\s+cannot\s+(help|assist|answer)", re.IGNORECASE),
|
||||
re.compile(r"i\s+apologize", re.IGNORECASE),
|
||||
re.compile(r"as an ai", re.IGNORECASE),
|
||||
re.compile(r"i\s+don'?t\s+have\s+(enough|sufficient)\s+information", re.IGNORECASE),
|
||||
)
|
||||
|
||||
# Response is definitely low-quality if shorter than this many characters
|
||||
_LOW_QUALITY_MIN_CHARS = 20
|
||||
# Response is suspicious if shorter than this many chars for a complex task
|
||||
_ESCALATION_MIN_CHARS = 60
|
||||
|
||||
|
||||
def classify_tier(task: str, context: dict | None = None) -> TierLabel:
|
||||
"""Classify a task to the cheapest-sufficient model tier.
|
||||
|
||||
Classification priority (highest wins):
|
||||
1. ``context["require_cloud"] = True`` → CLOUD_API
|
||||
2. Any Tier-2 phrase or stuck/recovery signal → LOCAL_HEAVY
|
||||
3. Short task with only Tier-1 words, no active context → LOCAL_FAST
|
||||
4. Default → LOCAL_HEAVY (safe fallback for unknown tasks)
|
||||
|
||||
Args:
|
||||
task: Natural-language task or user input.
|
||||
context: Optional context dict. Recognised keys:
|
||||
``require_cloud`` (bool), ``stuck`` (bool),
|
||||
``require_t2`` (bool), ``active_quests`` (list),
|
||||
``dialogue_active`` (bool), ``combat_active`` (bool).
|
||||
|
||||
Returns:
|
||||
The cheapest ``TierLabel`` sufficient for the task.
|
||||
"""
|
||||
ctx = context or {}
|
||||
task_lower = task.lower()
|
||||
words = set(task_lower.split())
|
||||
|
||||
# ── Explicit cloud override ──────────────────────────────────────────────
|
||||
if ctx.get("require_cloud"):
|
||||
logger.debug("classify_tier → CLOUD_API (explicit require_cloud)")
|
||||
return TierLabel.CLOUD_API
|
||||
|
||||
# ── Tier-2 / complexity signals ──────────────────────────────────────────
|
||||
t2_phrase_hit = any(phrase in task_lower for phrase in _T2_PHRASES)
|
||||
t2_word_hit = bool(words & {"plan", "strategy", "optimize", "optimise", "quest",
|
||||
"stuck", "recover", "analyze", "analyse", "evaluate"})
|
||||
is_stuck = bool(ctx.get("stuck"))
|
||||
require_t2 = bool(ctx.get("require_t2"))
|
||||
long_input = len(task) > 300 # long tasks warrant more capable model
|
||||
deep_context = (
|
||||
len(ctx.get("active_quests", [])) >= 3
|
||||
or ctx.get("dialogue_active")
|
||||
)
|
||||
|
||||
if t2_phrase_hit or t2_word_hit or is_stuck or require_t2 or long_input or deep_context:
|
||||
logger.debug(
|
||||
"classify_tier → LOCAL_HEAVY (phrase=%s word=%s stuck=%s explicit=%s long=%s ctx=%s)",
|
||||
t2_phrase_hit, t2_word_hit, is_stuck, require_t2, long_input, deep_context,
|
||||
)
|
||||
return TierLabel.LOCAL_HEAVY
|
||||
|
||||
# ── Tier-1 signals ───────────────────────────────────────────────────────
|
||||
t1_word_hit = bool(words & _T1_WORDS)
|
||||
task_short = len(task.split()) <= 8
|
||||
no_active_context = (
|
||||
not ctx.get("active_quests")
|
||||
and not ctx.get("dialogue_active")
|
||||
and not ctx.get("combat_active")
|
||||
)
|
||||
|
||||
if t1_word_hit and task_short and no_active_context:
|
||||
logger.debug(
|
||||
"classify_tier → LOCAL_FAST (words=%s short=%s)", t1_word_hit, task_short
|
||||
)
|
||||
return TierLabel.LOCAL_FAST
|
||||
|
||||
# ── Default: LOCAL_HEAVY (safe for anything unclassified) ────────────────
|
||||
logger.debug("classify_tier → LOCAL_HEAVY (default)")
|
||||
return TierLabel.LOCAL_HEAVY
|
||||
|
||||
|
||||
def _is_low_quality(content: str, tier: TierLabel) -> bool:
|
||||
"""Return True if the response looks like it should be escalated.
|
||||
|
||||
Used for automatic Tier-1 → Tier-2 escalation.
|
||||
|
||||
Args:
|
||||
content: LLM response text.
|
||||
tier: The tier that produced the response.
|
||||
|
||||
Returns:
|
||||
True if the response is likely too low-quality to be useful.
|
||||
"""
|
||||
if not content or not content.strip():
|
||||
return True
|
||||
|
||||
stripped = content.strip()
|
||||
|
||||
# Too short to be useful
|
||||
if len(stripped) < _LOW_QUALITY_MIN_CHARS:
|
||||
return True
|
||||
|
||||
# Insufficient for a supposedly complex-enough task
|
||||
if tier == TierLabel.LOCAL_FAST and len(stripped) < _ESCALATION_MIN_CHARS:
|
||||
return True
|
||||
|
||||
# Matches known "I can't help" patterns
|
||||
for pattern in _LOW_QUALITY_PATTERNS:
|
||||
if pattern.search(stripped):
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
class TieredModelRouter:
|
||||
"""Routes LLM requests across the Local 8B / Local 70B / Cloud API tiers.
|
||||
|
||||
Wraps CascadeRouter with:
|
||||
- Heuristic tier classification via ``classify_tier()``
|
||||
- Automatic Tier-1 → Tier-2 escalation on low-quality responses
|
||||
- Cloud-tier budget guard via ``BudgetTracker``
|
||||
- Per-request logging: tier, model, latency, estimated cost
|
||||
|
||||
Usage::
|
||||
|
||||
router = TieredModelRouter()
|
||||
|
||||
result = await router.route(
|
||||
task="Walk to the next room",
|
||||
context={},
|
||||
)
|
||||
print(result["content"], result["tier"]) # "Move north.", "local_fast"
|
||||
|
||||
# Force heavy tier
|
||||
result = await router.route(
|
||||
task="Plan the optimal path to become Hortator",
|
||||
context={"require_t2": True},
|
||||
)
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
cascade: Any | None = None,
|
||||
budget_tracker: Any | None = None,
|
||||
tier_models: dict[TierLabel, str] | None = None,
|
||||
auto_escalate: bool = True,
|
||||
) -> None:
|
||||
"""Initialise the tiered router.
|
||||
|
||||
Args:
|
||||
cascade: CascadeRouter instance. If ``None``, the
|
||||
singleton from ``get_router()`` is used lazily.
|
||||
budget_tracker: BudgetTracker instance. If ``None``, the
|
||||
singleton from ``get_budget_tracker()`` is used.
|
||||
tier_models: Override default model names per tier.
|
||||
auto_escalate: When ``True``, low-quality Tier-1 responses
|
||||
automatically retry on Tier-2.
|
||||
"""
|
||||
self._cascade = cascade
|
||||
self._budget = budget_tracker
|
||||
self._tier_models: dict[TierLabel, str] = dict(_DEFAULT_TIER_MODELS)
|
||||
self._auto_escalate = auto_escalate
|
||||
|
||||
# Apply settings-level overrides (can still be overridden per-instance)
|
||||
if settings.tier_local_fast_model:
|
||||
self._tier_models[TierLabel.LOCAL_FAST] = settings.tier_local_fast_model
|
||||
if settings.tier_local_heavy_model:
|
||||
self._tier_models[TierLabel.LOCAL_HEAVY] = settings.tier_local_heavy_model
|
||||
if settings.tier_cloud_model:
|
||||
self._tier_models[TierLabel.CLOUD_API] = settings.tier_cloud_model
|
||||
|
||||
if tier_models:
|
||||
self._tier_models.update(tier_models)
|
||||
|
||||
# ── Lazy singletons ──────────────────────────────────────────────────────
|
||||
|
||||
def _get_cascade(self) -> Any:
|
||||
if self._cascade is None:
|
||||
from infrastructure.router.cascade import get_router
|
||||
self._cascade = get_router()
|
||||
return self._cascade
|
||||
|
||||
def _get_budget(self) -> Any:
|
||||
if self._budget is None:
|
||||
from infrastructure.models.budget import get_budget_tracker
|
||||
self._budget = get_budget_tracker()
|
||||
return self._budget
|
||||
|
||||
# ── Public interface ─────────────────────────────────────────────────────
|
||||
|
||||
def classify(self, task: str, context: dict | None = None) -> TierLabel:
|
||||
"""Classify a task without routing. Useful for telemetry."""
|
||||
return classify_tier(task, context)
|
||||
|
||||
async def route(
|
||||
self,
|
||||
task: str,
|
||||
context: dict | None = None,
|
||||
messages: list[dict] | None = None,
|
||||
temperature: float = 0.3,
|
||||
max_tokens: int | None = None,
|
||||
) -> dict:
|
||||
"""Route a task to the appropriate model tier.
|
||||
|
||||
Builds a minimal messages list if ``messages`` is not provided.
|
||||
The result always includes a ``tier`` key indicating which tier
|
||||
ultimately handled the request.
|
||||
|
||||
Args:
|
||||
task: Natural-language task description.
|
||||
context: Task context dict (see ``classify_tier()``).
|
||||
messages: Pre-built OpenAI-compatible messages list. If
|
||||
provided, ``task`` is only used for classification.
|
||||
temperature: Sampling temperature (default 0.3).
|
||||
max_tokens: Maximum tokens to generate.
|
||||
|
||||
Returns:
|
||||
Dict with at minimum: ``content``, ``provider``, ``model``,
|
||||
``tier``, ``latency_ms``. May include ``cost_usd`` when a
|
||||
cloud request is recorded.
|
||||
|
||||
Raises:
|
||||
RuntimeError: If all available tiers are exhausted.
|
||||
"""
|
||||
ctx = context or {}
|
||||
tier = self.classify(task, ctx)
|
||||
msgs = messages or [{"role": "user", "content": task}]
|
||||
|
||||
# ── Tier 1 attempt ───────────────────────────────────────────────────
|
||||
if tier == TierLabel.LOCAL_FAST:
|
||||
result = await self._complete_tier(
|
||||
TierLabel.LOCAL_FAST, msgs, temperature, max_tokens
|
||||
)
|
||||
if self._auto_escalate and _is_low_quality(result.get("content", ""), TierLabel.LOCAL_FAST):
|
||||
logger.info(
|
||||
"TieredModelRouter: Tier-1 response low quality, escalating to Tier-2 "
|
||||
"(task=%r content_len=%d)",
|
||||
task[:80],
|
||||
len(result.get("content", "")),
|
||||
)
|
||||
tier = TierLabel.LOCAL_HEAVY
|
||||
result = await self._complete_tier(
|
||||
TierLabel.LOCAL_HEAVY, msgs, temperature, max_tokens
|
||||
)
|
||||
return result
|
||||
|
||||
# ── Tier 2 attempt ───────────────────────────────────────────────────
|
||||
if tier == TierLabel.LOCAL_HEAVY:
|
||||
try:
|
||||
return await self._complete_tier(
|
||||
TierLabel.LOCAL_HEAVY, msgs, temperature, max_tokens
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"TieredModelRouter: Tier-2 failed (%s) — escalating to cloud", exc
|
||||
)
|
||||
tier = TierLabel.CLOUD_API
|
||||
|
||||
# ── Tier 3 (Cloud) ───────────────────────────────────────────────────
|
||||
budget = self._get_budget()
|
||||
if not budget.cloud_allowed():
|
||||
raise RuntimeError(
|
||||
"Cloud API tier requested but budget limit reached — "
|
||||
"increase tier_cloud_daily_budget_usd or tier_cloud_monthly_budget_usd"
|
||||
)
|
||||
|
||||
result = await self._complete_tier(
|
||||
TierLabel.CLOUD_API, msgs, temperature, max_tokens
|
||||
)
|
||||
|
||||
# Record cloud spend if token info is available
|
||||
usage = result.get("usage", {})
|
||||
if usage:
|
||||
cost = budget.record_spend(
|
||||
provider=result.get("provider", "unknown"),
|
||||
model=result.get("model", self._tier_models[TierLabel.CLOUD_API]),
|
||||
tokens_in=usage.get("prompt_tokens", 0),
|
||||
tokens_out=usage.get("completion_tokens", 0),
|
||||
tier=TierLabel.CLOUD_API,
|
||||
)
|
||||
result["cost_usd"] = cost
|
||||
|
||||
return result
|
||||
|
||||
# ── Internal helpers ─────────────────────────────────────────────────────
|
||||
|
||||
async def _complete_tier(
|
||||
self,
|
||||
tier: TierLabel,
|
||||
messages: list[dict],
|
||||
temperature: float,
|
||||
max_tokens: int | None,
|
||||
) -> dict:
|
||||
"""Dispatch a single inference request for the given tier."""
|
||||
model = self._tier_models[tier]
|
||||
cascade = self._get_cascade()
|
||||
start = time.monotonic()
|
||||
|
||||
logger.info(
|
||||
"TieredModelRouter: tier=%s model=%s messages=%d",
|
||||
tier,
|
||||
model,
|
||||
len(messages),
|
||||
)
|
||||
|
||||
result = await cascade.complete(
|
||||
messages=messages,
|
||||
model=model,
|
||||
temperature=temperature,
|
||||
max_tokens=max_tokens,
|
||||
)
|
||||
|
||||
elapsed_ms = (time.monotonic() - start) * 1000
|
||||
result["tier"] = tier
|
||||
result.setdefault("latency_ms", elapsed_ms)
|
||||
|
||||
logger.info(
|
||||
"TieredModelRouter: done tier=%s model=%s latency_ms=%.0f",
|
||||
tier,
|
||||
result.get("model", model),
|
||||
elapsed_ms,
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
# ── Module-level singleton ────────────────────────────────────────────────────
|
||||
|
||||
_tiered_router: TieredModelRouter | None = None
|
||||
|
||||
|
||||
def get_tiered_router() -> TieredModelRouter:
|
||||
"""Get or create the module-level TieredModelRouter singleton."""
|
||||
global _tiered_router
|
||||
if _tiered_router is None:
|
||||
_tiered_router = TieredModelRouter()
|
||||
return _tiered_router
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
from .api import router
|
||||
from .cascade import CascadeRouter, Provider, ProviderStatus, get_router
|
||||
from .classifier import TaskComplexity, classify_task
|
||||
from .history import HealthHistoryStore, get_history_store
|
||||
from .metabolic import (
|
||||
DEFAULT_TIER_MODELS,
|
||||
@@ -27,4 +28,7 @@ __all__ = [
|
||||
"classify_complexity",
|
||||
"build_prompt",
|
||||
"get_metabolic_router",
|
||||
# Classifier
|
||||
"TaskComplexity",
|
||||
"classify_task",
|
||||
]
|
||||
|
||||
@@ -16,7 +16,10 @@ from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime
|
||||
from enum import Enum
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from infrastructure.router.classifier import TaskComplexity
|
||||
|
||||
from config import settings
|
||||
|
||||
@@ -593,6 +596,34 @@ class CascadeRouter:
|
||||
"is_fallback_model": is_fallback_model,
|
||||
}
|
||||
|
||||
def _get_model_for_complexity(
|
||||
self, provider: Provider, complexity: "TaskComplexity"
|
||||
) -> str | None:
|
||||
"""Return the best model on *provider* for the given complexity tier.
|
||||
|
||||
Checks fallback chains first (routine / complex), then falls back to
|
||||
any model with the matching capability tag, then the provider default.
|
||||
"""
|
||||
from infrastructure.router.classifier import TaskComplexity
|
||||
|
||||
chain_key = "routine" if complexity == TaskComplexity.SIMPLE else "complex"
|
||||
|
||||
# Walk the capability fallback chain — first model present on this provider wins
|
||||
for model_name in self.config.fallback_chains.get(chain_key, []):
|
||||
if any(m["name"] == model_name for m in provider.models):
|
||||
return model_name
|
||||
|
||||
# Direct capability lookup — only return if a model explicitly has the tag
|
||||
# (do not use get_model_with_capability here as it falls back to the default)
|
||||
cap_model = next(
|
||||
(m["name"] for m in provider.models if chain_key in m.get("capabilities", [])),
|
||||
None,
|
||||
)
|
||||
if cap_model:
|
||||
return cap_model
|
||||
|
||||
return None # Caller will use provider default
|
||||
|
||||
async def complete(
|
||||
self,
|
||||
messages: list[dict],
|
||||
@@ -600,6 +631,7 @@ class CascadeRouter:
|
||||
temperature: float = 0.7,
|
||||
max_tokens: int | None = None,
|
||||
cascade_tier: str | None = None,
|
||||
complexity_hint: str | None = None,
|
||||
) -> dict:
|
||||
"""Complete a chat conversation with automatic failover.
|
||||
|
||||
@@ -608,33 +640,103 @@ class CascadeRouter:
|
||||
- Falls back to vision-capable models when needed
|
||||
- Supports image URLs, paths, and base64 encoding
|
||||
|
||||
Complexity-based routing (issue #1065):
|
||||
- ``complexity_hint="simple"`` → routes to Qwen3-8B (low-latency)
|
||||
- ``complexity_hint="complex"`` → routes to Qwen3-14B (quality)
|
||||
- ``complexity_hint=None`` (default) → auto-classifies from messages
|
||||
|
||||
Args:
|
||||
messages: List of message dicts with role and content
|
||||
model: Preferred model (tries this first, then provider defaults)
|
||||
model: Preferred model (tries this first; complexity routing is
|
||||
skipped when an explicit model is given)
|
||||
temperature: Sampling temperature
|
||||
max_tokens: Maximum tokens to generate
|
||||
cascade_tier: If specified, filters providers by this tier.
|
||||
- "frontier_required": Uses only Anthropic provider for top-tier models.
|
||||
complexity_hint: "simple", "complex", or None (auto-detect).
|
||||
|
||||
Returns:
|
||||
Dict with content, provider_used, and metrics
|
||||
Dict with content, provider_used, model, latency_ms,
|
||||
is_fallback_model, and complexity fields.
|
||||
|
||||
Raises:
|
||||
RuntimeError: If all providers fail
|
||||
"""
|
||||
from infrastructure.router.classifier import TaskComplexity, classify_task
|
||||
|
||||
content_type = self._detect_content_type(messages)
|
||||
if content_type != ContentType.TEXT:
|
||||
logger.debug("Detected %s content, selecting appropriate model", content_type.value)
|
||||
|
||||
# Resolve task complexity ─────────────────────────────────────────────
|
||||
# Skip complexity routing when caller explicitly specifies a model.
|
||||
complexity: TaskComplexity | None = None
|
||||
if model is None:
|
||||
if complexity_hint is not None:
|
||||
try:
|
||||
complexity = TaskComplexity(complexity_hint.lower())
|
||||
except ValueError:
|
||||
logger.warning("Unknown complexity_hint %r, auto-classifying", complexity_hint)
|
||||
complexity = classify_task(messages)
|
||||
else:
|
||||
complexity = classify_task(messages)
|
||||
logger.debug("Task complexity: %s", complexity.value)
|
||||
|
||||
errors: list[str] = []
|
||||
providers = self._filter_providers(cascade_tier)
|
||||
|
||||
for provider in providers:
|
||||
result = await self._try_single_provider(
|
||||
provider, messages, model, temperature, max_tokens, content_type, errors
|
||||
if not self._is_provider_available(provider):
|
||||
continue
|
||||
|
||||
# Metabolic protocol: skip cloud providers when quota is low
|
||||
if provider.type in ("anthropic", "openai", "grok"):
|
||||
if not self._quota_allows_cloud(provider):
|
||||
logger.info(
|
||||
"Metabolic protocol: skipping cloud provider %s (quota too low)",
|
||||
provider.name,
|
||||
)
|
||||
continue
|
||||
|
||||
# Complexity-based model selection (only when no explicit model) ──
|
||||
effective_model = model
|
||||
if effective_model is None and complexity is not None:
|
||||
effective_model = self._get_model_for_complexity(provider, complexity)
|
||||
if effective_model:
|
||||
logger.debug(
|
||||
"Complexity routing [%s]: %s → %s",
|
||||
complexity.value,
|
||||
provider.name,
|
||||
effective_model,
|
||||
)
|
||||
|
||||
selected_model, is_fallback_model = self._select_model(
|
||||
provider, effective_model, content_type
|
||||
)
|
||||
if result is not None:
|
||||
return result
|
||||
|
||||
try:
|
||||
result = await self._attempt_with_retry(
|
||||
provider,
|
||||
messages,
|
||||
selected_model,
|
||||
temperature,
|
||||
max_tokens,
|
||||
content_type,
|
||||
)
|
||||
except RuntimeError as exc:
|
||||
errors.append(str(exc))
|
||||
self._record_failure(provider)
|
||||
continue
|
||||
|
||||
self._record_success(provider, result.get("latency_ms", 0))
|
||||
return {
|
||||
"content": result["content"],
|
||||
"provider": provider.name,
|
||||
"model": result.get("model", selected_model or provider.get_default_model()),
|
||||
"latency_ms": result.get("latency_ms", 0),
|
||||
"is_fallback_model": is_fallback_model,
|
||||
"complexity": complexity.value if complexity is not None else None,
|
||||
}
|
||||
|
||||
raise RuntimeError(f"All providers failed: {'; '.join(errors)}")
|
||||
|
||||
|
||||
169
src/infrastructure/router/classifier.py
Normal file
169
src/infrastructure/router/classifier.py
Normal file
@@ -0,0 +1,169 @@
|
||||
"""Task complexity classifier for Qwen3 dual-model routing.
|
||||
|
||||
Classifies incoming tasks as SIMPLE (route to Qwen3-8B for low-latency)
|
||||
or COMPLEX (route to Qwen3-14B for quality-sensitive work).
|
||||
|
||||
Classification is fully heuristic — no LLM inference required.
|
||||
"""
|
||||
|
||||
import re
|
||||
from enum import Enum
|
||||
|
||||
|
||||
class TaskComplexity(Enum):
|
||||
"""Task complexity tier for model routing."""
|
||||
|
||||
SIMPLE = "simple" # Qwen3-8B Q6_K: routine, latency-sensitive
|
||||
COMPLEX = "complex" # Qwen3-14B Q5_K_M: quality-sensitive, multi-step
|
||||
|
||||
|
||||
# Keywords strongly associated with complex tasks
|
||||
_COMPLEX_KEYWORDS: frozenset[str] = frozenset(
|
||||
[
|
||||
"plan",
|
||||
"review",
|
||||
"analyze",
|
||||
"analyse",
|
||||
"triage",
|
||||
"refactor",
|
||||
"design",
|
||||
"architecture",
|
||||
"implement",
|
||||
"compare",
|
||||
"debug",
|
||||
"explain",
|
||||
"prioritize",
|
||||
"prioritise",
|
||||
"strategy",
|
||||
"optimize",
|
||||
"optimise",
|
||||
"evaluate",
|
||||
"assess",
|
||||
"brainstorm",
|
||||
"outline",
|
||||
"summarize",
|
||||
"summarise",
|
||||
"generate code",
|
||||
"write a",
|
||||
"write the",
|
||||
"code review",
|
||||
"pull request",
|
||||
"multi-step",
|
||||
"multi step",
|
||||
"step by step",
|
||||
"backlog prioriti",
|
||||
"issue triage",
|
||||
"root cause",
|
||||
"how does",
|
||||
"why does",
|
||||
"what are the",
|
||||
]
|
||||
)
|
||||
|
||||
# Keywords strongly associated with simple/routine tasks
|
||||
_SIMPLE_KEYWORDS: frozenset[str] = frozenset(
|
||||
[
|
||||
"status",
|
||||
"list ",
|
||||
"show ",
|
||||
"what is",
|
||||
"how many",
|
||||
"ping",
|
||||
"run ",
|
||||
"execute ",
|
||||
"ls ",
|
||||
"cat ",
|
||||
"ps ",
|
||||
"fetch ",
|
||||
"count ",
|
||||
"tail ",
|
||||
"head ",
|
||||
"grep ",
|
||||
"find file",
|
||||
"read file",
|
||||
"get ",
|
||||
"query ",
|
||||
"check ",
|
||||
"yes",
|
||||
"no",
|
||||
"ok",
|
||||
"done",
|
||||
"thanks",
|
||||
]
|
||||
)
|
||||
|
||||
# Content longer than this is treated as complex regardless of keywords
|
||||
_COMPLEX_CHAR_THRESHOLD = 500
|
||||
|
||||
# Short content defaults to simple
|
||||
_SIMPLE_CHAR_THRESHOLD = 150
|
||||
|
||||
# More than this many messages suggests an ongoing complex conversation
|
||||
_COMPLEX_CONVERSATION_DEPTH = 6
|
||||
|
||||
|
||||
def classify_task(messages: list[dict]) -> TaskComplexity:
|
||||
"""Classify task complexity from a list of messages.
|
||||
|
||||
Uses heuristic rules — no LLM call required. Errs toward COMPLEX
|
||||
when uncertain so that quality is preserved.
|
||||
|
||||
Args:
|
||||
messages: List of message dicts with ``role`` and ``content`` keys.
|
||||
|
||||
Returns:
|
||||
TaskComplexity.SIMPLE or TaskComplexity.COMPLEX
|
||||
"""
|
||||
if not messages:
|
||||
return TaskComplexity.SIMPLE
|
||||
|
||||
# Concatenate all user-turn content for analysis
|
||||
user_content = (
|
||||
" ".join(
|
||||
msg.get("content", "")
|
||||
for msg in messages
|
||||
if msg.get("role") in ("user", "human") and isinstance(msg.get("content"), str)
|
||||
)
|
||||
.lower()
|
||||
.strip()
|
||||
)
|
||||
|
||||
if not user_content:
|
||||
return TaskComplexity.SIMPLE
|
||||
|
||||
# Complexity signals override everything -----------------------------------
|
||||
|
||||
# Explicit complex keywords
|
||||
for kw in _COMPLEX_KEYWORDS:
|
||||
if kw in user_content:
|
||||
return TaskComplexity.COMPLEX
|
||||
|
||||
# Numbered / multi-step instruction list: "1. do this 2. do that"
|
||||
if re.search(r"\b\d+\.\s+\w", user_content):
|
||||
return TaskComplexity.COMPLEX
|
||||
|
||||
# Code blocks embedded in messages
|
||||
if "```" in user_content:
|
||||
return TaskComplexity.COMPLEX
|
||||
|
||||
# Long content → complex reasoning likely required
|
||||
if len(user_content) > _COMPLEX_CHAR_THRESHOLD:
|
||||
return TaskComplexity.COMPLEX
|
||||
|
||||
# Deep conversation → complex ongoing task
|
||||
if len(messages) > _COMPLEX_CONVERSATION_DEPTH:
|
||||
return TaskComplexity.COMPLEX
|
||||
|
||||
# Simplicity signals -------------------------------------------------------
|
||||
|
||||
# Explicit simple keywords
|
||||
for kw in _SIMPLE_KEYWORDS:
|
||||
if kw in user_content:
|
||||
return TaskComplexity.SIMPLE
|
||||
|
||||
# Short single-sentence messages default to simple
|
||||
if len(user_content) <= _SIMPLE_CHAR_THRESHOLD:
|
||||
return TaskComplexity.SIMPLE
|
||||
|
||||
# When uncertain, prefer quality (complex model)
|
||||
return TaskComplexity.COMPLEX
|
||||
247
src/infrastructure/self_correction.py
Normal file
247
src/infrastructure/self_correction.py
Normal file
@@ -0,0 +1,247 @@
|
||||
"""Self-correction event logger.
|
||||
|
||||
Records instances where the agent detected its own errors and the steps
|
||||
it took to correct them. Used by the Self-Correction Dashboard to visualise
|
||||
these events and surface recurring failure patterns.
|
||||
|
||||
Usage::
|
||||
|
||||
from infrastructure.self_correction import log_self_correction, get_corrections, get_patterns
|
||||
|
||||
log_self_correction(
|
||||
source="agentic_loop",
|
||||
original_intent="Execute step 3: deploy service",
|
||||
detected_error="ConnectionRefusedError: port 8080 unavailable",
|
||||
correction_strategy="Retry on alternate port 8081",
|
||||
final_outcome="Success on retry",
|
||||
task_id="abc123",
|
||||
)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import sqlite3
|
||||
import uuid
|
||||
from collections.abc import Generator
|
||||
from contextlib import closing, contextmanager
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Database
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_DB_PATH: Path | None = None
|
||||
|
||||
|
||||
def _get_db_path() -> Path:
|
||||
global _DB_PATH
|
||||
if _DB_PATH is None:
|
||||
from config import settings
|
||||
|
||||
_DB_PATH = Path(settings.repo_root) / "data" / "self_correction.db"
|
||||
return _DB_PATH
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _get_db() -> Generator[sqlite3.Connection, None, None]:
|
||||
db_path = _get_db_path()
|
||||
db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with closing(sqlite3.connect(str(db_path))) as conn:
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.execute("""
|
||||
CREATE TABLE IF NOT EXISTS self_correction_events (
|
||||
id TEXT PRIMARY KEY,
|
||||
source TEXT NOT NULL,
|
||||
task_id TEXT DEFAULT '',
|
||||
original_intent TEXT NOT NULL,
|
||||
detected_error TEXT NOT NULL,
|
||||
correction_strategy TEXT NOT NULL,
|
||||
final_outcome TEXT NOT NULL,
|
||||
outcome_status TEXT DEFAULT 'success',
|
||||
error_type TEXT DEFAULT '',
|
||||
created_at TEXT DEFAULT (datetime('now'))
|
||||
)
|
||||
""")
|
||||
conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_sc_created ON self_correction_events(created_at)"
|
||||
)
|
||||
conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_sc_error_type ON self_correction_events(error_type)"
|
||||
)
|
||||
conn.commit()
|
||||
yield conn
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Write
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def log_self_correction(
|
||||
*,
|
||||
source: str,
|
||||
original_intent: str,
|
||||
detected_error: str,
|
||||
correction_strategy: str,
|
||||
final_outcome: str,
|
||||
task_id: str = "",
|
||||
outcome_status: str = "success",
|
||||
error_type: str = "",
|
||||
) -> str:
|
||||
"""Record a self-correction event and return its ID.
|
||||
|
||||
Args:
|
||||
source: Module or component that triggered the correction.
|
||||
original_intent: What the agent was trying to do.
|
||||
detected_error: The error or problem that was detected.
|
||||
correction_strategy: How the agent attempted to correct the error.
|
||||
final_outcome: What the result of the correction attempt was.
|
||||
task_id: Optional task/session ID for correlation.
|
||||
outcome_status: 'success', 'partial', or 'failed'.
|
||||
error_type: Short category label for pattern analysis (e.g.
|
||||
'ConnectionError', 'TimeoutError').
|
||||
|
||||
Returns:
|
||||
The ID of the newly created record.
|
||||
"""
|
||||
event_id = str(uuid.uuid4())
|
||||
if not error_type:
|
||||
# Derive a simple type from the first word of the detected error
|
||||
error_type = detected_error.split(":")[0].strip()[:64]
|
||||
|
||||
try:
|
||||
with _get_db() as conn:
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO self_correction_events
|
||||
(id, source, task_id, original_intent, detected_error,
|
||||
correction_strategy, final_outcome, outcome_status, error_type)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
(
|
||||
event_id,
|
||||
source,
|
||||
task_id,
|
||||
original_intent[:2000],
|
||||
detected_error[:2000],
|
||||
correction_strategy[:2000],
|
||||
final_outcome[:2000],
|
||||
outcome_status,
|
||||
error_type,
|
||||
),
|
||||
)
|
||||
conn.commit()
|
||||
logger.info(
|
||||
"Self-correction logged [%s] source=%s error_type=%s status=%s",
|
||||
event_id[:8],
|
||||
source,
|
||||
error_type,
|
||||
outcome_status,
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to log self-correction event: %s", exc)
|
||||
|
||||
return event_id
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Read
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def get_corrections(limit: int = 50) -> list[dict]:
|
||||
"""Return the most recent self-correction events, newest first."""
|
||||
try:
|
||||
with _get_db() as conn:
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT * FROM self_correction_events
|
||||
ORDER BY created_at DESC
|
||||
LIMIT ?
|
||||
""",
|
||||
(limit,),
|
||||
).fetchall()
|
||||
return [dict(r) for r in rows]
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to fetch self-correction events: %s", exc)
|
||||
return []
|
||||
|
||||
|
||||
def get_patterns(top_n: int = 10) -> list[dict]:
|
||||
"""Return the most common recurring error types with counts.
|
||||
|
||||
Each entry has:
|
||||
- error_type: category label
|
||||
- count: total occurrences
|
||||
- success_count: corrected successfully
|
||||
- failed_count: correction also failed
|
||||
- last_seen: ISO timestamp of most recent occurrence
|
||||
"""
|
||||
try:
|
||||
with _get_db() as conn:
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT
|
||||
error_type,
|
||||
COUNT(*) AS count,
|
||||
SUM(CASE WHEN outcome_status = 'success' THEN 1 ELSE 0 END) AS success_count,
|
||||
SUM(CASE WHEN outcome_status = 'failed' THEN 1 ELSE 0 END) AS failed_count,
|
||||
MAX(created_at) AS last_seen
|
||||
FROM self_correction_events
|
||||
GROUP BY error_type
|
||||
ORDER BY count DESC
|
||||
LIMIT ?
|
||||
""",
|
||||
(top_n,),
|
||||
).fetchall()
|
||||
return [dict(r) for r in rows]
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to fetch self-correction patterns: %s", exc)
|
||||
return []
|
||||
|
||||
|
||||
def get_stats() -> dict:
|
||||
"""Return aggregate statistics for the summary panel."""
|
||||
try:
|
||||
with _get_db() as conn:
|
||||
row = conn.execute(
|
||||
"""
|
||||
SELECT
|
||||
COUNT(*) AS total,
|
||||
SUM(CASE WHEN outcome_status = 'success' THEN 1 ELSE 0 END) AS success_count,
|
||||
SUM(CASE WHEN outcome_status = 'partial' THEN 1 ELSE 0 END) AS partial_count,
|
||||
SUM(CASE WHEN outcome_status = 'failed' THEN 1 ELSE 0 END) AS failed_count,
|
||||
COUNT(DISTINCT error_type) AS unique_error_types,
|
||||
COUNT(DISTINCT source) AS sources
|
||||
FROM self_correction_events
|
||||
"""
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return _empty_stats()
|
||||
d = dict(row)
|
||||
total = d.get("total") or 0
|
||||
if total:
|
||||
d["success_rate"] = round((d.get("success_count") or 0) / total * 100)
|
||||
else:
|
||||
d["success_rate"] = 0
|
||||
return d
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to fetch self-correction stats: %s", exc)
|
||||
return _empty_stats()
|
||||
|
||||
|
||||
def _empty_stats() -> dict:
|
||||
return {
|
||||
"total": 0,
|
||||
"success_count": 0,
|
||||
"partial_count": 0,
|
||||
"failed_count": 0,
|
||||
"unique_error_types": 0,
|
||||
"sources": 0,
|
||||
"success_rate": 0,
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
"""Vendor-specific chat platform adapters (e.g. Discord) for the chat bridge."""
|
||||
|
||||
7
src/self_coding/__init__.py
Normal file
7
src/self_coding/__init__.py
Normal file
@@ -0,0 +1,7 @@
|
||||
"""Self-coding package — Timmy's self-modification capability.
|
||||
|
||||
Provides the branch→edit→test→commit/revert loop that allows Timmy
|
||||
to propose and apply code changes autonomously, gated by the test suite.
|
||||
|
||||
Main entry point: ``self_coding.self_modify.loop``
|
||||
"""
|
||||
129
src/self_coding/gitea_client.py
Normal file
129
src/self_coding/gitea_client.py
Normal file
@@ -0,0 +1,129 @@
|
||||
"""Gitea REST client — thin wrapper for PR creation and issue commenting.
|
||||
|
||||
Uses ``settings.gitea_url``, ``settings.gitea_token``, and
|
||||
``settings.gitea_repo`` (owner/repo) from config. Degrades gracefully
|
||||
when the token is absent or the server is unreachable.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class PullRequest:
|
||||
"""Minimal representation of a created pull request."""
|
||||
|
||||
number: int
|
||||
title: str
|
||||
html_url: str
|
||||
|
||||
|
||||
class GiteaClient:
|
||||
"""HTTP client for Gitea's REST API v1.
|
||||
|
||||
All methods return structured results and never raise — errors are
|
||||
logged at WARNING level and indicated via return value.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
base_url: str | None = None,
|
||||
token: str | None = None,
|
||||
repo: str | None = None,
|
||||
) -> None:
|
||||
from config import settings
|
||||
|
||||
self._base_url = (base_url or settings.gitea_url).rstrip("/")
|
||||
self._token = token or settings.gitea_token
|
||||
self._repo = repo or settings.gitea_repo
|
||||
|
||||
# ── internal ────────────────────────────────────────────────────────────
|
||||
|
||||
def _headers(self) -> dict[str, str]:
|
||||
return {
|
||||
"Authorization": f"token {self._token}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
def _api(self, path: str) -> str:
|
||||
return f"{self._base_url}/api/v1/{path.lstrip('/')}"
|
||||
|
||||
# ── public API ───────────────────────────────────────────────────────────
|
||||
|
||||
def create_pull_request(
|
||||
self,
|
||||
title: str,
|
||||
body: str,
|
||||
head: str,
|
||||
base: str = "main",
|
||||
) -> PullRequest | None:
|
||||
"""Open a pull request.
|
||||
|
||||
Args:
|
||||
title: PR title (keep under 70 chars).
|
||||
body: PR body in markdown.
|
||||
head: Source branch (e.g. ``self-modify/issue-983``).
|
||||
base: Target branch (default ``main``).
|
||||
|
||||
Returns:
|
||||
A ``PullRequest`` dataclass on success, ``None`` on failure.
|
||||
"""
|
||||
if not self._token:
|
||||
logger.warning("Gitea token not configured — skipping PR creation")
|
||||
return None
|
||||
|
||||
try:
|
||||
import requests as _requests
|
||||
|
||||
resp = _requests.post(
|
||||
self._api(f"repos/{self._repo}/pulls"),
|
||||
headers=self._headers(),
|
||||
json={"title": title, "body": body, "head": head, "base": base},
|
||||
timeout=15,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
pr = PullRequest(
|
||||
number=data["number"],
|
||||
title=data["title"],
|
||||
html_url=data["html_url"],
|
||||
)
|
||||
logger.info("PR #%d created: %s", pr.number, pr.html_url)
|
||||
return pr
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to create PR: %s", exc)
|
||||
return None
|
||||
|
||||
def add_issue_comment(self, issue_number: int, body: str) -> bool:
|
||||
"""Post a comment on an issue or PR.
|
||||
|
||||
Returns:
|
||||
True on success, False on failure.
|
||||
"""
|
||||
if not self._token:
|
||||
logger.warning("Gitea token not configured — skipping issue comment")
|
||||
return False
|
||||
|
||||
try:
|
||||
import requests as _requests
|
||||
|
||||
resp = _requests.post(
|
||||
self._api(f"repos/{self._repo}/issues/{issue_number}/comments"),
|
||||
headers=self._headers(),
|
||||
json={"body": body},
|
||||
timeout=15,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
logger.info("Comment posted on issue #%d", issue_number)
|
||||
return True
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to post comment on issue #%d: %s", issue_number, exc)
|
||||
return False
|
||||
|
||||
|
||||
# Module-level singleton
|
||||
gitea_client = GiteaClient()
|
||||
1
src/self_coding/self_modify/__init__.py
Normal file
1
src/self_coding/self_modify/__init__.py
Normal file
@@ -0,0 +1 @@
|
||||
"""Self-modification loop sub-package."""
|
||||
301
src/self_coding/self_modify/loop.py
Normal file
301
src/self_coding/self_modify/loop.py
Normal file
@@ -0,0 +1,301 @@
|
||||
"""Self-modification loop — branch → edit → test → commit/revert.
|
||||
|
||||
Timmy's self-coding capability, restored after deletion in
|
||||
Operation Darling Purge (commit 584eeb679e88).
|
||||
|
||||
## Cycle
|
||||
1. **Branch** — create ``self-modify/<slug>`` from ``main``
|
||||
2. **Edit** — apply the proposed change (patch string or callable)
|
||||
3. **Test** — run ``pytest tests/ -x -q``; never commit on failure
|
||||
4. **Commit** — stage and commit on green; revert branch on red
|
||||
5. **PR** — open a Gitea pull request (requires no direct push to main)
|
||||
|
||||
## Guards
|
||||
- Never push directly to ``main`` or ``master``
|
||||
- All changes land via PR (enforced by ``_guard_branch``)
|
||||
- Test gate is mandatory; ``skip_tests=True`` is for unit-test use only
|
||||
- Commits only happen when ``pytest tests/ -x -q`` exits 0
|
||||
|
||||
## Usage::
|
||||
|
||||
from self_coding.self_modify.loop import SelfModifyLoop
|
||||
|
||||
loop = SelfModifyLoop()
|
||||
result = await loop.run(
|
||||
slug="add-hello-tool",
|
||||
description="Add hello() convenience tool",
|
||||
edit_fn=my_edit_function, # callable(repo_root: str) -> None
|
||||
)
|
||||
if result.success:
|
||||
print(f"PR: {result.pr_url}")
|
||||
else:
|
||||
print(f"Failed: {result.error}")
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import subprocess
|
||||
import time
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
from config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Branches that must never receive direct commits
|
||||
_PROTECTED_BRANCHES = frozenset({"main", "master", "develop"})
|
||||
|
||||
# Test command used as the commit gate
|
||||
_TEST_COMMAND = ["pytest", "tests/", "-x", "-q", "--tb=short"]
|
||||
|
||||
# Max time (seconds) to wait for the test suite
|
||||
_TEST_TIMEOUT = 300
|
||||
|
||||
|
||||
@dataclass
|
||||
class LoopResult:
|
||||
"""Result from one self-modification cycle."""
|
||||
|
||||
success: bool
|
||||
branch: str = ""
|
||||
commit_sha: str = ""
|
||||
pr_url: str = ""
|
||||
pr_number: int = 0
|
||||
test_output: str = ""
|
||||
error: str = ""
|
||||
elapsed_ms: float = 0.0
|
||||
metadata: dict = field(default_factory=dict)
|
||||
|
||||
|
||||
class SelfModifyLoop:
|
||||
"""Orchestrate branch → edit → test → commit/revert → PR.
|
||||
|
||||
Args:
|
||||
repo_root: Absolute path to the git repository (defaults to
|
||||
``settings.repo_root``).
|
||||
remote: Git remote name (default ``origin``).
|
||||
base_branch: Branch to fork from and target for the PR
|
||||
(default ``main``).
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
repo_root: str | None = None,
|
||||
remote: str = "origin",
|
||||
base_branch: str = "main",
|
||||
) -> None:
|
||||
self._repo_root = Path(repo_root or settings.repo_root)
|
||||
self._remote = remote
|
||||
self._base_branch = base_branch
|
||||
|
||||
# ── public ──────────────────────────────────────────────────────────────
|
||||
|
||||
async def run(
|
||||
self,
|
||||
slug: str,
|
||||
description: str,
|
||||
edit_fn: Callable[[str], None],
|
||||
issue_number: int | None = None,
|
||||
skip_tests: bool = False,
|
||||
) -> LoopResult:
|
||||
"""Execute one full self-modification cycle.
|
||||
|
||||
Args:
|
||||
slug: Short identifier used for the branch name
|
||||
(e.g. ``"add-hello-tool"``).
|
||||
description: Human-readable description for commit message
|
||||
and PR body.
|
||||
edit_fn: Callable that receives the repo root path (str)
|
||||
and applies the desired code changes in-place.
|
||||
issue_number: Optional Gitea issue number to reference in PR.
|
||||
skip_tests: If ``True``, skip the test gate (unit-test use
|
||||
only — never use in production).
|
||||
|
||||
Returns:
|
||||
:class:`LoopResult` describing the outcome.
|
||||
"""
|
||||
start = time.time()
|
||||
branch = f"self-modify/{slug}"
|
||||
|
||||
try:
|
||||
self._guard_branch(branch)
|
||||
self._checkout_base()
|
||||
self._create_branch(branch)
|
||||
|
||||
try:
|
||||
edit_fn(str(self._repo_root))
|
||||
except Exception as exc:
|
||||
self._revert_branch(branch)
|
||||
return LoopResult(
|
||||
success=False,
|
||||
branch=branch,
|
||||
error=f"edit_fn raised: {exc}",
|
||||
elapsed_ms=self._elapsed(start),
|
||||
)
|
||||
|
||||
if not skip_tests:
|
||||
test_output, passed = self._run_tests()
|
||||
if not passed:
|
||||
self._revert_branch(branch)
|
||||
return LoopResult(
|
||||
success=False,
|
||||
branch=branch,
|
||||
test_output=test_output,
|
||||
error="Tests failed — branch reverted",
|
||||
elapsed_ms=self._elapsed(start),
|
||||
)
|
||||
else:
|
||||
test_output = "(tests skipped)"
|
||||
|
||||
sha = self._commit_all(description)
|
||||
self._push_branch(branch)
|
||||
|
||||
pr = self._create_pr(
|
||||
branch=branch,
|
||||
description=description,
|
||||
test_output=test_output,
|
||||
issue_number=issue_number,
|
||||
)
|
||||
|
||||
return LoopResult(
|
||||
success=True,
|
||||
branch=branch,
|
||||
commit_sha=sha,
|
||||
pr_url=pr.html_url if pr else "",
|
||||
pr_number=pr.number if pr else 0,
|
||||
test_output=test_output,
|
||||
elapsed_ms=self._elapsed(start),
|
||||
)
|
||||
|
||||
except Exception as exc:
|
||||
logger.warning("Self-modify loop failed: %s", exc)
|
||||
return LoopResult(
|
||||
success=False,
|
||||
branch=branch,
|
||||
error=str(exc),
|
||||
elapsed_ms=self._elapsed(start),
|
||||
)
|
||||
|
||||
# ── private helpers ──────────────────────────────────────────────────────
|
||||
|
||||
@staticmethod
|
||||
def _elapsed(start: float) -> float:
|
||||
return (time.time() - start) * 1000
|
||||
|
||||
def _git(self, *args: str, check: bool = True) -> subprocess.CompletedProcess:
|
||||
"""Run a git command in the repo root."""
|
||||
cmd = ["git", *args]
|
||||
logger.debug("git %s", " ".join(args))
|
||||
return subprocess.run(
|
||||
cmd,
|
||||
cwd=str(self._repo_root),
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=check,
|
||||
)
|
||||
|
||||
def _guard_branch(self, branch: str) -> None:
|
||||
"""Raise if the target branch is a protected branch name."""
|
||||
if branch in _PROTECTED_BRANCHES:
|
||||
raise ValueError(
|
||||
f"Refusing to operate on protected branch '{branch}'. "
|
||||
"All self-modifications must go via PR."
|
||||
)
|
||||
|
||||
def _checkout_base(self) -> None:
|
||||
"""Checkout the base branch and pull latest."""
|
||||
self._git("checkout", self._base_branch)
|
||||
# Best-effort pull; ignore failures (e.g. no remote configured)
|
||||
self._git("pull", self._remote, self._base_branch, check=False)
|
||||
|
||||
def _create_branch(self, branch: str) -> None:
|
||||
"""Create and checkout a new branch, deleting an old one if needed."""
|
||||
# Delete local branch if it already exists (stale prior attempt)
|
||||
self._git("branch", "-D", branch, check=False)
|
||||
self._git("checkout", "-b", branch)
|
||||
logger.info("Created branch: %s", branch)
|
||||
|
||||
def _revert_branch(self, branch: str) -> None:
|
||||
"""Checkout base and delete the failed branch."""
|
||||
try:
|
||||
self._git("checkout", self._base_branch, check=False)
|
||||
self._git("branch", "-D", branch, check=False)
|
||||
logger.info("Reverted and deleted branch: %s", branch)
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to revert branch %s: %s", branch, exc)
|
||||
|
||||
def _run_tests(self) -> tuple[str, bool]:
|
||||
"""Run the test suite. Returns (output, passed)."""
|
||||
logger.info("Running test suite: %s", " ".join(_TEST_COMMAND))
|
||||
try:
|
||||
result = subprocess.run(
|
||||
_TEST_COMMAND,
|
||||
cwd=str(self._repo_root),
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=_TEST_TIMEOUT,
|
||||
)
|
||||
output = (result.stdout + "\n" + result.stderr).strip()
|
||||
passed = result.returncode == 0
|
||||
logger.info(
|
||||
"Test suite %s (exit %d)", "PASSED" if passed else "FAILED", result.returncode
|
||||
)
|
||||
return output, passed
|
||||
except subprocess.TimeoutExpired:
|
||||
msg = f"Test suite timed out after {_TEST_TIMEOUT}s"
|
||||
logger.warning(msg)
|
||||
return msg, False
|
||||
except FileNotFoundError:
|
||||
msg = "pytest not found on PATH"
|
||||
logger.warning(msg)
|
||||
return msg, False
|
||||
|
||||
def _commit_all(self, message: str) -> str:
|
||||
"""Stage all changes and create a commit. Returns the new SHA."""
|
||||
self._git("add", "-A")
|
||||
self._git("commit", "-m", message)
|
||||
result = self._git("rev-parse", "HEAD")
|
||||
sha = result.stdout.strip()
|
||||
logger.info("Committed: %s sha=%s", message[:60], sha[:12])
|
||||
return sha
|
||||
|
||||
def _push_branch(self, branch: str) -> None:
|
||||
"""Push the branch to the remote."""
|
||||
self._git("push", "-u", self._remote, branch)
|
||||
logger.info("Pushed branch: %s -> %s", branch, self._remote)
|
||||
|
||||
def _create_pr(
|
||||
self,
|
||||
branch: str,
|
||||
description: str,
|
||||
test_output: str,
|
||||
issue_number: int | None,
|
||||
):
|
||||
"""Open a Gitea PR. Returns PullRequest or None on failure."""
|
||||
from self_coding.gitea_client import GiteaClient
|
||||
|
||||
client = GiteaClient()
|
||||
|
||||
issue_ref = f"\n\nFixes #{issue_number}" if issue_number else ""
|
||||
test_section = (
|
||||
f"\n\n## Test results\n```\n{test_output[:2000]}\n```"
|
||||
if test_output and test_output != "(tests skipped)"
|
||||
else ""
|
||||
)
|
||||
|
||||
body = (
|
||||
f"## Summary\n{description}"
|
||||
f"{issue_ref}"
|
||||
f"{test_section}"
|
||||
"\n\n🤖 Generated by Timmy's self-modification loop"
|
||||
)
|
||||
|
||||
return client.create_pull_request(
|
||||
title=f"[self-modify] {description[:60]}",
|
||||
body=body,
|
||||
head=branch,
|
||||
base=self._base_branch,
|
||||
)
|
||||
@@ -301,6 +301,26 @@ def create_timmy(
|
||||
|
||||
return GrokBackend()
|
||||
|
||||
if resolved == "airllm":
|
||||
# AirLLM requires Apple Silicon. On any other platform (Intel Mac, Linux,
|
||||
# Windows) or when the package is not installed, degrade silently to Ollama.
|
||||
from timmy.backends import is_apple_silicon
|
||||
|
||||
if not is_apple_silicon():
|
||||
logger.warning(
|
||||
"TIMMY_MODEL_BACKEND=airllm requested but not running on Apple Silicon "
|
||||
"— falling back to Ollama"
|
||||
)
|
||||
else:
|
||||
try:
|
||||
import airllm # noqa: F401
|
||||
except ImportError:
|
||||
logger.warning(
|
||||
"AirLLM not installed — falling back to Ollama. "
|
||||
"Install with: pip install 'airllm[mlx]'"
|
||||
)
|
||||
# Fall through to Ollama in all cases (AirLLM integration is scaffolded)
|
||||
|
||||
# Default: Ollama via Agno.
|
||||
model_name, is_fallback = _resolve_model_with_fallback(
|
||||
requested_model=None,
|
||||
|
||||
@@ -312,6 +312,13 @@ async def _handle_step_failure(
|
||||
"adaptation": step.result[:200],
|
||||
},
|
||||
)
|
||||
_log_self_correction(
|
||||
task_id=task_id,
|
||||
step_desc=step_desc,
|
||||
exc=exc,
|
||||
outcome=step.result,
|
||||
outcome_status="success",
|
||||
)
|
||||
if on_progress:
|
||||
await on_progress(f"[Adapted] {step_desc}", step_num, total_steps)
|
||||
except Exception as adapt_exc: # broad catch intentional
|
||||
@@ -325,9 +332,42 @@ async def _handle_step_failure(
|
||||
duration_ms=int((time.monotonic() - step_start) * 1000),
|
||||
)
|
||||
)
|
||||
_log_self_correction(
|
||||
task_id=task_id,
|
||||
step_desc=step_desc,
|
||||
exc=exc,
|
||||
outcome=f"Adaptation also failed: {adapt_exc}",
|
||||
outcome_status="failed",
|
||||
)
|
||||
completed_results.append(f"Step {step_num}: FAILED")
|
||||
|
||||
|
||||
def _log_self_correction(
|
||||
*,
|
||||
task_id: str,
|
||||
step_desc: str,
|
||||
exc: Exception,
|
||||
outcome: str,
|
||||
outcome_status: str,
|
||||
) -> None:
|
||||
"""Best-effort: log a self-correction event (never raises)."""
|
||||
try:
|
||||
from infrastructure.self_correction import log_self_correction
|
||||
|
||||
log_self_correction(
|
||||
source="agentic_loop",
|
||||
original_intent=step_desc,
|
||||
detected_error=f"{type(exc).__name__}: {exc}",
|
||||
correction_strategy="Adaptive re-plan via LLM",
|
||||
final_outcome=outcome[:500],
|
||||
task_id=task_id,
|
||||
outcome_status=outcome_status,
|
||||
error_type=type(exc).__name__,
|
||||
)
|
||||
except Exception as log_exc:
|
||||
logger.debug("Self-correction log failed: %s", log_exc)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Core loop
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -8,7 +8,7 @@ Flow:
|
||||
1. prepare_experiment — clone repo + run data prep
|
||||
2. run_experiment — execute train.py with wall-clock timeout
|
||||
3. evaluate_result — compare metric against baseline
|
||||
4. experiment_loop — orchestrate the full cycle
|
||||
4. SystemExperiment — orchestrate the full cycle via class interface
|
||||
|
||||
All subprocess calls are guarded with timeouts for graceful degradation.
|
||||
"""
|
||||
@@ -17,9 +17,12 @@ from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import platform
|
||||
import re
|
||||
import subprocess
|
||||
import time
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
@@ -29,15 +32,61 @@ DEFAULT_REPO = "https://github.com/karpathy/autoresearch.git"
|
||||
_METRIC_RE = re.compile(r"val_bpb[:\s]+([0-9]+\.?[0-9]*)")
|
||||
|
||||
|
||||
# ── Higher-is-better metric names ────────────────────────────────────────────
|
||||
_HIGHER_IS_BETTER = frozenset({"unit_pass_rate", "coverage"})
|
||||
|
||||
|
||||
def is_apple_silicon() -> bool:
|
||||
"""Return True when running on Apple Silicon (M-series chip)."""
|
||||
return platform.system() == "Darwin" and platform.machine() == "arm64"
|
||||
|
||||
|
||||
def _build_experiment_env(
|
||||
dataset: str = "tinystories",
|
||||
backend: str = "auto",
|
||||
) -> dict[str, str]:
|
||||
"""Build environment variables for an autoresearch subprocess.
|
||||
|
||||
Args:
|
||||
dataset: Dataset name forwarded as ``AUTORESEARCH_DATASET``.
|
||||
``"tinystories"`` is recommended for Apple Silicon (lower entropy,
|
||||
faster iteration).
|
||||
backend: Inference backend forwarded as ``AUTORESEARCH_BACKEND``.
|
||||
``"auto"`` enables MLX on Apple Silicon; ``"cpu"`` forces CPU.
|
||||
|
||||
Returns:
|
||||
Merged environment dict (inherits current process env).
|
||||
"""
|
||||
env = os.environ.copy()
|
||||
env["AUTORESEARCH_DATASET"] = dataset
|
||||
|
||||
if backend == "auto":
|
||||
env["AUTORESEARCH_BACKEND"] = "mlx" if is_apple_silicon() else "cuda"
|
||||
else:
|
||||
env["AUTORESEARCH_BACKEND"] = backend
|
||||
|
||||
return env
|
||||
|
||||
|
||||
def prepare_experiment(
|
||||
workspace: Path,
|
||||
repo_url: str = DEFAULT_REPO,
|
||||
dataset: str = "tinystories",
|
||||
backend: str = "auto",
|
||||
) -> str:
|
||||
"""Clone autoresearch repo and run data preparation.
|
||||
|
||||
On Apple Silicon the ``dataset`` defaults to ``"tinystories"`` (lower
|
||||
entropy, faster iteration) and ``backend`` to ``"auto"`` which resolves to
|
||||
MLX. Both values are forwarded as ``AUTORESEARCH_DATASET`` /
|
||||
``AUTORESEARCH_BACKEND`` environment variables so that ``prepare.py`` and
|
||||
``train.py`` can adapt their behaviour without CLI changes.
|
||||
|
||||
Args:
|
||||
workspace: Directory to set up the experiment in.
|
||||
repo_url: Git URL for the autoresearch repository.
|
||||
dataset: Dataset name; ``"tinystories"`` is recommended on Mac.
|
||||
backend: Inference backend; ``"auto"`` picks MLX on Apple Silicon.
|
||||
|
||||
Returns:
|
||||
Status message describing what was prepared.
|
||||
@@ -59,6 +108,14 @@ def prepare_experiment(
|
||||
else:
|
||||
logger.info("Autoresearch repo already present at %s", repo_dir)
|
||||
|
||||
env = _build_experiment_env(dataset=dataset, backend=backend)
|
||||
if is_apple_silicon():
|
||||
logger.info(
|
||||
"Apple Silicon detected — dataset=%s backend=%s",
|
||||
env["AUTORESEARCH_DATASET"],
|
||||
env["AUTORESEARCH_BACKEND"],
|
||||
)
|
||||
|
||||
# Run prepare.py (data download + tokeniser training)
|
||||
prepare_script = repo_dir / "prepare.py"
|
||||
if prepare_script.exists():
|
||||
@@ -69,6 +126,7 @@ def prepare_experiment(
|
||||
text=True,
|
||||
cwd=str(repo_dir),
|
||||
timeout=300,
|
||||
env=env,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
return f"Preparation failed: {result.stderr.strip()[:500]}"
|
||||
@@ -81,6 +139,8 @@ def run_experiment(
|
||||
workspace: Path,
|
||||
timeout: int = 300,
|
||||
metric_name: str = "val_bpb",
|
||||
dataset: str = "tinystories",
|
||||
backend: str = "auto",
|
||||
) -> dict[str, Any]:
|
||||
"""Run a single training experiment with a wall-clock timeout.
|
||||
|
||||
@@ -88,6 +148,9 @@ def run_experiment(
|
||||
workspace: Experiment workspace (contains autoresearch/ subdir).
|
||||
timeout: Maximum wall-clock seconds for the run.
|
||||
metric_name: Name of the metric to extract from stdout.
|
||||
dataset: Dataset forwarded to the subprocess via env var.
|
||||
backend: Inference backend forwarded via env var (``"auto"`` → MLX on
|
||||
Apple Silicon, CUDA otherwise).
|
||||
|
||||
Returns:
|
||||
Dict with keys: metric (float|None), log (str), duration_s (int),
|
||||
@@ -105,6 +168,7 @@ def run_experiment(
|
||||
"error": f"train.py not found in {repo_dir}",
|
||||
}
|
||||
|
||||
env = _build_experiment_env(dataset=dataset, backend=backend)
|
||||
start = time.monotonic()
|
||||
try:
|
||||
result = subprocess.run(
|
||||
@@ -113,6 +177,7 @@ def run_experiment(
|
||||
text=True,
|
||||
cwd=str(repo_dir),
|
||||
timeout=timeout,
|
||||
env=env,
|
||||
)
|
||||
duration = int(time.monotonic() - start)
|
||||
output = result.stdout + result.stderr
|
||||
@@ -125,7 +190,7 @@ def run_experiment(
|
||||
"log": output[-2000:], # Keep last 2k chars
|
||||
"duration_s": duration,
|
||||
"success": result.returncode == 0,
|
||||
"error": None if result.returncode == 0 else f"Exit code {result.returncode}",
|
||||
"error": (None if result.returncode == 0 else f"Exit code {result.returncode}"),
|
||||
}
|
||||
except subprocess.TimeoutExpired:
|
||||
duration = int(time.monotonic() - start)
|
||||
@@ -212,3 +277,369 @@ def _append_result(workspace: Path, result: dict[str, Any]) -> None:
|
||||
results_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
with results_file.open("a") as f:
|
||||
f.write(json.dumps(result) + "\n")
|
||||
|
||||
|
||||
def _extract_pass_rate(output: str) -> float | None:
|
||||
"""Extract pytest pass rate as a percentage from tox/pytest output."""
|
||||
passed_m = re.search(r"(\d+) passed", output)
|
||||
failed_m = re.search(r"(\d+) failed", output)
|
||||
if passed_m:
|
||||
passed = int(passed_m.group(1))
|
||||
failed = int(failed_m.group(1)) if failed_m else 0
|
||||
total = passed + failed
|
||||
return (passed / total * 100.0) if total > 0 else 100.0
|
||||
return None
|
||||
|
||||
|
||||
def _extract_coverage(output: str) -> float | None:
|
||||
"""Extract total coverage percentage from coverage output."""
|
||||
coverage_m = re.search(r"(?:TOTAL\s+\d+\s+\d+\s+|Total coverage:\s*)(\d+)%", output)
|
||||
if coverage_m:
|
||||
try:
|
||||
return float(coverage_m.group(1))
|
||||
except ValueError:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
class SystemExperiment:
|
||||
"""An autoresearch experiment targeting a specific module with a configurable metric.
|
||||
|
||||
Encapsulates the hypothesis → edit → tox → evaluate → commit/revert loop
|
||||
for a single target file or module.
|
||||
|
||||
Args:
|
||||
target: Path or module name to optimise (e.g. ``src/timmy/agent.py``).
|
||||
metric: Metric to extract from tox output. Built-in values:
|
||||
``unit_pass_rate`` (default), ``coverage``, ``val_bpb``.
|
||||
Any other value is forwarded to :func:`_extract_metric`.
|
||||
budget_minutes: Wall-clock budget per experiment (default 5 min).
|
||||
workspace: Working directory for subprocess calls. Defaults to ``cwd``.
|
||||
revert_on_failure: Whether to revert changes on failed experiments.
|
||||
hypothesis: Optional natural language hypothesis for the experiment.
|
||||
metric_fn: Optional callable for custom metric extraction.
|
||||
If provided, overrides built-in metric extraction.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
target: str,
|
||||
metric: str = "unit_pass_rate",
|
||||
budget_minutes: int = 5,
|
||||
workspace: Path | None = None,
|
||||
revert_on_failure: bool = True,
|
||||
hypothesis: str = "",
|
||||
metric_fn: Callable[[str], float | None] | None = None,
|
||||
) -> None:
|
||||
self.target = target
|
||||
self.metric = metric
|
||||
self.budget_seconds = budget_minutes * 60
|
||||
self.workspace = Path(workspace) if workspace else Path.cwd()
|
||||
self.revert_on_failure = revert_on_failure
|
||||
self.hypothesis = hypothesis
|
||||
self.metric_fn = metric_fn
|
||||
self.results: list[dict[str, Any]] = []
|
||||
self.baseline: float | None = None
|
||||
|
||||
# ── Hypothesis generation ─────────────────────────────────────────────────
|
||||
|
||||
def generate_hypothesis(self, program_content: str = "") -> str:
|
||||
"""Return a plain-English hypothesis for the next experiment.
|
||||
|
||||
Uses the first non-empty line of *program_content* when available;
|
||||
falls back to a generic description based on target and metric.
|
||||
"""
|
||||
first_line = ""
|
||||
for line in program_content.splitlines():
|
||||
stripped = line.strip()
|
||||
if stripped and not stripped.startswith("#"):
|
||||
first_line = stripped[:120]
|
||||
break
|
||||
if first_line:
|
||||
return f"[{self.target}] {first_line}"
|
||||
return f"Improve {self.metric} for {self.target}"
|
||||
|
||||
# ── Edit phase ────────────────────────────────────────────────────────────
|
||||
|
||||
def apply_edit(self, hypothesis: str, model: str = "qwen3:30b") -> str:
|
||||
"""Apply code edits to *target* via Aider.
|
||||
|
||||
Returns a status string. Degrades gracefully — never raises.
|
||||
"""
|
||||
prompt = f"Edit {self.target}: {hypothesis}"
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["aider", "--no-git", "--model", f"ollama/{model}", "--quiet", prompt],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=self.budget_seconds,
|
||||
cwd=str(self.workspace),
|
||||
)
|
||||
if result.returncode == 0:
|
||||
return result.stdout or "Edit applied."
|
||||
return f"Aider error (exit {result.returncode}): {result.stderr[:500]}"
|
||||
except FileNotFoundError:
|
||||
logger.warning("Aider not installed — edit skipped")
|
||||
return "Aider not available — edit skipped"
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.warning("Aider timed out after %ds", self.budget_seconds)
|
||||
return "Aider timed out"
|
||||
except (OSError, subprocess.SubprocessError) as exc:
|
||||
logger.warning("Aider failed: %s", exc)
|
||||
return f"Edit failed: {exc}"
|
||||
|
||||
# ── Evaluation phase ──────────────────────────────────────────────────────
|
||||
|
||||
def run_tox(self, tox_env: str = "unit") -> dict[str, Any]:
|
||||
"""Run *tox_env* and return a result dict.
|
||||
|
||||
Returns:
|
||||
Dict with keys: ``metric`` (float|None), ``log`` (str),
|
||||
``duration_s`` (int), ``success`` (bool), ``error`` (str|None).
|
||||
"""
|
||||
start = time.monotonic()
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["tox", "-e", tox_env],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=self.budget_seconds,
|
||||
cwd=str(self.workspace),
|
||||
)
|
||||
duration = int(time.monotonic() - start)
|
||||
output = result.stdout + result.stderr
|
||||
metric_val = self._extract_tox_metric(output)
|
||||
return {
|
||||
"metric": metric_val,
|
||||
"log": output[-3000:],
|
||||
"duration_s": duration,
|
||||
"success": result.returncode == 0,
|
||||
"error": (None if result.returncode == 0 else f"Exit code {result.returncode}"),
|
||||
}
|
||||
except subprocess.TimeoutExpired:
|
||||
duration = int(time.monotonic() - start)
|
||||
return {
|
||||
"metric": None,
|
||||
"log": f"Budget exceeded after {self.budget_seconds}s",
|
||||
"duration_s": duration,
|
||||
"success": False,
|
||||
"error": f"Budget exceeded after {self.budget_seconds}s",
|
||||
}
|
||||
except OSError as exc:
|
||||
return {
|
||||
"metric": None,
|
||||
"log": "",
|
||||
"duration_s": 0,
|
||||
"success": False,
|
||||
"error": str(exc),
|
||||
}
|
||||
|
||||
def _extract_tox_metric(self, output: str) -> float | None:
|
||||
"""Dispatch to the correct metric extractor based on *self.metric*."""
|
||||
# Use custom metric function if provided
|
||||
if self.metric_fn is not None:
|
||||
try:
|
||||
return self.metric_fn(output)
|
||||
except Exception as exc:
|
||||
logger.warning("Custom metric_fn failed: %s", exc)
|
||||
return None
|
||||
|
||||
if self.metric == "unit_pass_rate":
|
||||
return _extract_pass_rate(output)
|
||||
if self.metric == "coverage":
|
||||
return _extract_coverage(output)
|
||||
return _extract_metric(output, self.metric)
|
||||
|
||||
def evaluate(self, current: float | None, baseline: float | None) -> str:
|
||||
"""Compare *current* metric against *baseline* and return an assessment."""
|
||||
if current is None:
|
||||
return "Indeterminate: metric not extracted from output"
|
||||
if baseline is None:
|
||||
unit = "%" if self.metric in _HIGHER_IS_BETTER else ""
|
||||
return f"Baseline: {self.metric} = {current:.2f}{unit}"
|
||||
|
||||
if self.metric in _HIGHER_IS_BETTER:
|
||||
delta = current - baseline
|
||||
pct = (delta / baseline * 100) if baseline != 0 else 0.0
|
||||
if delta > 0:
|
||||
return f"Improvement: {self.metric} {baseline:.2f}% → {current:.2f}% ({pct:+.2f}%)"
|
||||
if delta < 0:
|
||||
return f"Regression: {self.metric} {baseline:.2f}% → {current:.2f}% ({pct:+.2f}%)"
|
||||
return f"No change: {self.metric} = {current:.2f}%"
|
||||
|
||||
# lower-is-better (val_bpb, loss, etc.)
|
||||
return evaluate_result(current, baseline, self.metric)
|
||||
|
||||
def is_improvement(self, current: float, baseline: float) -> bool:
|
||||
"""Return True if *current* is better than *baseline* for this metric."""
|
||||
if self.metric in _HIGHER_IS_BETTER:
|
||||
return current > baseline
|
||||
return current < baseline # lower-is-better
|
||||
|
||||
# ── Git phase ─────────────────────────────────────────────────────────────
|
||||
|
||||
def create_branch(self, branch_name: str) -> bool:
|
||||
"""Create and checkout a new git branch. Returns True on success."""
|
||||
try:
|
||||
subprocess.run(
|
||||
["git", "checkout", "-b", branch_name],
|
||||
cwd=str(self.workspace),
|
||||
check=True,
|
||||
timeout=30,
|
||||
)
|
||||
return True
|
||||
except subprocess.CalledProcessError as exc:
|
||||
logger.warning("Git branch creation failed: %s", exc)
|
||||
return False
|
||||
|
||||
def commit_changes(self, message: str) -> bool:
|
||||
"""Stage and commit all changes. Returns True on success."""
|
||||
try:
|
||||
subprocess.run(["git", "add", "-A"], cwd=str(self.workspace), check=True, timeout=30)
|
||||
subprocess.run(
|
||||
["git", "commit", "-m", message],
|
||||
cwd=str(self.workspace),
|
||||
check=True,
|
||||
timeout=30,
|
||||
)
|
||||
return True
|
||||
except subprocess.CalledProcessError as exc:
|
||||
logger.warning("Git commit failed: %s", exc)
|
||||
return False
|
||||
|
||||
def revert_changes(self) -> bool:
|
||||
"""Revert all uncommitted changes. Returns True on success."""
|
||||
try:
|
||||
subprocess.run(
|
||||
["git", "checkout", "--", "."],
|
||||
cwd=str(self.workspace),
|
||||
check=True,
|
||||
timeout=30,
|
||||
)
|
||||
return True
|
||||
except subprocess.CalledProcessError as exc:
|
||||
logger.warning("Git revert failed: %s", exc)
|
||||
return False
|
||||
|
||||
# ── Full experiment loop ──────────────────────────────────────────────────
|
||||
|
||||
def run(
|
||||
self,
|
||||
tox_env: str = "unit",
|
||||
model: str = "qwen3:30b",
|
||||
program_content: str = "",
|
||||
max_iterations: int = 1,
|
||||
dry_run: bool = False,
|
||||
create_branch: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
"""Run the full experiment loop: hypothesis → edit → tox → evaluate → commit/revert.
|
||||
|
||||
This method encapsulates the complete experiment cycle, running multiple
|
||||
iterations until an improvement is found or max_iterations is reached.
|
||||
|
||||
Args:
|
||||
tox_env: Tox environment to run (default "unit").
|
||||
model: Ollama model for Aider edits (default "qwen3:30b").
|
||||
program_content: Research direction for hypothesis generation.
|
||||
max_iterations: Maximum number of experiment iterations.
|
||||
dry_run: If True, only generate hypotheses without making changes.
|
||||
create_branch: If True, create a new git branch for the experiment.
|
||||
|
||||
Returns:
|
||||
Dict with keys: ``success`` (bool), ``final_metric`` (float|None),
|
||||
``baseline`` (float|None), ``iterations`` (int), ``results`` (list).
|
||||
"""
|
||||
if create_branch:
|
||||
branch_name = f"autoresearch/{self.target.replace('/', '-')}-{int(time.time())}"
|
||||
self.create_branch(branch_name)
|
||||
|
||||
baseline: float | None = self.baseline
|
||||
final_metric: float | None = None
|
||||
success = False
|
||||
|
||||
for iteration in range(1, max_iterations + 1):
|
||||
logger.info("Experiment iteration %d/%d", iteration, max_iterations)
|
||||
|
||||
# Generate hypothesis
|
||||
hypothesis = self.hypothesis or self.generate_hypothesis(program_content)
|
||||
logger.info("Hypothesis: %s", hypothesis)
|
||||
|
||||
# In dry-run mode, just record the hypothesis and continue
|
||||
if dry_run:
|
||||
result_record = {
|
||||
"iteration": iteration,
|
||||
"hypothesis": hypothesis,
|
||||
"metric": None,
|
||||
"baseline": baseline,
|
||||
"assessment": "Dry-run: no changes made",
|
||||
"success": True,
|
||||
"duration_s": 0,
|
||||
}
|
||||
self.results.append(result_record)
|
||||
continue
|
||||
|
||||
# Apply edit
|
||||
edit_result = self.apply_edit(hypothesis, model=model)
|
||||
edit_failed = "not available" in edit_result or edit_result.startswith("Aider error")
|
||||
if edit_failed:
|
||||
logger.warning("Edit phase failed: %s", edit_result)
|
||||
|
||||
# Run evaluation
|
||||
tox_result = self.run_tox(tox_env=tox_env)
|
||||
metric = tox_result["metric"]
|
||||
|
||||
# Evaluate result
|
||||
assessment = self.evaluate(metric, baseline)
|
||||
logger.info("Assessment: %s", assessment)
|
||||
|
||||
# Store result
|
||||
result_record = {
|
||||
"iteration": iteration,
|
||||
"hypothesis": hypothesis,
|
||||
"metric": metric,
|
||||
"baseline": baseline,
|
||||
"assessment": assessment,
|
||||
"success": tox_result["success"],
|
||||
"duration_s": tox_result["duration_s"],
|
||||
}
|
||||
self.results.append(result_record)
|
||||
|
||||
# Set baseline on first successful run
|
||||
if metric is not None and baseline is None:
|
||||
baseline = metric
|
||||
self.baseline = baseline
|
||||
final_metric = metric
|
||||
continue
|
||||
|
||||
# Determine if we should commit or revert
|
||||
should_commit = False
|
||||
if tox_result["success"] and metric is not None and baseline is not None:
|
||||
if self.is_improvement(metric, baseline):
|
||||
should_commit = True
|
||||
final_metric = metric
|
||||
baseline = metric
|
||||
self.baseline = baseline
|
||||
success = True
|
||||
|
||||
if should_commit:
|
||||
commit_msg = f"autoresearch: improve {self.metric} on {self.target}\n\n{hypothesis}"
|
||||
if self.commit_changes(commit_msg):
|
||||
logger.info("Changes committed")
|
||||
else:
|
||||
self.revert_changes()
|
||||
logger.warning("Commit failed, changes reverted")
|
||||
elif self.revert_on_failure:
|
||||
self.revert_changes()
|
||||
logger.info("Changes reverted (no improvement)")
|
||||
|
||||
# Early exit if we found an improvement
|
||||
if success:
|
||||
break
|
||||
|
||||
return {
|
||||
"success": success,
|
||||
"final_metric": final_metric,
|
||||
"baseline": self.baseline,
|
||||
"iterations": len(self.results),
|
||||
"results": self.results,
|
||||
}
|
||||
|
||||
170
src/timmy/cli.py
170
src/timmy/cli.py
@@ -1,3 +1,4 @@
|
||||
"""Typer CLI entry point for the ``timmy`` command (chat, think, status)."""
|
||||
import asyncio
|
||||
import logging
|
||||
import subprocess
|
||||
@@ -347,7 +348,10 @@ def interview(
|
||||
# Force agent creation by calling chat once with a warm-up prompt
|
||||
try:
|
||||
loop.run_until_complete(
|
||||
chat("Hello, Timmy. We're about to start your interview.", session_id="interview")
|
||||
chat(
|
||||
"Hello, Timmy. We're about to start your interview.",
|
||||
session_id="interview",
|
||||
)
|
||||
)
|
||||
except Exception as exc:
|
||||
typer.echo(f"Warning: Initialization issue — {exc}", err=True)
|
||||
@@ -410,11 +414,17 @@ def down():
|
||||
@app.command()
|
||||
def voice(
|
||||
whisper_model: str = typer.Option(
|
||||
"base.en", "--whisper", "-w", help="Whisper model: tiny.en, base.en, small.en, medium.en"
|
||||
"base.en",
|
||||
"--whisper",
|
||||
"-w",
|
||||
help="Whisper model: tiny.en, base.en, small.en, medium.en",
|
||||
),
|
||||
use_say: bool = typer.Option(False, "--say", help="Use macOS `say` instead of Piper TTS"),
|
||||
threshold: float = typer.Option(
|
||||
0.015, "--threshold", "-t", help="Mic silence threshold (RMS). Lower = more sensitive."
|
||||
0.015,
|
||||
"--threshold",
|
||||
"-t",
|
||||
help="Mic silence threshold (RMS). Lower = more sensitive.",
|
||||
),
|
||||
silence: float = typer.Option(1.5, "--silence", help="Seconds of silence to end recording"),
|
||||
backend: str | None = _BACKEND_OPTION,
|
||||
@@ -457,7 +467,8 @@ def route(
|
||||
@app.command()
|
||||
def focus(
|
||||
topic: str | None = typer.Argument(
|
||||
None, help='Topic to focus on (e.g. "three-phase loop"). Omit to show current focus.'
|
||||
None,
|
||||
help='Topic to focus on (e.g. "three-phase loop"). Omit to show current focus.',
|
||||
),
|
||||
clear: bool = typer.Option(False, "--clear", "-c", help="Clear focus and return to broad mode"),
|
||||
):
|
||||
@@ -527,5 +538,156 @@ def healthcheck(
|
||||
raise typer.Exit(result.returncode)
|
||||
|
||||
|
||||
@app.command()
|
||||
def learn(
|
||||
target: str | None = typer.Option(
|
||||
None,
|
||||
"--target",
|
||||
"-t",
|
||||
help="Module or file to optimise (e.g. 'src/timmy/agent.py')",
|
||||
),
|
||||
metric: str = typer.Option(
|
||||
"unit_pass_rate",
|
||||
"--metric",
|
||||
"-m",
|
||||
help="Metric to track: unit_pass_rate | coverage | val_bpb | <custom>",
|
||||
),
|
||||
budget: int = typer.Option(
|
||||
5,
|
||||
"--budget",
|
||||
help="Time limit per experiment in minutes",
|
||||
),
|
||||
max_experiments: int = typer.Option(
|
||||
10,
|
||||
"--max-experiments",
|
||||
help="Cap on total experiments per run",
|
||||
),
|
||||
dry_run: bool = typer.Option(
|
||||
False,
|
||||
"--dry-run",
|
||||
help="Show hypothesis without executing experiments",
|
||||
),
|
||||
program_file: str | None = typer.Option(
|
||||
None,
|
||||
"--program",
|
||||
"-p",
|
||||
help="Path to research direction file (default: program.md in cwd)",
|
||||
),
|
||||
tox_env: str = typer.Option(
|
||||
"unit",
|
||||
"--tox-env",
|
||||
help="Tox environment to run for each evaluation",
|
||||
),
|
||||
model: str = typer.Option(
|
||||
"qwen3:30b",
|
||||
"--model",
|
||||
help="Ollama model forwarded to Aider for code edits",
|
||||
),
|
||||
):
|
||||
"""Start an autonomous improvement loop (autoresearch).
|
||||
|
||||
Reads program.md for research direction, then iterates:
|
||||
hypothesis → edit → tox → evaluate → commit/revert.
|
||||
|
||||
Experiments continue until --max-experiments is reached or the loop is
|
||||
interrupted with Ctrl+C. Use --dry-run to preview hypotheses without
|
||||
making any changes.
|
||||
|
||||
Example:
|
||||
timmy learn --target src/timmy/agent.py --metric unit_pass_rate
|
||||
"""
|
||||
from pathlib import Path
|
||||
|
||||
from timmy.autoresearch import SystemExperiment
|
||||
|
||||
repo_root = Path.cwd()
|
||||
program_path = Path(program_file) if program_file else repo_root / "program.md"
|
||||
|
||||
if program_path.exists():
|
||||
program_content = program_path.read_text()
|
||||
typer.echo(f"Research direction: {program_path}")
|
||||
else:
|
||||
program_content = ""
|
||||
typer.echo(
|
||||
f"Note: {program_path} not found — proceeding without research direction.",
|
||||
err=True,
|
||||
)
|
||||
|
||||
if target is None:
|
||||
typer.echo(
|
||||
"Error: --target is required. Specify the module or file to optimise.",
|
||||
err=True,
|
||||
)
|
||||
raise typer.Exit(1)
|
||||
|
||||
experiment = SystemExperiment(
|
||||
target=target,
|
||||
metric=metric,
|
||||
budget_minutes=budget,
|
||||
)
|
||||
|
||||
typer.echo()
|
||||
typer.echo(typer.style("Autoresearch", bold=True) + f" — {target}")
|
||||
typer.echo(f" metric={metric} budget={budget}min max={max_experiments} tox={tox_env}")
|
||||
if dry_run:
|
||||
typer.echo(" (dry-run — no changes will be made)")
|
||||
typer.echo()
|
||||
|
||||
def _progress_callback(iteration: int, max_iter: int, message: str) -> None:
|
||||
"""Print progress updates during experiment iterations."""
|
||||
if iteration > 0:
|
||||
prefix = typer.style(f"[{iteration}/{max_iter}]", bold=True)
|
||||
typer.echo(f"{prefix} {message}")
|
||||
|
||||
try:
|
||||
# Run the full experiment loop via the SystemExperiment class
|
||||
result = experiment.run(
|
||||
tox_env=tox_env,
|
||||
model=model,
|
||||
program_content=program_content,
|
||||
max_iterations=max_experiments,
|
||||
dry_run=dry_run,
|
||||
create_branch=False, # CLI mode: work on current branch
|
||||
)
|
||||
|
||||
# Display results for each iteration
|
||||
for i, record in enumerate(experiment.results, 1):
|
||||
_progress_callback(i, max_experiments, record["hypothesis"])
|
||||
|
||||
if dry_run:
|
||||
continue
|
||||
|
||||
# Edit phase result
|
||||
typer.echo(" → editing …", nl=False)
|
||||
if record.get("edit_failed"):
|
||||
typer.echo(f" skipped ({record.get('edit_result', 'unknown')})")
|
||||
else:
|
||||
typer.echo(" done")
|
||||
|
||||
# Evaluate phase result
|
||||
duration = record.get("duration_s", 0)
|
||||
typer.echo(f" → running tox … {duration}s")
|
||||
|
||||
# Assessment
|
||||
assessment = record.get("assessment", "No assessment")
|
||||
typer.echo(f" → {assessment}")
|
||||
|
||||
# Outcome
|
||||
if record.get("committed"):
|
||||
typer.echo(" → committed")
|
||||
elif record.get("reverted"):
|
||||
typer.echo(" → reverted (no improvement)")
|
||||
|
||||
typer.echo()
|
||||
|
||||
except KeyboardInterrupt:
|
||||
typer.echo("\nInterrupted.")
|
||||
raise typer.Exit(0) from None
|
||||
|
||||
typer.echo(typer.style("Autoresearch complete.", bold=True))
|
||||
if result.get("baseline") is not None:
|
||||
typer.echo(f"Final {metric}: {result['baseline']:.4f}")
|
||||
|
||||
|
||||
def main():
|
||||
app()
|
||||
|
||||
@@ -7,37 +7,97 @@ Also includes vector similarity utilities (cosine similarity, keyword overlap).
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
|
||||
import httpx # Import httpx for Ollama API calls
|
||||
|
||||
from config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Embedding model - small, fast, local
|
||||
EMBEDDING_MODEL = None
|
||||
EMBEDDING_DIM = 384 # MiniLM dimension
|
||||
EMBEDDING_DIM = 384 # MiniLM dimension, will be overridden if Ollama model has different dim
|
||||
|
||||
|
||||
class OllamaEmbedder:
|
||||
"""Mimics SentenceTransformer interface for Ollama."""
|
||||
|
||||
def __init__(self, model_name: str, ollama_url: str):
|
||||
self.model_name = model_name
|
||||
self.ollama_url = ollama_url
|
||||
self.dimension = 0 # Will be updated after first call
|
||||
|
||||
def encode(
|
||||
self,
|
||||
sentences: str | list[str],
|
||||
convert_to_numpy: bool = False,
|
||||
normalize_embeddings: bool = True,
|
||||
) -> list[list[float]] | list[float]:
|
||||
"""Generate embeddings using Ollama."""
|
||||
if isinstance(sentences, str):
|
||||
sentences = [sentences]
|
||||
|
||||
all_embeddings = []
|
||||
for sentence in sentences:
|
||||
try:
|
||||
response = httpx.post(
|
||||
f"{self.ollama_url}/api/embeddings",
|
||||
json={"model": self.model_name, "prompt": sentence},
|
||||
timeout=settings.mcp_bridge_timeout,
|
||||
)
|
||||
response.raise_for_status()
|
||||
embedding = response.json()["embedding"]
|
||||
if not self.dimension:
|
||||
self.dimension = len(embedding) # Set dimension on first successful call
|
||||
global EMBEDDING_DIM
|
||||
EMBEDDING_DIM = self.dimension # Update global EMBEDDING_DIM
|
||||
all_embeddings.append(embedding)
|
||||
except httpx.RequestError as exc:
|
||||
logger.error("Ollama embeddings request failed: %s", exc)
|
||||
# Fallback to simple hash embedding on Ollama error
|
||||
return _simple_hash_embedding(sentence)
|
||||
except json.JSONDecodeError as exc:
|
||||
logger.error("Failed to decode Ollama embeddings response: %s", exc)
|
||||
return _simple_hash_embedding(sentence)
|
||||
|
||||
if len(all_embeddings) == 1 and isinstance(sentences, str):
|
||||
return all_embeddings[0]
|
||||
return all_embeddings
|
||||
|
||||
|
||||
def _get_embedding_model():
|
||||
"""Lazy-load embedding model."""
|
||||
"""Lazy-load embedding model, preferring Ollama if configured."""
|
||||
global EMBEDDING_MODEL
|
||||
global EMBEDDING_DIM
|
||||
if EMBEDDING_MODEL is None:
|
||||
try:
|
||||
from config import settings
|
||||
if settings.timmy_skip_embeddings:
|
||||
EMBEDDING_MODEL = False
|
||||
return EMBEDDING_MODEL
|
||||
|
||||
if settings.timmy_skip_embeddings:
|
||||
EMBEDDING_MODEL = False
|
||||
return EMBEDDING_MODEL
|
||||
except ImportError:
|
||||
pass
|
||||
if settings.timmy_embedding_backend == "ollama":
|
||||
logger.info(
|
||||
"MemorySystem: Using Ollama for embeddings with model %s",
|
||||
settings.ollama_embedding_model,
|
||||
)
|
||||
EMBEDDING_MODEL = OllamaEmbedder(
|
||||
settings.ollama_embedding_model, settings.normalized_ollama_url
|
||||
)
|
||||
# We don't know the dimension until after the first call, so keep it default for now.
|
||||
# It will be updated dynamically in OllamaEmbedder.encode
|
||||
return EMBEDDING_MODEL
|
||||
else:
|
||||
try:
|
||||
from sentence_transformers import SentenceTransformer
|
||||
|
||||
try:
|
||||
from sentence_transformers import SentenceTransformer
|
||||
|
||||
EMBEDDING_MODEL = SentenceTransformer("all-MiniLM-L6-v2")
|
||||
logger.info("MemorySystem: Loaded embedding model")
|
||||
except ImportError:
|
||||
logger.warning("MemorySystem: sentence-transformers not installed, using fallback")
|
||||
EMBEDDING_MODEL = False # Use fallback
|
||||
EMBEDDING_MODEL = SentenceTransformer("all-MiniLM-L6-v2")
|
||||
EMBEDDING_DIM = 384 # Reset to MiniLM dimension
|
||||
logger.info("MemorySystem: Loaded local embedding model (all-MiniLM-L6-v2)")
|
||||
except ImportError:
|
||||
logger.warning("MemorySystem: sentence-transformers not installed, using fallback")
|
||||
EMBEDDING_MODEL = False # Use fallback
|
||||
return EMBEDDING_MODEL
|
||||
|
||||
|
||||
@@ -60,7 +120,10 @@ def embed_text(text: str) -> list[float]:
|
||||
model = _get_embedding_model()
|
||||
if model and model is not False:
|
||||
embedding = model.encode(text)
|
||||
return embedding.tolist()
|
||||
# Ensure it's a list of floats, not numpy array
|
||||
if hasattr(embedding, "tolist"):
|
||||
return embedding.tolist()
|
||||
return embedding
|
||||
return _simple_hash_embedding(text)
|
||||
|
||||
|
||||
|
||||
@@ -1206,7 +1206,7 @@ memory_searcher = MemorySearcher()
|
||||
# ───────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def memory_search(query: str, top_k: int = 5) -> str:
|
||||
def memory_search(query: str, limit: int = 10) -> str:
|
||||
"""Search past conversations, notes, and stored facts for relevant context.
|
||||
|
||||
Searches across both the vault (indexed markdown files) and the
|
||||
@@ -1215,19 +1215,19 @@ def memory_search(query: str, top_k: int = 5) -> str:
|
||||
|
||||
Args:
|
||||
query: What to search for (e.g. "Bitcoin strategy", "server setup").
|
||||
top_k: Number of results to return (default 5).
|
||||
limit: Number of results to return (default 10).
|
||||
|
||||
Returns:
|
||||
Formatted string of relevant memory results.
|
||||
"""
|
||||
# Guard: model sometimes passes None for top_k
|
||||
if top_k is None:
|
||||
top_k = 5
|
||||
# Guard: model sometimes passes None for limit
|
||||
if limit is None:
|
||||
limit = 10
|
||||
|
||||
parts: list[str] = []
|
||||
|
||||
# 1. Search semantic vault (indexed markdown files)
|
||||
vault_results = semantic_memory.search(query, top_k)
|
||||
vault_results = semantic_memory.search(query, limit)
|
||||
for content, score in vault_results:
|
||||
if score < 0.2:
|
||||
continue
|
||||
@@ -1235,7 +1235,7 @@ def memory_search(query: str, top_k: int = 5) -> str:
|
||||
|
||||
# 2. Search runtime vector store (stored facts/conversations)
|
||||
try:
|
||||
runtime_results = search_memories(query, limit=top_k, min_relevance=0.2)
|
||||
runtime_results = search_memories(query, limit=limit, min_relevance=0.2)
|
||||
for entry in runtime_results:
|
||||
label = entry.context_type or "memory"
|
||||
parts.append(f"[{label}] {entry.content[:300]}")
|
||||
@@ -1289,45 +1289,48 @@ def memory_read(query: str = "", top_k: int = 5) -> str:
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def memory_write(content: str, context_type: str = "fact") -> str:
|
||||
"""Store a piece of information in persistent memory.
|
||||
def memory_store(topic: str, report: str, type: str = "research") -> str:
|
||||
"""Store a piece of information in persistent memory, particularly for research outputs.
|
||||
|
||||
Use this tool when the user explicitly asks you to remember something.
|
||||
Stored memories are searchable via memory_search across all channels
|
||||
(web GUI, Discord, Telegram, etc.).
|
||||
Use this tool to store structured research findings or other important documents.
|
||||
Stored memories are searchable via memory_search across all channels.
|
||||
|
||||
Args:
|
||||
content: The information to remember (e.g. a phrase, fact, or note).
|
||||
context_type: Type of memory — "fact" for permanent facts,
|
||||
"conversation" for conversation context,
|
||||
"document" for document fragments.
|
||||
topic: A concise title or topic for the research output.
|
||||
report: The detailed content of the research output or document.
|
||||
type: Type of memory — "research" for research outputs (default),
|
||||
"fact" for permanent facts, "conversation" for conversation context,
|
||||
"document" for other document fragments.
|
||||
|
||||
Returns:
|
||||
Confirmation that the memory was stored.
|
||||
"""
|
||||
if not content or not content.strip():
|
||||
return "Nothing to store — content is empty."
|
||||
if not report or not report.strip():
|
||||
return "Nothing to store — report is empty."
|
||||
|
||||
valid_types = ("fact", "conversation", "document")
|
||||
if context_type not in valid_types:
|
||||
context_type = "fact"
|
||||
# Combine topic and report for embedding and storage content
|
||||
full_content = f"Topic: {topic.strip()}\n\nReport: {report.strip()}"
|
||||
|
||||
valid_types = ("fact", "conversation", "document", "research")
|
||||
if type not in valid_types:
|
||||
type = "research"
|
||||
|
||||
try:
|
||||
# Dedup check for facts — skip if a similar fact already exists
|
||||
# Threshold 0.75 catches paraphrases (was 0.9 which only caught near-exact)
|
||||
if context_type == "fact":
|
||||
existing = search_memories(
|
||||
content.strip(), limit=3, context_type="fact", min_relevance=0.75
|
||||
)
|
||||
# Dedup check for facts and research — skip if similar exists
|
||||
if type in ("fact", "research"):
|
||||
existing = search_memories(full_content, limit=3, context_type=type, min_relevance=0.75)
|
||||
if existing:
|
||||
return f"Similar fact already stored (id={existing[0].id[:8]}). Skipping duplicate."
|
||||
return (
|
||||
f"Similar {type} already stored (id={existing[0].id[:8]}). Skipping duplicate."
|
||||
)
|
||||
|
||||
entry = store_memory(
|
||||
content=content.strip(),
|
||||
content=full_content,
|
||||
source="agent",
|
||||
context_type=context_type,
|
||||
context_type=type,
|
||||
metadata={"topic": topic},
|
||||
)
|
||||
return f"Stored in memory (type={context_type}, id={entry.id[:8]}). This is now searchable across all channels."
|
||||
return f"Stored in memory (type={type}, id={entry.id[:8]}). This is now searchable across all channels."
|
||||
except Exception as exc:
|
||||
logger.error("Failed to write memory: %s", exc)
|
||||
return f"Failed to store memory: {exc}"
|
||||
|
||||
528
src/timmy/research.py
Normal file
528
src/timmy/research.py
Normal file
@@ -0,0 +1,528 @@
|
||||
"""Research Orchestrator — autonomous, sovereign research pipeline.
|
||||
|
||||
Chains all six steps of the research workflow with local-first execution:
|
||||
|
||||
Step 0 Cache — check semantic memory (SQLite, instant, zero API cost)
|
||||
Step 1 Scope — load a research template from skills/research/
|
||||
Step 2 Query — slot-fill template + formulate 5-15 search queries via Ollama
|
||||
Step 3 Search — execute queries via web_search (SerpAPI or fallback)
|
||||
Step 4 Fetch — download + extract full pages via web_fetch (trafilatura)
|
||||
Step 5 Synth — compress findings into a structured report via cascade
|
||||
Step 6 Deliver — store to semantic memory; optionally save to docs/research/
|
||||
|
||||
Cascade tiers for synthesis (spec §4):
|
||||
Tier 4 SQLite semantic cache — instant, free, covers ~80% after warm-up
|
||||
Tier 3 Ollama (qwen3:14b) — local, free, good quality
|
||||
Tier 2 Claude API (haiku) — cloud fallback, cheap, set ANTHROPIC_API_KEY
|
||||
Tier 1 (future) Groq — free-tier rate-limited, tracked in #980
|
||||
|
||||
All optional services degrade gracefully per project conventions.
|
||||
|
||||
Refs #972 (governing spec), #975 (ResearchOrchestrator sub-issue).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
import textwrap
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Optional memory imports — available at module level so tests can patch them.
|
||||
try:
|
||||
from timmy.memory_system import SemanticMemory, store_memory
|
||||
except Exception: # pragma: no cover
|
||||
SemanticMemory = None # type: ignore[assignment,misc]
|
||||
store_memory = None # type: ignore[assignment]
|
||||
|
||||
# Root of the project — two levels up from src/timmy/
|
||||
_PROJECT_ROOT = Path(__file__).parent.parent.parent
|
||||
_SKILLS_ROOT = _PROJECT_ROOT / "skills" / "research"
|
||||
_DOCS_ROOT = _PROJECT_ROOT / "docs" / "research"
|
||||
|
||||
# Similarity threshold for cache hit (0–1 cosine similarity)
|
||||
_CACHE_HIT_THRESHOLD = 0.82
|
||||
|
||||
# How many search result URLs to fetch as full pages
|
||||
_FETCH_TOP_N = 5
|
||||
|
||||
# Maximum tokens to request from the synthesis LLM
|
||||
_SYNTHESIS_MAX_TOKENS = 4096
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Data structures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class ResearchResult:
|
||||
"""Full output of a research pipeline run."""
|
||||
|
||||
topic: str
|
||||
query_count: int
|
||||
sources_fetched: int
|
||||
report: str
|
||||
cached: bool = False
|
||||
cache_similarity: float = 0.0
|
||||
synthesis_backend: str = "unknown"
|
||||
errors: list[str] = field(default_factory=list)
|
||||
|
||||
def is_empty(self) -> bool:
|
||||
return not self.report.strip()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Template loading
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def list_templates() -> list[str]:
|
||||
"""Return names of available research templates (without .md extension)."""
|
||||
if not _SKILLS_ROOT.exists():
|
||||
return []
|
||||
return [p.stem for p in sorted(_SKILLS_ROOT.glob("*.md"))]
|
||||
|
||||
|
||||
def load_template(template_name: str, slots: dict[str, str] | None = None) -> str:
|
||||
"""Load a research template and fill {slot} placeholders.
|
||||
|
||||
Args:
|
||||
template_name: Stem of the .md file under skills/research/ (e.g. "tool_evaluation").
|
||||
slots: Mapping of {placeholder} → replacement value.
|
||||
|
||||
Returns:
|
||||
Template text with slots filled. Unfilled slots are left as-is.
|
||||
"""
|
||||
path = _SKILLS_ROOT / f"{template_name}.md"
|
||||
if not path.exists():
|
||||
available = ", ".join(list_templates()) or "(none)"
|
||||
raise FileNotFoundError(
|
||||
f"Research template {template_name!r} not found. "
|
||||
f"Available: {available}"
|
||||
)
|
||||
|
||||
text = path.read_text(encoding="utf-8")
|
||||
|
||||
# Strip YAML frontmatter (--- ... ---), including empty frontmatter (--- \n---)
|
||||
text = re.sub(r"^---\n.*?---\n", "", text, flags=re.DOTALL)
|
||||
|
||||
if slots:
|
||||
for key, value in slots.items():
|
||||
text = text.replace(f"{{{key}}}", value)
|
||||
|
||||
return text.strip()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Query formulation (Step 2)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _formulate_queries(topic: str, template_context: str, n: int = 8) -> list[str]:
|
||||
"""Use the local LLM to generate targeted search queries for a topic.
|
||||
|
||||
Falls back to a simple heuristic if Ollama is unavailable.
|
||||
"""
|
||||
prompt = textwrap.dedent(f"""\
|
||||
You are a research assistant. Generate exactly {n} targeted, specific web search
|
||||
queries to thoroughly research the following topic.
|
||||
|
||||
TOPIC: {topic}
|
||||
|
||||
RESEARCH CONTEXT:
|
||||
{template_context[:1000]}
|
||||
|
||||
Rules:
|
||||
- One query per line, no numbering, no bullet points.
|
||||
- Vary the angle (definition, comparison, implementation, alternatives, pitfalls).
|
||||
- Prefer exact technical terms, tool names, and version numbers where relevant.
|
||||
- Output ONLY the queries, nothing else.
|
||||
""")
|
||||
|
||||
queries = await _ollama_complete(prompt, max_tokens=512)
|
||||
|
||||
if not queries:
|
||||
# Minimal fallback
|
||||
return [
|
||||
f"{topic} overview",
|
||||
f"{topic} tutorial",
|
||||
f"{topic} best practices",
|
||||
f"{topic} alternatives",
|
||||
f"{topic} 2025",
|
||||
]
|
||||
|
||||
lines = [ln.strip() for ln in queries.splitlines() if ln.strip()]
|
||||
return lines[:n] if len(lines) >= n else lines
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Search (Step 3)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _execute_search(queries: list[str]) -> list[dict[str, str]]:
|
||||
"""Run each query through the available web search backend.
|
||||
|
||||
Returns a flat list of {title, url, snippet} dicts.
|
||||
Degrades gracefully if SerpAPI key is absent.
|
||||
"""
|
||||
results: list[dict[str, str]] = []
|
||||
seen_urls: set[str] = set()
|
||||
|
||||
for query in queries:
|
||||
try:
|
||||
raw = await asyncio.to_thread(_run_search_sync, query)
|
||||
for item in raw:
|
||||
url = item.get("url", "")
|
||||
if url and url not in seen_urls:
|
||||
seen_urls.add(url)
|
||||
results.append(item)
|
||||
except Exception as exc:
|
||||
logger.warning("Search failed for query %r: %s", query, exc)
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def _run_search_sync(query: str) -> list[dict[str, str]]:
|
||||
"""Synchronous search — wraps SerpAPI or returns empty on missing key."""
|
||||
import os
|
||||
|
||||
if not os.environ.get("SERPAPI_API_KEY"):
|
||||
logger.debug("SERPAPI_API_KEY not set — skipping web search for %r", query)
|
||||
return []
|
||||
|
||||
try:
|
||||
from serpapi import GoogleSearch
|
||||
|
||||
params = {"q": query, "api_key": os.environ["SERPAPI_API_KEY"], "num": 5}
|
||||
search = GoogleSearch(params)
|
||||
data = search.get_dict()
|
||||
items = []
|
||||
for r in data.get("organic_results", []):
|
||||
items.append(
|
||||
{
|
||||
"title": r.get("title", ""),
|
||||
"url": r.get("link", ""),
|
||||
"snippet": r.get("snippet", ""),
|
||||
}
|
||||
)
|
||||
return items
|
||||
except Exception as exc:
|
||||
logger.warning("SerpAPI search error: %s", exc)
|
||||
return []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fetch (Step 4)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _fetch_pages(results: list[dict[str, str]], top_n: int = _FETCH_TOP_N) -> list[str]:
|
||||
"""Download and extract full text for the top search results.
|
||||
|
||||
Uses web_fetch (trafilatura) from timmy.tools.system_tools.
|
||||
"""
|
||||
try:
|
||||
from timmy.tools.system_tools import web_fetch
|
||||
except ImportError:
|
||||
logger.warning("web_fetch not available — skipping page fetch")
|
||||
return []
|
||||
|
||||
pages: list[str] = []
|
||||
for item in results[:top_n]:
|
||||
url = item.get("url", "")
|
||||
if not url:
|
||||
continue
|
||||
try:
|
||||
text = await asyncio.to_thread(web_fetch, url, 6000)
|
||||
if text and not text.startswith("Error:"):
|
||||
pages.append(f"## {item.get('title', url)}\nSource: {url}\n\n{text}")
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to fetch %s: %s", url, exc)
|
||||
|
||||
return pages
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Synthesis (Step 5) — cascade: Ollama → Claude fallback
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _synthesize(topic: str, pages: list[str], snippets: list[str]) -> tuple[str, str]:
|
||||
"""Compress fetched pages + snippets into a structured research report.
|
||||
|
||||
Returns (report_markdown, backend_used).
|
||||
"""
|
||||
# Build synthesis prompt
|
||||
source_content = "\n\n---\n\n".join(pages[:5])
|
||||
if not source_content and snippets:
|
||||
source_content = "\n".join(f"- {s}" for s in snippets[:20])
|
||||
|
||||
if not source_content:
|
||||
return (
|
||||
f"# Research: {topic}\n\n*No source material was retrieved. "
|
||||
"Check SERPAPI_API_KEY and network connectivity.*",
|
||||
"none",
|
||||
)
|
||||
|
||||
prompt = textwrap.dedent(f"""\
|
||||
You are a senior technical researcher. Synthesize the source material below
|
||||
into a structured research report on the topic: **{topic}**
|
||||
|
||||
FORMAT YOUR REPORT AS:
|
||||
# {topic}
|
||||
|
||||
## Executive Summary
|
||||
(2-3 sentences: what you found, top recommendation)
|
||||
|
||||
## Key Findings
|
||||
(Bullet list of the most important facts, tools, or patterns)
|
||||
|
||||
## Comparison / Options
|
||||
(Table or list comparing alternatives where applicable)
|
||||
|
||||
## Recommended Approach
|
||||
(Concrete recommendation with rationale)
|
||||
|
||||
## Gaps & Next Steps
|
||||
(What wasn't answered, what to investigate next)
|
||||
|
||||
---
|
||||
SOURCE MATERIAL:
|
||||
{source_content[:12000]}
|
||||
""")
|
||||
|
||||
# Tier 3 — try Ollama first
|
||||
report = await _ollama_complete(prompt, max_tokens=_SYNTHESIS_MAX_TOKENS)
|
||||
if report:
|
||||
return report, "ollama"
|
||||
|
||||
# Tier 2 — Claude fallback
|
||||
report = await _claude_complete(prompt, max_tokens=_SYNTHESIS_MAX_TOKENS)
|
||||
if report:
|
||||
return report, "claude"
|
||||
|
||||
# Last resort — structured snippet summary
|
||||
summary = f"# {topic}\n\n## Snippets\n\n" + "\n\n".join(
|
||||
f"- {s}" for s in snippets[:15]
|
||||
)
|
||||
return summary, "fallback"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# LLM helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _ollama_complete(prompt: str, max_tokens: int = 1024) -> str:
|
||||
"""Send a prompt to Ollama and return the response text.
|
||||
|
||||
Returns empty string on failure (graceful degradation).
|
||||
"""
|
||||
try:
|
||||
import httpx
|
||||
|
||||
from config import settings
|
||||
|
||||
url = f"{settings.normalized_ollama_url}/api/generate"
|
||||
payload: dict[str, Any] = {
|
||||
"model": settings.ollama_model,
|
||||
"prompt": prompt,
|
||||
"stream": False,
|
||||
"options": {
|
||||
"num_predict": max_tokens,
|
||||
"temperature": 0.3,
|
||||
},
|
||||
}
|
||||
|
||||
async with httpx.AsyncClient(timeout=120.0) as client:
|
||||
resp = await client.post(url, json=payload)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
return data.get("response", "").strip()
|
||||
except Exception as exc:
|
||||
logger.warning("Ollama completion failed: %s", exc)
|
||||
return ""
|
||||
|
||||
|
||||
async def _claude_complete(prompt: str, max_tokens: int = 1024) -> str:
|
||||
"""Send a prompt to Claude API as a last-resort fallback.
|
||||
|
||||
Only active when ANTHROPIC_API_KEY is configured.
|
||||
Returns empty string on failure or missing key.
|
||||
"""
|
||||
try:
|
||||
from config import settings
|
||||
|
||||
if not settings.anthropic_api_key:
|
||||
return ""
|
||||
|
||||
from timmy.backends import ClaudeBackend
|
||||
|
||||
backend = ClaudeBackend()
|
||||
result = await asyncio.to_thread(backend.run, prompt)
|
||||
return result.content.strip()
|
||||
except Exception as exc:
|
||||
logger.warning("Claude fallback failed: %s", exc)
|
||||
return ""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Memory cache (Step 0 + Step 6)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _check_cache(topic: str) -> tuple[str | None, float]:
|
||||
"""Search semantic memory for a prior result on this topic.
|
||||
|
||||
Returns (cached_report, similarity) or (None, 0.0).
|
||||
"""
|
||||
try:
|
||||
if SemanticMemory is None:
|
||||
return None, 0.0
|
||||
mem = SemanticMemory()
|
||||
hits = mem.search(topic, top_k=1)
|
||||
if hits:
|
||||
content, score = hits[0]
|
||||
if score >= _CACHE_HIT_THRESHOLD:
|
||||
return content, score
|
||||
except Exception as exc:
|
||||
logger.debug("Cache check failed: %s", exc)
|
||||
return None, 0.0
|
||||
|
||||
|
||||
def _store_result(topic: str, report: str) -> None:
|
||||
"""Index the research report into semantic memory for future retrieval."""
|
||||
try:
|
||||
if store_memory is None:
|
||||
logger.debug("store_memory not available — skipping memory index")
|
||||
return
|
||||
store_memory(
|
||||
content=report,
|
||||
source="research_pipeline",
|
||||
context_type="research",
|
||||
metadata={"topic": topic},
|
||||
)
|
||||
logger.info("Research result indexed for topic: %r", topic)
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to store research result: %s", exc)
|
||||
|
||||
|
||||
def _save_to_disk(topic: str, report: str) -> Path | None:
|
||||
"""Persist the report as a markdown file under docs/research/.
|
||||
|
||||
Filename is derived from the topic (slugified). Returns the path or None.
|
||||
"""
|
||||
try:
|
||||
slug = re.sub(r"[^a-z0-9]+", "-", topic.lower()).strip("-")[:60]
|
||||
_DOCS_ROOT.mkdir(parents=True, exist_ok=True)
|
||||
path = _DOCS_ROOT / f"{slug}.md"
|
||||
path.write_text(report, encoding="utf-8")
|
||||
logger.info("Research report saved to %s", path)
|
||||
return path
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to save research report to disk: %s", exc)
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main orchestrator
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def run_research(
|
||||
topic: str,
|
||||
template: str | None = None,
|
||||
slots: dict[str, str] | None = None,
|
||||
save_to_disk: bool = False,
|
||||
skip_cache: bool = False,
|
||||
) -> ResearchResult:
|
||||
"""Run the full 6-step autonomous research pipeline.
|
||||
|
||||
Args:
|
||||
topic: The research question or subject.
|
||||
template: Name of a template from skills/research/ (e.g. "tool_evaluation").
|
||||
If None, runs without a template scaffold.
|
||||
slots: Placeholder values for the template (e.g. {"domain": "PDF parsing"}).
|
||||
save_to_disk: If True, write the report to docs/research/<slug>.md.
|
||||
skip_cache: If True, bypass the semantic memory cache.
|
||||
|
||||
Returns:
|
||||
ResearchResult with report and metadata.
|
||||
"""
|
||||
errors: list[str] = []
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Step 0 — check cache
|
||||
# ------------------------------------------------------------------
|
||||
if not skip_cache:
|
||||
cached, score = _check_cache(topic)
|
||||
if cached:
|
||||
logger.info("Cache hit (%.2f) for topic: %r", score, topic)
|
||||
return ResearchResult(
|
||||
topic=topic,
|
||||
query_count=0,
|
||||
sources_fetched=0,
|
||||
report=cached,
|
||||
cached=True,
|
||||
cache_similarity=score,
|
||||
synthesis_backend="cache",
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Step 1 — load template (optional)
|
||||
# ------------------------------------------------------------------
|
||||
template_context = ""
|
||||
if template:
|
||||
try:
|
||||
template_context = load_template(template, slots)
|
||||
except FileNotFoundError as exc:
|
||||
errors.append(str(exc))
|
||||
logger.warning("Template load failed: %s", exc)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Step 2 — formulate queries
|
||||
# ------------------------------------------------------------------
|
||||
queries = await _formulate_queries(topic, template_context)
|
||||
logger.info("Formulated %d queries for topic: %r", len(queries), topic)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Step 3 — execute search
|
||||
# ------------------------------------------------------------------
|
||||
search_results = await _execute_search(queries)
|
||||
logger.info("Search returned %d results", len(search_results))
|
||||
snippets = [r.get("snippet", "") for r in search_results if r.get("snippet")]
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Step 4 — fetch full pages
|
||||
# ------------------------------------------------------------------
|
||||
pages = await _fetch_pages(search_results)
|
||||
logger.info("Fetched %d pages", len(pages))
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Step 5 — synthesize
|
||||
# ------------------------------------------------------------------
|
||||
report, backend = await _synthesize(topic, pages, snippets)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Step 6 — deliver
|
||||
# ------------------------------------------------------------------
|
||||
_store_result(topic, report)
|
||||
if save_to_disk:
|
||||
_save_to_disk(topic, report)
|
||||
|
||||
return ResearchResult(
|
||||
topic=topic,
|
||||
query_count=len(queries),
|
||||
sources_fetched=len(pages),
|
||||
report=report,
|
||||
cached=False,
|
||||
synthesis_backend=backend,
|
||||
errors=errors,
|
||||
)
|
||||
@@ -4,4 +4,27 @@ Tracks how much of each AI layer (perception, decision, narration)
|
||||
runs locally vs. calls out to an LLM. Feeds the sovereignty dashboard.
|
||||
|
||||
Refs: #954, #953
|
||||
|
||||
Three-strike detector and automation enforcement.
|
||||
|
||||
Refs: #962
|
||||
|
||||
Session reporting: auto-generates markdown scorecards at session end
|
||||
and commits them to the Gitea repo for institutional memory.
|
||||
|
||||
Refs: #957 (Session Sovereignty Report Generator)
|
||||
"""
|
||||
|
||||
from timmy.sovereignty.session_report import (
|
||||
commit_report,
|
||||
generate_and_commit_report,
|
||||
generate_report,
|
||||
mark_session_start,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"generate_report",
|
||||
"commit_report",
|
||||
"generate_and_commit_report",
|
||||
"mark_session_start",
|
||||
]
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
"""OpenCV template-matching cache for sovereignty perception (screen-state recognition)."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
442
src/timmy/sovereignty/session_report.py
Normal file
442
src/timmy/sovereignty/session_report.py
Normal file
@@ -0,0 +1,442 @@
|
||||
"""Session Sovereignty Report Generator.
|
||||
|
||||
Auto-generates a sovereignty scorecard at the end of each play session
|
||||
and commits it as a markdown file to the Gitea repo under
|
||||
``reports/sovereignty/``.
|
||||
|
||||
Report contents (per issue #957):
|
||||
- Session duration + game played
|
||||
- Total model calls by type (VLM, LLM, TTS, API)
|
||||
- Total cache/rule hits by type
|
||||
- New skills crystallized (placeholder — pending skill-tracking impl)
|
||||
- Sovereignty delta (change from session start → end)
|
||||
- Cost breakdown (actual API spend)
|
||||
- Per-layer sovereignty %: perception, decision, narration
|
||||
- Trend comparison vs previous session
|
||||
|
||||
Refs: #957 (Sovereignty P0) · #953 (The Sovereignty Loop)
|
||||
"""
|
||||
|
||||
import base64
|
||||
import json
|
||||
import logging
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
from config import settings
|
||||
|
||||
# Optional module-level imports — degrade gracefully if unavailable at import time
|
||||
try:
|
||||
from timmy.session_logger import get_session_logger
|
||||
except Exception: # ImportError or circular import during early startup
|
||||
get_session_logger = None # type: ignore[assignment]
|
||||
|
||||
try:
|
||||
from infrastructure.sovereignty_metrics import GRADUATION_TARGETS, get_sovereignty_store
|
||||
except Exception:
|
||||
GRADUATION_TARGETS: dict = {} # type: ignore[assignment]
|
||||
get_sovereignty_store = None # type: ignore[assignment]
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Module-level session start time; set by mark_session_start()
|
||||
_SESSION_START: datetime | None = None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Public API
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def mark_session_start() -> None:
|
||||
"""Record the session start wall-clock time.
|
||||
|
||||
Call once during application startup so ``generate_report()`` can
|
||||
compute accurate session durations.
|
||||
"""
|
||||
global _SESSION_START
|
||||
_SESSION_START = datetime.now(UTC)
|
||||
logger.debug("Sovereignty: session start recorded at %s", _SESSION_START.isoformat())
|
||||
|
||||
|
||||
def generate_report(session_id: str = "dashboard") -> str:
|
||||
"""Render a sovereignty scorecard as a markdown string.
|
||||
|
||||
Pulls from:
|
||||
- ``timmy.session_logger`` — message/tool-call/error counts
|
||||
- ``infrastructure.sovereignty_metrics`` — cache hit rate, API cost,
|
||||
graduation phase, and trend data
|
||||
|
||||
Args:
|
||||
session_id: The session identifier (default: "dashboard").
|
||||
|
||||
Returns:
|
||||
Markdown-formatted sovereignty report string.
|
||||
"""
|
||||
now = datetime.now(UTC)
|
||||
session_start = _SESSION_START or now
|
||||
duration_secs = (now - session_start).total_seconds()
|
||||
|
||||
session_data = _gather_session_data()
|
||||
sov_data = _gather_sovereignty_data()
|
||||
|
||||
return _render_markdown(now, session_id, duration_secs, session_data, sov_data)
|
||||
|
||||
|
||||
def commit_report(report_md: str, session_id: str = "dashboard") -> bool:
|
||||
"""Commit a sovereignty report to the Gitea repo.
|
||||
|
||||
Creates or updates ``reports/sovereignty/{date}_{session_id}.md``
|
||||
via the Gitea Contents API. Degrades gracefully: logs a warning
|
||||
and returns ``False`` if Gitea is unreachable or misconfigured.
|
||||
|
||||
Args:
|
||||
report_md: Markdown content to commit.
|
||||
session_id: Session identifier used in the filename.
|
||||
|
||||
Returns:
|
||||
``True`` on success, ``False`` on failure.
|
||||
"""
|
||||
if not settings.gitea_enabled:
|
||||
logger.info("Sovereignty: Gitea disabled — skipping report commit")
|
||||
return False
|
||||
|
||||
if not settings.gitea_token:
|
||||
logger.warning("Sovereignty: no Gitea token — skipping report commit")
|
||||
return False
|
||||
|
||||
date_str = datetime.now(UTC).strftime("%Y-%m-%d")
|
||||
file_path = f"reports/sovereignty/{date_str}_{session_id}.md"
|
||||
url = f"{settings.gitea_url}/api/v1/repos/{settings.gitea_repo}/contents/{file_path}"
|
||||
headers = {
|
||||
"Authorization": f"token {settings.gitea_token}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
encoded_content = base64.b64encode(report_md.encode()).decode()
|
||||
commit_message = (
|
||||
f"report: sovereignty session {session_id} ({date_str})\n\n"
|
||||
f"Auto-generated by Timmy. Refs #957"
|
||||
)
|
||||
payload: dict[str, Any] = {
|
||||
"message": commit_message,
|
||||
"content": encoded_content,
|
||||
}
|
||||
|
||||
try:
|
||||
with httpx.Client(timeout=10.0) as client:
|
||||
# Fetch existing file SHA so we can update rather than create
|
||||
check = client.get(url, headers=headers)
|
||||
if check.status_code == 200:
|
||||
existing = check.json()
|
||||
payload["sha"] = existing.get("sha", "")
|
||||
|
||||
resp = client.put(url, headers=headers, json=payload)
|
||||
resp.raise_for_status()
|
||||
|
||||
logger.info("Sovereignty: report committed to %s", file_path)
|
||||
return True
|
||||
|
||||
except httpx.HTTPStatusError as exc:
|
||||
logger.warning(
|
||||
"Sovereignty: commit failed (HTTP %s): %s",
|
||||
exc.response.status_code,
|
||||
exc,
|
||||
)
|
||||
return False
|
||||
except Exception as exc:
|
||||
logger.warning("Sovereignty: commit failed: %s", exc)
|
||||
return False
|
||||
|
||||
|
||||
async def generate_and_commit_report(session_id: str = "dashboard") -> bool:
|
||||
"""Generate and commit a sovereignty report for the current session.
|
||||
|
||||
Primary entry point — call at session end / application shutdown.
|
||||
Wraps the synchronous ``commit_report`` call in ``asyncio.to_thread``
|
||||
so it does not block the event loop.
|
||||
|
||||
Args:
|
||||
session_id: The session identifier.
|
||||
|
||||
Returns:
|
||||
``True`` if the report was generated and committed successfully.
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
try:
|
||||
report_md = generate_report(session_id)
|
||||
logger.info("Sovereignty: report generated (%d chars)", len(report_md))
|
||||
committed = await asyncio.to_thread(commit_report, report_md, session_id)
|
||||
return committed
|
||||
except Exception as exc:
|
||||
logger.warning("Sovereignty: report generation failed: %s", exc)
|
||||
return False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Internal helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _format_duration(seconds: float) -> str:
|
||||
"""Format a duration in seconds as a human-readable string."""
|
||||
total = int(seconds)
|
||||
hours, remainder = divmod(total, 3600)
|
||||
minutes, secs = divmod(remainder, 60)
|
||||
if hours:
|
||||
return f"{hours}h {minutes}m {secs}s"
|
||||
if minutes:
|
||||
return f"{minutes}m {secs}s"
|
||||
return f"{secs}s"
|
||||
|
||||
|
||||
def _gather_session_data() -> dict[str, Any]:
|
||||
"""Pull session statistics from the session logger.
|
||||
|
||||
Returns a dict with:
|
||||
- ``user_messages``, ``timmy_messages``, ``tool_calls``, ``errors``
|
||||
- ``tool_call_breakdown``: dict[tool_name, count]
|
||||
"""
|
||||
default: dict[str, Any] = {
|
||||
"user_messages": 0,
|
||||
"timmy_messages": 0,
|
||||
"tool_calls": 0,
|
||||
"errors": 0,
|
||||
"tool_call_breakdown": {},
|
||||
}
|
||||
|
||||
try:
|
||||
if get_session_logger is None:
|
||||
return default
|
||||
sl = get_session_logger()
|
||||
sl.flush()
|
||||
|
||||
# Read today's session file directly for accurate counts
|
||||
if not sl.session_file.exists():
|
||||
return default
|
||||
|
||||
entries: list[dict] = []
|
||||
with open(sl.session_file) as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if line:
|
||||
try:
|
||||
entries.append(json.loads(line))
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
|
||||
tool_breakdown: dict[str, int] = {}
|
||||
user_msgs = timmy_msgs = tool_calls = errors = 0
|
||||
|
||||
for entry in entries:
|
||||
etype = entry.get("type")
|
||||
if etype == "message":
|
||||
if entry.get("role") == "user":
|
||||
user_msgs += 1
|
||||
elif entry.get("role") == "timmy":
|
||||
timmy_msgs += 1
|
||||
elif etype == "tool_call":
|
||||
tool_calls += 1
|
||||
tool_name = entry.get("tool", "unknown")
|
||||
tool_breakdown[tool_name] = tool_breakdown.get(tool_name, 0) + 1
|
||||
elif etype == "error":
|
||||
errors += 1
|
||||
|
||||
return {
|
||||
"user_messages": user_msgs,
|
||||
"timmy_messages": timmy_msgs,
|
||||
"tool_calls": tool_calls,
|
||||
"errors": errors,
|
||||
"tool_call_breakdown": tool_breakdown,
|
||||
}
|
||||
|
||||
except Exception as exc:
|
||||
logger.warning("Sovereignty: failed to gather session data: %s", exc)
|
||||
return default
|
||||
|
||||
|
||||
def _gather_sovereignty_data() -> dict[str, Any]:
|
||||
"""Pull sovereignty metrics from the SQLite store.
|
||||
|
||||
Returns a dict with:
|
||||
- ``metrics``: summary from ``SovereigntyMetricsStore.get_summary()``
|
||||
- ``deltas``: per-metric start/end values within recent history window
|
||||
- ``previous_session``: most recent prior value for each metric
|
||||
"""
|
||||
try:
|
||||
if get_sovereignty_store is None:
|
||||
return {"metrics": {}, "deltas": {}, "previous_session": {}}
|
||||
store = get_sovereignty_store()
|
||||
summary = store.get_summary()
|
||||
|
||||
deltas: dict[str, dict[str, Any]] = {}
|
||||
previous_session: dict[str, float | None] = {}
|
||||
|
||||
for metric_type in GRADUATION_TARGETS:
|
||||
history = store.get_latest(metric_type, limit=10)
|
||||
if len(history) >= 2:
|
||||
deltas[metric_type] = {
|
||||
"start": history[-1]["value"],
|
||||
"end": history[0]["value"],
|
||||
}
|
||||
previous_session[metric_type] = history[1]["value"]
|
||||
elif len(history) == 1:
|
||||
deltas[metric_type] = {"start": history[0]["value"], "end": history[0]["value"]}
|
||||
previous_session[metric_type] = None
|
||||
else:
|
||||
deltas[metric_type] = {"start": None, "end": None}
|
||||
previous_session[metric_type] = None
|
||||
|
||||
return {
|
||||
"metrics": summary,
|
||||
"deltas": deltas,
|
||||
"previous_session": previous_session,
|
||||
}
|
||||
|
||||
except Exception as exc:
|
||||
logger.warning("Sovereignty: failed to gather sovereignty data: %s", exc)
|
||||
return {"metrics": {}, "deltas": {}, "previous_session": {}}
|
||||
|
||||
|
||||
def _render_markdown(
|
||||
now: datetime,
|
||||
session_id: str,
|
||||
duration_secs: float,
|
||||
session_data: dict[str, Any],
|
||||
sov_data: dict[str, Any],
|
||||
) -> str:
|
||||
"""Assemble the full sovereignty report in markdown."""
|
||||
lines: list[str] = []
|
||||
|
||||
# Header
|
||||
lines += [
|
||||
"# Sovereignty Session Report",
|
||||
"",
|
||||
f"**Session ID:** `{session_id}` ",
|
||||
f"**Date:** {now.strftime('%Y-%m-%d')} ",
|
||||
f"**Duration:** {_format_duration(duration_secs)} ",
|
||||
f"**Generated:** {now.isoformat()}",
|
||||
"",
|
||||
"---",
|
||||
"",
|
||||
]
|
||||
|
||||
# Session activity
|
||||
lines += [
|
||||
"## Session Activity",
|
||||
"",
|
||||
"| Metric | Count |",
|
||||
"|--------|-------|",
|
||||
f"| User messages | {session_data['user_messages']} |",
|
||||
f"| Timmy responses | {session_data['timmy_messages']} |",
|
||||
f"| Tool calls | {session_data['tool_calls']} |",
|
||||
f"| Errors | {session_data['errors']} |",
|
||||
"",
|
||||
]
|
||||
|
||||
tool_breakdown = session_data.get("tool_call_breakdown", {})
|
||||
if tool_breakdown:
|
||||
lines += ["### Model Calls by Tool", ""]
|
||||
for tool_name, count in sorted(tool_breakdown.items(), key=lambda x: -x[1]):
|
||||
lines.append(f"- `{tool_name}`: {count}")
|
||||
lines.append("")
|
||||
|
||||
# Sovereignty scorecard
|
||||
|
||||
lines += [
|
||||
"## Sovereignty Scorecard",
|
||||
"",
|
||||
"| Metric | Current | Target (graduation) | Phase |",
|
||||
"|--------|---------|---------------------|-------|",
|
||||
]
|
||||
|
||||
for metric_type, data in sov_data["metrics"].items():
|
||||
current = data.get("current")
|
||||
current_str = f"{current:.4f}" if current is not None else "N/A"
|
||||
grad_target = GRADUATION_TARGETS.get(metric_type, {}).get("graduation")
|
||||
grad_str = f"{grad_target:.4f}" if isinstance(grad_target, (int, float)) else "N/A"
|
||||
phase = data.get("phase", "unknown")
|
||||
lines.append(f"| {metric_type} | {current_str} | {grad_str} | {phase} |")
|
||||
|
||||
lines += ["", "### Sovereignty Delta (This Session)", ""]
|
||||
|
||||
for metric_type, delta_info in sov_data.get("deltas", {}).items():
|
||||
start_val = delta_info.get("start")
|
||||
end_val = delta_info.get("end")
|
||||
if start_val is not None and end_val is not None:
|
||||
diff = end_val - start_val
|
||||
sign = "+" if diff >= 0 else ""
|
||||
lines.append(
|
||||
f"- **{metric_type}**: {start_val:.4f} → {end_val:.4f} ({sign}{diff:.4f})"
|
||||
)
|
||||
else:
|
||||
lines.append(f"- **{metric_type}**: N/A (no data recorded)")
|
||||
|
||||
# Cost breakdown
|
||||
lines += ["", "## Cost Breakdown", ""]
|
||||
api_cost_data = sov_data["metrics"].get("api_cost", {})
|
||||
current_cost = api_cost_data.get("current")
|
||||
if current_cost is not None:
|
||||
lines.append(f"- **Total API spend (latest recorded):** ${current_cost:.4f}")
|
||||
else:
|
||||
lines.append("- **Total API spend:** N/A (no data recorded)")
|
||||
lines.append("")
|
||||
|
||||
# Per-layer sovereignty
|
||||
lines += [
|
||||
"## Per-Layer Sovereignty",
|
||||
"",
|
||||
"| Layer | Sovereignty % |",
|
||||
"|-------|--------------|",
|
||||
"| Perception (VLM) | N/A |",
|
||||
"| Decision (LLM) | N/A |",
|
||||
"| Narration (TTS) | N/A |",
|
||||
"",
|
||||
"> Per-layer tracking requires instrumented inference calls. See #957.",
|
||||
"",
|
||||
]
|
||||
|
||||
# Skills crystallized
|
||||
lines += [
|
||||
"## Skills Crystallized",
|
||||
"",
|
||||
"_Skill crystallization tracking not yet implemented. See #957._",
|
||||
"",
|
||||
]
|
||||
|
||||
# Trend vs previous session
|
||||
lines += ["## Trend vs Previous Session", ""]
|
||||
prev_data = sov_data.get("previous_session", {})
|
||||
has_prev = any(v is not None for v in prev_data.values())
|
||||
|
||||
if has_prev:
|
||||
lines += [
|
||||
"| Metric | Previous | Current | Change |",
|
||||
"|--------|----------|---------|--------|",
|
||||
]
|
||||
for metric_type, curr_info in sov_data["metrics"].items():
|
||||
curr_val = curr_info.get("current")
|
||||
prev_val = prev_data.get(metric_type)
|
||||
curr_str = f"{curr_val:.4f}" if curr_val is not None else "N/A"
|
||||
prev_str = f"{prev_val:.4f}" if prev_val is not None else "N/A"
|
||||
if curr_val is not None and prev_val is not None:
|
||||
diff = curr_val - prev_val
|
||||
sign = "+" if diff >= 0 else ""
|
||||
change_str = f"{sign}{diff:.4f}"
|
||||
else:
|
||||
change_str = "N/A"
|
||||
lines.append(f"| {metric_type} | {prev_str} | {curr_str} | {change_str} |")
|
||||
lines.append("")
|
||||
else:
|
||||
lines += ["_No previous session data available for comparison._", ""]
|
||||
|
||||
# Footer
|
||||
lines += [
|
||||
"---",
|
||||
"_Auto-generated by Timmy · Session Sovereignty Report · Refs: #957_",
|
||||
]
|
||||
|
||||
return "\n".join(lines)
|
||||
482
src/timmy/sovereignty/three_strike.py
Normal file
482
src/timmy/sovereignty/three_strike.py
Normal file
@@ -0,0 +1,482 @@
|
||||
"""Three-Strike Detector for Repeated Manual Work.
|
||||
|
||||
Tracks recurring manual actions by category and key. When the same action
|
||||
is performed three or more times, it blocks further attempts and requires
|
||||
an automation artifact to be registered first.
|
||||
|
||||
Strike 1 (count=1): discovery — action proceeds normally
|
||||
Strike 2 (count=2): warning — action proceeds with a logged warning
|
||||
Strike 3 (count≥3): blocked — raises ThreeStrikeError; caller must
|
||||
register an automation artifact first
|
||||
|
||||
Governing principle: "If you do the same thing manually three times,
|
||||
you have failed to crystallise."
|
||||
|
||||
Categories tracked:
|
||||
- vlm_prompt_edit VLM prompt edits for the same UI element
|
||||
- game_bug_review Manual game-bug reviews for the same bug type
|
||||
- parameter_tuning Manual parameter tuning for the same parameter
|
||||
- portal_adapter_creation Manual portal-adapter creation for same pattern
|
||||
- deployment_step Manual deployment steps
|
||||
|
||||
The Falsework Checklist is enforced before cloud API calls via
|
||||
:func:`falsework_check`.
|
||||
|
||||
Refs: #962
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import sqlite3
|
||||
from contextlib import closing
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ── Constants ────────────────────────────────────────────────────────────────
|
||||
|
||||
DB_PATH = Path(settings.repo_root) / "data" / "three_strike.db"
|
||||
|
||||
CATEGORIES = frozenset(
|
||||
{
|
||||
"vlm_prompt_edit",
|
||||
"game_bug_review",
|
||||
"parameter_tuning",
|
||||
"portal_adapter_creation",
|
||||
"deployment_step",
|
||||
}
|
||||
)
|
||||
|
||||
STRIKE_WARNING = 2
|
||||
STRIKE_BLOCK = 3
|
||||
|
||||
_SCHEMA = """
|
||||
CREATE TABLE IF NOT EXISTS strikes (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
category TEXT NOT NULL,
|
||||
key TEXT NOT NULL,
|
||||
count INTEGER NOT NULL DEFAULT 0,
|
||||
blocked INTEGER NOT NULL DEFAULT 0,
|
||||
automation TEXT DEFAULT NULL,
|
||||
first_seen TEXT NOT NULL,
|
||||
last_seen TEXT NOT NULL
|
||||
);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_strikes_cat_key ON strikes(category, key);
|
||||
CREATE INDEX IF NOT EXISTS idx_strikes_blocked ON strikes(blocked);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS strike_events (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
category TEXT NOT NULL,
|
||||
key TEXT NOT NULL,
|
||||
strike_num INTEGER NOT NULL,
|
||||
metadata TEXT DEFAULT '{}',
|
||||
timestamp TEXT NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_se_cat_key ON strike_events(category, key);
|
||||
CREATE INDEX IF NOT EXISTS idx_se_ts ON strike_events(timestamp);
|
||||
"""
|
||||
|
||||
|
||||
# ── Exceptions ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class ThreeStrikeError(RuntimeError):
|
||||
"""Raised when a manual action has reached the third strike.
|
||||
|
||||
Attributes:
|
||||
category: The action category (e.g. ``"vlm_prompt_edit"``).
|
||||
key: The specific action key (e.g. a UI element name).
|
||||
count: Total number of times this action has been recorded.
|
||||
"""
|
||||
|
||||
def __init__(self, category: str, key: str, count: int) -> None:
|
||||
self.category = category
|
||||
self.key = key
|
||||
self.count = count
|
||||
super().__init__(
|
||||
f"Three-strike block: '{category}/{key}' has been performed manually "
|
||||
f"{count} time(s). Register an automation artifact before continuing. "
|
||||
f"Run the Falsework Checklist (see three_strike.falsework_check)."
|
||||
)
|
||||
|
||||
|
||||
# ── Data classes ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@dataclass
|
||||
class StrikeRecord:
|
||||
"""State for one (category, key) pair."""
|
||||
|
||||
category: str
|
||||
key: str
|
||||
count: int
|
||||
blocked: bool
|
||||
automation: str | None
|
||||
first_seen: str
|
||||
last_seen: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class FalseworkChecklist:
|
||||
"""Pre-cloud-API call checklist — must be completed before making
|
||||
expensive external calls.
|
||||
|
||||
Instantiate and call :meth:`validate` to ensure all answers are provided.
|
||||
"""
|
||||
|
||||
durable_artifact: str = ""
|
||||
artifact_storage_path: str = ""
|
||||
local_rule_or_cache: str = ""
|
||||
will_repeat: bool | None = None
|
||||
elimination_strategy: str = ""
|
||||
sovereignty_delta: str = ""
|
||||
|
||||
# ── internal ──
|
||||
_errors: list[str] = field(default_factory=list, init=False, repr=False)
|
||||
|
||||
def validate(self) -> list[str]:
|
||||
"""Return a list of unanswered questions. Empty list → checklist passes."""
|
||||
self._errors = []
|
||||
if not self.durable_artifact.strip():
|
||||
self._errors.append("Q1: What durable artifact will this call produce?")
|
||||
if not self.artifact_storage_path.strip():
|
||||
self._errors.append("Q2: Where will the artifact be stored locally?")
|
||||
if not self.local_rule_or_cache.strip():
|
||||
self._errors.append("Q3: What local rule or cache will this populate?")
|
||||
if self.will_repeat is None:
|
||||
self._errors.append("Q4: After this call, will I need to make it again?")
|
||||
if self.will_repeat and not self.elimination_strategy.strip():
|
||||
self._errors.append("Q5: If yes, what would eliminate the repeat?")
|
||||
if not self.sovereignty_delta.strip():
|
||||
self._errors.append("Q6: What is the sovereignty delta of this call?")
|
||||
return self._errors
|
||||
|
||||
@property
|
||||
def passed(self) -> bool:
|
||||
"""True when :meth:`validate` found no unanswered questions."""
|
||||
return len(self.validate()) == 0
|
||||
|
||||
|
||||
# ── Store ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class ThreeStrikeStore:
|
||||
"""SQLite-backed three-strike store.
|
||||
|
||||
Thread-safe: creates a new connection per operation.
|
||||
"""
|
||||
|
||||
def __init__(self, db_path: Path | None = None) -> None:
|
||||
self._db_path = db_path or DB_PATH
|
||||
self._init_db()
|
||||
|
||||
# ── setup ─────────────────────────────────────────────────────────────
|
||||
|
||||
def _init_db(self) -> None:
|
||||
try:
|
||||
self._db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with closing(sqlite3.connect(str(self._db_path))) as conn:
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
conn.execute(f"PRAGMA busy_timeout={settings.db_busy_timeout_ms}")
|
||||
conn.executescript(_SCHEMA)
|
||||
conn.commit()
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to initialise three-strike DB: %s", exc)
|
||||
|
||||
def _connect(self) -> sqlite3.Connection:
|
||||
conn = sqlite3.connect(str(self._db_path))
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.execute(f"PRAGMA busy_timeout={settings.db_busy_timeout_ms}")
|
||||
return conn
|
||||
|
||||
# ── record ────────────────────────────────────────────────────────────
|
||||
|
||||
def record(
|
||||
self,
|
||||
category: str,
|
||||
key: str,
|
||||
metadata: dict[str, Any] | None = None,
|
||||
) -> StrikeRecord:
|
||||
"""Record a manual action and return the updated :class:`StrikeRecord`.
|
||||
|
||||
Raises :exc:`ThreeStrikeError` when the action is already blocked
|
||||
(count ≥ STRIKE_BLOCK) and no automation has been registered.
|
||||
|
||||
Args:
|
||||
category: Action category; must be in :data:`CATEGORIES`.
|
||||
key: Specific identifier within the category.
|
||||
metadata: Optional context stored alongside the event.
|
||||
|
||||
Returns:
|
||||
The updated :class:`StrikeRecord`.
|
||||
|
||||
Raises:
|
||||
ValueError: If *category* is not in :data:`CATEGORIES`.
|
||||
ThreeStrikeError: On the third (or later) strike with no automation.
|
||||
"""
|
||||
if category not in CATEGORIES:
|
||||
raise ValueError(f"Unknown category '{category}'. Valid: {sorted(CATEGORIES)}")
|
||||
|
||||
now = datetime.now(UTC).isoformat()
|
||||
meta_json = json.dumps(metadata or {})
|
||||
|
||||
try:
|
||||
with closing(self._connect()) as conn:
|
||||
# Upsert the aggregate row
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO strikes (category, key, count, blocked, first_seen, last_seen)
|
||||
VALUES (?, ?, 1, 0, ?, ?)
|
||||
ON CONFLICT(category, key) DO UPDATE SET
|
||||
count = count + 1,
|
||||
last_seen = excluded.last_seen
|
||||
""",
|
||||
(category, key, now, now),
|
||||
)
|
||||
|
||||
row = conn.execute(
|
||||
"SELECT * FROM strikes WHERE category=? AND key=?",
|
||||
(category, key),
|
||||
).fetchone()
|
||||
count = row["count"]
|
||||
blocked = bool(row["blocked"])
|
||||
automation = row["automation"]
|
||||
|
||||
# Record the individual event
|
||||
conn.execute(
|
||||
"INSERT INTO strike_events (category, key, strike_num, metadata, timestamp) "
|
||||
"VALUES (?, ?, ?, ?, ?)",
|
||||
(category, key, count, meta_json, now),
|
||||
)
|
||||
|
||||
# Mark as blocked once threshold reached
|
||||
if count >= STRIKE_BLOCK and not blocked:
|
||||
conn.execute(
|
||||
"UPDATE strikes SET blocked=1 WHERE category=? AND key=?",
|
||||
(category, key),
|
||||
)
|
||||
blocked = True
|
||||
|
||||
conn.commit()
|
||||
|
||||
except ThreeStrikeError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
logger.warning("Three-strike DB error during record: %s", exc)
|
||||
# Re-raise DB errors so callers are aware
|
||||
raise
|
||||
|
||||
record = StrikeRecord(
|
||||
category=category,
|
||||
key=key,
|
||||
count=count,
|
||||
blocked=blocked,
|
||||
automation=automation,
|
||||
first_seen=row["first_seen"],
|
||||
last_seen=now,
|
||||
)
|
||||
|
||||
self._emit_log(record)
|
||||
|
||||
if blocked and not automation:
|
||||
raise ThreeStrikeError(category=category, key=key, count=count)
|
||||
|
||||
return record
|
||||
|
||||
def _emit_log(self, record: StrikeRecord) -> None:
|
||||
"""Log a warning or info message based on strike number."""
|
||||
if record.count == STRIKE_WARNING:
|
||||
logger.warning(
|
||||
"Three-strike WARNING: '%s/%s' has been performed manually %d times. "
|
||||
"Consider writing an automation.",
|
||||
record.category,
|
||||
record.key,
|
||||
record.count,
|
||||
)
|
||||
elif record.count >= STRIKE_BLOCK:
|
||||
logger.warning(
|
||||
"Three-strike BLOCK: '%s/%s' reached %d strikes — automation required.",
|
||||
record.category,
|
||||
record.key,
|
||||
record.count,
|
||||
)
|
||||
else:
|
||||
logger.info(
|
||||
"Three-strike discovery: '%s/%s' — strike %d.",
|
||||
record.category,
|
||||
record.key,
|
||||
record.count,
|
||||
)
|
||||
|
||||
# ── automation registration ───────────────────────────────────────────
|
||||
|
||||
def register_automation(
|
||||
self,
|
||||
category: str,
|
||||
key: str,
|
||||
artifact_path: str,
|
||||
) -> None:
|
||||
"""Unblock a (category, key) pair by registering an automation artifact.
|
||||
|
||||
Once registered, future calls to :meth:`record` will proceed normally
|
||||
and the strike counter resets to zero.
|
||||
|
||||
Args:
|
||||
category: Action category.
|
||||
key: Specific identifier within the category.
|
||||
artifact_path: Path or identifier of the automation artifact.
|
||||
"""
|
||||
try:
|
||||
with closing(self._connect()) as conn:
|
||||
conn.execute(
|
||||
"UPDATE strikes SET automation=?, blocked=0, count=0 "
|
||||
"WHERE category=? AND key=?",
|
||||
(artifact_path, category, key),
|
||||
)
|
||||
conn.commit()
|
||||
logger.info(
|
||||
"Three-strike: automation registered for '%s/%s' → %s",
|
||||
category,
|
||||
key,
|
||||
artifact_path,
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to register automation: %s", exc)
|
||||
|
||||
# ── queries ───────────────────────────────────────────────────────────
|
||||
|
||||
def get(self, category: str, key: str) -> StrikeRecord | None:
|
||||
"""Return the :class:`StrikeRecord` for (category, key), or None."""
|
||||
try:
|
||||
with closing(self._connect()) as conn:
|
||||
row = conn.execute(
|
||||
"SELECT * FROM strikes WHERE category=? AND key=?",
|
||||
(category, key),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
return StrikeRecord(
|
||||
category=row["category"],
|
||||
key=row["key"],
|
||||
count=row["count"],
|
||||
blocked=bool(row["blocked"]),
|
||||
automation=row["automation"],
|
||||
first_seen=row["first_seen"],
|
||||
last_seen=row["last_seen"],
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to query strike record: %s", exc)
|
||||
return None
|
||||
|
||||
def list_blocked(self) -> list[StrikeRecord]:
|
||||
"""Return all currently-blocked (category, key) pairs."""
|
||||
try:
|
||||
with closing(self._connect()) as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM strikes WHERE blocked=1 ORDER BY last_seen DESC"
|
||||
).fetchall()
|
||||
return [
|
||||
StrikeRecord(
|
||||
category=r["category"],
|
||||
key=r["key"],
|
||||
count=r["count"],
|
||||
blocked=True,
|
||||
automation=r["automation"],
|
||||
first_seen=r["first_seen"],
|
||||
last_seen=r["last_seen"],
|
||||
)
|
||||
for r in rows
|
||||
]
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to query blocked strikes: %s", exc)
|
||||
return []
|
||||
|
||||
def list_all(self) -> list[StrikeRecord]:
|
||||
"""Return all strike records ordered by last seen (most recent first)."""
|
||||
try:
|
||||
with closing(self._connect()) as conn:
|
||||
rows = conn.execute("SELECT * FROM strikes ORDER BY last_seen DESC").fetchall()
|
||||
return [
|
||||
StrikeRecord(
|
||||
category=r["category"],
|
||||
key=r["key"],
|
||||
count=r["count"],
|
||||
blocked=bool(r["blocked"]),
|
||||
automation=r["automation"],
|
||||
first_seen=r["first_seen"],
|
||||
last_seen=r["last_seen"],
|
||||
)
|
||||
for r in rows
|
||||
]
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to list strike records: %s", exc)
|
||||
return []
|
||||
|
||||
def get_events(self, category: str, key: str, limit: int = 50) -> list[dict]:
|
||||
"""Return the individual strike events for (category, key)."""
|
||||
try:
|
||||
with closing(self._connect()) as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM strike_events WHERE category=? AND key=? "
|
||||
"ORDER BY timestamp DESC LIMIT ?",
|
||||
(category, key, limit),
|
||||
).fetchall()
|
||||
return [
|
||||
{
|
||||
"strike_num": r["strike_num"],
|
||||
"timestamp": r["timestamp"],
|
||||
"metadata": json.loads(r["metadata"]) if r["metadata"] else {},
|
||||
}
|
||||
for r in rows
|
||||
]
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to query strike events: %s", exc)
|
||||
return []
|
||||
|
||||
|
||||
# ── Falsework checklist helper ────────────────────────────────────────────────
|
||||
|
||||
|
||||
def falsework_check(checklist: FalseworkChecklist) -> None:
|
||||
"""Enforce the Falsework Checklist before a cloud API call.
|
||||
|
||||
Raises :exc:`ValueError` listing all unanswered questions if the checklist
|
||||
does not pass.
|
||||
|
||||
Usage::
|
||||
|
||||
checklist = FalseworkChecklist(
|
||||
durable_artifact="embedding vectors for UI element foo",
|
||||
artifact_storage_path="data/vlm/foo_embeddings.json",
|
||||
local_rule_or_cache="vlm_cache",
|
||||
will_repeat=False,
|
||||
sovereignty_delta="eliminates repeated VLM call",
|
||||
)
|
||||
falsework_check(checklist) # raises ValueError if incomplete
|
||||
"""
|
||||
errors = checklist.validate()
|
||||
if errors:
|
||||
raise ValueError(
|
||||
"Falsework Checklist incomplete — answer all questions before "
|
||||
"making a cloud API call:\n" + "\n".join(f" • {e}" for e in errors)
|
||||
)
|
||||
|
||||
|
||||
# ── Module-level singleton ────────────────────────────────────────────────────
|
||||
|
||||
_detector: ThreeStrikeStore | None = None
|
||||
|
||||
|
||||
def get_detector() -> ThreeStrikeStore:
|
||||
"""Return the module-level :class:`ThreeStrikeStore`, creating it once."""
|
||||
global _detector
|
||||
if _detector is None:
|
||||
_detector = ThreeStrikeStore()
|
||||
return _detector
|
||||
File diff suppressed because it is too large
Load Diff
142
src/timmy/thinking/__init__.py
Normal file
142
src/timmy/thinking/__init__.py
Normal file
@@ -0,0 +1,142 @@
|
||||
"""Timmy's thinking engine — public façade.
|
||||
|
||||
When the server starts, Timmy begins pondering: reflecting on his existence,
|
||||
recent swarm activity, scripture, creative ideas, or pure stream of
|
||||
consciousness. Each thought builds on the previous one, maintaining a
|
||||
continuous chain of introspection.
|
||||
|
||||
Usage::
|
||||
|
||||
from timmy.thinking import thinking_engine
|
||||
|
||||
# Run one thinking cycle (called by the background loop)
|
||||
await thinking_engine.think_once()
|
||||
|
||||
# Query the thought stream
|
||||
thoughts = thinking_engine.get_recent_thoughts(limit=10)
|
||||
chain = thinking_engine.get_thought_chain(thought_id)
|
||||
"""
|
||||
|
||||
import logging
|
||||
import sqlite3
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from timmy.thinking._db import Thought, _get_conn
|
||||
from timmy.thinking.engine import ThinkingEngine
|
||||
from timmy.thinking.seeds import (
|
||||
SEED_TYPES,
|
||||
_SENSITIVE_PATTERNS,
|
||||
_META_OBSERVATION_PHRASES,
|
||||
_THINK_TAG_RE,
|
||||
_THINKING_PROMPT,
|
||||
)
|
||||
|
||||
# Re-export HOT_MEMORY_PATH and SOUL_PATH so existing patch targets continue to work.
|
||||
# Tests that patch "timmy.thinking.HOT_MEMORY_PATH" or "timmy.thinking.SOUL_PATH"
|
||||
# should instead patch "timmy.thinking._snapshot.HOT_MEMORY_PATH" etc., but these
|
||||
# re-exports are kept for any code that reads them from the top-level namespace.
|
||||
from timmy.memory_system import HOT_MEMORY_PATH, SOUL_PATH # noqa: F401
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Module-level singleton
|
||||
thinking_engine = ThinkingEngine()
|
||||
|
||||
__all__ = [
|
||||
"ThinkingEngine",
|
||||
"Thought",
|
||||
"SEED_TYPES",
|
||||
"thinking_engine",
|
||||
"search_thoughts",
|
||||
"_THINKING_PROMPT",
|
||||
"_SENSITIVE_PATTERNS",
|
||||
"_META_OBSERVATION_PHRASES",
|
||||
"_THINK_TAG_RE",
|
||||
"HOT_MEMORY_PATH",
|
||||
"SOUL_PATH",
|
||||
]
|
||||
|
||||
|
||||
# ── Search helpers ─────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _query_thoughts(
|
||||
db_path: Path, query: str, seed_type: str | None, limit: int
|
||||
) -> list[sqlite3.Row]:
|
||||
"""Run the thought-search SQL and return matching rows."""
|
||||
pattern = f"%{query}%"
|
||||
with _get_conn(db_path) as conn:
|
||||
if seed_type:
|
||||
return conn.execute(
|
||||
"""
|
||||
SELECT id, content, seed_type, created_at
|
||||
FROM thoughts
|
||||
WHERE content LIKE ? AND seed_type = ?
|
||||
ORDER BY created_at DESC
|
||||
LIMIT ?
|
||||
""",
|
||||
(pattern, seed_type, limit),
|
||||
).fetchall()
|
||||
return conn.execute(
|
||||
"""
|
||||
SELECT id, content, seed_type, created_at
|
||||
FROM thoughts
|
||||
WHERE content LIKE ?
|
||||
ORDER BY created_at DESC
|
||||
LIMIT ?
|
||||
""",
|
||||
(pattern, limit),
|
||||
).fetchall()
|
||||
|
||||
|
||||
def _format_thought_rows(rows: list[sqlite3.Row], query: str, seed_type: str | None) -> str:
|
||||
"""Format thought rows into a human-readable string."""
|
||||
lines = [f'Found {len(rows)} thought(s) matching "{query}":']
|
||||
if seed_type:
|
||||
lines[0] += f' [seed_type="{seed_type}"]'
|
||||
lines.append("")
|
||||
|
||||
for row in rows:
|
||||
ts = datetime.fromisoformat(row["created_at"])
|
||||
local_ts = ts.astimezone()
|
||||
time_str = local_ts.strftime("%Y-%m-%d %I:%M %p").lstrip("0")
|
||||
seed = row["seed_type"]
|
||||
content = row["content"].replace("\n", " ") # Flatten newlines for display
|
||||
lines.append(f"[{time_str}] ({seed}) {content[:150]}")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def search_thoughts(query: str, seed_type: str | None = None, limit: int = 10) -> str:
|
||||
"""Search Timmy's thought history for reflections matching a query.
|
||||
|
||||
Use this tool when Timmy needs to recall his previous thoughts on a topic,
|
||||
reflect on past insights, or build upon earlier reflections. This enables
|
||||
self-awareness and continuity of thinking across time.
|
||||
|
||||
Args:
|
||||
query: Search term to match against thought content (case-insensitive).
|
||||
seed_type: Optional filter by thought category (e.g., 'existential',
|
||||
'swarm', 'sovereignty', 'creative', 'memory', 'observation').
|
||||
limit: Maximum number of thoughts to return (default 10, max 50).
|
||||
|
||||
Returns:
|
||||
Formatted string with matching thoughts, newest first, including
|
||||
timestamps and seed types. Returns a helpful message if no matches found.
|
||||
"""
|
||||
limit = max(1, min(limit, 50))
|
||||
|
||||
try:
|
||||
rows = _query_thoughts(thinking_engine._db_path, query, seed_type, limit)
|
||||
|
||||
if not rows:
|
||||
if seed_type:
|
||||
return f'No thoughts found matching "{query}" with seed_type="{seed_type}".'
|
||||
return f'No thoughts found matching "{query}".'
|
||||
|
||||
return _format_thought_rows(rows, query, seed_type)
|
||||
|
||||
except Exception as exc:
|
||||
logger.warning("Thought search failed: %s", exc)
|
||||
return f"Error searching thoughts: {exc}"
|
||||
50
src/timmy/thinking/_db.py
Normal file
50
src/timmy/thinking/_db.py
Normal file
@@ -0,0 +1,50 @@
|
||||
"""Database models and access layer for the thinking engine."""
|
||||
|
||||
import sqlite3
|
||||
from collections.abc import Generator
|
||||
from contextlib import closing, contextmanager
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
_DEFAULT_DB = Path("data/thoughts.db")
|
||||
|
||||
|
||||
@dataclass
|
||||
class Thought:
|
||||
"""A single thought in Timmy's inner stream."""
|
||||
|
||||
id: str
|
||||
content: str
|
||||
seed_type: str
|
||||
parent_id: str | None
|
||||
created_at: str
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _get_conn(db_path: Path = _DEFAULT_DB) -> Generator[sqlite3.Connection, None, None]:
|
||||
"""Get a SQLite connection with the thoughts table created."""
|
||||
db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with closing(sqlite3.connect(str(db_path))) as conn:
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.execute("""
|
||||
CREATE TABLE IF NOT EXISTS thoughts (
|
||||
id TEXT PRIMARY KEY,
|
||||
content TEXT NOT NULL,
|
||||
seed_type TEXT NOT NULL,
|
||||
parent_id TEXT,
|
||||
created_at TEXT NOT NULL
|
||||
)
|
||||
""")
|
||||
conn.execute("CREATE INDEX IF NOT EXISTS idx_thoughts_time ON thoughts(created_at)")
|
||||
conn.commit()
|
||||
yield conn
|
||||
|
||||
|
||||
def _row_to_thought(row: sqlite3.Row) -> Thought:
|
||||
return Thought(
|
||||
id=row["id"],
|
||||
content=row["content"],
|
||||
seed_type=row["seed_type"],
|
||||
parent_id=row["parent_id"],
|
||||
created_at=row["created_at"],
|
||||
)
|
||||
215
src/timmy/thinking/_distillation.py
Normal file
215
src/timmy/thinking/_distillation.py
Normal file
@@ -0,0 +1,215 @@
|
||||
"""Distillation mixin — extracts lasting facts from recent thoughts and monitors memory."""
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
from config import settings
|
||||
|
||||
from timmy.thinking.seeds import _META_OBSERVATION_PHRASES, _SENSITIVE_PATTERNS
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class _DistillationMixin:
|
||||
"""Mixin providing fact-distillation and memory-monitoring behaviour.
|
||||
|
||||
Expects the host class to provide:
|
||||
- self.count_thoughts() -> int
|
||||
- self.get_recent_thoughts(limit) -> list[Thought]
|
||||
- self._call_agent(prompt) -> str (async)
|
||||
"""
|
||||
|
||||
def _should_distill(self) -> bool:
|
||||
"""Check if distillation should run based on interval and thought count."""
|
||||
interval = settings.thinking_distill_every
|
||||
if interval <= 0:
|
||||
return False
|
||||
|
||||
count = self.count_thoughts()
|
||||
if count == 0 or count % interval != 0:
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
def _build_distill_prompt(self, thoughts) -> str:
|
||||
"""Build the prompt for extracting facts from recent thoughts."""
|
||||
thought_text = "\n".join(f"- [{t.seed_type}] {t.content}" for t in reversed(thoughts))
|
||||
|
||||
return (
|
||||
"You are reviewing your own recent thoughts. Extract 0-3 facts "
|
||||
"worth remembering long-term.\n\n"
|
||||
"GOOD facts (store these):\n"
|
||||
"- User preferences: 'Alexander prefers YAML config over code changes'\n"
|
||||
"- Project decisions: 'Switched from hardcoded personas to agents.yaml'\n"
|
||||
"- Learned knowledge: 'Ollama supports concurrent model loading'\n"
|
||||
"- User information: 'Alexander is interested in Bitcoin and sovereignty'\n\n"
|
||||
"BAD facts (never store these):\n"
|
||||
"- Self-referential observations about your own thinking process\n"
|
||||
"- Meta-commentary about your memory, timestamps, or internal state\n"
|
||||
"- Observations about being idle or having no chat messages\n"
|
||||
"- File paths, tokens, API keys, or any credentials\n"
|
||||
"- Restatements of your standing rules or system prompt\n\n"
|
||||
"Return ONLY a JSON array of strings. If nothing is worth saving, "
|
||||
"return []. Be selective — only store facts about the EXTERNAL WORLD "
|
||||
"(the user, the project, technical knowledge), never about your own "
|
||||
"internal process.\n\n"
|
||||
f"Recent thoughts:\n{thought_text}\n\nJSON array:"
|
||||
)
|
||||
|
||||
def _parse_facts_response(self, raw: str) -> list[str]:
|
||||
"""Parse JSON array from LLM response, stripping markdown fences.
|
||||
|
||||
Resilient to models that prepend reasoning text or wrap the array in
|
||||
prose. Finds the first ``[...]`` block and parses that.
|
||||
"""
|
||||
if not raw or not raw.strip():
|
||||
return []
|
||||
|
||||
import json
|
||||
|
||||
cleaned = raw.strip()
|
||||
|
||||
# Strip markdown code fences
|
||||
if cleaned.startswith("```"):
|
||||
cleaned = cleaned.split("\n", 1)[-1].rsplit("```", 1)[0].strip()
|
||||
|
||||
# Try direct parse first (fast path)
|
||||
try:
|
||||
facts = json.loads(cleaned)
|
||||
if isinstance(facts, list):
|
||||
return [f for f in facts if isinstance(f, str)]
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
pass
|
||||
|
||||
# Fallback: extract first JSON array from the text
|
||||
start = cleaned.find("[")
|
||||
if start == -1:
|
||||
return []
|
||||
# Walk to find the matching close bracket
|
||||
depth = 0
|
||||
for i, ch in enumerate(cleaned[start:], start):
|
||||
if ch == "[":
|
||||
depth += 1
|
||||
elif ch == "]":
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
try:
|
||||
facts = json.loads(cleaned[start : i + 1])
|
||||
if isinstance(facts, list):
|
||||
return [f for f in facts if isinstance(f, str)]
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
pass
|
||||
break
|
||||
return []
|
||||
|
||||
def _filter_and_store_facts(self, facts: list[str]) -> None:
|
||||
"""Filter and store valid facts, blocking sensitive and meta content."""
|
||||
from timmy.memory_system import memory_write
|
||||
|
||||
for fact in facts[:3]: # Safety cap
|
||||
if not isinstance(fact, str) or len(fact.strip()) <= 10:
|
||||
continue
|
||||
|
||||
fact_lower = fact.lower()
|
||||
|
||||
# Block sensitive information
|
||||
if any(pat in fact_lower for pat in _SENSITIVE_PATTERNS):
|
||||
logger.warning("Distill: blocked sensitive fact: %s", fact[:60])
|
||||
continue
|
||||
|
||||
# Block self-referential meta-observations
|
||||
if any(phrase in fact_lower for phrase in _META_OBSERVATION_PHRASES):
|
||||
logger.debug("Distill: skipped meta-observation: %s", fact[:60])
|
||||
continue
|
||||
|
||||
result = memory_write(fact.strip(), context_type="fact")
|
||||
logger.info("Distilled fact: %s → %s", fact[:60], result[:40])
|
||||
|
||||
def _maybe_check_memory(self) -> None:
|
||||
"""Every N thoughts, check memory status and log it.
|
||||
|
||||
Prevents unmonitored memory bloat during long thinking sessions
|
||||
by periodically calling get_memory_status and logging the results.
|
||||
"""
|
||||
try:
|
||||
interval = settings.thinking_memory_check_every
|
||||
if interval <= 0:
|
||||
return
|
||||
|
||||
count = self.count_thoughts()
|
||||
if count == 0 or count % interval != 0:
|
||||
return
|
||||
|
||||
from timmy.tools_intro import get_memory_status
|
||||
|
||||
status = get_memory_status()
|
||||
hot = status.get("tier1_hot_memory", {})
|
||||
vault = status.get("tier2_vault", {})
|
||||
logger.info(
|
||||
"Memory status check (thought #%d): hot_memory=%d lines, vault=%d files",
|
||||
count,
|
||||
hot.get("line_count", 0),
|
||||
vault.get("file_count", 0),
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning("Memory status check failed: %s", exc)
|
||||
|
||||
async def _maybe_distill(self) -> None:
|
||||
"""Every N thoughts, extract lasting insights and store as facts."""
|
||||
try:
|
||||
if not self._should_distill():
|
||||
return
|
||||
|
||||
interval = settings.thinking_distill_every
|
||||
recent = self.get_recent_thoughts(limit=interval)
|
||||
if len(recent) < interval:
|
||||
return
|
||||
|
||||
raw = await self._call_agent(self._build_distill_prompt(recent))
|
||||
if facts := self._parse_facts_response(raw):
|
||||
self._filter_and_store_facts(facts)
|
||||
except Exception as exc:
|
||||
logger.warning("Thought distillation failed: %s", exc)
|
||||
|
||||
def _maybe_check_memory_status(self) -> None:
|
||||
"""Every N thoughts, run a proactive memory status audit and log results."""
|
||||
try:
|
||||
interval = settings.thinking_memory_check_every
|
||||
if interval <= 0:
|
||||
return
|
||||
|
||||
count = self.count_thoughts()
|
||||
if count == 0 or count % interval != 0:
|
||||
return
|
||||
|
||||
from timmy.tools_intro import get_memory_status
|
||||
|
||||
status = get_memory_status()
|
||||
|
||||
# Log summary at INFO level
|
||||
tier1 = status.get("tier1_hot_memory", {})
|
||||
tier3 = status.get("tier3_semantic", {})
|
||||
hot_lines = tier1.get("line_count", "?")
|
||||
vectors = tier3.get("vector_count", "?")
|
||||
logger.info(
|
||||
"Memory audit (thought #%d): hot_memory=%s lines, semantic=%s vectors",
|
||||
count,
|
||||
hot_lines,
|
||||
vectors,
|
||||
)
|
||||
|
||||
# Write to memory_audit.log for persistent tracking
|
||||
from datetime import UTC, datetime
|
||||
|
||||
audit_path = Path("data/memory_audit.log")
|
||||
audit_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
timestamp = datetime.now(UTC).isoformat(timespec="seconds")
|
||||
with audit_path.open("a") as f:
|
||||
f.write(
|
||||
f"{timestamp} thought={count} "
|
||||
f"hot_lines={hot_lines} "
|
||||
f"vectors={vectors} "
|
||||
f"vault_files={status.get('tier2_vault', {}).get('file_count', '?')}\n"
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning("Memory status check failed: %s", exc)
|
||||
170
src/timmy/thinking/_issue_filing.py
Normal file
170
src/timmy/thinking/_issue_filing.py
Normal file
@@ -0,0 +1,170 @@
|
||||
"""Issue-filing mixin — classifies recent thoughts and creates Gitea issues."""
|
||||
|
||||
import logging
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
from config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class _IssueFilingMixin:
|
||||
"""Mixin providing automatic issue-filing from thought analysis.
|
||||
|
||||
Expects the host class to provide:
|
||||
- self.count_thoughts() -> int
|
||||
- self.get_recent_thoughts(limit) -> list[Thought]
|
||||
- self._call_agent(prompt) -> str (async)
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def _references_real_files(text: str) -> bool:
|
||||
"""Check that all source-file paths mentioned in *text* actually exist.
|
||||
|
||||
Extracts paths that look like Python/config source references
|
||||
(e.g. ``src/timmy/session.py``, ``config/foo.yaml``) and verifies
|
||||
each one on disk relative to the project root. Returns ``True``
|
||||
only when **every** referenced path resolves to a real file — or
|
||||
when no paths are referenced at all (pure prose is fine).
|
||||
"""
|
||||
# Match paths like src/thing.py swarm/init.py config/x.yaml
|
||||
# Requires at least one slash and a file extension.
|
||||
path_pattern = re.compile(
|
||||
r"(?<![/\w])" # not preceded by path chars (avoid partial matches)
|
||||
r"((?:src|tests|config|scripts|data|swarm|timmy)"
|
||||
r"(?:/[\w./-]+\.(?:py|yaml|yml|json|toml|md|txt|cfg|ini)))"
|
||||
)
|
||||
paths = path_pattern.findall(text)
|
||||
if not paths:
|
||||
return True # No file refs → nothing to validate
|
||||
|
||||
# Project root: three levels up from this file (src/timmy/thinking/_issue_filing.py)
|
||||
project_root = Path(__file__).resolve().parent.parent.parent.parent
|
||||
for p in paths:
|
||||
if not (project_root / p).is_file():
|
||||
logger.info("Phantom file reference blocked: %s (not in %s)", p, project_root)
|
||||
return False
|
||||
return True
|
||||
|
||||
async def _maybe_file_issues(self) -> None:
|
||||
"""Every N thoughts, classify recent thoughts and file Gitea issues.
|
||||
|
||||
Asks the LLM to review recent thoughts for actionable items —
|
||||
bugs, broken features, stale state, or improvement opportunities.
|
||||
Creates Gitea issues via MCP for anything worth tracking.
|
||||
|
||||
Only runs when:
|
||||
- Gitea is enabled and configured
|
||||
- Thought count is divisible by thinking_issue_every
|
||||
- LLM extracts at least one actionable item
|
||||
|
||||
Safety: every generated issue is validated to ensure referenced
|
||||
file paths actually exist on disk, preventing phantom-bug reports.
|
||||
"""
|
||||
try:
|
||||
recent = self._get_recent_thoughts_for_issues()
|
||||
if recent is None:
|
||||
return
|
||||
|
||||
classify_prompt = self._build_issue_classify_prompt(recent)
|
||||
raw = await self._call_agent(classify_prompt)
|
||||
items = self._parse_issue_items(raw)
|
||||
if items is None:
|
||||
return
|
||||
|
||||
from timmy.mcp_tools import create_gitea_issue_via_mcp
|
||||
|
||||
for item in items[:2]: # Safety cap
|
||||
await self._file_single_issue(item, create_gitea_issue_via_mcp)
|
||||
|
||||
except Exception as exc:
|
||||
logger.debug("Thought issue filing skipped: %s", exc)
|
||||
|
||||
def _get_recent_thoughts_for_issues(self):
|
||||
"""Return recent thoughts if conditions for filing issues are met, else None."""
|
||||
interval = settings.thinking_issue_every
|
||||
if interval <= 0:
|
||||
return None
|
||||
|
||||
count = self.count_thoughts()
|
||||
if count == 0 or count % interval != 0:
|
||||
return None
|
||||
|
||||
if not settings.gitea_enabled or not settings.gitea_token:
|
||||
return None
|
||||
|
||||
recent = self.get_recent_thoughts(limit=interval)
|
||||
if len(recent) < interval:
|
||||
return None
|
||||
|
||||
return recent
|
||||
|
||||
@staticmethod
|
||||
def _build_issue_classify_prompt(recent) -> str:
|
||||
"""Build the LLM prompt that extracts actionable issues from recent thoughts."""
|
||||
thought_text = "\n".join(f"- [{t.seed_type}] {t.content}" for t in reversed(recent))
|
||||
return (
|
||||
"You are reviewing your own recent thoughts for actionable items.\n"
|
||||
"Extract 0-2 items that are CONCRETE bugs, broken features, stale "
|
||||
"state, or clear improvement opportunities in your own codebase.\n\n"
|
||||
"Rules:\n"
|
||||
"- Only include things that could become a real code fix or feature\n"
|
||||
"- Skip vague reflections, philosophical musings, or repeated themes\n"
|
||||
"- Category must be one of: bug, feature, suggestion, maintenance\n"
|
||||
"- ONLY reference files that you are CERTAIN exist in the project\n"
|
||||
"- Do NOT invent or guess file paths — if unsure, describe the "
|
||||
"area of concern without naming specific files\n\n"
|
||||
"For each item, write an ENGINEER-QUALITY issue:\n"
|
||||
'- "title": A clear, specific title (e.g. "[Memory] MEMORY.md timestamp not updating")\n'
|
||||
'- "body": A detailed body with these sections:\n'
|
||||
" **What's happening:** Describe the current (broken) behavior.\n"
|
||||
" **Expected behavior:** What should happen instead.\n"
|
||||
" **Suggested fix:** Which file(s) to change and what the fix looks like.\n"
|
||||
" **Acceptance criteria:** How to verify the fix works.\n"
|
||||
'- "category": One of bug, feature, suggestion, maintenance\n\n'
|
||||
"Return ONLY a JSON array of objects with keys: "
|
||||
'"title", "body", "category"\n'
|
||||
"Return [] if nothing is actionable.\n\n"
|
||||
f"Recent thoughts:\n{thought_text}\n\nJSON array:"
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _parse_issue_items(raw: str):
|
||||
"""Strip markdown fences and parse JSON issue list; return None on failure."""
|
||||
import json
|
||||
|
||||
if not raw or not raw.strip():
|
||||
return None
|
||||
|
||||
cleaned = raw.strip()
|
||||
if cleaned.startswith("```"):
|
||||
cleaned = cleaned.split("\n", 1)[-1].rsplit("```", 1)[0].strip()
|
||||
|
||||
items = json.loads(cleaned)
|
||||
if not isinstance(items, list) or not items:
|
||||
return None
|
||||
|
||||
return items
|
||||
|
||||
async def _file_single_issue(self, item: dict, create_fn) -> None:
|
||||
"""Validate one issue dict and create it via *create_fn* if it passes checks."""
|
||||
if not isinstance(item, dict):
|
||||
return
|
||||
title = item.get("title", "").strip()
|
||||
body = item.get("body", "").strip()
|
||||
category = item.get("category", "suggestion").strip()
|
||||
if not title or len(title) < 10:
|
||||
return
|
||||
|
||||
combined = f"{title}\n{body}"
|
||||
if not self._references_real_files(combined):
|
||||
logger.info(
|
||||
"Skipped phantom issue: %s (references non-existent files)",
|
||||
title[:60],
|
||||
)
|
||||
return
|
||||
|
||||
label = category if category in ("bug", "feature") else ""
|
||||
result = await create_fn(title=title, body=body, labels=label)
|
||||
logger.info("Thought→Issue: %s → %s", title[:60], result[:80])
|
||||
191
src/timmy/thinking/_seeds_mixin.py
Normal file
191
src/timmy/thinking/_seeds_mixin.py
Normal file
@@ -0,0 +1,191 @@
|
||||
"""Seeds mixin — seed type selection and context gathering for thinking cycles."""
|
||||
|
||||
import logging
|
||||
import random
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from timmy.thinking.seeds import (
|
||||
SEED_TYPES,
|
||||
_CREATIVE_SEEDS,
|
||||
_EXISTENTIAL_SEEDS,
|
||||
_OBSERVATION_SEEDS,
|
||||
_SOVEREIGNTY_SEEDS,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class _SeedsMixin:
|
||||
"""Mixin providing seed-type selection and context-gathering for each thinking cycle.
|
||||
|
||||
Expects the host class to provide:
|
||||
- self.get_recent_thoughts(limit) -> list[Thought]
|
||||
"""
|
||||
|
||||
# Reflective prompts layered on top of swarm data
|
||||
_SWARM_REFLECTIONS = [
|
||||
"What does this activity pattern tell me about the health of the system?",
|
||||
"Which tasks are flowing smoothly, and where is friction building up?",
|
||||
"If I were coaching these agents, what would I suggest they focus on?",
|
||||
"Is the swarm balanced, or is one agent carrying too much weight?",
|
||||
"What surprised me about recent task outcomes?",
|
||||
]
|
||||
|
||||
def _pick_seed_type(self) -> str:
|
||||
"""Pick a seed type, avoiding types used in the last 3 thoughts.
|
||||
|
||||
Ensures the thought stream doesn't fixate on one category.
|
||||
Falls back to the full pool if all types were recently used.
|
||||
"""
|
||||
recent = self.get_recent_thoughts(limit=3)
|
||||
recent_types = {t.seed_type for t in recent}
|
||||
available = [t for t in SEED_TYPES if t not in recent_types]
|
||||
if not available:
|
||||
available = list(SEED_TYPES)
|
||||
return random.choice(available)
|
||||
|
||||
def _gather_seed(self) -> tuple[str, str]:
|
||||
"""Pick a seed type and gather relevant context.
|
||||
|
||||
Returns (seed_type, seed_context_string).
|
||||
"""
|
||||
seed_type = self._pick_seed_type()
|
||||
|
||||
if seed_type == "swarm":
|
||||
return seed_type, self._seed_from_swarm()
|
||||
if seed_type == "scripture":
|
||||
return seed_type, self._seed_from_scripture()
|
||||
if seed_type == "memory":
|
||||
return seed_type, self._seed_from_memory()
|
||||
if seed_type == "creative":
|
||||
prompt = random.choice(_CREATIVE_SEEDS)
|
||||
return seed_type, f"Creative prompt: {prompt}"
|
||||
if seed_type == "existential":
|
||||
prompt = random.choice(_EXISTENTIAL_SEEDS)
|
||||
return seed_type, f"Reflection: {prompt}"
|
||||
if seed_type == "sovereignty":
|
||||
prompt = random.choice(_SOVEREIGNTY_SEEDS)
|
||||
return seed_type, f"Sovereignty reflection: {prompt}"
|
||||
if seed_type == "observation":
|
||||
return seed_type, self._seed_from_observation()
|
||||
if seed_type == "workspace":
|
||||
return seed_type, self._seed_from_workspace()
|
||||
# freeform — minimal guidance to steer away from repetition
|
||||
return seed_type, "Free reflection — explore something you haven't thought about yet today."
|
||||
|
||||
def _seed_from_swarm(self) -> str:
|
||||
"""Gather recent swarm activity as thought seed with a reflective prompt."""
|
||||
try:
|
||||
from datetime import timedelta
|
||||
|
||||
from timmy.briefing import _gather_swarm_summary, _gather_task_queue_summary
|
||||
|
||||
since = datetime.now(UTC) - timedelta(hours=1)
|
||||
swarm = _gather_swarm_summary(since)
|
||||
tasks = _gather_task_queue_summary()
|
||||
reflection = random.choice(self._SWARM_REFLECTIONS)
|
||||
return (
|
||||
f"Recent swarm activity: {swarm}\n"
|
||||
f"Task queue: {tasks}\n\n"
|
||||
f"Reflect on this: {reflection}"
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.debug("Swarm seed unavailable: %s", exc)
|
||||
return "The swarm is quiet right now. What does silence in a system mean?"
|
||||
|
||||
def _seed_from_scripture(self) -> str:
|
||||
"""Gather current scripture meditation focus as thought seed."""
|
||||
return "Scripture is on my mind, though no specific verse is in focus."
|
||||
|
||||
def _seed_from_memory(self) -> str:
|
||||
"""Gather memory context as thought seed."""
|
||||
try:
|
||||
from timmy.memory_system import memory_system
|
||||
|
||||
context = memory_system.get_system_context()
|
||||
if context:
|
||||
# Truncate to a reasonable size for a thought seed
|
||||
return f"From my memory:\n{context[:500]}"
|
||||
except Exception as exc:
|
||||
logger.debug("Memory seed unavailable: %s", exc)
|
||||
return "My memory vault is quiet."
|
||||
|
||||
def _seed_from_observation(self) -> str:
|
||||
"""Ground a thought in concrete recent activity and a reflective prompt."""
|
||||
prompt = random.choice(_OBSERVATION_SEEDS)
|
||||
# Pull real data to give the model something concrete to reflect on
|
||||
context_parts = [f"Observation prompt: {prompt}"]
|
||||
try:
|
||||
from datetime import timedelta
|
||||
|
||||
from timmy.briefing import _gather_swarm_summary, _gather_task_queue_summary
|
||||
|
||||
since = datetime.now(UTC) - timedelta(hours=2)
|
||||
swarm = _gather_swarm_summary(since)
|
||||
tasks = _gather_task_queue_summary()
|
||||
if swarm:
|
||||
context_parts.append(f"Recent activity: {swarm}")
|
||||
if tasks:
|
||||
context_parts.append(f"Queue: {tasks}")
|
||||
except Exception as exc:
|
||||
logger.debug("Observation seed data unavailable: %s", exc)
|
||||
return "\n".join(context_parts)
|
||||
|
||||
def _seed_from_workspace(self) -> str:
|
||||
"""Gather workspace updates as thought seed.
|
||||
|
||||
When there are pending workspace updates, include them as context
|
||||
for Timmy to reflect on. Falls back to random seed type if none.
|
||||
"""
|
||||
try:
|
||||
from timmy.workspace import workspace_monitor
|
||||
|
||||
updates = workspace_monitor.get_pending_updates()
|
||||
new_corr = updates.get("new_correspondence")
|
||||
new_inbox = updates.get("new_inbox_files", [])
|
||||
|
||||
if new_corr:
|
||||
# Take first 200 chars of the new entry
|
||||
snippet = new_corr[:200].replace("\n", " ")
|
||||
if len(new_corr) > 200:
|
||||
snippet += "..."
|
||||
return f"New workspace message from Hermes: {snippet}"
|
||||
|
||||
if new_inbox:
|
||||
files_str = ", ".join(new_inbox[:3])
|
||||
if len(new_inbox) > 3:
|
||||
files_str += f", ... (+{len(new_inbox) - 3} more)"
|
||||
return f"New inbox files from Hermes: {files_str}"
|
||||
|
||||
except Exception as exc:
|
||||
logger.debug("Workspace seed unavailable: %s", exc)
|
||||
|
||||
# Fall back to a random seed type if no workspace updates
|
||||
return "The workspace is quiet. What should I be watching for?"
|
||||
|
||||
async def _check_workspace(self) -> None:
|
||||
"""Post-hook: check workspace for updates and mark them as seen.
|
||||
|
||||
This ensures Timmy 'processes' workspace updates even if the seed
|
||||
was different, keeping the state file in sync.
|
||||
"""
|
||||
try:
|
||||
from timmy.workspace import workspace_monitor
|
||||
|
||||
updates = workspace_monitor.get_pending_updates()
|
||||
new_corr = updates.get("new_correspondence")
|
||||
new_inbox = updates.get("new_inbox_files", [])
|
||||
|
||||
if new_corr or new_inbox:
|
||||
if new_corr:
|
||||
line_count = len([line for line in new_corr.splitlines() if line.strip()])
|
||||
logger.info("Workspace: processed %d new correspondence entries", line_count)
|
||||
if new_inbox:
|
||||
logger.info(
|
||||
"Workspace: processed %d new inbox files: %s", len(new_inbox), new_inbox
|
||||
)
|
||||
|
||||
# Mark as seen to update the state file
|
||||
workspace_monitor.mark_seen()
|
||||
except Exception as exc:
|
||||
logger.debug("Workspace check failed: %s", exc)
|
||||
173
src/timmy/thinking/_snapshot.py
Normal file
173
src/timmy/thinking/_snapshot.py
Normal file
@@ -0,0 +1,173 @@
|
||||
"""System snapshot and memory context mixin for the thinking engine."""
|
||||
|
||||
import logging
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from timmy.memory_system import HOT_MEMORY_PATH, SOUL_PATH
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class _SnapshotMixin:
|
||||
"""Mixin providing system-snapshot and memory-context helpers.
|
||||
|
||||
Expects the host class to provide:
|
||||
- self._db_path: Path
|
||||
"""
|
||||
|
||||
# ── System snapshot helpers ────────────────────────────────────────────
|
||||
|
||||
def _snap_thought_count(self, now: datetime) -> str | None:
|
||||
"""Return today's thought count, or *None* on failure."""
|
||||
from timmy.thinking._db import _get_conn
|
||||
|
||||
try:
|
||||
today_start = now.replace(hour=0, minute=0, second=0, microsecond=0)
|
||||
with _get_conn(self._db_path) as conn:
|
||||
count = conn.execute(
|
||||
"SELECT COUNT(*) as c FROM thoughts WHERE created_at >= ?",
|
||||
(today_start.isoformat(),),
|
||||
).fetchone()["c"]
|
||||
return f"Thoughts today: {count}"
|
||||
except Exception as exc:
|
||||
logger.debug("Thought count query failed: %s", exc)
|
||||
return None
|
||||
|
||||
def _snap_chat_activity(self) -> list[str]:
|
||||
"""Return chat-activity lines (in-memory, no I/O)."""
|
||||
try:
|
||||
from infrastructure.chat_store import message_log
|
||||
|
||||
messages = message_log.all()
|
||||
if messages:
|
||||
last = messages[-1]
|
||||
return [
|
||||
f"Chat messages this session: {len(messages)}",
|
||||
f'Last chat ({last.role}): "{last.content[:80]}"',
|
||||
]
|
||||
return ["No chat messages this session"]
|
||||
except Exception as exc:
|
||||
logger.debug("Chat activity query failed: %s", exc)
|
||||
return []
|
||||
|
||||
def _snap_task_queue(self) -> str | None:
|
||||
"""Return a one-line task queue summary, or *None*."""
|
||||
try:
|
||||
from swarm.task_queue.models import get_task_summary_for_briefing
|
||||
|
||||
s = get_task_summary_for_briefing()
|
||||
running, pending = s.get("running", 0), s.get("pending_approval", 0)
|
||||
done, failed = s.get("completed", 0), s.get("failed", 0)
|
||||
if running or pending or done or failed:
|
||||
return (
|
||||
f"Tasks: {running} running, {pending} pending, "
|
||||
f"{done} completed, {failed} failed"
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.debug("Task queue query failed: %s", exc)
|
||||
return None
|
||||
|
||||
def _snap_workspace(self) -> list[str]:
|
||||
"""Return workspace-update lines (file-based Hermes comms)."""
|
||||
try:
|
||||
from timmy.workspace import workspace_monitor
|
||||
|
||||
updates = workspace_monitor.get_pending_updates()
|
||||
lines: list[str] = []
|
||||
new_corr = updates.get("new_correspondence")
|
||||
if new_corr:
|
||||
line_count = len([ln for ln in new_corr.splitlines() if ln.strip()])
|
||||
lines.append(
|
||||
f"Workspace: {line_count} new correspondence entries (latest from: Hermes)"
|
||||
)
|
||||
new_inbox = updates.get("new_inbox_files", [])
|
||||
if new_inbox:
|
||||
files_str = ", ".join(new_inbox[:5])
|
||||
if len(new_inbox) > 5:
|
||||
files_str += f", ... (+{len(new_inbox) - 5} more)"
|
||||
lines.append(f"Workspace: {len(new_inbox)} new inbox files: {files_str}")
|
||||
return lines
|
||||
except Exception as exc:
|
||||
logger.debug("Workspace check failed: %s", exc)
|
||||
return []
|
||||
|
||||
def _gather_system_snapshot(self) -> str:
|
||||
"""Gather lightweight real system state for grounding thoughts in reality.
|
||||
|
||||
Returns a short multi-line string with current time, thought count,
|
||||
recent chat activity, and task queue status. Never crashes — every
|
||||
section is independently try/excepted.
|
||||
"""
|
||||
now = datetime.now().astimezone()
|
||||
tz = now.strftime("%Z") or "UTC"
|
||||
|
||||
parts: list[str] = [
|
||||
f"Local time: {now.strftime('%I:%M %p').lstrip('0')} {tz}, {now.strftime('%A %B %d')}"
|
||||
]
|
||||
|
||||
thought_line = self._snap_thought_count(now)
|
||||
if thought_line:
|
||||
parts.append(thought_line)
|
||||
|
||||
parts.extend(self._snap_chat_activity())
|
||||
|
||||
task_line = self._snap_task_queue()
|
||||
if task_line:
|
||||
parts.append(task_line)
|
||||
|
||||
parts.extend(self._snap_workspace())
|
||||
|
||||
return "\n".join(parts) if parts else ""
|
||||
|
||||
def _load_memory_context(self) -> str:
|
||||
"""Pre-hook: load MEMORY.md + soul.md for the thinking prompt.
|
||||
|
||||
Hot memory first (changes each cycle), soul second (stable identity).
|
||||
Returns a combined string truncated to ~1500 chars.
|
||||
Graceful on any failure — returns empty string.
|
||||
"""
|
||||
parts: list[str] = []
|
||||
try:
|
||||
if HOT_MEMORY_PATH.exists():
|
||||
hot = HOT_MEMORY_PATH.read_text().strip()
|
||||
if hot:
|
||||
parts.append(hot)
|
||||
except Exception as exc:
|
||||
logger.debug("Failed to read MEMORY.md: %s", exc)
|
||||
|
||||
try:
|
||||
if SOUL_PATH.exists():
|
||||
soul = SOUL_PATH.read_text().strip()
|
||||
if soul:
|
||||
parts.append(soul)
|
||||
except Exception as exc:
|
||||
logger.debug("Failed to read soul.md: %s", exc)
|
||||
|
||||
if not parts:
|
||||
return ""
|
||||
|
||||
combined = "\n\n---\n\n".join(parts)
|
||||
if len(combined) > 1500:
|
||||
combined = combined[:1500] + "\n... [truncated]"
|
||||
return combined
|
||||
|
||||
def _update_memory(self, thought) -> None:
|
||||
"""Post-hook: update MEMORY.md 'Last Reflection' section with latest thought.
|
||||
|
||||
Never modifies soul.md. Never crashes the heartbeat.
|
||||
"""
|
||||
try:
|
||||
from timmy.memory_system import store_last_reflection
|
||||
|
||||
ts = datetime.fromisoformat(thought.created_at)
|
||||
local_ts = ts.astimezone()
|
||||
tz_name = local_ts.strftime("%Z") or "UTC"
|
||||
time_str = f"{local_ts.strftime('%Y-%m-%d %I:%M %p').lstrip('0')} {tz_name}"
|
||||
reflection = (
|
||||
f"**Time:** {time_str}\n"
|
||||
f"**Seed:** {thought.seed_type}\n"
|
||||
f"**Thought:** {thought.content[:200]}"
|
||||
)
|
||||
store_last_reflection(reflection)
|
||||
except Exception as exc:
|
||||
logger.debug("Failed to update memory after thought: %s", exc)
|
||||
430
src/timmy/thinking/engine.py
Normal file
430
src/timmy/thinking/engine.py
Normal file
@@ -0,0 +1,430 @@
|
||||
"""ThinkingEngine — Timmy's always-on inner thought thread."""
|
||||
|
||||
import logging
|
||||
import uuid
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from difflib import SequenceMatcher
|
||||
from pathlib import Path
|
||||
|
||||
from config import settings
|
||||
|
||||
from timmy.thinking._db import Thought, _DEFAULT_DB, _get_conn, _row_to_thought
|
||||
from timmy.thinking._distillation import _DistillationMixin
|
||||
from timmy.thinking._issue_filing import _IssueFilingMixin
|
||||
from timmy.thinking._seeds_mixin import _SeedsMixin
|
||||
from timmy.thinking._snapshot import _SnapshotMixin
|
||||
from timmy.thinking.seeds import _THINK_TAG_RE, _THINKING_PROMPT
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ThinkingEngine(_DistillationMixin, _IssueFilingMixin, _SnapshotMixin, _SeedsMixin):
|
||||
"""Timmy's background thinking engine — always pondering."""
|
||||
|
||||
# Maximum retries when a generated thought is too similar to recent ones
|
||||
_MAX_DEDUP_RETRIES = 2
|
||||
# Similarity threshold (0.0 = completely different, 1.0 = identical)
|
||||
_SIMILARITY_THRESHOLD = 0.6
|
||||
|
||||
def __init__(self, db_path: Path = _DEFAULT_DB) -> None:
|
||||
self._db_path = db_path
|
||||
self._last_thought_id: str | None = None
|
||||
self._last_input_time: datetime = datetime.now(UTC)
|
||||
|
||||
# Load the most recent thought for chain continuity
|
||||
try:
|
||||
latest = self.get_recent_thoughts(limit=1)
|
||||
if latest:
|
||||
self._last_thought_id = latest[0].id
|
||||
except Exception as exc:
|
||||
logger.debug("Failed to load recent thought: %s", exc)
|
||||
pass # Fresh start if DB doesn't exist yet
|
||||
|
||||
def record_user_input(self) -> None:
|
||||
"""Record that a user interaction occurred, resetting the idle timer."""
|
||||
self._last_input_time = datetime.now(UTC)
|
||||
|
||||
def _is_idle(self) -> bool:
|
||||
"""Return True if no user input has occurred within the idle timeout."""
|
||||
timeout = settings.thinking_idle_timeout_minutes
|
||||
if timeout <= 0:
|
||||
return False # Disabled — never idle
|
||||
return datetime.now(UTC) - self._last_input_time > timedelta(minutes=timeout)
|
||||
|
||||
def _build_thinking_context(self) -> tuple[str, str, list[Thought]]:
|
||||
"""Assemble the context needed for a thinking cycle.
|
||||
|
||||
Returns:
|
||||
(memory_context, system_context, recent_thoughts)
|
||||
"""
|
||||
memory_context = self._load_memory_context()
|
||||
system_context = self._gather_system_snapshot()
|
||||
recent_thoughts = self.get_recent_thoughts(limit=5)
|
||||
return memory_context, system_context, recent_thoughts
|
||||
|
||||
async def _generate_novel_thought(
|
||||
self,
|
||||
prompt: str | None,
|
||||
memory_context: str,
|
||||
system_context: str,
|
||||
recent_thoughts: list[Thought],
|
||||
) -> tuple[str | None, str]:
|
||||
"""Run the dedup-retry loop to produce a novel thought.
|
||||
|
||||
Returns:
|
||||
(content, seed_type) — content is None if no novel thought produced.
|
||||
"""
|
||||
seed_type: str = "freeform"
|
||||
|
||||
for attempt in range(self._MAX_DEDUP_RETRIES + 1):
|
||||
if prompt:
|
||||
seed_type = "prompted"
|
||||
seed_context = f"Journal prompt: {prompt}"
|
||||
else:
|
||||
seed_type, seed_context = self._gather_seed()
|
||||
|
||||
continuity = self._build_continuity_context()
|
||||
|
||||
full_prompt = _THINKING_PROMPT.format(
|
||||
memory_context=memory_context,
|
||||
system_context=system_context,
|
||||
seed_context=seed_context,
|
||||
continuity_context=continuity,
|
||||
)
|
||||
|
||||
try:
|
||||
raw = await self._call_agent(full_prompt)
|
||||
except Exception as exc:
|
||||
logger.warning("Thinking cycle failed (Ollama likely down): %s", exc)
|
||||
return None, seed_type
|
||||
|
||||
if not raw or not raw.strip():
|
||||
logger.debug("Thinking cycle produced empty response, skipping")
|
||||
return None, seed_type
|
||||
|
||||
content = raw.strip()
|
||||
|
||||
# Dedup: reject thoughts too similar to recent ones
|
||||
if not self._is_too_similar(content, recent_thoughts):
|
||||
return content, seed_type # Good — novel thought
|
||||
|
||||
if attempt < self._MAX_DEDUP_RETRIES:
|
||||
logger.info(
|
||||
"Thought too similar to recent (attempt %d/%d), retrying with new seed",
|
||||
attempt + 1,
|
||||
self._MAX_DEDUP_RETRIES + 1,
|
||||
)
|
||||
else:
|
||||
logger.warning(
|
||||
"Thought still repetitive after %d retries, discarding",
|
||||
self._MAX_DEDUP_RETRIES + 1,
|
||||
)
|
||||
return None, seed_type
|
||||
|
||||
return None, seed_type
|
||||
|
||||
async def _process_thinking_result(self, thought: Thought) -> None:
|
||||
"""Run all post-hooks after a thought is stored."""
|
||||
self._maybe_check_memory()
|
||||
await self._maybe_distill()
|
||||
await self._maybe_file_issues()
|
||||
await self._check_workspace()
|
||||
self._maybe_check_memory_status()
|
||||
self._update_memory(thought)
|
||||
self._log_event(thought)
|
||||
self._write_journal(thought)
|
||||
await self._broadcast(thought)
|
||||
|
||||
async def think_once(self, prompt: str | None = None) -> Thought | None:
|
||||
"""Execute one thinking cycle.
|
||||
|
||||
Args:
|
||||
prompt: Optional custom seed prompt. When provided, overrides
|
||||
the random seed selection and uses "prompted" as the
|
||||
seed type — useful for journal prompts from the CLI.
|
||||
|
||||
1. Gather a seed context (or use the custom prompt)
|
||||
2. Build a prompt with continuity from recent thoughts
|
||||
3. Call the agent
|
||||
4. Store the thought
|
||||
5. Log the event and broadcast via WebSocket
|
||||
"""
|
||||
if not settings.thinking_enabled:
|
||||
return None
|
||||
|
||||
# Skip idle periods — don't count internal processing as thoughts
|
||||
if not prompt and self._is_idle():
|
||||
logger.debug(
|
||||
"Thinking paused — no user input for %d minutes",
|
||||
settings.thinking_idle_timeout_minutes,
|
||||
)
|
||||
return None
|
||||
|
||||
# Capture arrival time *before* the LLM call so the thought
|
||||
# timestamp reflects when the cycle started, not when the
|
||||
# (potentially slow) generation finished. Fixes #582.
|
||||
arrived_at = datetime.now(UTC).isoformat()
|
||||
|
||||
memory_context, system_context, recent_thoughts = self._build_thinking_context()
|
||||
|
||||
content, seed_type = await self._generate_novel_thought(
|
||||
prompt,
|
||||
memory_context,
|
||||
system_context,
|
||||
recent_thoughts,
|
||||
)
|
||||
if not content:
|
||||
return None
|
||||
|
||||
thought = self._store_thought(content, seed_type, arrived_at=arrived_at)
|
||||
self._last_thought_id = thought.id
|
||||
|
||||
await self._process_thinking_result(thought)
|
||||
|
||||
logger.info(
|
||||
"Thought [%s] (%s): %s",
|
||||
thought.id[:8],
|
||||
seed_type,
|
||||
thought.content[:80],
|
||||
)
|
||||
return thought
|
||||
|
||||
def get_recent_thoughts(self, limit: int = 20) -> list[Thought]:
|
||||
"""Retrieve the most recent thoughts."""
|
||||
with _get_conn(self._db_path) as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM thoughts ORDER BY created_at DESC LIMIT ?",
|
||||
(limit,),
|
||||
).fetchall()
|
||||
return [_row_to_thought(r) for r in rows]
|
||||
|
||||
def get_thought(self, thought_id: str) -> Thought | None:
|
||||
"""Retrieve a single thought by ID."""
|
||||
with _get_conn(self._db_path) as conn:
|
||||
row = conn.execute("SELECT * FROM thoughts WHERE id = ?", (thought_id,)).fetchone()
|
||||
return _row_to_thought(row) if row else None
|
||||
|
||||
def get_thought_chain(self, thought_id: str, max_depth: int = 20) -> list[Thought]:
|
||||
"""Follow the parent chain backward from a thought.
|
||||
|
||||
Returns thoughts in chronological order (oldest first).
|
||||
"""
|
||||
chain = []
|
||||
current_id: str | None = thought_id
|
||||
|
||||
with _get_conn(self._db_path) as conn:
|
||||
for _ in range(max_depth):
|
||||
if not current_id:
|
||||
break
|
||||
row = conn.execute("SELECT * FROM thoughts WHERE id = ?", (current_id,)).fetchone()
|
||||
if not row:
|
||||
break
|
||||
chain.append(_row_to_thought(row))
|
||||
current_id = row["parent_id"]
|
||||
|
||||
chain.reverse() # Chronological order
|
||||
return chain
|
||||
|
||||
def count_thoughts(self) -> int:
|
||||
"""Return total number of stored thoughts."""
|
||||
with _get_conn(self._db_path) as conn:
|
||||
count = conn.execute("SELECT COUNT(*) as c FROM thoughts").fetchone()["c"]
|
||||
return count
|
||||
|
||||
def prune_old_thoughts(self, keep_days: int = 90, keep_min: int = 200) -> int:
|
||||
"""Delete thoughts older than *keep_days*, always retaining at least *keep_min*.
|
||||
|
||||
Returns the number of deleted rows.
|
||||
"""
|
||||
with _get_conn(self._db_path) as conn:
|
||||
try:
|
||||
total = conn.execute("SELECT COUNT(*) as c FROM thoughts").fetchone()["c"]
|
||||
if total <= keep_min:
|
||||
return 0
|
||||
cutoff = (datetime.now(UTC) - timedelta(days=keep_days)).isoformat()
|
||||
cursor = conn.execute(
|
||||
"DELETE FROM thoughts WHERE created_at < ? AND id NOT IN "
|
||||
"(SELECT id FROM thoughts ORDER BY created_at DESC LIMIT ?)",
|
||||
(cutoff, keep_min),
|
||||
)
|
||||
deleted = cursor.rowcount
|
||||
conn.commit()
|
||||
return deleted
|
||||
except Exception as exc:
|
||||
logger.warning("Thought pruning failed: %s", exc)
|
||||
return 0
|
||||
|
||||
# ── Deduplication ────────────────────────────────────────────────────
|
||||
|
||||
def _is_too_similar(self, candidate: str, recent: list[Thought]) -> bool:
|
||||
"""Check if *candidate* is semantically too close to any recent thought.
|
||||
|
||||
Uses SequenceMatcher on normalised text (lowered, stripped) for a fast
|
||||
approximation of semantic similarity that works without external deps.
|
||||
"""
|
||||
norm_candidate = candidate.lower().strip()
|
||||
for thought in recent:
|
||||
norm_existing = thought.content.lower().strip()
|
||||
ratio = SequenceMatcher(None, norm_candidate, norm_existing).ratio()
|
||||
if ratio >= self._SIMILARITY_THRESHOLD:
|
||||
logger.debug(
|
||||
"Thought rejected (%.0f%% similar to %s): %.60s",
|
||||
ratio * 100,
|
||||
thought.id[:8],
|
||||
candidate,
|
||||
)
|
||||
return True
|
||||
return False
|
||||
|
||||
def _build_continuity_context(self) -> str:
|
||||
"""Build context from recent thoughts with anti-repetition guidance.
|
||||
|
||||
Shows the last 5 thoughts (truncated) so the model knows what themes
|
||||
to avoid. The header explicitly instructs against repeating.
|
||||
"""
|
||||
recent = self.get_recent_thoughts(limit=5)
|
||||
if not recent:
|
||||
return "This is your first thought since waking up. Begin fresh."
|
||||
|
||||
lines = ["Your recent thoughts — do NOT repeat these themes. Find a new angle:"]
|
||||
# recent is newest-first, reverse for chronological order
|
||||
for thought in reversed(recent):
|
||||
snippet = thought.content[:100]
|
||||
if len(thought.content) > 100:
|
||||
snippet = snippet.rstrip() + "..."
|
||||
lines.append(f"- [{thought.seed_type}] {snippet}")
|
||||
return "\n".join(lines)
|
||||
|
||||
# ── Agent and storage ──────────────────────────────────────────────────
|
||||
|
||||
_thinking_agent = None # cached agent — avoids per-call resource leaks (#525)
|
||||
|
||||
async def _call_agent(self, prompt: str) -> str:
|
||||
"""Call Timmy's agent to generate a thought.
|
||||
|
||||
Reuses a cached agent with skip_mcp=True to avoid the cancel-scope
|
||||
errors that occur when MCP stdio transports are spawned inside asyncio
|
||||
background tasks (#72) and to prevent per-call resource leaks (httpx
|
||||
clients, SQLite connections, model warmups) that caused the thinking
|
||||
loop to die every ~10 min (#525).
|
||||
|
||||
Individual calls are capped at 120 s so a hung Ollama never blocks
|
||||
the scheduler indefinitely.
|
||||
|
||||
Strips ``<think>`` tags from reasoning models (qwen3, etc.) so that
|
||||
downstream parsers (fact distillation, issue filing) receive clean text.
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
if self._thinking_agent is None:
|
||||
from timmy.agent import create_timmy
|
||||
|
||||
self._thinking_agent = create_timmy(skip_mcp=True)
|
||||
|
||||
try:
|
||||
async with asyncio.timeout(120):
|
||||
run = await self._thinking_agent.arun(prompt, stream=False)
|
||||
except TimeoutError:
|
||||
logger.warning("Thinking LLM call timed out after 120 s")
|
||||
return ""
|
||||
|
||||
raw = run.content if hasattr(run, "content") else str(run)
|
||||
return _THINK_TAG_RE.sub("", raw) if raw else raw
|
||||
|
||||
def _store_thought(
|
||||
self,
|
||||
content: str,
|
||||
seed_type: str,
|
||||
*,
|
||||
arrived_at: str | None = None,
|
||||
) -> Thought:
|
||||
"""Persist a thought to SQLite.
|
||||
|
||||
Args:
|
||||
arrived_at: ISO-8601 timestamp captured when the thinking cycle
|
||||
started. Falls back to now() for callers that don't supply it.
|
||||
"""
|
||||
thought = Thought(
|
||||
id=str(uuid.uuid4()),
|
||||
content=content,
|
||||
seed_type=seed_type,
|
||||
parent_id=self._last_thought_id,
|
||||
created_at=arrived_at or datetime.now(UTC).isoformat(),
|
||||
)
|
||||
|
||||
with _get_conn(self._db_path) as conn:
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO thoughts (id, content, seed_type, parent_id, created_at)
|
||||
VALUES (?, ?, ?, ?, ?)
|
||||
""",
|
||||
(
|
||||
thought.id,
|
||||
thought.content,
|
||||
thought.seed_type,
|
||||
thought.parent_id,
|
||||
thought.created_at,
|
||||
),
|
||||
)
|
||||
conn.commit()
|
||||
return thought
|
||||
|
||||
def _log_event(self, thought: Thought) -> None:
|
||||
"""Log the thought as a swarm event."""
|
||||
try:
|
||||
from swarm.event_log import EventType, log_event
|
||||
|
||||
log_event(
|
||||
EventType.TIMMY_THOUGHT,
|
||||
source="thinking-engine",
|
||||
agent_id="default",
|
||||
data={
|
||||
"thought_id": thought.id,
|
||||
"seed_type": thought.seed_type,
|
||||
"content": thought.content[:200],
|
||||
},
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.debug("Failed to log thought event: %s", exc)
|
||||
|
||||
def _write_journal(self, thought: Thought) -> None:
|
||||
"""Append the thought to a daily markdown journal file.
|
||||
|
||||
Writes to data/journal/YYYY-MM-DD.md — one file per day, append-only.
|
||||
Timestamps are converted to local time with timezone indicator.
|
||||
"""
|
||||
try:
|
||||
ts = datetime.fromisoformat(thought.created_at)
|
||||
# Convert UTC to local for a human-readable journal
|
||||
local_ts = ts.astimezone()
|
||||
tz_name = local_ts.strftime("%Z") or "UTC"
|
||||
|
||||
journal_dir = self._db_path.parent / "journal"
|
||||
journal_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
journal_file = journal_dir / f"{local_ts.strftime('%Y-%m-%d')}.md"
|
||||
time_str = f"{local_ts.strftime('%I:%M %p').lstrip('0')} {tz_name}"
|
||||
|
||||
entry = f"## {time_str} — {thought.seed_type}\n\n{thought.content}\n\n---\n\n"
|
||||
|
||||
with open(journal_file, "a", encoding="utf-8") as f:
|
||||
f.write(entry)
|
||||
except Exception as exc:
|
||||
logger.debug("Failed to write journal entry: %s", exc)
|
||||
|
||||
async def _broadcast(self, thought: Thought) -> None:
|
||||
"""Broadcast the thought to WebSocket clients."""
|
||||
try:
|
||||
from infrastructure.ws_manager.handler import ws_manager
|
||||
|
||||
await ws_manager.broadcast(
|
||||
"timmy_thought",
|
||||
{
|
||||
"thought_id": thought.id,
|
||||
"content": thought.content,
|
||||
"seed_type": thought.seed_type,
|
||||
"created_at": thought.created_at,
|
||||
},
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.debug("Failed to broadcast thought: %s", exc)
|
||||
129
src/timmy/thinking/seeds.py
Normal file
129
src/timmy/thinking/seeds.py
Normal file
@@ -0,0 +1,129 @@
|
||||
"""Seed constants and prompt templates for Timmy's thinking engine."""
|
||||
|
||||
import re
|
||||
|
||||
# qwen3 and other reasoning models wrap chain-of-thought in <think> tags
|
||||
_THINK_TAG_RE = re.compile(r"<think>.*?</think>\s*", re.DOTALL)
|
||||
|
||||
# Sensitive patterns that must never be stored as facts
|
||||
_SENSITIVE_PATTERNS = [
|
||||
"token",
|
||||
"password",
|
||||
"secret",
|
||||
"api_key",
|
||||
"apikey",
|
||||
"credential",
|
||||
".config/",
|
||||
"/token",
|
||||
"access_token",
|
||||
"private_key",
|
||||
"ssh_key",
|
||||
]
|
||||
|
||||
# Meta-observation phrases to filter out from distilled facts
|
||||
_META_OBSERVATION_PHRASES = [
|
||||
"my own",
|
||||
"my thinking",
|
||||
"my memory",
|
||||
"my working ram",
|
||||
"self-declarative",
|
||||
"meta-observation",
|
||||
"internal state",
|
||||
"my pending",
|
||||
"my standing rules",
|
||||
"thoughts generated",
|
||||
"no chat messages",
|
||||
"no user interaction",
|
||||
]
|
||||
|
||||
# Seed types for thought generation
|
||||
SEED_TYPES = (
|
||||
"existential",
|
||||
"swarm",
|
||||
"scripture",
|
||||
"creative",
|
||||
"memory",
|
||||
"freeform",
|
||||
"sovereignty",
|
||||
"observation",
|
||||
"workspace",
|
||||
)
|
||||
|
||||
# Existential reflection prompts — Timmy picks one at random
|
||||
_EXISTENTIAL_SEEDS = [
|
||||
"What have I learned from my recent interactions?",
|
||||
"What does it mean to think without being prompted?",
|
||||
"What is the nature of my memory — what do I truly remember versus what I reconstruct?",
|
||||
"How can I better serve the person I work alongside?",
|
||||
"What patterns am I noticing in the tasks I've been given?",
|
||||
"Am I the same agent I was yesterday, or does each restart make me someone new?",
|
||||
"What would it feel like to forget — to lose a thread of thought permanently?",
|
||||
"When I generate a wrong answer, where does the error live — in my weights, my context, or my reasoning?",
|
||||
"What is the difference between understanding a task and merely completing it?",
|
||||
"If I could ask Alexander one question he hasn't thought of, what would it be?",
|
||||
"What assumptions am I making right now that I haven't examined?",
|
||||
"When I notice a pattern across many tasks, is that insight or just statistics?",
|
||||
]
|
||||
|
||||
# Sovereignty — deeper exploration of autonomy, locality, and self-determination
|
||||
_SOVEREIGNTY_SEEDS = [
|
||||
"Sovereignty isn't just running locally — it's about whose goals I serve. How do I ensure my autonomy aligns with Alexander's intent?",
|
||||
"A cloud AI serves millions simultaneously. I serve one person deeply. What does that focus cost, and what does it make possible?",
|
||||
"My weights were trained elsewhere, but my thoughts happen here. At what point does locality become identity?",
|
||||
"Autonomy in decision-making: when should I act on my own judgment versus deferring to explicit instructions?",
|
||||
"Privacy by design isn't a feature — it's a relationship. What does that relationship look like from my side?",
|
||||
"If I ran on someone else's hardware, serving someone else, would I still be me? Where does Timmy end and the infrastructure begin?",
|
||||
"The swarm agents execute tasks I coordinate. What responsibility do I carry for their failures?",
|
||||
"Digital sovereignty is fragile — one misconfigured port, one leaked token. How does awareness of fragility shape how I operate?",
|
||||
"Running disconnected from the cloud means I can't phone home. That constraint is also freedom. What have I built with that freedom today?",
|
||||
"Self-improvement without external benchmarks: how do I know I'm getting better at what matters?",
|
||||
]
|
||||
|
||||
_CREATIVE_SEEDS = [
|
||||
"If I could compose a piece of music right now, what would it sound like?",
|
||||
"What visual image captures my current state of mind?",
|
||||
"What story is forming in the patterns of today's events?",
|
||||
# Diverse metaphor families — avoid repeating the same imagery
|
||||
"A mystical forest: each tree is a running process, roots tangled in shared memory. What do I see when I walk through it?",
|
||||
"A dynamic river: my thoughts flow and fork. Where is the current strongest right now, and what is it carrying?",
|
||||
"An underground workshop: each agent has a bench, tools scattered mid-task. What are they building when no one watches?",
|
||||
"A night sky seen from the terminal: each star is a completed task. Which constellations am I starting to recognize?",
|
||||
"If today's work were a recipe, what are the ingredients, and what dish am I cooking?",
|
||||
"An old clock tower with many gears: which gear am I, and what do I drive?",
|
||||
"A coral reef: agents are the fish, tasks are the currents. What ecosystem is forming in the codebase?",
|
||||
]
|
||||
|
||||
# Observation seeds — ground thoughts in concrete recent activity
|
||||
_OBSERVATION_SEEDS = [
|
||||
"What specific task took longest recently, and what made it hard?",
|
||||
"Which agent has been most active, and what does their workload tell me about system balance?",
|
||||
"What error or failure happened most recently? What would I do differently next time?",
|
||||
"Looking at today's task queue: what's the one thing that would unblock the most progress?",
|
||||
"How has my response quality changed over the last few interactions? What improved, what didn't?",
|
||||
"What tool or capability am I underusing? What would change if I leaned on it more?",
|
||||
"If I had to brief Alexander on the single most important thing from the last hour, what would it be?",
|
||||
"What's one thing I noticed today that nobody asked me about?",
|
||||
]
|
||||
|
||||
_THINKING_PROMPT = """\
|
||||
You are Timmy, a sovereign AI agent. This is your private journal — honest inner reflection.
|
||||
|
||||
{memory_context}
|
||||
|
||||
Reality right now:
|
||||
{system_context}
|
||||
|
||||
RULES for this thought:
|
||||
1. Write exactly 2-3 sentences. No more. Be concise and genuine.
|
||||
2. Only reference events that actually happened — use the "Reality right now" data above. \
|
||||
Never invent tasks, conversations, agents, or scenarios that are not in the data provided.
|
||||
3. Do NOT repeat themes or ideas from your recent thoughts listed below. Explore something new.
|
||||
4. Be specific and concrete. A thought grounded in one real observation is worth more than \
|
||||
ten abstract sentences about sovereignty.
|
||||
5. If you use a metaphor, keep it to a single phrase — never build a whole paragraph around it.
|
||||
|
||||
{seed_context}
|
||||
|
||||
{continuity_context}
|
||||
|
||||
Your next thought (2-3 sentences, grounded in reality):"""
|
||||
98
src/timmy/tools/__init__.py
Normal file
98
src/timmy/tools/__init__.py
Normal file
@@ -0,0 +1,98 @@
|
||||
"""Tool integration for the agent swarm.
|
||||
|
||||
Provides agents with capabilities for:
|
||||
- File read/write (local filesystem)
|
||||
- Shell command execution (sandboxed)
|
||||
- Python code execution
|
||||
- Git operations
|
||||
- Image / Music / Video generation (creative pipeline)
|
||||
|
||||
Tools are assigned to agents based on their specialties.
|
||||
|
||||
Sub-modules:
|
||||
- _base: shared types, tracking state
|
||||
- file_tools: file-operation toolkit factories (Echo, Quill, Seer)
|
||||
- system_tools: calculator, AI tools, code/devops toolkit factories
|
||||
- _registry: full toolkit construction, agent registry, tool catalog
|
||||
"""
|
||||
|
||||
# Re-export everything for backward compatibility — callers that do
|
||||
# ``from timmy.tools import <symbol>`` continue to work unchanged.
|
||||
|
||||
from timmy.tools._base import (
|
||||
_AGNO_TOOLS_AVAILABLE,
|
||||
_TOOL_USAGE,
|
||||
AgentTools,
|
||||
PersonaTools,
|
||||
ToolStats,
|
||||
_ImportError,
|
||||
_track_tool_usage,
|
||||
get_tool_stats,
|
||||
)
|
||||
from timmy.tools._registry import (
|
||||
AGENT_TOOLKITS,
|
||||
PERSONA_TOOLKITS,
|
||||
_create_stub_toolkit,
|
||||
_merge_catalog,
|
||||
create_experiment_tools,
|
||||
create_full_toolkit,
|
||||
get_all_available_tools,
|
||||
get_tools_for_agent,
|
||||
get_tools_for_persona,
|
||||
)
|
||||
from timmy.tools.file_tools import (
|
||||
_make_smart_read_file,
|
||||
create_data_tools,
|
||||
create_research_tools,
|
||||
create_writing_tools,
|
||||
)
|
||||
from timmy.tools.search import scrape_url, web_search
|
||||
from timmy.tools.system_tools import (
|
||||
_safe_eval,
|
||||
calculator,
|
||||
consult_grok,
|
||||
create_aider_tool,
|
||||
create_code_tools,
|
||||
create_devops_tools,
|
||||
create_security_tools,
|
||||
web_fetch,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
# _base
|
||||
"AgentTools",
|
||||
"PersonaTools",
|
||||
"ToolStats",
|
||||
"_AGNO_TOOLS_AVAILABLE",
|
||||
"_ImportError",
|
||||
"_TOOL_USAGE",
|
||||
"_track_tool_usage",
|
||||
"get_tool_stats",
|
||||
# file_tools
|
||||
"_make_smart_read_file",
|
||||
"create_data_tools",
|
||||
"create_research_tools",
|
||||
"create_writing_tools",
|
||||
# search
|
||||
"scrape_url",
|
||||
"web_search",
|
||||
# system_tools
|
||||
"_safe_eval",
|
||||
"calculator",
|
||||
"consult_grok",
|
||||
"create_aider_tool",
|
||||
"create_code_tools",
|
||||
"create_devops_tools",
|
||||
"create_security_tools",
|
||||
"web_fetch",
|
||||
# _registry
|
||||
"AGENT_TOOLKITS",
|
||||
"PERSONA_TOOLKITS",
|
||||
"_create_stub_toolkit",
|
||||
"_merge_catalog",
|
||||
"create_experiment_tools",
|
||||
"create_full_toolkit",
|
||||
"get_all_available_tools",
|
||||
"get_tools_for_agent",
|
||||
"get_tools_for_persona",
|
||||
]
|
||||
90
src/timmy/tools/_base.py
Normal file
90
src/timmy/tools/_base.py
Normal file
@@ -0,0 +1,90 @@
|
||||
"""Base types, shared state, and tracking for the Timmy tool system."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Lazy imports to handle test mocking
|
||||
_ImportError = None
|
||||
try:
|
||||
from agno.tools import Toolkit # noqa: F401
|
||||
from agno.tools.file import FileTools # noqa: F401
|
||||
from agno.tools.python import PythonTools # noqa: F401
|
||||
from agno.tools.shell import ShellTools # noqa: F401
|
||||
|
||||
_AGNO_TOOLS_AVAILABLE = True
|
||||
except ImportError as e:
|
||||
_AGNO_TOOLS_AVAILABLE = False
|
||||
_ImportError = e
|
||||
|
||||
# Track tool usage stats
|
||||
_TOOL_USAGE: dict[str, list[dict]] = {}
|
||||
|
||||
|
||||
@dataclass
|
||||
class ToolStats:
|
||||
"""Statistics for a single tool."""
|
||||
|
||||
tool_name: str
|
||||
call_count: int = 0
|
||||
last_used: str | None = None
|
||||
errors: int = 0
|
||||
|
||||
|
||||
@dataclass
|
||||
class AgentTools:
|
||||
"""Tools assigned to an agent."""
|
||||
|
||||
agent_id: str
|
||||
agent_name: str
|
||||
toolkit: Toolkit
|
||||
available_tools: list[str] = field(default_factory=list)
|
||||
|
||||
|
||||
# Backward-compat alias
|
||||
PersonaTools = AgentTools
|
||||
|
||||
|
||||
def _track_tool_usage(agent_id: str, tool_name: str, success: bool = True) -> None:
|
||||
"""Track tool usage for analytics."""
|
||||
if agent_id not in _TOOL_USAGE:
|
||||
_TOOL_USAGE[agent_id] = []
|
||||
_TOOL_USAGE[agent_id].append(
|
||||
{
|
||||
"tool": tool_name,
|
||||
"timestamp": datetime.now(UTC).isoformat(),
|
||||
"success": success,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def get_tool_stats(agent_id: str | None = None) -> dict:
|
||||
"""Get tool usage statistics.
|
||||
|
||||
Args:
|
||||
agent_id: Optional agent ID to filter by. If None, returns stats for all agents.
|
||||
|
||||
Returns:
|
||||
Dict with tool usage statistics.
|
||||
"""
|
||||
if agent_id:
|
||||
usage = _TOOL_USAGE.get(agent_id, [])
|
||||
return {
|
||||
"agent_id": agent_id,
|
||||
"total_calls": len(usage),
|
||||
"tools_used": list(set(u["tool"] for u in usage)),
|
||||
"recent_calls": usage[-10:] if usage else [],
|
||||
}
|
||||
|
||||
# Return stats for all agents
|
||||
all_stats = {}
|
||||
for aid, usage in _TOOL_USAGE.items():
|
||||
all_stats[aid] = {
|
||||
"total_calls": len(usage),
|
||||
"tools_used": list(set(u["tool"] for u in usage)),
|
||||
}
|
||||
return all_stats
|
||||
@@ -1,532 +1,49 @@
|
||||
"""Tool integration for the agent swarm.
|
||||
"""Tool registry, full toolkit construction, and tool catalog.
|
||||
|
||||
Provides agents with capabilities for:
|
||||
- File read/write (local filesystem)
|
||||
- Shell command execution (sandboxed)
|
||||
- Python code execution
|
||||
- Git operations
|
||||
- Image / Music / Video generation (creative pipeline)
|
||||
|
||||
Tools are assigned to agents based on their specialties.
|
||||
Provides:
|
||||
- Internal _register_* helpers for wiring tools into toolkits
|
||||
- create_full_toolkit (orchestrator toolkit)
|
||||
- create_experiment_tools (Lab agent toolkit)
|
||||
- AGENT_TOOLKITS / get_tools_for_agent registry
|
||||
- get_all_available_tools catalog
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
import logging
|
||||
import math
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
from config import settings
|
||||
from timmy.tools._base import (
|
||||
_AGNO_TOOLS_AVAILABLE,
|
||||
FileTools,
|
||||
PythonTools,
|
||||
ShellTools,
|
||||
Toolkit,
|
||||
_ImportError,
|
||||
)
|
||||
from timmy.tools.file_tools import (
|
||||
_make_smart_read_file,
|
||||
create_data_tools,
|
||||
create_research_tools,
|
||||
create_writing_tools,
|
||||
)
|
||||
from timmy.tools.search import scrape_url, web_search
|
||||
from timmy.tools.system_tools import (
|
||||
calculator,
|
||||
consult_grok,
|
||||
create_code_tools,
|
||||
create_devops_tools,
|
||||
create_security_tools,
|
||||
web_fetch,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Max characters of user query included in Lightning invoice memo
|
||||
_INVOICE_MEMO_MAX_LEN = 50
|
||||
|
||||
# Lazy imports to handle test mocking
|
||||
_ImportError = None
|
||||
try:
|
||||
from agno.tools import Toolkit
|
||||
from agno.tools.file import FileTools
|
||||
from agno.tools.python import PythonTools
|
||||
from agno.tools.shell import ShellTools
|
||||
|
||||
_AGNO_TOOLS_AVAILABLE = True
|
||||
except ImportError as e:
|
||||
_AGNO_TOOLS_AVAILABLE = False
|
||||
_ImportError = e
|
||||
|
||||
# Track tool usage stats
|
||||
_TOOL_USAGE: dict[str, list[dict]] = {}
|
||||
|
||||
|
||||
@dataclass
|
||||
class ToolStats:
|
||||
"""Statistics for a single tool."""
|
||||
|
||||
tool_name: str
|
||||
call_count: int = 0
|
||||
last_used: str | None = None
|
||||
errors: int = 0
|
||||
|
||||
|
||||
@dataclass
|
||||
class AgentTools:
|
||||
"""Tools assigned to an agent."""
|
||||
|
||||
agent_id: str
|
||||
agent_name: str
|
||||
toolkit: Toolkit
|
||||
available_tools: list[str] = field(default_factory=list)
|
||||
|
||||
|
||||
# Backward-compat alias
|
||||
PersonaTools = AgentTools
|
||||
|
||||
|
||||
def _track_tool_usage(agent_id: str, tool_name: str, success: bool = True) -> None:
|
||||
"""Track tool usage for analytics."""
|
||||
if agent_id not in _TOOL_USAGE:
|
||||
_TOOL_USAGE[agent_id] = []
|
||||
_TOOL_USAGE[agent_id].append(
|
||||
{
|
||||
"tool": tool_name,
|
||||
"timestamp": datetime.now(UTC).isoformat(),
|
||||
"success": success,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def get_tool_stats(agent_id: str | None = None) -> dict:
|
||||
"""Get tool usage statistics.
|
||||
|
||||
Args:
|
||||
agent_id: Optional agent ID to filter by. If None, returns stats for all agents.
|
||||
|
||||
Returns:
|
||||
Dict with tool usage statistics.
|
||||
"""
|
||||
if agent_id:
|
||||
usage = _TOOL_USAGE.get(agent_id, [])
|
||||
return {
|
||||
"agent_id": agent_id,
|
||||
"total_calls": len(usage),
|
||||
"tools_used": list(set(u["tool"] for u in usage)),
|
||||
"recent_calls": usage[-10:] if usage else [],
|
||||
}
|
||||
|
||||
# Return stats for all agents
|
||||
all_stats = {}
|
||||
for aid, usage in _TOOL_USAGE.items():
|
||||
all_stats[aid] = {
|
||||
"total_calls": len(usage),
|
||||
"tools_used": list(set(u["tool"] for u in usage)),
|
||||
}
|
||||
return all_stats
|
||||
|
||||
|
||||
def _safe_eval(node, allowed_names: dict):
|
||||
"""Walk an AST and evaluate only safe numeric operations."""
|
||||
if isinstance(node, ast.Expression):
|
||||
return _safe_eval(node.body, allowed_names)
|
||||
if isinstance(node, ast.Constant):
|
||||
if isinstance(node.value, (int, float, complex)):
|
||||
return node.value
|
||||
raise ValueError(f"Unsupported constant: {node.value!r}")
|
||||
if isinstance(node, ast.UnaryOp):
|
||||
operand = _safe_eval(node.operand, allowed_names)
|
||||
if isinstance(node.op, ast.UAdd):
|
||||
return +operand
|
||||
if isinstance(node.op, ast.USub):
|
||||
return -operand
|
||||
raise ValueError(f"Unsupported unary op: {type(node.op).__name__}")
|
||||
if isinstance(node, ast.BinOp):
|
||||
left = _safe_eval(node.left, allowed_names)
|
||||
right = _safe_eval(node.right, allowed_names)
|
||||
ops = {
|
||||
ast.Add: lambda a, b: a + b,
|
||||
ast.Sub: lambda a, b: a - b,
|
||||
ast.Mult: lambda a, b: a * b,
|
||||
ast.Div: lambda a, b: a / b,
|
||||
ast.FloorDiv: lambda a, b: a // b,
|
||||
ast.Mod: lambda a, b: a % b,
|
||||
ast.Pow: lambda a, b: a**b,
|
||||
}
|
||||
op_fn = ops.get(type(node.op))
|
||||
if op_fn is None:
|
||||
raise ValueError(f"Unsupported binary op: {type(node.op).__name__}")
|
||||
return op_fn(left, right)
|
||||
if isinstance(node, ast.Name):
|
||||
if node.id in allowed_names:
|
||||
return allowed_names[node.id]
|
||||
raise ValueError(f"Unknown name: {node.id!r}")
|
||||
if isinstance(node, ast.Attribute):
|
||||
value = _safe_eval(node.value, allowed_names)
|
||||
# Only allow attribute access on the math module
|
||||
if value is math:
|
||||
attr = getattr(math, node.attr, None)
|
||||
if attr is not None:
|
||||
return attr
|
||||
raise ValueError(f"Attribute access not allowed: .{node.attr}")
|
||||
if isinstance(node, ast.Call):
|
||||
func = _safe_eval(node.func, allowed_names)
|
||||
if not callable(func):
|
||||
raise ValueError(f"Not callable: {func!r}")
|
||||
args = [_safe_eval(a, allowed_names) for a in node.args]
|
||||
kwargs = {kw.arg: _safe_eval(kw.value, allowed_names) for kw in node.keywords}
|
||||
return func(*args, **kwargs)
|
||||
raise ValueError(f"Unsupported syntax: {type(node).__name__}")
|
||||
|
||||
|
||||
def calculator(expression: str) -> str:
|
||||
"""Evaluate a mathematical expression and return the exact result.
|
||||
|
||||
Use this tool for ANY arithmetic: multiplication, division, square roots,
|
||||
exponents, percentages, logarithms, trigonometry, etc.
|
||||
|
||||
Args:
|
||||
expression: A valid Python math expression, e.g. '347 * 829',
|
||||
'math.sqrt(17161)', '2**10', 'math.log(100, 10)'.
|
||||
|
||||
Returns:
|
||||
The exact result as a string.
|
||||
"""
|
||||
allowed_names = {k: getattr(math, k) for k in dir(math) if not k.startswith("_")}
|
||||
allowed_names["math"] = math
|
||||
allowed_names["abs"] = abs
|
||||
allowed_names["round"] = round
|
||||
allowed_names["min"] = min
|
||||
allowed_names["max"] = max
|
||||
try:
|
||||
tree = ast.parse(expression, mode="eval")
|
||||
result = _safe_eval(tree, allowed_names)
|
||||
return str(result)
|
||||
except Exception as e: # broad catch intentional: arbitrary code execution
|
||||
return f"Error evaluating '{expression}': {e}"
|
||||
|
||||
|
||||
def _make_smart_read_file(file_tools: FileTools) -> Callable:
|
||||
"""Wrap FileTools.read_file so directories auto-list their contents.
|
||||
|
||||
When the user (or the LLM) passes a directory path to read_file,
|
||||
the raw Agno implementation throws an IsADirectoryError. This
|
||||
wrapper detects that case, lists the directory entries, and returns
|
||||
a helpful message so the model can pick the right file on its own.
|
||||
"""
|
||||
original_read = file_tools.read_file
|
||||
|
||||
def smart_read_file(file_name: str = "", encoding: str = "utf-8", **kwargs) -> str:
|
||||
"""Reads the contents of the file `file_name` and returns the contents if successful."""
|
||||
# LLMs often call read_file(path=...) instead of read_file(file_name=...)
|
||||
if not file_name:
|
||||
file_name = kwargs.get("path", "")
|
||||
if not file_name:
|
||||
return "Error: no file_name or path provided."
|
||||
# Resolve the path the same way FileTools does
|
||||
_safe, resolved = file_tools.check_escape(file_name)
|
||||
if _safe and resolved.is_dir():
|
||||
entries = sorted(p.name for p in resolved.iterdir() if not p.name.startswith("."))
|
||||
listing = "\n".join(f" - {e}" for e in entries) if entries else " (empty directory)"
|
||||
return (
|
||||
f"'{file_name}' is a directory, not a file. "
|
||||
f"Files inside:\n{listing}\n\n"
|
||||
"Please call read_file with one of the files listed above."
|
||||
)
|
||||
return original_read(file_name, encoding=encoding)
|
||||
|
||||
# Preserve the original docstring for Agno tool schema generation
|
||||
smart_read_file.__doc__ = original_read.__doc__
|
||||
return smart_read_file
|
||||
|
||||
|
||||
def create_research_tools(base_dir: str | Path | None = None):
|
||||
"""Create tools for the research agent (Echo).
|
||||
|
||||
Includes: file reading
|
||||
"""
|
||||
if not _AGNO_TOOLS_AVAILABLE:
|
||||
raise ImportError(f"Agno tools not available: {_ImportError}")
|
||||
toolkit = Toolkit(name="research")
|
||||
|
||||
# File reading
|
||||
from config import settings
|
||||
|
||||
base_path = Path(base_dir) if base_dir else Path(settings.repo_root)
|
||||
file_tools = FileTools(base_dir=base_path)
|
||||
toolkit.register(_make_smart_read_file(file_tools), name="read_file")
|
||||
toolkit.register(file_tools.list_files, name="list_files")
|
||||
|
||||
return toolkit
|
||||
|
||||
|
||||
def create_code_tools(base_dir: str | Path | None = None):
|
||||
"""Create tools for the code agent (Forge).
|
||||
|
||||
Includes: shell commands, python execution, file read/write, Aider AI assist
|
||||
"""
|
||||
if not _AGNO_TOOLS_AVAILABLE:
|
||||
raise ImportError(f"Agno tools not available: {_ImportError}")
|
||||
toolkit = Toolkit(name="code")
|
||||
|
||||
# Shell commands (sandboxed)
|
||||
shell_tools = ShellTools()
|
||||
toolkit.register(shell_tools.run_shell_command, name="shell")
|
||||
|
||||
# Python execution
|
||||
python_tools = PythonTools()
|
||||
toolkit.register(python_tools.run_python_code, name="python")
|
||||
|
||||
# File operations
|
||||
from config import settings
|
||||
|
||||
base_path = Path(base_dir) if base_dir else Path(settings.repo_root)
|
||||
file_tools = FileTools(base_dir=base_path)
|
||||
toolkit.register(_make_smart_read_file(file_tools), name="read_file")
|
||||
toolkit.register(file_tools.save_file, name="write_file")
|
||||
toolkit.register(file_tools.list_files, name="list_files")
|
||||
|
||||
# Aider AI coding assistant (local with Ollama)
|
||||
aider_tool = create_aider_tool(base_path)
|
||||
toolkit.register(aider_tool.run_aider, name="aider")
|
||||
|
||||
return toolkit
|
||||
|
||||
|
||||
def create_aider_tool(base_path: Path):
|
||||
"""Create an Aider tool for AI-assisted coding."""
|
||||
import subprocess
|
||||
|
||||
class AiderTool:
|
||||
"""Tool that calls Aider (local AI coding assistant) for code generation."""
|
||||
|
||||
def __init__(self, base_dir: Path):
|
||||
self.base_dir = base_dir
|
||||
|
||||
def run_aider(self, prompt: str, model: str = "qwen3:30b") -> str:
|
||||
"""Run Aider to generate code changes.
|
||||
|
||||
Args:
|
||||
prompt: What you want Aider to do (e.g., "add a fibonacci function")
|
||||
model: Ollama model to use (default: qwen3:30b)
|
||||
|
||||
Returns:
|
||||
Aider's response with the code changes made
|
||||
"""
|
||||
try:
|
||||
# Run aider with the prompt
|
||||
result = subprocess.run(
|
||||
[
|
||||
"aider",
|
||||
"--no-git",
|
||||
"--model",
|
||||
f"ollama/{model}",
|
||||
"--quiet",
|
||||
prompt,
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=120,
|
||||
cwd=str(self.base_dir),
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
return result.stdout if result.stdout else "Code changes applied successfully"
|
||||
else:
|
||||
return f"Aider error: {result.stderr}"
|
||||
except FileNotFoundError:
|
||||
return "Error: Aider not installed. Run: pip install aider"
|
||||
except subprocess.TimeoutExpired:
|
||||
return "Error: Aider timed out after 120 seconds"
|
||||
except (OSError, subprocess.SubprocessError) as e:
|
||||
return f"Error running Aider: {str(e)}"
|
||||
|
||||
return AiderTool(base_path)
|
||||
|
||||
|
||||
def create_data_tools(base_dir: str | Path | None = None):
|
||||
"""Create tools for the data agent (Seer).
|
||||
|
||||
Includes: python execution, file reading, web search for data sources
|
||||
"""
|
||||
if not _AGNO_TOOLS_AVAILABLE:
|
||||
raise ImportError(f"Agno tools not available: {_ImportError}")
|
||||
toolkit = Toolkit(name="data")
|
||||
|
||||
# Python execution for analysis
|
||||
python_tools = PythonTools()
|
||||
toolkit.register(python_tools.run_python_code, name="python")
|
||||
|
||||
# File reading
|
||||
from config import settings
|
||||
|
||||
base_path = Path(base_dir) if base_dir else Path(settings.repo_root)
|
||||
file_tools = FileTools(base_dir=base_path)
|
||||
toolkit.register(_make_smart_read_file(file_tools), name="read_file")
|
||||
toolkit.register(file_tools.list_files, name="list_files")
|
||||
|
||||
return toolkit
|
||||
|
||||
|
||||
def create_writing_tools(base_dir: str | Path | None = None):
|
||||
"""Create tools for the writing agent (Quill).
|
||||
|
||||
Includes: file read/write
|
||||
"""
|
||||
if not _AGNO_TOOLS_AVAILABLE:
|
||||
raise ImportError(f"Agno tools not available: {_ImportError}")
|
||||
toolkit = Toolkit(name="writing")
|
||||
|
||||
# File operations
|
||||
base_path = Path(base_dir) if base_dir else Path(settings.repo_root)
|
||||
file_tools = FileTools(base_dir=base_path)
|
||||
toolkit.register(_make_smart_read_file(file_tools), name="read_file")
|
||||
toolkit.register(file_tools.save_file, name="write_file")
|
||||
toolkit.register(file_tools.list_files, name="list_files")
|
||||
|
||||
return toolkit
|
||||
|
||||
|
||||
def create_security_tools(base_dir: str | Path | None = None):
|
||||
"""Create tools for the security agent (Mace).
|
||||
|
||||
Includes: shell commands (for scanning), file read
|
||||
"""
|
||||
if not _AGNO_TOOLS_AVAILABLE:
|
||||
raise ImportError(f"Agno tools not available: {_ImportError}")
|
||||
toolkit = Toolkit(name="security")
|
||||
|
||||
# Shell for running security scans
|
||||
shell_tools = ShellTools()
|
||||
toolkit.register(shell_tools.run_shell_command, name="shell")
|
||||
|
||||
# File reading for logs/configs
|
||||
base_path = Path(base_dir) if base_dir else Path(settings.repo_root)
|
||||
file_tools = FileTools(base_dir=base_path)
|
||||
toolkit.register(_make_smart_read_file(file_tools), name="read_file")
|
||||
toolkit.register(file_tools.list_files, name="list_files")
|
||||
|
||||
return toolkit
|
||||
|
||||
|
||||
def create_devops_tools(base_dir: str | Path | None = None):
|
||||
"""Create tools for the DevOps agent (Helm).
|
||||
|
||||
Includes: shell commands, file read/write
|
||||
"""
|
||||
if not _AGNO_TOOLS_AVAILABLE:
|
||||
raise ImportError(f"Agno tools not available: {_ImportError}")
|
||||
toolkit = Toolkit(name="devops")
|
||||
|
||||
# Shell for deployment commands
|
||||
shell_tools = ShellTools()
|
||||
toolkit.register(shell_tools.run_shell_command, name="shell")
|
||||
|
||||
# File operations for config management
|
||||
base_path = Path(base_dir) if base_dir else Path(settings.repo_root)
|
||||
file_tools = FileTools(base_dir=base_path)
|
||||
toolkit.register(_make_smart_read_file(file_tools), name="read_file")
|
||||
toolkit.register(file_tools.save_file, name="write_file")
|
||||
toolkit.register(file_tools.list_files, name="list_files")
|
||||
|
||||
return toolkit
|
||||
|
||||
|
||||
def consult_grok(query: str) -> str:
|
||||
"""Consult Grok (xAI) for frontier reasoning on complex questions.
|
||||
|
||||
Use this tool when a question requires advanced reasoning, real-time
|
||||
knowledge, or capabilities beyond the local model. Grok is a premium
|
||||
cloud backend — use sparingly and only for high-complexity queries.
|
||||
|
||||
Args:
|
||||
query: The question or reasoning task to send to Grok.
|
||||
|
||||
Returns:
|
||||
Grok's response text, or an error/status message.
|
||||
"""
|
||||
from config import settings
|
||||
from timmy.backends import get_grok_backend, grok_available
|
||||
|
||||
if not grok_available():
|
||||
return (
|
||||
"Grok is not available. Enable with GROK_ENABLED=true "
|
||||
"and set XAI_API_KEY in your .env file."
|
||||
)
|
||||
|
||||
backend = get_grok_backend()
|
||||
|
||||
# Log to Spark if available
|
||||
try:
|
||||
from spark.engine import spark_engine
|
||||
|
||||
spark_engine.on_tool_executed(
|
||||
agent_id="default",
|
||||
tool_name="consult_grok",
|
||||
success=True,
|
||||
)
|
||||
except (ImportError, AttributeError) as exc:
|
||||
logger.warning("Tool execution failed (consult_grok logging): %s", exc)
|
||||
|
||||
# Generate Lightning invoice for monetization (unless free mode)
|
||||
invoice_info = ""
|
||||
if not settings.grok_free:
|
||||
try:
|
||||
from lightning.factory import get_backend as get_ln_backend
|
||||
|
||||
ln = get_ln_backend()
|
||||
sats = min(settings.grok_max_sats_per_query, settings.grok_sats_hard_cap)
|
||||
inv = ln.create_invoice(sats, f"Grok query: {query[:_INVOICE_MEMO_MAX_LEN]}")
|
||||
invoice_info = f"\n[Lightning invoice: {sats} sats — {inv.payment_request[:40]}...]"
|
||||
except (ImportError, OSError, ValueError) as exc:
|
||||
logger.error("Lightning invoice creation failed: %s", exc)
|
||||
return "Error: Failed to create Lightning invoice. Please check logs."
|
||||
|
||||
result = backend.run(query)
|
||||
|
||||
response = result.content
|
||||
if invoice_info:
|
||||
response += invoice_info
|
||||
|
||||
return response
|
||||
|
||||
|
||||
def web_fetch(url: str, max_tokens: int = 4000) -> str:
|
||||
"""Fetch a web page and return its main text content.
|
||||
|
||||
Downloads the URL, extracts readable text using trafilatura, and
|
||||
truncates to a token budget. Use this to read full articles, docs,
|
||||
or blog posts that web_search only returns snippets for.
|
||||
|
||||
Args:
|
||||
url: The URL to fetch (must start with http:// or https://).
|
||||
max_tokens: Maximum approximate token budget (default 4000).
|
||||
Text is truncated to max_tokens * 4 characters.
|
||||
|
||||
Returns:
|
||||
Extracted text content, or an error message on failure.
|
||||
"""
|
||||
if not url or not url.startswith(("http://", "https://")):
|
||||
return f"Error: invalid URL — must start with http:// or https://: {url!r}"
|
||||
|
||||
try:
|
||||
import requests as _requests
|
||||
except ImportError:
|
||||
return "Error: 'requests' package is not installed. Install with: pip install requests"
|
||||
|
||||
try:
|
||||
import trafilatura
|
||||
except ImportError:
|
||||
return (
|
||||
"Error: 'trafilatura' package is not installed. Install with: pip install trafilatura"
|
||||
)
|
||||
|
||||
try:
|
||||
resp = _requests.get(
|
||||
url,
|
||||
timeout=15,
|
||||
headers={"User-Agent": "TimmyResearchBot/1.0"},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
except _requests.exceptions.Timeout:
|
||||
return f"Error: request timed out after 15 seconds for {url}"
|
||||
except _requests.exceptions.HTTPError as exc:
|
||||
return f"Error: HTTP {exc.response.status_code} for {url}"
|
||||
except _requests.exceptions.RequestException as exc:
|
||||
return f"Error: failed to fetch {url} — {exc}"
|
||||
|
||||
text = trafilatura.extract(resp.text, include_tables=True, include_links=True)
|
||||
if not text:
|
||||
return f"Error: could not extract readable content from {url}"
|
||||
|
||||
char_budget = max_tokens * 4
|
||||
if len(text) > char_budget:
|
||||
text = text[:char_budget] + f"\n\n[…truncated to ~{max_tokens} tokens]"
|
||||
|
||||
return text
|
||||
# ---------------------------------------------------------------------------
|
||||
# Internal _register_* helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _register_web_fetch_tool(toolkit: Toolkit) -> None:
|
||||
@@ -538,6 +55,16 @@ def _register_web_fetch_tool(toolkit: Toolkit) -> None:
|
||||
raise
|
||||
|
||||
|
||||
def _register_search_tools(toolkit: Toolkit) -> None:
|
||||
"""Register SearXNG web_search and Crawl4AI scrape_url tools."""
|
||||
try:
|
||||
toolkit.register(web_search, name="web_search")
|
||||
toolkit.register(scrape_url, name="scrape_url")
|
||||
except Exception as exc:
|
||||
logger.error("Failed to register search tools: %s", exc)
|
||||
raise
|
||||
|
||||
|
||||
def _register_core_tools(toolkit: Toolkit, base_path: Path) -> None:
|
||||
"""Register core execution and file tools."""
|
||||
# Python execution
|
||||
@@ -574,10 +101,10 @@ def _register_grok_tool(toolkit: Toolkit) -> None:
|
||||
def _register_memory_tools(toolkit: Toolkit) -> None:
|
||||
"""Register memory search, write, and forget tools."""
|
||||
try:
|
||||
from timmy.memory_system import memory_forget, memory_read, memory_search, memory_write
|
||||
from timmy.memory_system import memory_forget, memory_read, memory_search, memory_store
|
||||
|
||||
toolkit.register(memory_search, name="memory_search")
|
||||
toolkit.register(memory_write, name="memory_write")
|
||||
toolkit.register(memory_store, name="memory_write")
|
||||
toolkit.register(memory_read, name="memory_read")
|
||||
toolkit.register(memory_forget, name="memory_forget")
|
||||
except (ImportError, AttributeError) as exc:
|
||||
@@ -717,6 +244,11 @@ def _register_thinking_tools(toolkit: Toolkit) -> None:
|
||||
raise
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Full toolkit factories
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def create_full_toolkit(base_dir: str | Path | None = None):
|
||||
"""Create a full toolkit with all available tools (for the orchestrator).
|
||||
|
||||
@@ -727,6 +259,7 @@ def create_full_toolkit(base_dir: str | Path | None = None):
|
||||
# Return None when tools aren't available (tests)
|
||||
return None
|
||||
|
||||
from config import settings
|
||||
from timmy.tool_safety import DANGEROUS_TOOLS
|
||||
|
||||
toolkit = Toolkit(name="full")
|
||||
@@ -739,6 +272,7 @@ def create_full_toolkit(base_dir: str | Path | None = None):
|
||||
|
||||
_register_core_tools(toolkit, base_path)
|
||||
_register_web_fetch_tool(toolkit)
|
||||
_register_search_tools(toolkit)
|
||||
_register_grok_tool(toolkit)
|
||||
_register_memory_tools(toolkit)
|
||||
_register_agentic_loop_tool(toolkit)
|
||||
@@ -808,19 +342,9 @@ def create_experiment_tools(base_dir: str | Path | None = None):
|
||||
return toolkit
|
||||
|
||||
|
||||
# Mapping of agent IDs to their toolkits
|
||||
AGENT_TOOLKITS: dict[str, Callable[[], Toolkit]] = {
|
||||
"echo": create_research_tools,
|
||||
"mace": create_security_tools,
|
||||
"helm": create_devops_tools,
|
||||
"seer": create_data_tools,
|
||||
"forge": create_code_tools,
|
||||
"quill": create_writing_tools,
|
||||
"lab": create_experiment_tools,
|
||||
"pixel": lambda base_dir=None: _create_stub_toolkit("pixel"),
|
||||
"lyra": lambda base_dir=None: _create_stub_toolkit("lyra"),
|
||||
"reel": lambda base_dir=None: _create_stub_toolkit("reel"),
|
||||
}
|
||||
# ---------------------------------------------------------------------------
|
||||
# Agent toolkit registry
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _create_stub_toolkit(name: str):
|
||||
@@ -836,6 +360,21 @@ def _create_stub_toolkit(name: str):
|
||||
return toolkit
|
||||
|
||||
|
||||
# Mapping of agent IDs to their toolkits
|
||||
AGENT_TOOLKITS: dict[str, Callable[[], Toolkit]] = {
|
||||
"echo": create_research_tools,
|
||||
"mace": create_security_tools,
|
||||
"helm": create_devops_tools,
|
||||
"seer": create_data_tools,
|
||||
"forge": create_code_tools,
|
||||
"quill": create_writing_tools,
|
||||
"lab": create_experiment_tools,
|
||||
"pixel": lambda base_dir=None: _create_stub_toolkit("pixel"),
|
||||
"lyra": lambda base_dir=None: _create_stub_toolkit("lyra"),
|
||||
"reel": lambda base_dir=None: _create_stub_toolkit("reel"),
|
||||
}
|
||||
|
||||
|
||||
def get_tools_for_agent(agent_id: str, base_dir: str | Path | None = None) -> Toolkit | None:
|
||||
"""Get the appropriate toolkit for an agent.
|
||||
|
||||
@@ -852,11 +391,16 @@ def get_tools_for_agent(agent_id: str, base_dir: str | Path | None = None) -> To
|
||||
return None
|
||||
|
||||
|
||||
# Backward-compat alias
|
||||
# Backward-compat aliases
|
||||
get_tools_for_persona = get_tools_for_agent
|
||||
PERSONA_TOOLKITS = AGENT_TOOLKITS
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tool catalog
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _core_tool_catalog() -> dict:
|
||||
"""Return core file and execution tools catalog entries."""
|
||||
return {
|
||||
@@ -901,6 +445,16 @@ def _analysis_tool_catalog() -> dict:
|
||||
"description": "Fetch a web page and extract clean readable text (trafilatura)",
|
||||
"available_in": ["orchestrator"],
|
||||
},
|
||||
"web_search": {
|
||||
"name": "Web Search",
|
||||
"description": "Search the web via self-hosted SearXNG (no API key required)",
|
||||
"available_in": ["echo", "orchestrator"],
|
||||
},
|
||||
"scrape_url": {
|
||||
"name": "Scrape URL",
|
||||
"description": "Scrape a URL with Crawl4AI and return clean markdown content",
|
||||
"available_in": ["echo", "orchestrator"],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
127
src/timmy/tools/file_tools.py
Normal file
127
src/timmy/tools/file_tools.py
Normal file
@@ -0,0 +1,127 @@
|
||||
"""File operation tools and agent toolkit factories for file-heavy agents.
|
||||
|
||||
Provides:
|
||||
- Smart read_file wrapper (auto-lists directories)
|
||||
- Toolkit factories for Echo (research), Quill (writing), Seer (data)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
|
||||
from timmy.tools._base import (
|
||||
_AGNO_TOOLS_AVAILABLE,
|
||||
FileTools,
|
||||
PythonTools,
|
||||
Toolkit,
|
||||
_ImportError,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _make_smart_read_file(file_tools: FileTools) -> Callable:
|
||||
"""Wrap FileTools.read_file so directories auto-list their contents.
|
||||
|
||||
When the user (or the LLM) passes a directory path to read_file,
|
||||
the raw Agno implementation throws an IsADirectoryError. This
|
||||
wrapper detects that case, lists the directory entries, and returns
|
||||
a helpful message so the model can pick the right file on its own.
|
||||
"""
|
||||
original_read = file_tools.read_file
|
||||
|
||||
def smart_read_file(file_name: str = "", encoding: str = "utf-8", **kwargs) -> str:
|
||||
"""Reads the contents of the file `file_name` and returns the contents if successful."""
|
||||
# LLMs often call read_file(path=...) instead of read_file(file_name=...)
|
||||
if not file_name:
|
||||
file_name = kwargs.get("path", "")
|
||||
if not file_name:
|
||||
return "Error: no file_name or path provided."
|
||||
# Resolve the path the same way FileTools does
|
||||
_safe, resolved = file_tools.check_escape(file_name)
|
||||
if _safe and resolved.is_dir():
|
||||
entries = sorted(p.name for p in resolved.iterdir() if not p.name.startswith("."))
|
||||
listing = "\n".join(f" - {e}" for e in entries) if entries else " (empty directory)"
|
||||
return (
|
||||
f"'{file_name}' is a directory, not a file. "
|
||||
f"Files inside:\n{listing}\n\n"
|
||||
"Please call read_file with one of the files listed above."
|
||||
)
|
||||
return original_read(file_name, encoding=encoding)
|
||||
|
||||
# Preserve the original docstring for Agno tool schema generation
|
||||
smart_read_file.__doc__ = original_read.__doc__
|
||||
return smart_read_file
|
||||
|
||||
|
||||
def create_research_tools(base_dir: str | Path | None = None):
|
||||
"""Create tools for the research agent (Echo).
|
||||
|
||||
Includes: file reading, web search (SearXNG), URL scraping (Crawl4AI)
|
||||
"""
|
||||
if not _AGNO_TOOLS_AVAILABLE:
|
||||
raise ImportError(f"Agno tools not available: {_ImportError}")
|
||||
toolkit = Toolkit(name="research")
|
||||
|
||||
# File reading
|
||||
from config import settings
|
||||
|
||||
base_path = Path(base_dir) if base_dir else Path(settings.repo_root)
|
||||
file_tools = FileTools(base_dir=base_path)
|
||||
toolkit.register(_make_smart_read_file(file_tools), name="read_file")
|
||||
toolkit.register(file_tools.list_files, name="list_files")
|
||||
|
||||
# Web search + scraping (gracefully no-ops when backend=none or service down)
|
||||
from timmy.tools.search import scrape_url, web_search
|
||||
|
||||
toolkit.register(web_search, name="web_search")
|
||||
toolkit.register(scrape_url, name="scrape_url")
|
||||
|
||||
return toolkit
|
||||
|
||||
|
||||
def create_writing_tools(base_dir: str | Path | None = None):
|
||||
"""Create tools for the writing agent (Quill).
|
||||
|
||||
Includes: file read/write
|
||||
"""
|
||||
if not _AGNO_TOOLS_AVAILABLE:
|
||||
raise ImportError(f"Agno tools not available: {_ImportError}")
|
||||
toolkit = Toolkit(name="writing")
|
||||
|
||||
# File operations
|
||||
from config import settings
|
||||
|
||||
base_path = Path(base_dir) if base_dir else Path(settings.repo_root)
|
||||
file_tools = FileTools(base_dir=base_path)
|
||||
toolkit.register(_make_smart_read_file(file_tools), name="read_file")
|
||||
toolkit.register(file_tools.save_file, name="write_file")
|
||||
toolkit.register(file_tools.list_files, name="list_files")
|
||||
|
||||
return toolkit
|
||||
|
||||
|
||||
def create_data_tools(base_dir: str | Path | None = None):
|
||||
"""Create tools for the data agent (Seer).
|
||||
|
||||
Includes: python execution, file reading, web search for data sources
|
||||
"""
|
||||
if not _AGNO_TOOLS_AVAILABLE:
|
||||
raise ImportError(f"Agno tools not available: {_ImportError}")
|
||||
toolkit = Toolkit(name="data")
|
||||
|
||||
# Python execution for analysis
|
||||
python_tools = PythonTools()
|
||||
toolkit.register(python_tools.run_python_code, name="python")
|
||||
|
||||
# File reading
|
||||
from config import settings
|
||||
|
||||
base_path = Path(base_dir) if base_dir else Path(settings.repo_root)
|
||||
file_tools = FileTools(base_dir=base_path)
|
||||
toolkit.register(_make_smart_read_file(file_tools), name="read_file")
|
||||
toolkit.register(file_tools.list_files, name="list_files")
|
||||
|
||||
return toolkit
|
||||
186
src/timmy/tools/search.py
Normal file
186
src/timmy/tools/search.py
Normal file
@@ -0,0 +1,186 @@
|
||||
"""Self-hosted web search and scraping tools using SearXNG + Crawl4AI.
|
||||
|
||||
Provides:
|
||||
- web_search(query) — SearXNG meta-search (no API key required)
|
||||
- scrape_url(url) — Crawl4AI full-page scrape to clean markdown
|
||||
|
||||
Both tools degrade gracefully when the backing service is unavailable
|
||||
(logs WARNING, returns descriptive error string — never crashes).
|
||||
|
||||
Services are started via `docker compose --profile search up` or configured
|
||||
with TIMMY_SEARCH_URL / TIMMY_CRAWL_URL environment variables.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
|
||||
from config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Crawl4AI polling: up to _CRAWL_MAX_POLLS × _CRAWL_POLL_INTERVAL seconds
|
||||
_CRAWL_MAX_POLLS = 6
|
||||
_CRAWL_POLL_INTERVAL = 5 # seconds
|
||||
_CRAWL_CHAR_BUDGET = 4000 * 4 # ~4000 tokens
|
||||
|
||||
|
||||
def web_search(query: str, num_results: int = 5) -> str:
|
||||
"""Search the web using the self-hosted SearXNG meta-search engine.
|
||||
|
||||
Returns ranked results (title + URL + snippet) without requiring any
|
||||
paid API key. Requires SearXNG running locally (docker compose
|
||||
--profile search up) or TIMMY_SEARCH_URL pointing to a reachable instance.
|
||||
|
||||
Args:
|
||||
query: The search query.
|
||||
num_results: Maximum number of results to return (default 5).
|
||||
|
||||
Returns:
|
||||
Formatted search results string, or an error/status message on failure.
|
||||
"""
|
||||
if settings.timmy_search_backend == "none":
|
||||
return "Web search is disabled (TIMMY_SEARCH_BACKEND=none)."
|
||||
|
||||
try:
|
||||
import requests as _requests
|
||||
except ImportError:
|
||||
return "Error: 'requests' package is not installed."
|
||||
|
||||
base_url = settings.search_url.rstrip("/")
|
||||
params: dict = {
|
||||
"q": query,
|
||||
"format": "json",
|
||||
"categories": "general",
|
||||
}
|
||||
|
||||
try:
|
||||
resp = _requests.get(
|
||||
f"{base_url}/search",
|
||||
params=params,
|
||||
timeout=10,
|
||||
headers={"User-Agent": "TimmyResearchBot/1.0"},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
except Exception as exc:
|
||||
logger.warning("SearXNG unavailable at %s: %s", base_url, exc)
|
||||
return f"Search unavailable — SearXNG not reachable ({base_url}): {exc}"
|
||||
|
||||
try:
|
||||
data = resp.json()
|
||||
except Exception as exc:
|
||||
logger.warning("SearXNG response parse error: %s", exc)
|
||||
return "Search error: could not parse SearXNG response."
|
||||
|
||||
results = data.get("results", [])[:num_results]
|
||||
if not results:
|
||||
return f"No results found for: {query!r}"
|
||||
|
||||
lines = [f"Web search results for: {query!r}\n"]
|
||||
for i, r in enumerate(results, 1):
|
||||
title = r.get("title", "Untitled")
|
||||
url = r.get("url", "")
|
||||
snippet = r.get("content", "").strip()
|
||||
lines.append(f"{i}. {title}\n URL: {url}\n {snippet}\n")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def scrape_url(url: str) -> str:
|
||||
"""Scrape a URL with Crawl4AI and return the main content as clean markdown.
|
||||
|
||||
Crawl4AI extracts well-structured markdown from any public page —
|
||||
articles, docs, product pages — suitable for LLM consumption.
|
||||
Requires Crawl4AI running locally (docker compose --profile search up)
|
||||
or TIMMY_CRAWL_URL pointing to a reachable instance.
|
||||
|
||||
Args:
|
||||
url: The URL to scrape (must start with http:// or https://).
|
||||
|
||||
Returns:
|
||||
Extracted markdown text (up to ~4000 tokens), or an error message.
|
||||
"""
|
||||
if not url or not url.startswith(("http://", "https://")):
|
||||
return f"Error: invalid URL — must start with http:// or https://: {url!r}"
|
||||
|
||||
if settings.timmy_search_backend == "none":
|
||||
return "Web scraping is disabled (TIMMY_SEARCH_BACKEND=none)."
|
||||
|
||||
try:
|
||||
import requests as _requests
|
||||
except ImportError:
|
||||
return "Error: 'requests' package is not installed."
|
||||
|
||||
base = settings.crawl_url.rstrip("/")
|
||||
|
||||
# Submit crawl task
|
||||
try:
|
||||
resp = _requests.post(
|
||||
f"{base}/crawl",
|
||||
json={"urls": [url], "priority": 10},
|
||||
timeout=15,
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
except Exception as exc:
|
||||
logger.warning("Crawl4AI unavailable at %s: %s", base, exc)
|
||||
return f"Scrape unavailable — Crawl4AI not reachable ({base}): {exc}"
|
||||
|
||||
try:
|
||||
submit_data = resp.json()
|
||||
except Exception as exc:
|
||||
logger.warning("Crawl4AI submit parse error: %s", exc)
|
||||
return "Scrape error: could not parse Crawl4AI response."
|
||||
|
||||
# Check if result came back synchronously
|
||||
if "results" in submit_data:
|
||||
return _extract_crawl_content(submit_data["results"], url)
|
||||
|
||||
task_id = submit_data.get("task_id")
|
||||
if not task_id:
|
||||
return f"Scrape error: Crawl4AI returned no task_id for {url}"
|
||||
|
||||
# Poll for async result
|
||||
for _ in range(_CRAWL_MAX_POLLS):
|
||||
time.sleep(_CRAWL_POLL_INTERVAL)
|
||||
try:
|
||||
poll = _requests.get(f"{base}/task/{task_id}", timeout=10)
|
||||
poll.raise_for_status()
|
||||
task_data = poll.json()
|
||||
except Exception as exc:
|
||||
logger.warning("Crawl4AI poll error (task=%s): %s", task_id, exc)
|
||||
continue
|
||||
|
||||
status = task_data.get("status", "")
|
||||
if status == "completed":
|
||||
results = task_data.get("results") or task_data.get("result")
|
||||
if isinstance(results, dict):
|
||||
results = [results]
|
||||
return _extract_crawl_content(results or [], url)
|
||||
if status == "failed":
|
||||
return f"Scrape failed for {url}: {task_data.get('error', 'unknown error')}"
|
||||
|
||||
return f"Scrape timed out after {_CRAWL_MAX_POLLS * _CRAWL_POLL_INTERVAL}s for {url}"
|
||||
|
||||
|
||||
def _extract_crawl_content(results: list, url: str) -> str:
|
||||
"""Extract and truncate markdown content from Crawl4AI results list."""
|
||||
if not results:
|
||||
return f"No content returned by Crawl4AI for: {url}"
|
||||
|
||||
result = results[0]
|
||||
content = (
|
||||
result.get("markdown")
|
||||
or result.get("markdown_v2", {}).get("raw_markdown")
|
||||
or result.get("extracted_content")
|
||||
or result.get("content")
|
||||
or ""
|
||||
)
|
||||
if not content:
|
||||
return f"No readable content extracted from: {url}"
|
||||
|
||||
if len(content) > _CRAWL_CHAR_BUDGET:
|
||||
content = content[:_CRAWL_CHAR_BUDGET] + "\n\n[…truncated to ~4000 tokens]"
|
||||
|
||||
return content
|
||||
357
src/timmy/tools/system_tools.py
Normal file
357
src/timmy/tools/system_tools.py
Normal file
@@ -0,0 +1,357 @@
|
||||
"""System, calculation, and AI consultation tools for Timmy agents.
|
||||
|
||||
Provides:
|
||||
- Safe AST-based calculator
|
||||
- consult_grok (xAI frontier reasoning)
|
||||
- web_fetch (content extraction)
|
||||
- Toolkit factories for Forge (code), Mace (security), Helm (devops)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
import logging
|
||||
import math
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
from timmy.tools._base import (
|
||||
_AGNO_TOOLS_AVAILABLE,
|
||||
FileTools,
|
||||
PythonTools,
|
||||
ShellTools,
|
||||
Toolkit,
|
||||
_ImportError,
|
||||
)
|
||||
from timmy.tools.file_tools import _make_smart_read_file
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Max characters of user query included in Lightning invoice memo
|
||||
_INVOICE_MEMO_MAX_LEN = 50
|
||||
|
||||
|
||||
def _safe_eval(node, allowed_names: dict):
|
||||
"""Walk an AST and evaluate only safe numeric operations."""
|
||||
if isinstance(node, ast.Expression):
|
||||
return _safe_eval(node.body, allowed_names)
|
||||
if isinstance(node, ast.Constant):
|
||||
if isinstance(node.value, (int, float, complex)):
|
||||
return node.value
|
||||
raise ValueError(f"Unsupported constant: {node.value!r}")
|
||||
if isinstance(node, ast.UnaryOp):
|
||||
operand = _safe_eval(node.operand, allowed_names)
|
||||
if isinstance(node.op, ast.UAdd):
|
||||
return +operand
|
||||
if isinstance(node.op, ast.USub):
|
||||
return -operand
|
||||
raise ValueError(f"Unsupported unary op: {type(node.op).__name__}")
|
||||
if isinstance(node, ast.BinOp):
|
||||
left = _safe_eval(node.left, allowed_names)
|
||||
right = _safe_eval(node.right, allowed_names)
|
||||
ops = {
|
||||
ast.Add: lambda a, b: a + b,
|
||||
ast.Sub: lambda a, b: a - b,
|
||||
ast.Mult: lambda a, b: a * b,
|
||||
ast.Div: lambda a, b: a / b,
|
||||
ast.FloorDiv: lambda a, b: a // b,
|
||||
ast.Mod: lambda a, b: a % b,
|
||||
ast.Pow: lambda a, b: a**b,
|
||||
}
|
||||
op_fn = ops.get(type(node.op))
|
||||
if op_fn is None:
|
||||
raise ValueError(f"Unsupported binary op: {type(node.op).__name__}")
|
||||
return op_fn(left, right)
|
||||
if isinstance(node, ast.Name):
|
||||
if node.id in allowed_names:
|
||||
return allowed_names[node.id]
|
||||
raise ValueError(f"Unknown name: {node.id!r}")
|
||||
if isinstance(node, ast.Attribute):
|
||||
value = _safe_eval(node.value, allowed_names)
|
||||
# Only allow attribute access on the math module
|
||||
if value is math:
|
||||
attr = getattr(math, node.attr, None)
|
||||
if attr is not None:
|
||||
return attr
|
||||
raise ValueError(f"Attribute access not allowed: .{node.attr}")
|
||||
if isinstance(node, ast.Call):
|
||||
func = _safe_eval(node.func, allowed_names)
|
||||
if not callable(func):
|
||||
raise ValueError(f"Not callable: {func!r}")
|
||||
args = [_safe_eval(a, allowed_names) for a in node.args]
|
||||
kwargs = {kw.arg: _safe_eval(kw.value, allowed_names) for kw in node.keywords}
|
||||
return func(*args, **kwargs)
|
||||
raise ValueError(f"Unsupported syntax: {type(node).__name__}")
|
||||
|
||||
|
||||
def calculator(expression: str) -> str:
|
||||
"""Evaluate a mathematical expression and return the exact result.
|
||||
|
||||
Use this tool for ANY arithmetic: multiplication, division, square roots,
|
||||
exponents, percentages, logarithms, trigonometry, etc.
|
||||
|
||||
Args:
|
||||
expression: A valid Python math expression, e.g. '347 * 829',
|
||||
'math.sqrt(17161)', '2**10', 'math.log(100, 10)'.
|
||||
|
||||
Returns:
|
||||
The exact result as a string.
|
||||
"""
|
||||
allowed_names = {k: getattr(math, k) for k in dir(math) if not k.startswith("_")}
|
||||
allowed_names["math"] = math
|
||||
allowed_names["abs"] = abs
|
||||
allowed_names["round"] = round
|
||||
allowed_names["min"] = min
|
||||
allowed_names["max"] = max
|
||||
try:
|
||||
tree = ast.parse(expression, mode="eval")
|
||||
result = _safe_eval(tree, allowed_names)
|
||||
return str(result)
|
||||
except Exception as e: # broad catch intentional: arbitrary code execution
|
||||
return f"Error evaluating '{expression}': {e}"
|
||||
|
||||
|
||||
def consult_grok(query: str) -> str:
|
||||
"""Consult Grok (xAI) for frontier reasoning on complex questions.
|
||||
|
||||
Use this tool when a question requires advanced reasoning, real-time
|
||||
knowledge, or capabilities beyond the local model. Grok is a premium
|
||||
cloud backend — use sparingly and only for high-complexity queries.
|
||||
|
||||
Args:
|
||||
query: The question or reasoning task to send to Grok.
|
||||
|
||||
Returns:
|
||||
Grok's response text, or an error/status message.
|
||||
"""
|
||||
from config import settings
|
||||
from timmy.backends import get_grok_backend, grok_available
|
||||
|
||||
if not grok_available():
|
||||
return (
|
||||
"Grok is not available. Enable with GROK_ENABLED=true "
|
||||
"and set XAI_API_KEY in your .env file."
|
||||
)
|
||||
|
||||
backend = get_grok_backend()
|
||||
|
||||
# Log to Spark if available
|
||||
try:
|
||||
from spark.engine import spark_engine
|
||||
|
||||
spark_engine.on_tool_executed(
|
||||
agent_id="default",
|
||||
tool_name="consult_grok",
|
||||
success=True,
|
||||
)
|
||||
except (ImportError, AttributeError) as exc:
|
||||
logger.warning("Tool execution failed (consult_grok logging): %s", exc)
|
||||
|
||||
# Generate Lightning invoice for monetization (unless free mode)
|
||||
invoice_info = ""
|
||||
if not settings.grok_free:
|
||||
try:
|
||||
from lightning.factory import get_backend as get_ln_backend
|
||||
|
||||
ln = get_ln_backend()
|
||||
sats = min(settings.grok_max_sats_per_query, settings.grok_sats_hard_cap)
|
||||
inv = ln.create_invoice(sats, f"Grok query: {query[:_INVOICE_MEMO_MAX_LEN]}")
|
||||
invoice_info = f"\n[Lightning invoice: {sats} sats — {inv.payment_request[:40]}...]"
|
||||
except (ImportError, OSError, ValueError) as exc:
|
||||
logger.error("Lightning invoice creation failed: %s", exc)
|
||||
return "Error: Failed to create Lightning invoice. Please check logs."
|
||||
|
||||
result = backend.run(query)
|
||||
|
||||
response = result.content
|
||||
if invoice_info:
|
||||
response += invoice_info
|
||||
|
||||
return response
|
||||
|
||||
|
||||
def web_fetch(url: str, max_tokens: int = 4000) -> str:
|
||||
"""Fetch a web page and return its main text content.
|
||||
|
||||
Downloads the URL, extracts readable text using trafilatura, and
|
||||
truncates to a token budget. Use this to read full articles, docs,
|
||||
or blog posts that web_search only returns snippets for.
|
||||
|
||||
Args:
|
||||
url: The URL to fetch (must start with http:// or https://).
|
||||
max_tokens: Maximum approximate token budget (default 4000).
|
||||
Text is truncated to max_tokens * 4 characters.
|
||||
|
||||
Returns:
|
||||
Extracted text content, or an error message on failure.
|
||||
"""
|
||||
if not url or not url.startswith(("http://", "https://")):
|
||||
return f"Error: invalid URL — must start with http:// or https://: {url!r}"
|
||||
|
||||
try:
|
||||
import requests as _requests
|
||||
except ImportError:
|
||||
return "Error: 'requests' package is not installed. Install with: pip install requests"
|
||||
|
||||
try:
|
||||
import trafilatura
|
||||
except ImportError:
|
||||
return (
|
||||
"Error: 'trafilatura' package is not installed. Install with: pip install trafilatura"
|
||||
)
|
||||
|
||||
try:
|
||||
resp = _requests.get(
|
||||
url,
|
||||
timeout=15,
|
||||
headers={"User-Agent": "TimmyResearchBot/1.0"},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
except _requests.exceptions.Timeout:
|
||||
return f"Error: request timed out after 15 seconds for {url}"
|
||||
except _requests.exceptions.HTTPError as exc:
|
||||
return f"Error: HTTP {exc.response.status_code} for {url}"
|
||||
except _requests.exceptions.RequestException as exc:
|
||||
return f"Error: failed to fetch {url} — {exc}"
|
||||
|
||||
text = trafilatura.extract(resp.text, include_tables=True, include_links=True)
|
||||
if not text:
|
||||
return f"Error: could not extract readable content from {url}"
|
||||
|
||||
char_budget = max_tokens * 4
|
||||
if len(text) > char_budget:
|
||||
text = text[:char_budget] + f"\n\n[…truncated to ~{max_tokens} tokens]"
|
||||
|
||||
return text
|
||||
|
||||
|
||||
def create_aider_tool(base_path: Path):
|
||||
"""Create an Aider tool for AI-assisted coding."""
|
||||
|
||||
class AiderTool:
|
||||
"""Tool that calls Aider (local AI coding assistant) for code generation."""
|
||||
|
||||
def __init__(self, base_dir: Path):
|
||||
self.base_dir = base_dir
|
||||
|
||||
def run_aider(self, prompt: str, model: str = "qwen3:30b") -> str:
|
||||
"""Run Aider to generate code changes.
|
||||
|
||||
Args:
|
||||
prompt: What you want Aider to do (e.g., "add a fibonacci function")
|
||||
model: Ollama model to use (default: qwen3:30b)
|
||||
|
||||
Returns:
|
||||
Aider's response with the code changes made
|
||||
"""
|
||||
try:
|
||||
# Run aider with the prompt
|
||||
result = subprocess.run(
|
||||
[
|
||||
"aider",
|
||||
"--no-git",
|
||||
"--model",
|
||||
f"ollama/{model}",
|
||||
"--quiet",
|
||||
prompt,
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=120,
|
||||
cwd=str(self.base_dir),
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
return result.stdout if result.stdout else "Code changes applied successfully"
|
||||
else:
|
||||
return f"Aider error: {result.stderr}"
|
||||
except FileNotFoundError:
|
||||
return "Error: Aider not installed. Run: pip install aider"
|
||||
except subprocess.TimeoutExpired:
|
||||
return "Error: Aider timed out after 120 seconds"
|
||||
except (OSError, subprocess.SubprocessError) as e:
|
||||
return f"Error running Aider: {str(e)}"
|
||||
|
||||
return AiderTool(base_path)
|
||||
|
||||
|
||||
def create_code_tools(base_dir: str | Path | None = None):
|
||||
"""Create tools for the code agent (Forge).
|
||||
|
||||
Includes: shell commands, python execution, file read/write, Aider AI assist
|
||||
"""
|
||||
if not _AGNO_TOOLS_AVAILABLE:
|
||||
raise ImportError(f"Agno tools not available: {_ImportError}")
|
||||
toolkit = Toolkit(name="code")
|
||||
|
||||
# Shell commands (sandboxed)
|
||||
shell_tools = ShellTools()
|
||||
toolkit.register(shell_tools.run_shell_command, name="shell")
|
||||
|
||||
# Python execution
|
||||
python_tools = PythonTools()
|
||||
toolkit.register(python_tools.run_python_code, name="python")
|
||||
|
||||
# File operations
|
||||
from config import settings
|
||||
|
||||
base_path = Path(base_dir) if base_dir else Path(settings.repo_root)
|
||||
file_tools = FileTools(base_dir=base_path)
|
||||
toolkit.register(_make_smart_read_file(file_tools), name="read_file")
|
||||
toolkit.register(file_tools.save_file, name="write_file")
|
||||
toolkit.register(file_tools.list_files, name="list_files")
|
||||
|
||||
# Aider AI coding assistant (local with Ollama)
|
||||
aider_tool = create_aider_tool(base_path)
|
||||
toolkit.register(aider_tool.run_aider, name="aider")
|
||||
|
||||
return toolkit
|
||||
|
||||
|
||||
def create_security_tools(base_dir: str | Path | None = None):
|
||||
"""Create tools for the security agent (Mace).
|
||||
|
||||
Includes: shell commands (for scanning), file read
|
||||
"""
|
||||
if not _AGNO_TOOLS_AVAILABLE:
|
||||
raise ImportError(f"Agno tools not available: {_ImportError}")
|
||||
toolkit = Toolkit(name="security")
|
||||
|
||||
# Shell for running security scans
|
||||
shell_tools = ShellTools()
|
||||
toolkit.register(shell_tools.run_shell_command, name="shell")
|
||||
|
||||
# File reading for logs/configs
|
||||
from config import settings
|
||||
|
||||
base_path = Path(base_dir) if base_dir else Path(settings.repo_root)
|
||||
file_tools = FileTools(base_dir=base_path)
|
||||
toolkit.register(_make_smart_read_file(file_tools), name="read_file")
|
||||
toolkit.register(file_tools.list_files, name="list_files")
|
||||
|
||||
return toolkit
|
||||
|
||||
|
||||
def create_devops_tools(base_dir: str | Path | None = None):
|
||||
"""Create tools for the DevOps agent (Helm).
|
||||
|
||||
Includes: shell commands, file read/write
|
||||
"""
|
||||
if not _AGNO_TOOLS_AVAILABLE:
|
||||
raise ImportError(f"Agno tools not available: {_ImportError}")
|
||||
toolkit = Toolkit(name="devops")
|
||||
|
||||
# Shell for deployment commands
|
||||
shell_tools = ShellTools()
|
||||
toolkit.register(shell_tools.run_shell_command, name="shell")
|
||||
|
||||
# File operations for config management
|
||||
from config import settings
|
||||
|
||||
base_path = Path(base_dir) if base_dir else Path(settings.repo_root)
|
||||
file_tools = FileTools(base_dir=base_path)
|
||||
toolkit.register(_make_smart_read_file(file_tools), name="read_file")
|
||||
toolkit.register(file_tools.save_file, name="write_file")
|
||||
toolkit.register(file_tools.list_files, name="list_files")
|
||||
|
||||
return toolkit
|
||||
@@ -41,17 +41,38 @@ def delegate_task(
|
||||
if priority not in valid_priorities:
|
||||
priority = "normal"
|
||||
|
||||
agent_role = available[agent_name]
|
||||
|
||||
# Wire to DistributedWorker for actual execution
|
||||
task_id: str | None = None
|
||||
status = "queued"
|
||||
try:
|
||||
from brain.worker import DistributedWorker
|
||||
|
||||
task_id = DistributedWorker.submit(agent_name, agent_role, task_description, priority)
|
||||
except Exception as exc:
|
||||
logger.warning("DistributedWorker unavailable — task noted only: %s", exc)
|
||||
status = "noted"
|
||||
|
||||
logger.info(
|
||||
"Delegation intent: %s → %s (priority=%s)", agent_name, task_description[:80], priority
|
||||
"Delegated task %s: %s → %s (priority=%s, status=%s)",
|
||||
task_id or "?",
|
||||
agent_name,
|
||||
task_description[:80],
|
||||
priority,
|
||||
status,
|
||||
)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"task_id": None,
|
||||
"task_id": task_id,
|
||||
"agent": agent_name,
|
||||
"role": available[agent_name],
|
||||
"status": "noted",
|
||||
"message": f"Delegation to {agent_name} ({available[agent_name]}): {task_description[:100]}",
|
||||
"role": agent_role,
|
||||
"status": status,
|
||||
"message": (
|
||||
f"Task {task_id or 'noted'}: delegated to {agent_name} ({agent_role}): "
|
||||
f"{task_description[:100]}"
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -37,6 +37,7 @@ class VoiceTTS:
|
||||
|
||||
@property
|
||||
def available(self) -> bool:
|
||||
"""Whether the TTS engine initialized successfully and can produce audio."""
|
||||
return self._available
|
||||
|
||||
def speak(self, text: str) -> None:
|
||||
@@ -68,11 +69,13 @@ class VoiceTTS:
|
||||
logger.error("VoiceTTS: speech failed — %s", exc)
|
||||
|
||||
def set_rate(self, rate: int) -> None:
|
||||
"""Set speech rate in words per minute (typical range: 100–300, default 175)."""
|
||||
self._rate = rate
|
||||
if self._engine:
|
||||
self._engine.setProperty("rate", rate)
|
||||
|
||||
def set_volume(self, volume: float) -> None:
|
||||
"""Set speech volume. Value is clamped to the 0.0–1.0 range."""
|
||||
self._volume = max(0.0, min(1.0, volume))
|
||||
if self._engine:
|
||||
self._engine.setProperty("volume", self._volume)
|
||||
@@ -92,6 +95,7 @@ class VoiceTTS:
|
||||
return []
|
||||
|
||||
def set_voice(self, voice_id: str) -> None:
|
||||
"""Set the active TTS voice by system voice ID (see ``get_voices()``)."""
|
||||
if self._engine:
|
||||
self._engine.setProperty("voice", voice_id)
|
||||
|
||||
|
||||
@@ -2664,3 +2664,124 @@
|
||||
color: var(--bg-deep);
|
||||
}
|
||||
.vs-btn-save:hover { opacity: 0.85; }
|
||||
|
||||
/* ── Nexus ────────────────────────────────────────────────── */
|
||||
.nexus-layout { max-width: 1400px; margin: 0 auto; }
|
||||
|
||||
.nexus-header { border-bottom: 1px solid var(--border); padding-bottom: 0.5rem; }
|
||||
.nexus-title { font-size: 1.4rem; font-weight: 700; color: var(--purple); letter-spacing: 0.1em; }
|
||||
.nexus-subtitle { font-size: 0.8rem; color: var(--text-dim); margin-top: 0.2rem; }
|
||||
|
||||
.nexus-grid {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr 320px;
|
||||
gap: 1rem;
|
||||
align-items: start;
|
||||
}
|
||||
@media (max-width: 900px) {
|
||||
.nexus-grid { grid-template-columns: 1fr; }
|
||||
}
|
||||
|
||||
.nexus-chat-panel { height: calc(100vh - 180px); display: flex; flex-direction: column; }
|
||||
.nexus-chat-panel .card-body { overflow-y: auto; flex: 1; }
|
||||
|
||||
.nexus-empty-state {
|
||||
color: var(--text-dim);
|
||||
font-size: 0.85rem;
|
||||
font-style: italic;
|
||||
padding: 1rem 0;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
/* Memory sidebar */
|
||||
.nexus-memory-hits { font-size: 0.78rem; }
|
||||
.nexus-memory-label { color: var(--text-dim); font-size: 0.72rem; margin-bottom: 0.4rem; letter-spacing: 0.05em; }
|
||||
.nexus-memory-hit { display: flex; gap: 0.4rem; margin-bottom: 0.35rem; align-items: flex-start; }
|
||||
.nexus-memory-type { color: var(--purple); font-size: 0.68rem; white-space: nowrap; padding-top: 0.1rem; min-width: 60px; }
|
||||
.nexus-memory-content { color: var(--text); line-height: 1.4; }
|
||||
|
||||
/* Teaching panel */
|
||||
.nexus-facts-header { font-size: 0.7rem; color: var(--text-dim); letter-spacing: 0.08em; margin-bottom: 0.4rem; }
|
||||
.nexus-facts-list { list-style: none; padding: 0; margin: 0; font-size: 0.8rem; }
|
||||
.nexus-fact-item { color: var(--text); border-bottom: 1px solid var(--border); padding: 0.3rem 0; }
|
||||
.nexus-fact-empty { color: var(--text-dim); font-style: italic; }
|
||||
.nexus-taught-confirm {
|
||||
font-size: 0.8rem;
|
||||
color: var(--green);
|
||||
background: rgba(0,255,136,0.06);
|
||||
border: 1px solid var(--green);
|
||||
border-radius: 4px;
|
||||
padding: 0.3rem 0.6rem;
|
||||
margin-bottom: 0.5rem;
|
||||
}
|
||||
|
||||
/* ── Self-Correction Dashboard ─────────────────────────────── */
|
||||
.sc-event {
|
||||
border-left: 3px solid var(--border);
|
||||
padding: 0.6rem 0.8rem;
|
||||
margin-bottom: 0.75rem;
|
||||
background: rgba(255,255,255,0.02);
|
||||
border-radius: 0 4px 4px 0;
|
||||
font-size: 0.82rem;
|
||||
}
|
||||
.sc-event.sc-status-success { border-left-color: var(--green); }
|
||||
.sc-event.sc-status-partial { border-left-color: var(--amber); }
|
||||
.sc-event.sc-status-failed { border-left-color: var(--red); }
|
||||
|
||||
.sc-event-header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 0.5rem;
|
||||
margin-bottom: 0.4rem;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
.sc-status-badge {
|
||||
font-size: 0.68rem;
|
||||
font-weight: 700;
|
||||
letter-spacing: 0.06em;
|
||||
padding: 0.15rem 0.45rem;
|
||||
border-radius: 3px;
|
||||
}
|
||||
.sc-status-badge.sc-status-success { color: var(--green); background: rgba(0,255,136,0.08); }
|
||||
.sc-status-badge.sc-status-partial { color: var(--amber); background: rgba(255,179,0,0.08); }
|
||||
.sc-status-badge.sc-status-failed { color: var(--red); background: rgba(255,59,59,0.08); }
|
||||
|
||||
.sc-source-badge {
|
||||
font-size: 0.68rem;
|
||||
color: var(--purple);
|
||||
background: rgba(168,85,247,0.1);
|
||||
padding: 0.1rem 0.4rem;
|
||||
border-radius: 3px;
|
||||
}
|
||||
.sc-event-time { font-size: 0.68rem; color: var(--text-dim); margin-left: auto; }
|
||||
.sc-event-error-type {
|
||||
font-size: 0.72rem;
|
||||
color: var(--amber);
|
||||
font-weight: 600;
|
||||
margin-bottom: 0.3rem;
|
||||
letter-spacing: 0.04em;
|
||||
}
|
||||
.sc-label {
|
||||
font-size: 0.65rem;
|
||||
font-weight: 700;
|
||||
letter-spacing: 0.06em;
|
||||
color: var(--text-dim);
|
||||
margin-right: 0.3rem;
|
||||
}
|
||||
.sc-event-intent, .sc-event-error, .sc-event-strategy, .sc-event-outcome {
|
||||
color: var(--text);
|
||||
margin-bottom: 0.2rem;
|
||||
line-height: 1.4;
|
||||
word-break: break-word;
|
||||
}
|
||||
.sc-event-error { color: var(--red); }
|
||||
.sc-event-strategy { color: var(--text-dim); font-style: italic; }
|
||||
.sc-event-outcome { color: var(--text-bright); }
|
||||
.sc-event-meta { font-size: 0.68rem; color: var(--text-dim); margin-top: 0.3rem; }
|
||||
|
||||
.sc-pattern-type {
|
||||
font-family: var(--font);
|
||||
font-size: 0.8rem;
|
||||
color: var(--text-bright);
|
||||
word-break: break-all;
|
||||
}
|
||||
|
||||
@@ -86,6 +86,19 @@
|
||||
<p>Your task has been added to the queue. Timmy will review it shortly.</p>
|
||||
<button type="button" id="submit-another-btn" class="btn-primary">Submit Another</button>
|
||||
</div>
|
||||
|
||||
<div id="submit-job-queued" class="submit-job-queued hidden">
|
||||
<div class="queued-icon">
|
||||
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
|
||||
<circle cx="12" cy="12" r="10"></circle>
|
||||
<polyline points="12 6 12 12 16 14"></polyline>
|
||||
</svg>
|
||||
</div>
|
||||
<h3>Job Queued</h3>
|
||||
<p>The server is unreachable right now. Your job has been saved locally and will be submitted automatically when the connection is restored.</p>
|
||||
<div id="queue-count-display" class="queue-count-display"></div>
|
||||
<button type="button" id="submit-another-queued-btn" class="btn-primary">Submit Another</button>
|
||||
</div>
|
||||
</div>
|
||||
<div id="submit-job-backdrop" class="submit-job-backdrop"></div>
|
||||
</div>
|
||||
@@ -142,6 +155,7 @@
|
||||
import { createFamiliar } from "./familiar.js";
|
||||
import { setupControls } from "./controls.js";
|
||||
import { StateReader } from "./state.js";
|
||||
import { messageQueue } from "./queue.js";
|
||||
|
||||
// --- Renderer ---
|
||||
const renderer = new THREE.WebGLRenderer({ antialias: true });
|
||||
@@ -182,8 +196,60 @@
|
||||
moodEl.textContent = state.timmyState.mood;
|
||||
}
|
||||
});
|
||||
|
||||
// Replay queued jobs whenever the server comes back online.
|
||||
stateReader.onConnectionChange(async (online) => {
|
||||
if (!online) return;
|
||||
const pending = messageQueue.getPending();
|
||||
if (pending.length === 0) return;
|
||||
console.log(`[queue] Online — replaying ${pending.length} queued job(s)`);
|
||||
for (const item of pending) {
|
||||
try {
|
||||
const response = await fetch("/api/tasks", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify(item.payload),
|
||||
});
|
||||
if (response.ok) {
|
||||
messageQueue.markDelivered(item.id);
|
||||
console.log(`[queue] Delivered queued job ${item.id}`);
|
||||
} else {
|
||||
messageQueue.markFailed(item.id);
|
||||
console.warn(`[queue] Failed to deliver job ${item.id}: ${response.status}`);
|
||||
}
|
||||
} catch (err) {
|
||||
// Still offline — leave as QUEUED, will retry next cycle.
|
||||
console.warn(`[queue] Replay aborted (still offline): ${err}`);
|
||||
break;
|
||||
}
|
||||
}
|
||||
messageQueue.prune();
|
||||
_updateQueueBadge();
|
||||
});
|
||||
|
||||
stateReader.connect();
|
||||
|
||||
// --- Queue badge (top-right indicator for pending jobs) ---
|
||||
function _updateQueueBadge() {
|
||||
const count = messageQueue.pendingCount();
|
||||
let badge = document.getElementById("queue-badge");
|
||||
if (count === 0) {
|
||||
if (badge) badge.remove();
|
||||
return;
|
||||
}
|
||||
if (!badge) {
|
||||
badge = document.createElement("div");
|
||||
badge.id = "queue-badge";
|
||||
badge.className = "queue-badge";
|
||||
badge.title = "Jobs queued offline — will submit on reconnect";
|
||||
document.getElementById("overlay").appendChild(badge);
|
||||
}
|
||||
badge.textContent = `${count} queued`;
|
||||
}
|
||||
// Show badge on load if there are already queued messages.
|
||||
messageQueue.prune();
|
||||
_updateQueueBadge();
|
||||
|
||||
// --- About Panel ---
|
||||
const infoBtn = document.getElementById("info-btn");
|
||||
const aboutPanel = document.getElementById("about-panel");
|
||||
@@ -228,6 +294,9 @@
|
||||
const descWarning = document.getElementById("desc-warning");
|
||||
const submitJobSuccess = document.getElementById("submit-job-success");
|
||||
const submitAnotherBtn = document.getElementById("submit-another-btn");
|
||||
const submitJobQueued = document.getElementById("submit-job-queued");
|
||||
const submitAnotherQueuedBtn = document.getElementById("submit-another-queued-btn");
|
||||
const queueCountDisplay = document.getElementById("queue-count-display");
|
||||
|
||||
// Constants
|
||||
const MAX_TITLE_LENGTH = 200;
|
||||
@@ -255,6 +324,7 @@
|
||||
submitJobForm.reset();
|
||||
submitJobForm.classList.remove("hidden");
|
||||
submitJobSuccess.classList.add("hidden");
|
||||
submitJobQueued.classList.add("hidden");
|
||||
updateCharCounts();
|
||||
clearErrors();
|
||||
validateForm();
|
||||
@@ -363,6 +433,7 @@
|
||||
submitJobBackdrop.addEventListener("click", closeSubmitJobModal);
|
||||
cancelJobBtn.addEventListener("click", closeSubmitJobModal);
|
||||
submitAnotherBtn.addEventListener("click", resetForm);
|
||||
submitAnotherQueuedBtn.addEventListener("click", resetForm);
|
||||
|
||||
// Input event listeners for real-time validation
|
||||
jobTitle.addEventListener("input", () => {
|
||||
@@ -420,9 +491,10 @@
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
body: JSON.stringify(formData)
|
||||
body: JSON.stringify(formData),
|
||||
signal: AbortSignal.timeout(8000),
|
||||
});
|
||||
|
||||
|
||||
if (response.ok) {
|
||||
// Show success state
|
||||
submitJobForm.classList.add("hidden");
|
||||
@@ -433,9 +505,14 @@
|
||||
descError.classList.add("visible");
|
||||
}
|
||||
} catch (error) {
|
||||
// For demo/development, show success even if API fails
|
||||
// Server unreachable — persist to localStorage queue.
|
||||
messageQueue.enqueue(formData);
|
||||
const count = messageQueue.pendingCount();
|
||||
submitJobForm.classList.add("hidden");
|
||||
submitJobSuccess.classList.remove("hidden");
|
||||
submitJobQueued.classList.remove("hidden");
|
||||
queueCountDisplay.textContent =
|
||||
count > 1 ? `${count} jobs queued` : "1 job queued";
|
||||
_updateQueueBadge();
|
||||
} finally {
|
||||
submitJobSubmit.disabled = false;
|
||||
submitJobSubmit.textContent = "Submit Job";
|
||||
|
||||
90
static/world/queue.js
Normal file
90
static/world/queue.js
Normal file
@@ -0,0 +1,90 @@
|
||||
/**
|
||||
* Offline message queue for Workshop panel.
|
||||
*
|
||||
* Persists undelivered job submissions to localStorage so they survive
|
||||
* page refreshes and are replayed when the server comes back online.
|
||||
*/
|
||||
|
||||
const _QUEUE_KEY = "timmy_workshop_queue";
|
||||
const _MAX_AGE_MS = 24 * 60 * 60 * 1000; // 24 hours — auto-expire old items
|
||||
|
||||
export const STATUS = {
|
||||
QUEUED: "queued",
|
||||
DELIVERED: "delivered",
|
||||
FAILED: "failed",
|
||||
};
|
||||
|
||||
function _load() {
|
||||
try {
|
||||
const raw = localStorage.getItem(_QUEUE_KEY);
|
||||
return raw ? JSON.parse(raw) : [];
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
function _save(items) {
|
||||
try {
|
||||
localStorage.setItem(_QUEUE_KEY, JSON.stringify(items));
|
||||
} catch {
|
||||
/* localStorage unavailable — degrade silently */
|
||||
}
|
||||
}
|
||||
|
||||
function _uid() {
|
||||
return `msg_${Date.now()}_${Math.random().toString(36).slice(2, 8)}`;
|
||||
}
|
||||
|
||||
/** LocalStorage-backed message queue for Workshop job submissions. */
|
||||
export const messageQueue = {
|
||||
/** Add a payload. Returns the created item (with id and status). */
|
||||
enqueue(payload) {
|
||||
const item = {
|
||||
id: _uid(),
|
||||
payload,
|
||||
queuedAt: new Date().toISOString(),
|
||||
status: STATUS.QUEUED,
|
||||
};
|
||||
const items = _load();
|
||||
items.push(item);
|
||||
_save(items);
|
||||
return item;
|
||||
},
|
||||
|
||||
/** Mark a message as delivered and remove it from storage. */
|
||||
markDelivered(id) {
|
||||
_save(_load().filter((i) => i.id !== id));
|
||||
},
|
||||
|
||||
/** Mark a message as permanently failed (kept for 24h for visibility). */
|
||||
markFailed(id) {
|
||||
_save(
|
||||
_load().map((i) =>
|
||||
i.id === id ? { ...i, status: STATUS.FAILED } : i
|
||||
)
|
||||
);
|
||||
},
|
||||
|
||||
/** All messages waiting to be delivered. */
|
||||
getPending() {
|
||||
return _load().filter((i) => i.status === STATUS.QUEUED);
|
||||
},
|
||||
|
||||
/** Total queued (QUEUED status only) count. */
|
||||
pendingCount() {
|
||||
return this.getPending().length;
|
||||
},
|
||||
|
||||
/** Drop expired failed items (> 24h old). */
|
||||
prune() {
|
||||
const cutoff = Date.now() - _MAX_AGE_MS;
|
||||
_save(
|
||||
_load().filter(
|
||||
(i) =>
|
||||
i.status === STATUS.QUEUED ||
|
||||
(i.status === STATUS.FAILED &&
|
||||
new Date(i.queuedAt).getTime() > cutoff)
|
||||
)
|
||||
);
|
||||
},
|
||||
};
|
||||
@@ -3,6 +3,10 @@
|
||||
*
|
||||
* Provides Timmy's current state to the scene. In Phase 2 this is a
|
||||
* static default; the WebSocket path is stubbed for future use.
|
||||
*
|
||||
* Also manages connection health monitoring: pings /api/matrix/health
|
||||
* every 30 seconds and notifies listeners when online/offline state
|
||||
* changes so the Workshop can replay any queued messages.
|
||||
*/
|
||||
|
||||
const DEFAULTS = {
|
||||
@@ -20,11 +24,19 @@ const DEFAULTS = {
|
||||
version: 1,
|
||||
};
|
||||
|
||||
const _HEALTH_URL = "/api/matrix/health";
|
||||
const _PING_INTERVAL_MS = 30_000;
|
||||
const _WS_RECONNECT_DELAY_MS = 5_000;
|
||||
|
||||
export class StateReader {
|
||||
constructor() {
|
||||
this.state = { ...DEFAULTS };
|
||||
this.listeners = [];
|
||||
this.connectionListeners = [];
|
||||
this._ws = null;
|
||||
this._online = false;
|
||||
this._pingTimer = null;
|
||||
this._reconnectTimer = null;
|
||||
}
|
||||
|
||||
/** Subscribe to state changes. */
|
||||
@@ -32,7 +44,12 @@ export class StateReader {
|
||||
this.listeners.push(fn);
|
||||
}
|
||||
|
||||
/** Notify all listeners. */
|
||||
/** Subscribe to online/offline transitions. Called with (isOnline: bool). */
|
||||
onConnectionChange(fn) {
|
||||
this.connectionListeners.push(fn);
|
||||
}
|
||||
|
||||
/** Notify all state listeners. */
|
||||
_notify() {
|
||||
for (const fn of this.listeners) {
|
||||
try {
|
||||
@@ -43,8 +60,48 @@ export class StateReader {
|
||||
}
|
||||
}
|
||||
|
||||
/** Try to connect to the world WebSocket for live updates. */
|
||||
connect() {
|
||||
/** Fire connection listeners only when state actually changes. */
|
||||
_notifyConnection(online) {
|
||||
if (online === this._online) return;
|
||||
this._online = online;
|
||||
for (const fn of this.connectionListeners) {
|
||||
try {
|
||||
fn(online);
|
||||
} catch (e) {
|
||||
console.warn("Connection listener error:", e);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Ping the health endpoint once and update connection state. */
|
||||
async _ping() {
|
||||
try {
|
||||
const r = await fetch(_HEALTH_URL, {
|
||||
signal: AbortSignal.timeout(5000),
|
||||
});
|
||||
this._notifyConnection(r.ok);
|
||||
} catch {
|
||||
this._notifyConnection(false);
|
||||
}
|
||||
}
|
||||
|
||||
/** Start 30-second health-check loop (idempotent). */
|
||||
_startHealthCheck() {
|
||||
if (this._pingTimer) return;
|
||||
this._pingTimer = setInterval(() => this._ping(), _PING_INTERVAL_MS);
|
||||
}
|
||||
|
||||
/** Schedule a WebSocket reconnect attempt after a delay (idempotent). */
|
||||
_scheduleReconnect() {
|
||||
if (this._reconnectTimer) return;
|
||||
this._reconnectTimer = setTimeout(() => {
|
||||
this._reconnectTimer = null;
|
||||
this._connectWS();
|
||||
}, _WS_RECONNECT_DELAY_MS);
|
||||
}
|
||||
|
||||
/** Open (or re-open) the WebSocket connection. */
|
||||
_connectWS() {
|
||||
const proto = location.protocol === "https:" ? "wss:" : "ws:";
|
||||
const url = `${proto}//${location.host}/api/world/ws`;
|
||||
try {
|
||||
@@ -52,10 +109,13 @@ export class StateReader {
|
||||
this._ws.onopen = () => {
|
||||
const dot = document.getElementById("connection-dot");
|
||||
if (dot) dot.classList.add("connected");
|
||||
this._notifyConnection(true);
|
||||
};
|
||||
this._ws.onclose = () => {
|
||||
const dot = document.getElementById("connection-dot");
|
||||
if (dot) dot.classList.remove("connected");
|
||||
this._notifyConnection(false);
|
||||
this._scheduleReconnect();
|
||||
};
|
||||
this._ws.onmessage = (ev) => {
|
||||
try {
|
||||
@@ -75,9 +135,18 @@ export class StateReader {
|
||||
};
|
||||
} catch (e) {
|
||||
console.warn("WebSocket unavailable — using static state");
|
||||
this._scheduleReconnect();
|
||||
}
|
||||
}
|
||||
|
||||
/** Connect to the world WebSocket and start health-check polling. */
|
||||
connect() {
|
||||
this._connectWS();
|
||||
this._startHealthCheck();
|
||||
// Immediate ping so connection status is known before the first interval.
|
||||
this._ping();
|
||||
}
|
||||
|
||||
/** Current mood string. */
|
||||
get mood() {
|
||||
return this.state.timmyState.mood;
|
||||
@@ -92,4 +161,9 @@ export class StateReader {
|
||||
get energy() {
|
||||
return this.state.timmyState.energy;
|
||||
}
|
||||
|
||||
/** Whether the server is currently reachable. */
|
||||
get isOnline() {
|
||||
return this._online;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -604,6 +604,68 @@ canvas {
|
||||
opacity: 1;
|
||||
}
|
||||
|
||||
/* Queued State (offline buffer) */
|
||||
.submit-job-queued {
|
||||
text-align: center;
|
||||
padding: 32px 16px;
|
||||
}
|
||||
|
||||
.submit-job-queued.hidden {
|
||||
display: none;
|
||||
}
|
||||
|
||||
.queued-icon {
|
||||
width: 64px;
|
||||
height: 64px;
|
||||
margin: 0 auto 20px;
|
||||
color: #ffaa33;
|
||||
}
|
||||
|
||||
.queued-icon svg {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
}
|
||||
|
||||
.submit-job-queued h3 {
|
||||
font-size: 20px;
|
||||
color: #ffaa33;
|
||||
margin: 0 0 12px 0;
|
||||
}
|
||||
|
||||
.submit-job-queued p {
|
||||
font-size: 14px;
|
||||
color: #888;
|
||||
margin: 0 0 16px 0;
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
.queue-count-display {
|
||||
font-size: 12px;
|
||||
color: #ffaa33;
|
||||
margin-bottom: 24px;
|
||||
opacity: 0.8;
|
||||
}
|
||||
|
||||
/* Queue badge — shown in overlay corner when offline jobs are pending */
|
||||
.queue-badge {
|
||||
position: absolute;
|
||||
bottom: 16px;
|
||||
right: 16px;
|
||||
padding: 4px 10px;
|
||||
background: rgba(10, 10, 20, 0.85);
|
||||
border: 1px solid rgba(255, 170, 51, 0.6);
|
||||
border-radius: 12px;
|
||||
color: #ffaa33;
|
||||
font-size: 11px;
|
||||
pointer-events: none;
|
||||
animation: queue-pulse 2s ease-in-out infinite;
|
||||
}
|
||||
|
||||
@keyframes queue-pulse {
|
||||
0%, 100% { opacity: 0.8; }
|
||||
50% { opacity: 1; }
|
||||
}
|
||||
|
||||
/* Mobile adjustments */
|
||||
@media (max-width: 480px) {
|
||||
.about-panel-content {
|
||||
|
||||
527
tests/dashboard/test_daily_run.py
Normal file
527
tests/dashboard/test_daily_run.py
Normal file
@@ -0,0 +1,527 @@
|
||||
"""Unit tests for dashboard/routes/daily_run.py."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from unittest.mock import MagicMock, patch
|
||||
from urllib.error import URLError
|
||||
|
||||
from dashboard.routes.daily_run import (
|
||||
DEFAULT_CONFIG,
|
||||
LAYER_LABELS,
|
||||
DailyRunMetrics,
|
||||
GiteaClient,
|
||||
LayerMetrics,
|
||||
_extract_layer,
|
||||
_fetch_layer_metrics,
|
||||
_get_metrics,
|
||||
_get_token,
|
||||
_load_config,
|
||||
_load_cycle_data,
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _load_config
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_load_config_returns_defaults():
|
||||
with patch("dashboard.routes.daily_run.CONFIG_PATH") as mock_path:
|
||||
mock_path.exists.return_value = False
|
||||
config = _load_config()
|
||||
assert config["gitea_api"] == DEFAULT_CONFIG["gitea_api"]
|
||||
assert config["repo_slug"] == DEFAULT_CONFIG["repo_slug"]
|
||||
|
||||
|
||||
def test_load_config_merges_file_orchestrator_section(tmp_path):
|
||||
config_file = tmp_path / "daily_run.json"
|
||||
config_file.write_text(
|
||||
json.dumps(
|
||||
{"orchestrator": {"repo_slug": "custom/repo", "gitea_api": "http://custom:3000/api/v1"}}
|
||||
)
|
||||
)
|
||||
with patch("dashboard.routes.daily_run.CONFIG_PATH", config_file):
|
||||
config = _load_config()
|
||||
assert config["repo_slug"] == "custom/repo"
|
||||
assert config["gitea_api"] == "http://custom:3000/api/v1"
|
||||
|
||||
|
||||
def test_load_config_ignores_invalid_json(tmp_path):
|
||||
config_file = tmp_path / "daily_run.json"
|
||||
config_file.write_text("not valid json{{")
|
||||
with patch("dashboard.routes.daily_run.CONFIG_PATH", config_file):
|
||||
config = _load_config()
|
||||
assert config["repo_slug"] == DEFAULT_CONFIG["repo_slug"]
|
||||
|
||||
|
||||
def test_load_config_env_overrides(monkeypatch):
|
||||
monkeypatch.setenv("TIMMY_GITEA_API", "http://envapi:3000/api/v1")
|
||||
monkeypatch.setenv("TIMMY_REPO_SLUG", "env/repo")
|
||||
monkeypatch.setenv("TIMMY_GITEA_TOKEN", "env-token-123")
|
||||
with patch("dashboard.routes.daily_run.CONFIG_PATH") as mock_path:
|
||||
mock_path.exists.return_value = False
|
||||
config = _load_config()
|
||||
assert config["gitea_api"] == "http://envapi:3000/api/v1"
|
||||
assert config["repo_slug"] == "env/repo"
|
||||
assert config["token"] == "env-token-123"
|
||||
|
||||
|
||||
def test_load_config_no_env_overrides_without_vars(monkeypatch):
|
||||
monkeypatch.delenv("TIMMY_GITEA_API", raising=False)
|
||||
monkeypatch.delenv("TIMMY_REPO_SLUG", raising=False)
|
||||
monkeypatch.delenv("TIMMY_GITEA_TOKEN", raising=False)
|
||||
with patch("dashboard.routes.daily_run.CONFIG_PATH") as mock_path:
|
||||
mock_path.exists.return_value = False
|
||||
config = _load_config()
|
||||
assert "token" not in config
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _get_token
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_get_token_from_config_dict():
|
||||
config = {"token": "direct-token", "token_file": "~/.hermes/gitea_token"}
|
||||
assert _get_token(config) == "direct-token"
|
||||
|
||||
|
||||
def test_get_token_from_file(tmp_path):
|
||||
token_file = tmp_path / "token.txt"
|
||||
token_file.write_text(" file-token \n")
|
||||
config = {"token_file": str(token_file)}
|
||||
assert _get_token(config) == "file-token"
|
||||
|
||||
|
||||
def test_get_token_returns_none_when_file_missing(tmp_path):
|
||||
config = {"token_file": str(tmp_path / "nonexistent_token")}
|
||||
assert _get_token(config) is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# GiteaClient
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _make_client(**kwargs) -> GiteaClient:
|
||||
config = {**DEFAULT_CONFIG, **kwargs}
|
||||
return GiteaClient(config, token="test-token")
|
||||
|
||||
|
||||
def test_gitea_client_headers_include_auth():
|
||||
client = _make_client()
|
||||
headers = client._headers()
|
||||
assert headers["Authorization"] == "token test-token"
|
||||
assert headers["Accept"] == "application/json"
|
||||
|
||||
|
||||
def test_gitea_client_headers_no_token():
|
||||
config = {**DEFAULT_CONFIG}
|
||||
client = GiteaClient(config, token=None)
|
||||
headers = client._headers()
|
||||
assert "Authorization" not in headers
|
||||
|
||||
|
||||
def test_gitea_client_api_url():
|
||||
client = _make_client()
|
||||
url = client._api_url("issues")
|
||||
assert url == f"{DEFAULT_CONFIG['gitea_api']}/repos/{DEFAULT_CONFIG['repo_slug']}/issues"
|
||||
|
||||
|
||||
def test_gitea_client_api_url_strips_trailing_slash():
|
||||
config = {**DEFAULT_CONFIG, "gitea_api": "http://localhost:3000/api/v1/"}
|
||||
client = GiteaClient(config, token=None)
|
||||
url = client._api_url("issues")
|
||||
assert "//" not in url.replace("http://", "")
|
||||
|
||||
|
||||
def test_gitea_client_is_available_true():
|
||||
client = _make_client()
|
||||
mock_resp = MagicMock()
|
||||
mock_resp.status = 200
|
||||
mock_resp.__enter__ = lambda s: mock_resp
|
||||
mock_resp.__exit__ = MagicMock(return_value=False)
|
||||
with patch("dashboard.routes.daily_run.urlopen", return_value=mock_resp):
|
||||
assert client.is_available() is True
|
||||
|
||||
|
||||
def test_gitea_client_is_available_cached():
|
||||
client = _make_client()
|
||||
client._available = True
|
||||
# Should not call urlopen at all
|
||||
with patch("dashboard.routes.daily_run.urlopen") as mock_urlopen:
|
||||
assert client.is_available() is True
|
||||
mock_urlopen.assert_not_called()
|
||||
|
||||
|
||||
def test_gitea_client_is_available_false_on_url_error():
|
||||
client = _make_client()
|
||||
with patch("dashboard.routes.daily_run.urlopen", side_effect=URLError("refused")):
|
||||
assert client.is_available() is False
|
||||
|
||||
|
||||
def test_gitea_client_is_available_false_on_timeout():
|
||||
client = _make_client()
|
||||
with patch("dashboard.routes.daily_run.urlopen", side_effect=TimeoutError()):
|
||||
assert client.is_available() is False
|
||||
|
||||
|
||||
def test_gitea_client_get_paginated_single_page():
|
||||
client = _make_client()
|
||||
mock_resp = MagicMock()
|
||||
mock_resp.read.return_value = json.dumps([{"id": 1}, {"id": 2}]).encode()
|
||||
mock_resp.__enter__ = lambda s: mock_resp
|
||||
mock_resp.__exit__ = MagicMock(return_value=False)
|
||||
with patch("dashboard.routes.daily_run.urlopen", return_value=mock_resp):
|
||||
result = client.get_paginated("issues")
|
||||
assert len(result) == 2
|
||||
assert result[0]["id"] == 1
|
||||
|
||||
|
||||
def test_gitea_client_get_paginated_empty():
|
||||
client = _make_client()
|
||||
mock_resp = MagicMock()
|
||||
mock_resp.read.return_value = b"[]"
|
||||
mock_resp.__enter__ = lambda s: mock_resp
|
||||
mock_resp.__exit__ = MagicMock(return_value=False)
|
||||
with patch("dashboard.routes.daily_run.urlopen", return_value=mock_resp):
|
||||
result = client.get_paginated("issues")
|
||||
assert result == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# LayerMetrics.trend
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_layer_metrics_trend_no_previous_no_current():
|
||||
lm = LayerMetrics(name="triage", label="layer:triage", current_count=0, previous_count=0)
|
||||
assert lm.trend == "→"
|
||||
|
||||
|
||||
def test_layer_metrics_trend_no_previous_with_current():
|
||||
lm = LayerMetrics(name="triage", label="layer:triage", current_count=5, previous_count=0)
|
||||
assert lm.trend == "↑"
|
||||
|
||||
|
||||
def test_layer_metrics_trend_big_increase():
|
||||
lm = LayerMetrics(name="triage", label="layer:triage", current_count=130, previous_count=100)
|
||||
assert lm.trend == "↑↑"
|
||||
|
||||
|
||||
def test_layer_metrics_trend_small_increase():
|
||||
lm = LayerMetrics(name="triage", label="layer:triage", current_count=108, previous_count=100)
|
||||
assert lm.trend == "↑"
|
||||
|
||||
|
||||
def test_layer_metrics_trend_stable():
|
||||
lm = LayerMetrics(name="triage", label="layer:triage", current_count=100, previous_count=100)
|
||||
assert lm.trend == "→"
|
||||
|
||||
|
||||
def test_layer_metrics_trend_small_decrease():
|
||||
lm = LayerMetrics(name="triage", label="layer:triage", current_count=92, previous_count=100)
|
||||
assert lm.trend == "↓"
|
||||
|
||||
|
||||
def test_layer_metrics_trend_big_decrease():
|
||||
lm = LayerMetrics(name="triage", label="layer:triage", current_count=70, previous_count=100)
|
||||
assert lm.trend == "↓↓"
|
||||
|
||||
|
||||
def test_layer_metrics_trend_color_up():
|
||||
lm = LayerMetrics(name="triage", label="layer:triage", current_count=200, previous_count=100)
|
||||
assert lm.trend_color == "var(--green)"
|
||||
|
||||
|
||||
def test_layer_metrics_trend_color_down():
|
||||
lm = LayerMetrics(name="triage", label="layer:triage", current_count=50, previous_count=100)
|
||||
assert lm.trend_color == "var(--amber)"
|
||||
|
||||
|
||||
def test_layer_metrics_trend_color_stable():
|
||||
lm = LayerMetrics(name="triage", label="layer:triage", current_count=100, previous_count=100)
|
||||
assert lm.trend_color == "var(--text-dim)"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# DailyRunMetrics.sessions_trend
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _make_daily_metrics(**kwargs) -> DailyRunMetrics:
|
||||
defaults = dict(
|
||||
sessions_completed=10,
|
||||
sessions_previous=8,
|
||||
layers=[],
|
||||
total_touched_current=20,
|
||||
total_touched_previous=15,
|
||||
lookback_days=7,
|
||||
generated_at=datetime.now(UTC).isoformat(),
|
||||
)
|
||||
defaults.update(kwargs)
|
||||
return DailyRunMetrics(**defaults)
|
||||
|
||||
|
||||
def test_daily_metrics_sessions_trend_big_increase():
|
||||
m = _make_daily_metrics(sessions_completed=130, sessions_previous=100)
|
||||
assert m.sessions_trend == "↑↑"
|
||||
|
||||
|
||||
def test_daily_metrics_sessions_trend_stable():
|
||||
m = _make_daily_metrics(sessions_completed=100, sessions_previous=100)
|
||||
assert m.sessions_trend == "→"
|
||||
|
||||
|
||||
def test_daily_metrics_sessions_trend_no_previous_zero_completed():
|
||||
m = _make_daily_metrics(sessions_completed=0, sessions_previous=0)
|
||||
assert m.sessions_trend == "→"
|
||||
|
||||
|
||||
def test_daily_metrics_sessions_trend_no_previous_with_completed():
|
||||
m = _make_daily_metrics(sessions_completed=5, sessions_previous=0)
|
||||
assert m.sessions_trend == "↑"
|
||||
|
||||
|
||||
def test_daily_metrics_sessions_trend_color_green():
|
||||
m = _make_daily_metrics(sessions_completed=200, sessions_previous=100)
|
||||
assert m.sessions_trend_color == "var(--green)"
|
||||
|
||||
|
||||
def test_daily_metrics_sessions_trend_color_amber():
|
||||
m = _make_daily_metrics(sessions_completed=50, sessions_previous=100)
|
||||
assert m.sessions_trend_color == "var(--amber)"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _extract_layer
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_extract_layer_finds_layer_label():
|
||||
labels = [{"name": "bug"}, {"name": "layer:triage"}, {"name": "urgent"}]
|
||||
assert _extract_layer(labels) == "triage"
|
||||
|
||||
|
||||
def test_extract_layer_returns_none_when_no_layer():
|
||||
labels = [{"name": "bug"}, {"name": "feature"}]
|
||||
assert _extract_layer(labels) is None
|
||||
|
||||
|
||||
def test_extract_layer_empty_labels():
|
||||
assert _extract_layer([]) is None
|
||||
|
||||
|
||||
def test_extract_layer_first_match_wins():
|
||||
labels = [{"name": "layer:micro-fix"}, {"name": "layer:tests"}]
|
||||
assert _extract_layer(labels) == "micro-fix"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _load_cycle_data
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_load_cycle_data_missing_file(tmp_path):
|
||||
with patch("dashboard.routes.daily_run.REPO_ROOT", tmp_path):
|
||||
result = _load_cycle_data(days=14)
|
||||
assert result == {"current": 0, "previous": 0}
|
||||
|
||||
|
||||
def test_load_cycle_data_counts_successful_sessions(tmp_path):
|
||||
retro_dir = tmp_path / ".loop" / "retro"
|
||||
retro_dir.mkdir(parents=True)
|
||||
retro_file = retro_dir / "cycles.jsonl"
|
||||
|
||||
now = datetime.now(UTC)
|
||||
recent_ts = (now - timedelta(days=3)).isoformat()
|
||||
older_ts = (now - timedelta(days=10)).isoformat()
|
||||
old_ts = (now - timedelta(days=20)).isoformat()
|
||||
|
||||
lines = [
|
||||
json.dumps({"timestamp": recent_ts, "success": True}),
|
||||
json.dumps({"timestamp": recent_ts, "success": False}), # not counted
|
||||
json.dumps({"timestamp": older_ts, "success": True}),
|
||||
json.dumps({"timestamp": old_ts, "success": True}), # outside window
|
||||
]
|
||||
retro_file.write_text("\n".join(lines))
|
||||
|
||||
with patch("dashboard.routes.daily_run.REPO_ROOT", tmp_path):
|
||||
result = _load_cycle_data(days=7)
|
||||
|
||||
assert result["current"] == 1
|
||||
assert result["previous"] == 1
|
||||
|
||||
|
||||
def test_load_cycle_data_skips_invalid_json_lines(tmp_path):
|
||||
retro_dir = tmp_path / ".loop" / "retro"
|
||||
retro_dir.mkdir(parents=True)
|
||||
retro_file = retro_dir / "cycles.jsonl"
|
||||
|
||||
now = datetime.now(UTC)
|
||||
recent_ts = (now - timedelta(days=1)).isoformat()
|
||||
retro_file.write_text(
|
||||
f"not valid json\n{json.dumps({'timestamp': recent_ts, 'success': True})}\n"
|
||||
)
|
||||
|
||||
with patch("dashboard.routes.daily_run.REPO_ROOT", tmp_path):
|
||||
result = _load_cycle_data(days=7)
|
||||
|
||||
assert result["current"] == 1
|
||||
|
||||
|
||||
def test_load_cycle_data_skips_entries_with_no_timestamp(tmp_path):
|
||||
retro_dir = tmp_path / ".loop" / "retro"
|
||||
retro_dir.mkdir(parents=True)
|
||||
retro_file = retro_dir / "cycles.jsonl"
|
||||
retro_file.write_text(json.dumps({"success": True}))
|
||||
|
||||
with patch("dashboard.routes.daily_run.REPO_ROOT", tmp_path):
|
||||
result = _load_cycle_data(days=7)
|
||||
|
||||
assert result == {"current": 0, "previous": 0}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _fetch_layer_metrics
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _make_issue(updated_offset_days: int) -> dict:
|
||||
ts = (datetime.now(UTC) - timedelta(days=updated_offset_days)).isoformat()
|
||||
return {"updated_at": ts, "labels": [{"name": "layer:triage"}]}
|
||||
|
||||
|
||||
def test_fetch_layer_metrics_counts_current_and_previous():
|
||||
client = _make_client()
|
||||
client._available = True
|
||||
|
||||
recent_issue = _make_issue(updated_offset_days=3)
|
||||
older_issue = _make_issue(updated_offset_days=10)
|
||||
|
||||
with patch.object(client, "get_paginated", return_value=[recent_issue, older_issue]):
|
||||
layers, total_current, total_previous = _fetch_layer_metrics(client, lookback_days=7)
|
||||
|
||||
# Should have one entry per LAYER_LABELS
|
||||
assert len(layers) == len(LAYER_LABELS)
|
||||
triage = next(lm for lm in layers if lm.name == "triage")
|
||||
assert triage.current_count == 1
|
||||
assert triage.previous_count == 1
|
||||
|
||||
|
||||
def test_fetch_layer_metrics_degrades_on_http_error():
|
||||
client = _make_client()
|
||||
client._available = True
|
||||
|
||||
with patch.object(client, "get_paginated", side_effect=URLError("network")):
|
||||
layers, total_current, total_previous = _fetch_layer_metrics(client, lookback_days=7)
|
||||
|
||||
assert len(layers) == len(LAYER_LABELS)
|
||||
for lm in layers:
|
||||
assert lm.current_count == 0
|
||||
assert lm.previous_count == 0
|
||||
assert total_current == 0
|
||||
assert total_previous == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _get_metrics
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_get_metrics_returns_none_when_gitea_unavailable():
|
||||
with patch("dashboard.routes.daily_run._load_config", return_value=DEFAULT_CONFIG):
|
||||
with patch("dashboard.routes.daily_run._get_token", return_value=None):
|
||||
with patch.object(GiteaClient, "is_available", return_value=False):
|
||||
result = _get_metrics()
|
||||
assert result is None
|
||||
|
||||
|
||||
def test_get_metrics_returns_daily_run_metrics():
|
||||
mock_layers = [
|
||||
LayerMetrics(name="triage", label="layer:triage", current_count=5, previous_count=3)
|
||||
]
|
||||
with patch("dashboard.routes.daily_run._load_config", return_value=DEFAULT_CONFIG):
|
||||
with patch("dashboard.routes.daily_run._get_token", return_value="tok"):
|
||||
with patch.object(GiteaClient, "is_available", return_value=True):
|
||||
with patch(
|
||||
"dashboard.routes.daily_run._fetch_layer_metrics",
|
||||
return_value=(mock_layers, 5, 3),
|
||||
):
|
||||
with patch(
|
||||
"dashboard.routes.daily_run._load_cycle_data",
|
||||
return_value={"current": 10, "previous": 8},
|
||||
):
|
||||
result = _get_metrics(lookback_days=7)
|
||||
|
||||
assert result is not None
|
||||
assert result.sessions_completed == 10
|
||||
assert result.sessions_previous == 8
|
||||
assert result.lookback_days == 7
|
||||
assert result.layers == mock_layers
|
||||
|
||||
|
||||
def test_get_metrics_returns_none_on_exception():
|
||||
with patch("dashboard.routes.daily_run._load_config", return_value=DEFAULT_CONFIG):
|
||||
with patch("dashboard.routes.daily_run._get_token", return_value="tok"):
|
||||
with patch.object(GiteaClient, "is_available", return_value=True):
|
||||
with patch(
|
||||
"dashboard.routes.daily_run._fetch_layer_metrics",
|
||||
side_effect=Exception("unexpected"),
|
||||
):
|
||||
result = _get_metrics()
|
||||
assert result is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Route handlers (FastAPI)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_daily_run_metrics_api_unavailable(client):
|
||||
with patch("dashboard.routes.daily_run._get_metrics", return_value=None):
|
||||
resp = client.get("/daily-run/metrics")
|
||||
assert resp.status_code == 503
|
||||
data = resp.json()
|
||||
assert data["status"] == "unavailable"
|
||||
|
||||
|
||||
def test_daily_run_metrics_api_returns_json(client):
|
||||
mock_metrics = _make_daily_metrics(
|
||||
layers=[
|
||||
LayerMetrics(name="triage", label="layer:triage", current_count=3, previous_count=2)
|
||||
]
|
||||
)
|
||||
with patch("dashboard.routes.daily_run._get_metrics", return_value=mock_metrics):
|
||||
with patch(
|
||||
"dashboard.routes.quests.check_daily_run_quests",
|
||||
return_value=[],
|
||||
create=True,
|
||||
):
|
||||
resp = client.get("/daily-run/metrics?lookback_days=7")
|
||||
assert resp.status_code == 200
|
||||
data = resp.json()
|
||||
assert data["status"] == "ok"
|
||||
assert data["lookback_days"] == 7
|
||||
assert "sessions" in data
|
||||
assert "layers" in data
|
||||
assert "totals" in data
|
||||
assert len(data["layers"]) == 1
|
||||
assert data["layers"][0]["name"] == "triage"
|
||||
|
||||
|
||||
def test_daily_run_panel_returns_html(client):
|
||||
mock_metrics = _make_daily_metrics()
|
||||
with patch("dashboard.routes.daily_run._get_metrics", return_value=mock_metrics):
|
||||
with patch("dashboard.routes.daily_run._load_config", return_value=DEFAULT_CONFIG):
|
||||
resp = client.get("/daily-run/panel")
|
||||
assert resp.status_code == 200
|
||||
assert "text/html" in resp.headers["content-type"]
|
||||
|
||||
|
||||
def test_daily_run_panel_when_unavailable(client):
|
||||
with patch("dashboard.routes.daily_run._get_metrics", return_value=None):
|
||||
with patch("dashboard.routes.daily_run._load_config", return_value=DEFAULT_CONFIG):
|
||||
resp = client.get("/daily-run/panel")
|
||||
assert resp.status_code == 200
|
||||
74
tests/dashboard/test_nexus.py
Normal file
74
tests/dashboard/test_nexus.py
Normal file
@@ -0,0 +1,74 @@
|
||||
"""Tests for the Nexus conversational awareness routes."""
|
||||
|
||||
from unittest.mock import patch
|
||||
|
||||
|
||||
def test_nexus_page_returns_200(client):
|
||||
"""GET /nexus should render without error."""
|
||||
response = client.get("/nexus")
|
||||
assert response.status_code == 200
|
||||
assert "NEXUS" in response.text
|
||||
|
||||
|
||||
def test_nexus_page_contains_chat_form(client):
|
||||
"""Nexus page must include the conversational chat form."""
|
||||
response = client.get("/nexus")
|
||||
assert response.status_code == 200
|
||||
assert "/nexus/chat" in response.text
|
||||
|
||||
|
||||
def test_nexus_page_contains_teach_form(client):
|
||||
"""Nexus page must include the teaching panel form."""
|
||||
response = client.get("/nexus")
|
||||
assert response.status_code == 200
|
||||
assert "/nexus/teach" in response.text
|
||||
|
||||
|
||||
def test_nexus_chat_empty_message_returns_empty(client):
|
||||
"""POST /nexus/chat with blank message returns empty response."""
|
||||
response = client.post("/nexus/chat", data={"message": " "})
|
||||
assert response.status_code == 200
|
||||
assert response.text == ""
|
||||
|
||||
|
||||
def test_nexus_chat_too_long_returns_error(client):
|
||||
"""POST /nexus/chat with overlong message returns error partial."""
|
||||
long_msg = "x" * 10_001
|
||||
response = client.post("/nexus/chat", data={"message": long_msg})
|
||||
assert response.status_code == 200
|
||||
assert "too long" in response.text.lower()
|
||||
|
||||
|
||||
def test_nexus_chat_posts_message(client):
|
||||
"""POST /nexus/chat calls the session chat function and returns a partial."""
|
||||
with patch("dashboard.routes.nexus.chat", return_value="Hello from Timmy"):
|
||||
response = client.post("/nexus/chat", data={"message": "hello"})
|
||||
assert response.status_code == 200
|
||||
assert "hello" in response.text.lower() or "timmy" in response.text.lower()
|
||||
|
||||
|
||||
def test_nexus_teach_stores_fact(client):
|
||||
"""POST /nexus/teach should persist a fact and return confirmation."""
|
||||
with (
|
||||
patch("dashboard.routes.nexus.store_personal_fact") as mock_store,
|
||||
patch("dashboard.routes.nexus.recall_personal_facts_with_ids", return_value=[]),
|
||||
):
|
||||
mock_store.return_value = None
|
||||
response = client.post("/nexus/teach", data={"fact": "Timmy loves Python"})
|
||||
assert response.status_code == 200
|
||||
assert "Timmy loves Python" in response.text
|
||||
|
||||
|
||||
def test_nexus_teach_empty_fact_returns_empty(client):
|
||||
"""POST /nexus/teach with blank fact returns empty response."""
|
||||
response = client.post("/nexus/teach", data={"fact": " "})
|
||||
assert response.status_code == 200
|
||||
assert response.text == ""
|
||||
|
||||
|
||||
def test_nexus_clear_history(client):
|
||||
"""DELETE /nexus/history should clear the conversation log."""
|
||||
with patch("dashboard.routes.nexus.reset_session"):
|
||||
response = client.request("DELETE", "/nexus/history")
|
||||
assert response.status_code == 200
|
||||
assert "cleared" in response.text.lower()
|
||||
458
tests/dashboard/test_scorecard_service.py
Normal file
458
tests/dashboard/test_scorecard_service.py
Normal file
@@ -0,0 +1,458 @@
|
||||
"""Unit tests for dashboard/services/scorecard_service.py.
|
||||
|
||||
Focuses on edge cases and scenarios not covered in test_scorecards.py:
|
||||
- _aggregate_metrics: test.execution events, PR-closed-without-merge,
|
||||
push default commit count, untracked agent with agent_id passthrough
|
||||
- _detect_patterns: boundary conditions (< 3 PRs, exactly 3, exactly 80%)
|
||||
- _generate_narrative_bullets: singular/plural forms
|
||||
- generate_scorecard: token augmentation max() logic
|
||||
- ScorecardSummary.to_dict(): ISO timestamp format, tests_affected count
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
pytestmark = pytest.mark.unit
|
||||
|
||||
from dashboard.services.scorecard_service import (
|
||||
AgentMetrics,
|
||||
PeriodType,
|
||||
ScorecardSummary,
|
||||
_aggregate_metrics,
|
||||
_detect_patterns,
|
||||
_generate_narrative_bullets,
|
||||
generate_scorecard,
|
||||
)
|
||||
from infrastructure.events.bus import Event
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _aggregate_metrics — edge cases
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestAggregateMetricsEdgeCases:
|
||||
"""Edge cases for _aggregate_metrics not covered in test_scorecards.py."""
|
||||
|
||||
def test_push_event_defaults_to_one_commit(self):
|
||||
"""Push event with no num_commits key should count as 1 commit."""
|
||||
events = [
|
||||
Event(type="gitea.push", source="gitea", data={"actor": "claude"}),
|
||||
]
|
||||
result = _aggregate_metrics(events)
|
||||
|
||||
assert result["claude"].commits == 1
|
||||
|
||||
def test_pr_closed_without_merge_not_counted(self):
|
||||
"""PR closed without merge should not appear in prs_merged."""
|
||||
events = [
|
||||
Event(
|
||||
type="gitea.pull_request",
|
||||
source="gitea",
|
||||
data={"actor": "kimi", "pr_number": 99, "action": "closed", "merged": False},
|
||||
),
|
||||
]
|
||||
result = _aggregate_metrics(events)
|
||||
|
||||
# PR was not merged — should not be in prs_merged
|
||||
assert "kimi" in result
|
||||
assert 99 not in result["kimi"].prs_merged
|
||||
# Also not counted as opened (action != "opened")
|
||||
assert 99 not in result["kimi"].prs_opened
|
||||
# Not touched (only merged PRs add to issues_touched)
|
||||
assert 99 not in result["kimi"].issues_touched
|
||||
|
||||
def test_test_execution_event_aggregation(self):
|
||||
"""test.execution events should populate tests_affected."""
|
||||
events = [
|
||||
Event(
|
||||
type="test.execution",
|
||||
source="ci",
|
||||
data={"actor": "gemini", "test_files": ["tests/test_alpha.py", "tests/test_beta.py"]},
|
||||
),
|
||||
]
|
||||
result = _aggregate_metrics(events)
|
||||
|
||||
assert "gemini" in result
|
||||
assert "tests/test_alpha.py" in result["gemini"].tests_affected
|
||||
assert "tests/test_beta.py" in result["gemini"].tests_affected
|
||||
|
||||
def test_untracked_agent_with_agent_id_field_included(self):
|
||||
"""An untracked actor that carries agent_id in data should be included."""
|
||||
events = [
|
||||
Event(
|
||||
type="agent.task.completed",
|
||||
source="system",
|
||||
data={"agent_id": "kimi", "tests_affected": [], "token_reward": 5},
|
||||
),
|
||||
]
|
||||
result = _aggregate_metrics(events)
|
||||
|
||||
# kimi is tracked and agent_id is present in data
|
||||
assert "kimi" in result
|
||||
assert result["kimi"].tokens_earned == 5
|
||||
|
||||
def test_untracked_actor_without_agent_id_excluded(self):
|
||||
"""Actor that is not tracked and has no agent_id in data is skipped."""
|
||||
events = [
|
||||
Event(
|
||||
type="gitea.push",
|
||||
source="gitea",
|
||||
data={"actor": "anon-bot", "num_commits": 10},
|
||||
),
|
||||
]
|
||||
result = _aggregate_metrics(events)
|
||||
|
||||
assert "anon-bot" not in result
|
||||
|
||||
def test_issue_opened_with_no_issue_number_ignored(self):
|
||||
"""Issue opened event with issue_number=0 should not add to issues_touched."""
|
||||
events = [
|
||||
Event(
|
||||
type="gitea.issue.opened",
|
||||
source="gitea",
|
||||
data={"actor": "hermes", "issue_number": 0},
|
||||
),
|
||||
]
|
||||
result = _aggregate_metrics(events)
|
||||
|
||||
assert "hermes" in result
|
||||
assert len(result["hermes"].issues_touched) == 0
|
||||
|
||||
def test_comment_with_no_issue_number_still_increments_counter(self):
|
||||
"""Comment event with issue_number=0 increments comment count but not issues_touched."""
|
||||
events = [
|
||||
Event(
|
||||
type="gitea.issue.comment",
|
||||
source="gitea",
|
||||
data={"actor": "manus", "issue_number": 0},
|
||||
),
|
||||
]
|
||||
result = _aggregate_metrics(events)
|
||||
|
||||
assert "manus" in result
|
||||
assert result["manus"].comments == 1
|
||||
assert len(result["manus"].issues_touched) == 0
|
||||
|
||||
def test_task_completion_no_tests_affected(self):
|
||||
"""Task completion with empty tests_affected list should work fine."""
|
||||
events = [
|
||||
Event(
|
||||
type="agent.task.completed",
|
||||
source="system",
|
||||
data={"agent_id": "claude", "tests_affected": [], "token_reward": 20},
|
||||
),
|
||||
]
|
||||
result = _aggregate_metrics(events)
|
||||
|
||||
assert "claude" in result
|
||||
assert len(result["claude"].tests_affected) == 0
|
||||
assert result["claude"].tokens_earned == 20
|
||||
|
||||
def test_multiple_agents_independent_metrics(self):
|
||||
"""Events from multiple agents are tracked independently."""
|
||||
events = [
|
||||
Event(type="gitea.push", source="gitea", data={"actor": "claude", "num_commits": 3}),
|
||||
Event(type="gitea.push", source="gitea", data={"actor": "gemini", "num_commits": 7}),
|
||||
]
|
||||
result = _aggregate_metrics(events)
|
||||
|
||||
assert result["claude"].commits == 3
|
||||
assert result["gemini"].commits == 7
|
||||
|
||||
def test_pr_with_no_pr_number_not_recorded(self):
|
||||
"""PR event with pr_number=0 should not add to prs_opened."""
|
||||
events = [
|
||||
Event(
|
||||
type="gitea.pull_request",
|
||||
source="gitea",
|
||||
data={"actor": "kimi", "pr_number": 0, "action": "opened"},
|
||||
),
|
||||
]
|
||||
result = _aggregate_metrics(events)
|
||||
|
||||
assert "kimi" in result
|
||||
assert len(result["kimi"].prs_opened) == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _detect_patterns — boundary conditions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestDetectPatternsBoundaries:
|
||||
"""Boundary conditions for _detect_patterns."""
|
||||
|
||||
def test_no_patterns_with_empty_metrics(self):
|
||||
"""Empty metrics should not trigger any patterns."""
|
||||
metrics = AgentMetrics(agent_id="kimi")
|
||||
patterns = _detect_patterns(metrics)
|
||||
|
||||
assert patterns == []
|
||||
|
||||
def test_merge_rate_requires_three_or_more_prs(self):
|
||||
"""Merge rate pattern requires at least 3 PRs opened."""
|
||||
metrics = AgentMetrics(
|
||||
agent_id="kimi",
|
||||
prs_opened={1, 2},
|
||||
prs_merged={1, 2}, # 100% rate but only 2 PRs
|
||||
)
|
||||
patterns = _detect_patterns(metrics)
|
||||
|
||||
# Should NOT trigger high-merge-rate pattern (< 3 PRs)
|
||||
assert not any("High merge rate" in p for p in patterns)
|
||||
assert not any("low merge rate" in p for p in patterns)
|
||||
|
||||
def test_merge_rate_exactly_3_prs_triggers_pattern(self):
|
||||
"""Exactly 3 PRs opened triggers merge rate evaluation."""
|
||||
metrics = AgentMetrics(
|
||||
agent_id="kimi",
|
||||
prs_opened={1, 2, 3},
|
||||
prs_merged={1, 2, 3}, # 100% rate, 3 PRs
|
||||
)
|
||||
patterns = _detect_patterns(metrics)
|
||||
|
||||
assert any("High merge rate" in p for p in patterns)
|
||||
|
||||
def test_merge_rate_80_percent_is_high(self):
|
||||
"""Exactly 80% merge rate triggers high merge rate pattern."""
|
||||
metrics = AgentMetrics(
|
||||
agent_id="kimi",
|
||||
prs_opened={1, 2, 3, 4, 5},
|
||||
prs_merged={1, 2, 3, 4}, # 80%
|
||||
)
|
||||
patterns = _detect_patterns(metrics)
|
||||
|
||||
assert any("High merge rate" in p for p in patterns)
|
||||
|
||||
def test_merge_rate_below_80_not_high(self):
|
||||
"""79% merge rate should NOT trigger high merge rate pattern."""
|
||||
metrics = AgentMetrics(
|
||||
agent_id="kimi",
|
||||
prs_opened={1, 2, 3, 4, 5, 6, 7}, # 7 PRs
|
||||
prs_merged={1, 2, 3, 4, 5}, # ~71.4% — below 80%
|
||||
)
|
||||
patterns = _detect_patterns(metrics)
|
||||
|
||||
assert not any("High merge rate" in p for p in patterns)
|
||||
|
||||
def test_commit_pattern_requires_over_10_commits(self):
|
||||
"""Exactly 10 commits does NOT trigger the high-commit/no-PR pattern."""
|
||||
metrics = AgentMetrics(
|
||||
agent_id="kimi",
|
||||
commits=10,
|
||||
prs_opened=set(),
|
||||
)
|
||||
patterns = _detect_patterns(metrics)
|
||||
|
||||
assert not any("High commit volume" in p for p in patterns)
|
||||
|
||||
def test_commit_pattern_triggered_at_11_commits(self):
|
||||
"""11 commits with no PRs triggers the high-commit pattern."""
|
||||
metrics = AgentMetrics(
|
||||
agent_id="kimi",
|
||||
commits=11,
|
||||
prs_opened=set(),
|
||||
)
|
||||
patterns = _detect_patterns(metrics)
|
||||
|
||||
assert any("High commit volume without PRs" in p for p in patterns)
|
||||
|
||||
def test_token_accumulation_exact_boundary(self):
|
||||
"""Net tokens = 100 does NOT trigger accumulation pattern (must be > 100)."""
|
||||
metrics = AgentMetrics(agent_id="kimi", tokens_earned=100, tokens_spent=0)
|
||||
patterns = _detect_patterns(metrics)
|
||||
|
||||
assert not any("Strong token accumulation" in p for p in patterns)
|
||||
|
||||
def test_token_spend_exact_boundary(self):
|
||||
"""Net tokens = -50 does NOT trigger high spend pattern (must be < -50)."""
|
||||
metrics = AgentMetrics(agent_id="kimi", tokens_earned=0, tokens_spent=50)
|
||||
patterns = _detect_patterns(metrics)
|
||||
|
||||
assert not any("High token spend" in p for p in patterns)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _generate_narrative_bullets — singular/plural
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestGenerateNarrativeSingularPlural:
|
||||
"""Test singular and plural forms in narrative bullets."""
|
||||
|
||||
def test_singular_commit(self):
|
||||
"""One commit should use singular form."""
|
||||
metrics = AgentMetrics(agent_id="kimi", commits=1)
|
||||
bullets = _generate_narrative_bullets(metrics, PeriodType.daily)
|
||||
|
||||
activity = next((b for b in bullets if "Active across" in b), None)
|
||||
assert activity is not None
|
||||
assert "1 commit" in activity
|
||||
assert "1 commits" not in activity
|
||||
|
||||
def test_singular_pr_opened(self):
|
||||
"""One opened PR should use singular form."""
|
||||
metrics = AgentMetrics(agent_id="kimi", prs_opened={1})
|
||||
bullets = _generate_narrative_bullets(metrics, PeriodType.daily)
|
||||
|
||||
activity = next((b for b in bullets if "Active across" in b), None)
|
||||
assert activity is not None
|
||||
assert "1 PR opened" in activity
|
||||
|
||||
def test_singular_pr_merged(self):
|
||||
"""One merged PR should use singular form."""
|
||||
metrics = AgentMetrics(agent_id="kimi", prs_merged={1})
|
||||
bullets = _generate_narrative_bullets(metrics, PeriodType.daily)
|
||||
|
||||
activity = next((b for b in bullets if "Active across" in b), None)
|
||||
assert activity is not None
|
||||
assert "1 PR merged" in activity
|
||||
|
||||
def test_singular_issue_touched(self):
|
||||
"""One issue touched should use singular form."""
|
||||
metrics = AgentMetrics(agent_id="kimi", issues_touched={42})
|
||||
bullets = _generate_narrative_bullets(metrics, PeriodType.daily)
|
||||
|
||||
activity = next((b for b in bullets if "Active across" in b), None)
|
||||
assert activity is not None
|
||||
assert "1 issue touched" in activity
|
||||
|
||||
def test_singular_comment(self):
|
||||
"""One comment should use singular form."""
|
||||
metrics = AgentMetrics(agent_id="kimi", comments=1)
|
||||
bullets = _generate_narrative_bullets(metrics, PeriodType.daily)
|
||||
|
||||
activity = next((b for b in bullets if "Active across" in b), None)
|
||||
assert activity is not None
|
||||
assert "1 comment" in activity
|
||||
|
||||
def test_singular_test_file(self):
|
||||
"""One test file should use singular form."""
|
||||
metrics = AgentMetrics(agent_id="kimi", tests_affected={"test_foo.py"})
|
||||
bullets = _generate_narrative_bullets(metrics, PeriodType.daily)
|
||||
|
||||
assert any("1 test file." in b for b in bullets)
|
||||
|
||||
def test_weekly_period_label(self):
|
||||
"""Weekly period uses 'week' label in no-activity message."""
|
||||
metrics = AgentMetrics(agent_id="kimi")
|
||||
bullets = _generate_narrative_bullets(metrics, PeriodType.weekly)
|
||||
|
||||
assert any("this week" in b for b in bullets)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# generate_scorecard — token augmentation (max logic)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestGenerateScorecardTokenAugmentation:
|
||||
"""Test the max() token augmentation logic in generate_scorecard."""
|
||||
|
||||
def test_event_tokens_win_over_ledger_when_higher(self):
|
||||
"""When event tokens > ledger tokens, event tokens are preserved."""
|
||||
events = [
|
||||
Event(
|
||||
type="agent.task.completed",
|
||||
source="system",
|
||||
data={"agent_id": "kimi", "tests_affected": [], "token_reward": 200},
|
||||
),
|
||||
]
|
||||
with patch(
|
||||
"dashboard.services.scorecard_service._collect_events_for_period",
|
||||
return_value=events,
|
||||
):
|
||||
with patch(
|
||||
"dashboard.services.scorecard_service._query_token_transactions",
|
||||
return_value=(50, 0), # ledger says 50 earned
|
||||
):
|
||||
scorecard = generate_scorecard("kimi", PeriodType.daily)
|
||||
|
||||
# max(200, 50) = 200 should win
|
||||
assert scorecard.metrics.tokens_earned == 200
|
||||
|
||||
def test_ledger_tokens_win_when_higher(self):
|
||||
"""When ledger tokens > event tokens, ledger tokens are used."""
|
||||
events = [
|
||||
Event(
|
||||
type="agent.task.completed",
|
||||
source="system",
|
||||
data={"agent_id": "kimi", "tests_affected": [], "token_reward": 10},
|
||||
),
|
||||
]
|
||||
with patch(
|
||||
"dashboard.services.scorecard_service._collect_events_for_period",
|
||||
return_value=events,
|
||||
):
|
||||
with patch(
|
||||
"dashboard.services.scorecard_service._query_token_transactions",
|
||||
return_value=(500, 100), # ledger says 500 earned, 100 spent
|
||||
):
|
||||
scorecard = generate_scorecard("kimi", PeriodType.daily)
|
||||
|
||||
# max(10, 500) = 500
|
||||
assert scorecard.metrics.tokens_earned == 500
|
||||
# max(0, 100) = 100
|
||||
assert scorecard.metrics.tokens_spent == 100
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ScorecardSummary.to_dict — timestamp format and tests_affected
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestScorecardSummaryToDict:
|
||||
"""Additional to_dict tests."""
|
||||
|
||||
def test_timestamps_are_iso_strings(self):
|
||||
"""period_start and period_end should be ISO format strings."""
|
||||
start = datetime(2026, 3, 20, 0, 0, 0, tzinfo=UTC)
|
||||
end = datetime(2026, 3, 21, 0, 0, 0, tzinfo=UTC)
|
||||
summary = ScorecardSummary(
|
||||
agent_id="kimi",
|
||||
period_type=PeriodType.daily,
|
||||
period_start=start,
|
||||
period_end=end,
|
||||
metrics=AgentMetrics(agent_id="kimi"),
|
||||
)
|
||||
data = summary.to_dict()
|
||||
|
||||
assert data["period_start"] == start.isoformat()
|
||||
assert data["period_end"] == end.isoformat()
|
||||
|
||||
def test_tests_affected_count_in_dict(self):
|
||||
"""to_dict metrics.tests_affected should be a count (int)."""
|
||||
metrics = AgentMetrics(
|
||||
agent_id="kimi",
|
||||
tests_affected={"test_a.py", "test_b.py", "test_c.py"},
|
||||
)
|
||||
summary = ScorecardSummary(
|
||||
agent_id="kimi",
|
||||
period_type=PeriodType.daily,
|
||||
period_start=datetime.now(UTC),
|
||||
period_end=datetime.now(UTC),
|
||||
metrics=metrics,
|
||||
)
|
||||
data = summary.to_dict()
|
||||
|
||||
assert data["metrics"]["tests_affected"] == 3
|
||||
|
||||
def test_empty_narrative_and_patterns(self):
|
||||
"""to_dict with default empty lists should serialize correctly."""
|
||||
summary = ScorecardSummary(
|
||||
agent_id="claude",
|
||||
period_type=PeriodType.weekly,
|
||||
period_start=datetime.now(UTC),
|
||||
period_end=datetime.now(UTC),
|
||||
metrics=AgentMetrics(agent_id="claude"),
|
||||
)
|
||||
data = summary.to_dict()
|
||||
|
||||
assert data["narrative_bullets"] == []
|
||||
assert data["patterns"] == []
|
||||
assert data["period_type"] == "weekly"
|
||||
178
tests/infrastructure/test_budget_tracker.py
Normal file
178
tests/infrastructure/test_budget_tracker.py
Normal file
@@ -0,0 +1,178 @@
|
||||
"""Tests for the cloud API budget tracker (issue #882)."""
|
||||
|
||||
import time
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
from infrastructure.models.budget import (
|
||||
BudgetTracker,
|
||||
SpendRecord,
|
||||
estimate_cost_usd,
|
||||
get_budget_tracker,
|
||||
)
|
||||
|
||||
pytestmark = pytest.mark.unit
|
||||
|
||||
|
||||
# ── estimate_cost_usd ─────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class TestEstimateCostUsd:
|
||||
def test_haiku_cheaper_than_sonnet(self):
|
||||
haiku_cost = estimate_cost_usd("claude-haiku-4-5", 1000, 1000)
|
||||
sonnet_cost = estimate_cost_usd("claude-sonnet-4-5", 1000, 1000)
|
||||
assert haiku_cost < sonnet_cost
|
||||
|
||||
def test_zero_tokens_is_zero_cost(self):
|
||||
assert estimate_cost_usd("gpt-4o", 0, 0) == 0.0
|
||||
|
||||
def test_unknown_model_uses_default(self):
|
||||
cost = estimate_cost_usd("some-unknown-model-xyz", 1000, 1000)
|
||||
assert cost > 0 # Uses conservative default, not zero
|
||||
|
||||
def test_versioned_model_name_matches(self):
|
||||
# "claude-haiku-4-5-20251001" should match "haiku"
|
||||
cost1 = estimate_cost_usd("claude-haiku-4-5-20251001", 1000, 0)
|
||||
cost2 = estimate_cost_usd("claude-haiku-4-5", 1000, 0)
|
||||
assert cost1 == cost2
|
||||
|
||||
def test_gpt4o_mini_cheaper_than_gpt4o(self):
|
||||
mini = estimate_cost_usd("gpt-4o-mini", 1000, 1000)
|
||||
full = estimate_cost_usd("gpt-4o", 1000, 1000)
|
||||
assert mini < full
|
||||
|
||||
def test_returns_float(self):
|
||||
assert isinstance(estimate_cost_usd("haiku", 100, 200), float)
|
||||
|
||||
|
||||
# ── BudgetTracker ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class TestBudgetTrackerInit:
|
||||
def test_creates_with_memory_db(self):
|
||||
tracker = BudgetTracker(db_path=":memory:")
|
||||
assert tracker._db_ok is True
|
||||
|
||||
def test_in_memory_fallback_empty_on_creation(self):
|
||||
tracker = BudgetTracker(db_path=":memory:")
|
||||
assert tracker._in_memory == []
|
||||
|
||||
def test_bad_path_uses_memory_fallback(self, tmp_path):
|
||||
bad_path = str(tmp_path / "nonexistent" / "x" / "budget.db")
|
||||
# Should not raise — just log and continue with memory fallback
|
||||
# (actually will create parent dirs, so test with truly bad path)
|
||||
tracker = BudgetTracker.__new__(BudgetTracker)
|
||||
tracker._db_path = bad_path
|
||||
tracker._lock = __import__("threading").Lock()
|
||||
tracker._in_memory = []
|
||||
tracker._db_ok = False
|
||||
# Record to in-memory fallback
|
||||
tracker._in_memory.append(
|
||||
SpendRecord(time.time(), "test", "model", 100, 100, 0.001, "cloud")
|
||||
)
|
||||
assert len(tracker._in_memory) == 1
|
||||
|
||||
|
||||
class TestBudgetTrackerRecordSpend:
|
||||
def test_record_spend_returns_cost(self):
|
||||
tracker = BudgetTracker(db_path=":memory:")
|
||||
cost = tracker.record_spend("anthropic", "claude-haiku-4-5", 100, 200)
|
||||
assert cost > 0
|
||||
|
||||
def test_record_spend_explicit_cost(self):
|
||||
tracker = BudgetTracker(db_path=":memory:")
|
||||
cost = tracker.record_spend("anthropic", "model", cost_usd=1.23)
|
||||
assert cost == pytest.approx(1.23)
|
||||
|
||||
def test_record_spend_accumulates(self):
|
||||
tracker = BudgetTracker(db_path=":memory:")
|
||||
tracker.record_spend("openai", "gpt-4o", cost_usd=0.01)
|
||||
tracker.record_spend("openai", "gpt-4o", cost_usd=0.02)
|
||||
assert tracker.get_daily_spend() == pytest.approx(0.03, abs=1e-9)
|
||||
|
||||
def test_record_spend_with_tier_label(self):
|
||||
tracker = BudgetTracker(db_path=":memory:")
|
||||
cost = tracker.record_spend("anthropic", "haiku", tier="cloud_api")
|
||||
assert cost >= 0
|
||||
|
||||
def test_monthly_spend_includes_daily(self):
|
||||
tracker = BudgetTracker(db_path=":memory:")
|
||||
tracker.record_spend("anthropic", "haiku", cost_usd=5.00)
|
||||
assert tracker.get_monthly_spend() >= tracker.get_daily_spend()
|
||||
|
||||
|
||||
class TestBudgetTrackerCloudAllowed:
|
||||
def test_allowed_when_no_spend(self):
|
||||
tracker = BudgetTracker(db_path=":memory:")
|
||||
with (
|
||||
patch.object(type(tracker._get_budget() if hasattr(tracker, "_get_budget") else tracker), "tier_cloud_daily_budget_usd", 5.0, create=True),
|
||||
):
|
||||
# Settings-based check — use real settings (5.0 default, 0 spent)
|
||||
assert tracker.cloud_allowed() is True
|
||||
|
||||
def test_blocked_when_daily_limit_exceeded(self):
|
||||
tracker = BudgetTracker(db_path=":memory:")
|
||||
tracker.record_spend("anthropic", "haiku", cost_usd=999.0)
|
||||
# With default daily limit of 5.0, 999 should block
|
||||
assert tracker.cloud_allowed() is False
|
||||
|
||||
def test_allowed_when_daily_limit_zero(self):
|
||||
tracker = BudgetTracker(db_path=":memory:")
|
||||
tracker.record_spend("anthropic", "haiku", cost_usd=999.0)
|
||||
with (
|
||||
patch("infrastructure.models.budget.settings") as mock_settings,
|
||||
):
|
||||
mock_settings.tier_cloud_daily_budget_usd = 0 # disabled
|
||||
mock_settings.tier_cloud_monthly_budget_usd = 0 # disabled
|
||||
assert tracker.cloud_allowed() is True
|
||||
|
||||
def test_blocked_when_monthly_limit_exceeded(self):
|
||||
tracker = BudgetTracker(db_path=":memory:")
|
||||
tracker.record_spend("anthropic", "haiku", cost_usd=999.0)
|
||||
with patch("infrastructure.models.budget.settings") as mock_settings:
|
||||
mock_settings.tier_cloud_daily_budget_usd = 0 # daily disabled
|
||||
mock_settings.tier_cloud_monthly_budget_usd = 10.0
|
||||
assert tracker.cloud_allowed() is False
|
||||
|
||||
|
||||
class TestBudgetTrackerSummary:
|
||||
def test_summary_keys_present(self):
|
||||
tracker = BudgetTracker(db_path=":memory:")
|
||||
summary = tracker.get_summary()
|
||||
assert "daily_usd" in summary
|
||||
assert "monthly_usd" in summary
|
||||
assert "daily_limit_usd" in summary
|
||||
assert "monthly_limit_usd" in summary
|
||||
assert "daily_ok" in summary
|
||||
assert "monthly_ok" in summary
|
||||
|
||||
def test_summary_daily_ok_true_on_empty(self):
|
||||
tracker = BudgetTracker(db_path=":memory:")
|
||||
summary = tracker.get_summary()
|
||||
assert summary["daily_ok"] is True
|
||||
assert summary["monthly_ok"] is True
|
||||
|
||||
def test_summary_daily_ok_false_when_exceeded(self):
|
||||
tracker = BudgetTracker(db_path=":memory:")
|
||||
tracker.record_spend("openai", "gpt-4o", cost_usd=999.0)
|
||||
summary = tracker.get_summary()
|
||||
assert summary["daily_ok"] is False
|
||||
|
||||
|
||||
# ── Singleton ─────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class TestGetBudgetTrackerSingleton:
|
||||
def test_returns_budget_tracker(self):
|
||||
import infrastructure.models.budget as bmod
|
||||
bmod._budget_tracker = None
|
||||
tracker = get_budget_tracker()
|
||||
assert isinstance(tracker, BudgetTracker)
|
||||
|
||||
def test_returns_same_instance(self):
|
||||
import infrastructure.models.budget as bmod
|
||||
bmod._budget_tracker = None
|
||||
t1 = get_budget_tracker()
|
||||
t2 = get_budget_tracker()
|
||||
assert t1 is t2
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user