Timmy_Foundation/timmy-home
15
0
Fork 0
Code Issues 214 Pull Requests 50 Actions Packages Projects Releases Wiki Activity

Merge pull request 'Cut over stale ops docs to current workflow' (#399) from codex/workflow-docs-cutover into main

Browse Source
This commit was merged in pull request #399.
This commit is contained in:
Allegro
2026-04-04 22:25:57 +00:00
parent dba3e90893 4effb5a20e
commit 04213924d0
4 changed files with 254 additions and 566 deletions
Download Patch File Download Diff File Expand all files Collapse all files

236
docs/DEPLOYMENT_CHECKLIST.md
View File

@@ -1,197 +1,87 @@
# Uni-Wizard v4 — Deployment Checklist # Hermes Sidecar Deployment Checklist
## Pre-Deployment Updated: April 4, 2026
- [ ] VPS provisioned (Ubuntu 22.04 LTS recommended) This checklist is for the current local-first Timmy stack, not the archived `uni-wizard` deployment path.
- [ ] SSH access configured
- [ ] Firewall rules set (ports 22, 80, 443, 3000, 8643)
- [ ] Domain/DNS configured (optional)
- [ ] SSL certificates ready (optional)
## Base System ## Base Assumptions
- [ ] Update system packages - Hermes is already installed and runnable locally.
- `timmy-config` is the sidecar repo applied onto `~/.hermes`.
- `timmy-home` is the workspace repo living under `~/.timmy`.
- Local inference is reachable through the active provider surface Timmy is using.
## Repo Setup
- [ ] Clone `timmy-home` to `~/.timmy`
- [ ] Clone `timmy-config` to `~/.timmy/timmy-config`
- [ ] Confirm both repos are on the intended branch
## Sidecar Deploy
- [ ] Run:
```bash ```bash
sudo apt update && sudo apt upgrade -y cd ~/.timmy/timmy-config
./deploy.sh
``` ```
- [ ] Install base dependencies - [ ] Confirm `~/.hermes/config.yaml` matches the expected overlay
```bash - [ ] Confirm `SOUL.md` and sidecar config are in place
sudo apt install -y python3 python3-pip python3-venv sqlite3 curl git
```
- [ ] Create timmy user
```bash
sudo useradd -m -s /bin/bash timmy
```
- [ ] Configure sudo access (if needed)
## Gitea Setup ## Hermes Readiness
- [ ] Gitea installed and running - [ ] Hermes CLI works from the expected Python environment
- [ ] Repository created: `Timmy_Foundation/timmy-home` - [ ] Gateway is reachable
- [ ] API token generated - [ ] Sessions are being recorded under `~/.hermes/sessions`
- [ ] Webhooks configured (optional) - [ ] `model_health.json` updates successfully
- [ ] Test API access
```bash
curl -H "Authorization: token TOKEN" http://localhost:3000/api/v1/user
```
## Uni-Wizard Installation ## Workflow Tooling
- [ ] Clone repository - [ ] `~/.hermes/bin/ops-panel.sh` runs
```bash - [ ] `~/.hermes/bin/ops-gitea.sh` runs
sudo -u timmy git clone http://143.198.27.163:3000/Timmy_Foundation/timmy-home.git /opt/timmy/repo - [ ] `~/.hermes/bin/ops-helpers.sh` can be sourced
``` - [ ] `~/.hermes/bin/pipeline-freshness.sh` runs
- [ ] Run setup script - [ ] `~/.hermes/bin/timmy-dashboard` runs
```bash
sudo ./scripts/setup-uni-wizard.sh
```
- [ ] Verify installation
```bash
/opt/timmy/venv/bin/python -c "from uni_wizard import Harness; print('OK')"
```
## Configuration ## Heartbeat and Briefings
- [ ] Edit config file - [ ] `~/.timmy/heartbeat/last_tick.json` is updating
```bash - [ ] daily heartbeat logs are being appended
sudo nano /opt/timmy/config/uni-wizard.yaml - [ ] morning briefings are being generated if scheduled
```
- [ ] Set Gitea API token
- [ ] Configure house identity
- [ ] Set log level (INFO for production)
- [ ] Verify config syntax
```bash
/opt/timmy/venv/bin/python -c "import yaml; yaml.safe_load(open('/opt/timmy/config/uni-wizard.yaml'))"
```
## LLM Setup (if using local inference) ## Archive Pipeline
- [ ] llama.cpp installed - [ ] `~/.timmy/twitter-archive/PROJECT.md` exists
- [ ] Model downloaded (e.g., Hermes-4 14B) - [ ] raw archive location is configured locally
- [ ] Model placed in `/opt/timmy/models/` - [ ] extraction works without checking raw data into git
- [ ] llama-server configured - [ ] `checkpoint.json` advances after a batch
- [ ] Test inference - [ ] DPO artifacts land under `~/.timmy/twitter-archive/training/dpo/`
```bash - [ ] `pipeline-freshness.sh` does not show runaway lag
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "hermes4", "messages": [{"role": "user", "content": "Hello"}]}'
```
## Service Startup ## Gitea Workflow
- [ ] Start Uni-Wizard - [ ] Gitea token is present in a supported token path
```bash - [ ] review queue can be listed
sudo systemctl start uni-wizard - [ ] unassigned issues can be listed
``` - [ ] PR creation works from an agent branch
- [ ] Start health daemon
```bash
sudo systemctl start timmy-health
```
- [ ] Start task router
```bash
sudo systemctl start timmy-task-router
```
- [ ] Enable auto-start
```bash
sudo systemctl enable uni-wizard timmy-health timmy-task-router
```
## Verification ## Final Verification
- [ ] Check service status - [ ] local model smoke test succeeds
```bash - [ ] one archive batch completes successfully
sudo systemctl status uni-wizard - [ ] one PR can be opened and reviewed
``` - [ ] no stale loop-era scripts or docs are being treated as active truth
- [ ] View logs
```bash
sudo journalctl -u uni-wizard -f
```
- [ ] Test health endpoint
```bash
curl http://localhost:8082/health
```
- [ ] Test tool execution
```bash
/opt/timmy/venv/bin/uni-wizard execute system_info
```
- [ ] Verify Gitea polling
```bash
tail -f /opt/timmy/logs/task-router.log | grep "Polling"
```
## Syncthing Mesh (if using multiple VPS) ## Rollback
- [ ] Syncthing installed on all nodes If the sidecar deploy breaks behavior:
- [ ] Devices paired
- [ ] Folders shared
- `/opt/timmy/logs/`
- `/opt/timmy/data/`
- [ ] Test sync
```bash
touch /opt/timmy/logs/test && ssh other-vps "ls /opt/timmy/logs/test"
```
## Security
- [ ] Firewall configured
```bash
sudo ufw status
```
- [ ] Fail2ban installed (optional)
- [ ] Log rotation configured
```bash
sudo logrotate -d /etc/logrotate.d/uni-wizard
```
- [ ] Backup strategy in place
- [ ] Secrets not in git
```bash
grep -r "password\|token\|secret" /opt/timmy/repo/
```
## Monitoring
- [ ] Health checks responding
- [ ] Metrics being collected
- [ ] Alerts configured (optional)
- [ ] Log aggregation setup (optional)
## Post-Deployment
- [ ] Document any custom configuration
- [ ] Update runbooks
- [ ] Notify team
- [ ] Schedule first review (1 week)
## Rollback Plan
If deployment fails:
```bash ```bash
# Stop services cd ~/.timmy/timmy-config
sudo systemctl stop uni-wizard timmy-health timmy-task-router git status
git log --oneline -5
# Disable auto-start
sudo systemctl disable uni-wizard timmy-health timmy-task-router
# Restore from backup (if available)
# ...
# Or reset to clean state
sudo rm -rf /opt/timmy/
sudo userdel timmy
``` ```
## Success Criteria Then:
- restore the previous known-good sidecar commit
- [ ] All services running (`systemctl is-active` returns "active") - redeploy
- [ ] Health endpoint returns 200 - confirm Hermes health, heartbeat, and pipeline freshness again
- [ ] Can execute tools via CLI
- [ ] Gitea integration working (issues being polled)
- [ ] Logs being written without errors
- [ ] No critical errors in first 24 hours
---
**Deployed by:** _______________
**Date:** _______________
**VPS:** _______________

177
docs/OPERATIONS_DASHBOARD.md
View File

@@ -1,129 +1,112 @@
# Timmy Operations Dashboard # Timmy Operations Dashboard
**Generated:** March 30, 2026 Updated: April 4, 2026
**Generated by:** Allegro (Tempo-and-Dispatch) Purpose: a current-state reference for how the system is actually operated now.
--- This is no longer a `uni-wizard` dashboard.
The active architecture is:
- Timmy local workspace in `~/.timmy`
- Hermes harness in `~/.hermes`
- `timmy-config` as the identity and orchestration sidecar
- Gitea as the review and coordination surface
## 🎯 Current Sprint Status ## Core Jobs
### Open Issues by Priority Everything should map to one of these:
- Heartbeat: perceive, reflect, remember, decide, act, learn
- Harness: local models, Hermes sessions, tools, memory, training loop
- Portal Interface: the game/world-facing layer
| Priority | Count | Issues | ## Current Operating Surfaces
|----------|-------|--------|
| P0 (Critical) | 0 | — |
| P1 (High) | 3 | #99, #103, #94 |
| P2 (Medium) | 8 | #101, #97, #95, #93, #92, #91, #90, #87 |
| P3 (Low) | 6 | #86, #85, #84, #83, #72, others |
### Issue #94 Epic: Grand Timmy — The Uniwizard ### Local Paths
**Status:** In Progress - Timmy workspace: `~/.timmy`
**Completion:** ~40% - Timmy config repo: `~/.timmy/timmy-config`
- Hermes home: `~/.hermes`
- Twitter archive workspace: `~/.timmy/twitter-archive`
#### Completed ### Review Surface
- ✅ Uni-Wizard v4 architecture (4-pass evolution)
- ✅ Three-House separation (Timmy/Ezra/Bezalel)
- ✅ Self-improving intelligence engine
- ✅ Pattern database and adaptive policies
- ✅ Hermes bridge for telemetry
#### In Progress - Major changes go through PRs
- 🔄 Backend registry (#95) - Timmy is the principal reviewer for governing and sensitive changes
- 🔄 Caching layer (#103) - Allegro is the review and dispatch partner for queue hygiene, routing, and tempo
- 🔄 Wizard dissolution (#99)
#### Pending ### Workflow Scripts
- ⏳ RAG pipeline (#93)
- ⏳ Telemetry dashboard (#91)
- ⏳ Auto-grading (#92)
- ⏳ Evennia world shell (#83, #84)
--- - `~/.hermes/bin/ops-panel.sh`
- `~/.hermes/bin/ops-gitea.sh`
- `~/.hermes/bin/ops-helpers.sh`
- `~/.hermes/bin/pipeline-freshness.sh`
- `~/.hermes/bin/timmy-dashboard`
## 🏛️ House Assignments ## Daily Health Signals
| House | Status | Current Work | These are the signals that matter most:
|-------|--------|--------------| - Hermes gateway reachable
| **Timmy** | 🟢 Active | Local sovereign, reviewing PRs | - local inference surface responding
| **Ezra** | 🟢 Active | Research on LLM routing (#101) | - heartbeat ticks continuing
| **Bezalel** | 🟡 Standby | Awaiting implementation tasks | - Gitea reachable
| **Allegro** | 🟢 Active | Tempo-and-dispatch, Gitea bridge | - review queue not backing up
- session export / DPO freshness not lagging
- Twitter archive pipeline checkpoint advancing
--- ## Current Team Shape
## 📊 System Health ### Direction and Review
### VPS Fleet Status - Timmy: sovereignty, architecture, release judgment
- Allegro: dispatch, queue hygiene, Gitea bridge
| Host | IP | Role | Status | ### Research and Memory
|------|-----|------|--------|
| Allegro | 143.198.27.163 | Tempo-and-Dispatch | 🟢 Online |
| Ezra | TBD | Archivist/Research | ⚪ Not deployed |
| Bezalel | TBD | Artificer/Builder | ⚪ Not deployed |
### Services - Perplexity: research triage, integration evaluation
- Ezra: archival memory, RCA, onboarding doctrine
- KimiClaw: long-context reading and synthesis
| Service | Status | Notes | ### Execution
|---------|--------|-------|
| Gitea | 🟢 Running | 19 open issues |
| Hermes | 🟡 Configured | Awaiting model setup |
| Overnight Loop | 🔴 Stopped | Issue #72 reported |
| Uni-Wizard | 🟢 Ready | PR created |
--- - Codex Agent: workflow hardening, cleanup, migration verification
- Groq: fast bounded implementation
- Manus: moderate-scope follow-through
- Claude: hard refactors and deep implementation
- Gemini: frontier architecture and long-range design
- Grok: adversarial review and edge cases
## 🔄 Recent Activity ## Recommended Checks
### Last 24 Hours ### Start of Day
1. **Uni-Wizard v4 Completed** — Four-pass architecture evolution 1. Open the review queue and unassigned queue.
2. **PR Created** — feature/uni-wizard-v4-production 2. Check `pipeline-freshness.sh`.
3. **Allegro Lane Narrowed** — Focused on Gitea/Hermes bridge 3. Check the latest heartbeat tick.
4. **Issue #72 Reported** — Overnight loop not running 4. Check whether archive checkpoints and DPO artifacts advanced.
### Pending Actions ### Before Merging
1. Deploy Ezra VPS (archivist/research) 1. Confirm the PR is aligned with Heartbeat, Harness, or Portal.
2. Deploy Bezalel VPS (artificer/builder) 2. Confirm verification is real, not implied.
3. Start overnight loop 3. Confirm the change does not silently cross repo boundaries.
4. Configure Syncthing mesh 4. Confirm the change does not revive deprecated loop-era behavior.
5. Implement caching layer (#103)
--- ### End of Day
## 🎯 Recommendations 1. Check for duplicate issues and duplicate PR momentum.
2. Check whether Timmy is carrying routine queue work that Allegro should own.
3. Check whether builders were given work inside their real lanes.
### Immediate (Next 24h) ## Anti-Patterns
1. **Review Uni-Wizard v4 PR** — Ready for merge Avoid:
2. **Start Overnight Loop** — If operational approval given - treating archived dashboard-era issues as the live roadmap
3. **Deploy Ezra VPS** — For research tasks - using stale docs that assume `uni-wizard` is still the center
- routing work by habit instead of by current lane
- letting open loops multiply faster than they are reviewed
### Short-term (This Week) ## Success Condition
1. Implement caching layer (#103) — High impact The system is healthy when:
2. Build backend registry (#95) — Enables routing - work is routed cleanly
3. Create telemetry dashboard (#91) — Visibility - review is keeping pace
- private learning loops are producing artifacts
### Medium-term (This Month) - Timmy is spending time on sovereignty and judgment rather than queue untangling
1. Complete Grand Timmy epic (#94)
2. Dissolve wizard identities (#99)
3. Deploy Evennia world shell (#83, #84)
---
## 📈 Metrics
| Metric | Current | Target |
|--------|---------|--------|
| Issues Open | 19 | < 10 |
| PRs Open | 1 | — |
| VPS Online | 1/3 | 3/3 |
| Loop Cycles | 0 | 100/day |
---
*Dashboard updated: March 30, 2026*
*Next update: March 31, 2026*

255
docs/QUICK_REFERENCE.md
View File

@@ -1,220 +1,89 @@
# Uni-Wizard v4 — Quick Reference # Timmy Workflow Quick Reference
## Installation Updated: April 4, 2026
## What Lives Where
- `~/.timmy`: Timmy's workspace, lived data, heartbeat, archive artifacts
- `~/.timmy/timmy-config`: Timmy's identity and orchestration sidecar repo
- `~/.hermes`: Hermes harness, sessions, config overlay, helper scripts
## Most Useful Commands
### Workflow Status
```bash ```bash
# Run setup script ~/.hermes/bin/ops-panel.sh
sudo ./scripts/setup-uni-wizard.sh ~/.hermes/bin/ops-gitea.sh
~/.hermes/bin/timmy-dashboard
# Or manual install
cd uni-wizard/v4
pip install -e .
``` ```
## Basic Usage ### Workflow Helpers
```python
from uni_wizard import Harness, House, Mode
# Create harness
harness = Harness(house=House.TIMMY, mode=Mode.INTELLIGENT)
# Execute tool
result = harness.execute("git_status", repo_path="/path/to/repo")
# Check prediction
print(f"Predicted success: {result.provenance.prediction:.0%}")
# Get result
if result.success:
print(result.data)
else:
print(f"Error: {result.error}")
```
## Command Line
```bash ```bash
# Simple execution source ~/.hermes/bin/ops-helpers.sh
uni-wizard execute git_status --repo-path /path ops-help
ops-review-queue
# With specific house ops-unassigned all
uni-wizard execute git_status --house ezra --mode intelligent ops-queue codex-agent all
# Batch execution
uni-wizard batch tasks.json
# Check health
uni-wizard health
# View stats
uni-wizard stats
``` ```
## Houses ### Pipeline Freshness
| House | Role | Best For |
|-------|------|----------|
| `House.TIMMY` | Sovereign | Final decisions, critical ops |
| `House.EZRA` | Archivist | Reading, analysis, documentation |
| `House.BEZALEL` | Artificer | Building, testing, implementation |
| `House.ALLEGRO` | Dispatch | Routing, connectivity, tempo |
## Modes
| Mode | Use When | Features |
|------|----------|----------|
| `Mode.SIMPLE` | Scripts, quick tasks | Direct execution, no overhead |
| `Mode.INTELLIGENT` | Production work | Predictions, learning, adaptation |
| `Mode.SOVEREIGN` | Critical decisions | Full provenance, approval gates |
## Common Tasks
### Check System Status
```python
result = harness.execute("system_info")
print(result.data)
```
### Git Operations
```python
# Status
result = harness.execute("git_status", repo_path="/path")
# Log
result = harness.execute("git_log", repo_path="/path", max_count=10)
# Pull
result = harness.execute("git_pull", repo_path="/path")
```
### Health Check
```python
result = harness.execute("health_check")
print(f"Status: {result.data['status']}")
```
### Batch Operations
```python
tasks = [
{"tool": "git_status", "params": {"repo_path": "/path1"}},
{"tool": "git_status", "params": {"repo_path": "/path2"}},
{"tool": "system_info", "params": {}}
]
results = harness.execute_batch(tasks)
```
## Service Management
```bash ```bash
# Start services ~/.hermes/bin/pipeline-freshness.sh
sudo systemctl start uni-wizard
sudo systemctl start timmy-health
sudo systemctl start timmy-task-router
# Check status
sudo systemctl status uni-wizard
# View logs
sudo journalctl -u uni-wizard -f
tail -f /opt/timmy/logs/uni-wizard.log
# Restart
sudo systemctl restart uni-wizard
``` ```
## Troubleshooting ### Archive Pipeline
### Service Won't Start
```bash ```bash
# Check logs python3 - <<'PY'
journalctl -u uni-wizard -n 50 import json, sys
sys.path.insert(0, '/Users/apayne/.timmy/timmy-config')
# Verify config from tasks import _archive_pipeline_health_impl
cat /opt/timmy/config/uni-wizard.yaml print(json.dumps(_archive_pipeline_health_impl(), indent=2))
PY
# Test manually
python -m uni_wizard health
``` ```
### No Predictions ```bash
- Check pattern database exists: `ls /opt/timmy/data/patterns.db` python3 - <<'PY'
- Verify learning is enabled in config import json, sys
- Run a few tasks to build patterns sys.path.insert(0, '/Users/apayne/.timmy/timmy-config')
from tasks import _know_thy_father_impl
### Gitea Integration Failing print(json.dumps(_know_thy_father_impl(), indent=2))
- Verify API token in config PY
- Check Gitea URL is accessible
- Test: `curl http://143.198.27.163:3000/api/v1/version`
## Configuration
Location: `/opt/timmy/config/uni-wizard.yaml`
```yaml
house: timmy
mode: intelligent
enable_learning: true
pattern_db: /opt/timmy/data/patterns.db
log_level: INFO
gitea:
url: http://143.198.27.163:3000
token: YOUR_TOKEN_HERE
poll_interval: 300
hermes:
stream_enabled: true
db_path: /root/.hermes/state.db
``` ```
## API Reference ### Manual Dispatch Prompt
### Harness Methods ```bash
~/.hermes/bin/agent-dispatch.sh groq 542 Timmy_Foundation/the-nexus
```python
# Execute single tool
harness.execute(tool_name, **params) -> ExecutionResult
# Execute async
await harness.execute_async(tool_name, **params) -> ExecutionResult
# Execute batch
harness.execute_batch(tasks) -> List[ExecutionResult]
# Get prediction
harness.predict(tool_name, params) -> Prediction
# Get stats
harness.get_stats() -> Dict
# Get patterns
harness.get_patterns() -> Dict
``` ```
### ExecutionResult Fields ## Best Files to Check
```python ### Operational State
result.success # bool
result.data # Any
result.error # Optional[str]
result.provenance # Provenance
result.suggestions # List[str]
```
### Provenance Fields - `~/.timmy/heartbeat/last_tick.json`
- `~/.hermes/model_health.json`
- `~/.timmy/twitter-archive/checkpoint.json`
- `~/.timmy/twitter-archive/metrics/progress.json`
```python ### Archive Feedback
provenance.house # str
provenance.tool # str
provenance.mode # str
provenance.prediction # float
provenance.execution_time_ms # float
provenance.input_hash # str
provenance.output_hash # str
```
--- - `~/.timmy/twitter-archive/notes/`
- `~/.timmy/twitter-archive/knowledge/profile.json`
- `~/.timmy/twitter-archive/training/dpo/`
*For full documentation, see ARCHITECTURE.md* ### Review and Queue
- Gitea PR queue
- Gitea unassigned issues
- Timmy/Allegro assigned review queue
## Rules of Thumb
- If it changes identity or orchestration, review it carefully in `timmy-config`.
- If it changes lived outputs or training inputs, it probably belongs in `timmy-home`.
- If it only “sounds right” but is not proven by runtime state, it is not verified.
- If a change is major, package it as a PR for Timmy review.

152
docs/SCORECARD.md
View File

@@ -1,125 +1,71 @@
# Scorecard Generator Documentation # Workflow Scorecard
## Overview Updated: April 4, 2026
The Scorecard Generator analyzes overnight loop JSONL data and produces comprehensive reports with statistics, trends, and recommendations. The old overnight `uni-wizard` scorecard is no longer the primary operational metric.
The current scorecard should measure whether Timmy's real workflow is healthy.
## Usage ## What To Score
### Basic Usage ### Queue Health
```bash - unassigned issue count
# Generate scorecard from default input directory - PRs waiting on Timmy or Allegro review
python uni-wizard/scripts/generate_scorecard.py - overloaded assignees
- duplicate issue / duplicate PR pressure
# Specify custom input/output directories ### Runtime Health
python uni-wizard/scripts/generate_scorecard.py \
--input ~/shared/overnight-loop \
--output ~/timmy/reports
```
### Cron Setup - Hermes gateway reachable
- local provider responding
- latest heartbeat tick present
- model health reporting accurately
```bash ### Learning Loop Health
# Generate scorecard every morning at 6 AM
0 6 * * * /root/timmy/venv/bin/python /root/timmy/uni-wizard/scripts/generate_scorecard.py
```
## Input Format - archive checkpoint advancing
- notes and knowledge artifacts being emitted
- DPO files growing
- freshness lag between sessions and exports
JSONL files in `~/shared/overnight-loop/*.jsonl`: ## Suggested Daily Questions
```json 1. Did review keep pace with execution today?
{"task": "read-soul", "status": "pass", "duration_s": 19.7, "timestamp": "2026-03-29T21:54:12Z"} 2. Did any builder receive work outside their lane?
{"task": "check-health", "status": "fail", "duration_s": 5.2, "error": "timeout", "timestamp": "2026-03-29T22:15:33Z"} 3. Did Timmy spend time on judgment rather than routine queue cleanup?
``` 4. Did the private learning pipeline produce usable artifacts?
5. Did any stale doc, helper, or default try to pull the system back into old habits?
Fields: ## Useful Inputs
- `task`: Task identifier
- `status`: "pass" or "fail"
- `duration_s`: Execution time in seconds
- `timestamp`: ISO 8601 timestamp
- `error`: Error message (for failed tasks)
## Output - `~/.timmy/heartbeat/ticks_YYYYMMDD.jsonl`
- `~/.timmy/metrics/local_YYYYMMDD.jsonl`
- `~/.timmy/twitter-archive/checkpoint.json`
- `~/.timmy/twitter-archive/metrics/progress.json`
- Gitea open PR queue
- Gitea unassigned issue queue
### JSON Report ## Suggested Ratings
`~/timmy/reports/scorecard_YYYYMMDD.json`: ### Queue Discipline
```json - Strong: review and dispatch are keeping up, little duplicate churn
{ - Mixed: queue moves, but ambiguity or duplication is increasing
"generated_at": "2026-03-30T06:00:00Z", - Weak: review is backlogged or agents are being misrouted
"summary": {
"total_tasks": 100,
"passed": 95,
"failed": 5,
"pass_rate": 95.0,
"duration_stats": {
"avg": 12.5,
"median": 10.2,
"p95": 45.0,
"min": 1.2,
"max": 120.5
}
},
"by_task": {...},
"by_hour": {...},
"errors": {...},
"recommendations": [...]
}
```
### Markdown Report ### Runtime Reliability
`~/timmy/reports/scorecard_YYYYMMDD.md`: - Strong: heartbeat, Hermes, and provider surfaces all healthy
- Mixed: intermittent downtime or weak health signals
- Weak: major surfaces untrusted or stale
- Executive summary with pass/fail counts ### Learning Throughput
- Duration statistics (avg, median, p95)
- Per-task breakdown with pass rates
- Hourly timeline showing performance trends
- Error analysis with frequency counts
- Actionable recommendations
## Report Interpretation - Strong: checkpoint advances, DPO output accumulates, eval gates are visible
- Mixed: some artifacts land, but freshness or checkpointing lags
- Weak: sessions occur without export, or learning artifacts stall
### Pass Rate Thresholds ## The Goal
| Pass Rate | Status | Action | The point of the scorecard is not to admire activity.
|-----------|--------|--------| The point is to tell whether the system is becoming more reviewable, more sovereign, and more capable of learning from lived work.
| 95%+ | ✅ Excellent | Continue current operations |
| 85-94% | ⚠️ Good | Monitor for degradation |
| 70-84% | ⚠️ Fair | Review failing tasks |
| <70% | ❌ Poor | Immediate investigation required |
### Duration Guidelines
| Duration | Assessment |
|----------|------------|
| <5s | Fast |
| 5-15s | Normal |
| 15-30s | Slow |
| >30s | Very slow - consider optimization |
## Troubleshooting
### No JSONL files found
```bash
# Check input directory
ls -la ~/shared/overnight-loop/
# Ensure Syncthing is syncing
systemctl status syncthing@root
```
### Malformed lines
The generator skips malformed lines with a warning. Check the JSONL files for syntax errors.
### Empty reports
If no data exists, verify:
1. Overnight loop is running and writing JSONL
2. File permissions allow reading
3. Input path is correct