Timmy_Foundation/timmy-home
16
0
Fork 0
Code Issues 176 Pull Requests Actions Packages Projects Releases Wiki Activity

Cut over stale ops docs to current workflow

Browse Source
This commit is contained in:
Alexander Whitestone
2026-04-04 15:21:29 -04:00
parent d716800ea9
commit 4effb5a20e
4 changed files with 254 additions and 566 deletions
Download Patch File Download Diff File Expand all files Collapse all files

236
docs/DEPLOYMENT_CHECKLIST.md
View File

@@ -1,197 +1,87 @@
# Uni-Wizard v4 — Deployment Checklist
# Hermes Sidecar Deployment Checklist
## Pre-Deployment
Updated: April 4, 2026
- [ ] VPS provisioned (Ubuntu 22.04 LTS recommended)
- [ ] SSH access configured
- [ ] Firewall rules set (ports 22, 80, 443, 3000, 8643)
- [ ] Domain/DNS configured (optional)
- [ ] SSL certificates ready (optional)
This checklist is for the current local-first Timmy stack, not the archived `uni-wizard` deployment path.
## Base System
## Base Assumptions
- [ ] Update system packages
- Hermes is already installed and runnable locally.
- `timmy-config` is the sidecar repo applied onto `~/.hermes`.
- `timmy-home` is the workspace repo living under `~/.timmy`.
- Local inference is reachable through the active provider surface Timmy is using.
## Repo Setup
- [ ] Clone `timmy-home` to `~/.timmy`
- [ ] Clone `timmy-config` to `~/.timmy/timmy-config`
- [ ] Confirm both repos are on the intended branch
## Sidecar Deploy
- [ ] Run:
```bash
sudo apt update && sudo apt upgrade -y
cd ~/.timmy/timmy-config
./deploy.sh
```
- [ ] Install base dependencies
```bash
sudo apt install -y python3 python3-pip python3-venv sqlite3 curl git
```
- [ ] Create timmy user
```bash
sudo useradd -m -s /bin/bash timmy
```
- [ ] Configure sudo access (if needed)
- [ ] Confirm `~/.hermes/config.yaml` matches the expected overlay
- [ ] Confirm `SOUL.md` and sidecar config are in place
## Gitea Setup
## Hermes Readiness
- [ ] Gitea installed and running
- [ ] Repository created: `Timmy_Foundation/timmy-home`
- [ ] API token generated
- [ ] Webhooks configured (optional)
- [ ] Test API access
```bash
curl -H "Authorization: token TOKEN" http://localhost:3000/api/v1/user
```
- [ ] Hermes CLI works from the expected Python environment
- [ ] Gateway is reachable
- [ ] Sessions are being recorded under `~/.hermes/sessions`
- [ ] `model_health.json` updates successfully
## Uni-Wizard Installation
## Workflow Tooling
- [ ] Clone repository
```bash
sudo -u timmy git clone http://143.198.27.163:3000/Timmy_Foundation/timmy-home.git /opt/timmy/repo
```
- [ ] Run setup script
```bash
sudo ./scripts/setup-uni-wizard.sh
```
- [ ] Verify installation
```bash
/opt/timmy/venv/bin/python -c "from uni_wizard import Harness; print('OK')"
```
- [ ] `~/.hermes/bin/ops-panel.sh` runs
- [ ] `~/.hermes/bin/ops-gitea.sh` runs
- [ ] `~/.hermes/bin/ops-helpers.sh` can be sourced
- [ ] `~/.hermes/bin/pipeline-freshness.sh` runs
- [ ] `~/.hermes/bin/timmy-dashboard` runs
## Configuration
## Heartbeat and Briefings
- [ ] Edit config file
```bash
sudo nano /opt/timmy/config/uni-wizard.yaml
```
- [ ] Set Gitea API token
- [ ] Configure house identity
- [ ] Set log level (INFO for production)
- [ ] Verify config syntax
```bash
/opt/timmy/venv/bin/python -c "import yaml; yaml.safe_load(open('/opt/timmy/config/uni-wizard.yaml'))"
```
- [ ] `~/.timmy/heartbeat/last_tick.json` is updating
- [ ] daily heartbeat logs are being appended
- [ ] morning briefings are being generated if scheduled
## LLM Setup (if using local inference)
## Archive Pipeline
- [ ] llama.cpp installed
- [ ] Model downloaded (e.g., Hermes-4 14B)
- [ ] Model placed in `/opt/timmy/models/`
- [ ] llama-server configured
- [ ] Test inference
```bash
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "hermes4", "messages": [{"role": "user", "content": "Hello"}]}'
```
- [ ] `~/.timmy/twitter-archive/PROJECT.md` exists
- [ ] raw archive location is configured locally
- [ ] extraction works without checking raw data into git
- [ ] `checkpoint.json` advances after a batch
- [ ] DPO artifacts land under `~/.timmy/twitter-archive/training/dpo/`
- [ ] `pipeline-freshness.sh` does not show runaway lag
## Service Startup
## Gitea Workflow
- [ ] Start Uni-Wizard
```bash
sudo systemctl start uni-wizard
```
- [ ] Start health daemon
```bash
sudo systemctl start timmy-health
```
- [ ] Start task router
```bash
sudo systemctl start timmy-task-router
```
- [ ] Enable auto-start
```bash
sudo systemctl enable uni-wizard timmy-health timmy-task-router
```
- [ ] Gitea token is present in a supported token path
- [ ] review queue can be listed
- [ ] unassigned issues can be listed
- [ ] PR creation works from an agent branch
## Verification
## Final Verification
- [ ] Check service status
```bash
sudo systemctl status uni-wizard
```
- [ ] View logs
```bash
sudo journalctl -u uni-wizard -f
```
- [ ] Test health endpoint
```bash
curl http://localhost:8082/health
```
- [ ] Test tool execution
```bash
/opt/timmy/venv/bin/uni-wizard execute system_info
```
- [ ] Verify Gitea polling
```bash
tail -f /opt/timmy/logs/task-router.log | grep "Polling"
```
- [ ] local model smoke test succeeds
- [ ] one archive batch completes successfully
- [ ] one PR can be opened and reviewed
- [ ] no stale loop-era scripts or docs are being treated as active truth
## Syncthing Mesh (if using multiple VPS)
## Rollback
- [ ] Syncthing installed on all nodes
- [ ] Devices paired
- [ ] Folders shared
- `/opt/timmy/logs/`
- `/opt/timmy/data/`
- [ ] Test sync
```bash
touch /opt/timmy/logs/test && ssh other-vps "ls /opt/timmy/logs/test"
```
## Security
- [ ] Firewall configured
```bash
sudo ufw status
```
- [ ] Fail2ban installed (optional)
- [ ] Log rotation configured
```bash
sudo logrotate -d /etc/logrotate.d/uni-wizard
```
- [ ] Backup strategy in place
- [ ] Secrets not in git
```bash
grep -r "password\|token\|secret" /opt/timmy/repo/
```
## Monitoring
- [ ] Health checks responding
- [ ] Metrics being collected
- [ ] Alerts configured (optional)
- [ ] Log aggregation setup (optional)
## Post-Deployment
- [ ] Document any custom configuration
- [ ] Update runbooks
- [ ] Notify team
- [ ] Schedule first review (1 week)
## Rollback Plan
If deployment fails:
If the sidecar deploy breaks behavior:
```bash
# Stop services
sudo systemctl stop uni-wizard timmy-health timmy-task-router
# Disable auto-start
sudo systemctl disable uni-wizard timmy-health timmy-task-router
# Restore from backup (if available)
# ...
# Or reset to clean state
sudo rm -rf /opt/timmy/
sudo userdel timmy
cd ~/.timmy/timmy-config
git status
git log --oneline -5
```
## Success Criteria
- [ ] All services running (`systemctl is-active` returns "active")
- [ ] Health endpoint returns 200
- [ ] Can execute tools via CLI
- [ ] Gitea integration working (issues being polled)
- [ ] Logs being written without errors
- [ ] No critical errors in first 24 hours
---
**Deployed by:** _______________
**Date:** _______________
**VPS:** _______________
Then:
- restore the previous known-good sidecar commit
- redeploy
- confirm Hermes health, heartbeat, and pipeline freshness again

177
docs/OPERATIONS_DASHBOARD.md
View File

@@ -1,129 +1,112 @@
# Timmy Operations Dashboard
**Generated:** March 30, 2026
**Generated by:** Allegro (Tempo-and-Dispatch)
Updated: April 4, 2026
Purpose: a current-state reference for how the system is actually operated now.
---
This is no longer a `uni-wizard` dashboard.
The active architecture is:
- Timmy local workspace in `~/.timmy`
- Hermes harness in `~/.hermes`
- `timmy-config` as the identity and orchestration sidecar
- Gitea as the review and coordination surface
## 🎯 Current Sprint Status
## Core Jobs
### Open Issues by Priority
Everything should map to one of these:
- Heartbeat: perceive, reflect, remember, decide, act, learn
- Harness: local models, Hermes sessions, tools, memory, training loop
- Portal Interface: the game/world-facing layer
| Priority | Count | Issues |
|----------|-------|--------|
| P0 (Critical) | 0 | — |
| P1 (High) | 3 | #99, #103, #94 |
| P2 (Medium) | 8 | #101, #97, #95, #93, #92, #91, #90, #87 |
| P3 (Low) | 6 | #86, #85, #84, #83, #72, others |
## Current Operating Surfaces
### Issue #94 Epic: Grand Timmy — The Uniwizard
### Local Paths
**Status:** In Progress
**Completion:** ~40%
- Timmy workspace: `~/.timmy`
- Timmy config repo: `~/.timmy/timmy-config`
- Hermes home: `~/.hermes`
- Twitter archive workspace: `~/.timmy/twitter-archive`
#### Completed
- ✅ Uni-Wizard v4 architecture (4-pass evolution)
- ✅ Three-House separation (Timmy/Ezra/Bezalel)
- ✅ Self-improving intelligence engine
- ✅ Pattern database and adaptive policies
- ✅ Hermes bridge for telemetry
### Review Surface
#### In Progress
- 🔄 Backend registry (#95)
- 🔄 Caching layer (#103)
- 🔄 Wizard dissolution (#99)
- Major changes go through PRs
- Timmy is the principal reviewer for governing and sensitive changes
- Allegro is the review and dispatch partner for queue hygiene, routing, and tempo
#### Pending
- ⏳ RAG pipeline (#93)
- ⏳ Telemetry dashboard (#91)
- ⏳ Auto-grading (#92)
- ⏳ Evennia world shell (#83, #84)
### Workflow Scripts
---
- `~/.hermes/bin/ops-panel.sh`
- `~/.hermes/bin/ops-gitea.sh`
- `~/.hermes/bin/ops-helpers.sh`
- `~/.hermes/bin/pipeline-freshness.sh`
- `~/.hermes/bin/timmy-dashboard`
## 🏛️ House Assignments
## Daily Health Signals
| House | Status | Current Work |
|-------|--------|--------------|
| **Timmy** | 🟢 Active | Local sovereign, reviewing PRs |
| **Ezra** | 🟢 Active | Research on LLM routing (#101) |
| **Bezalel** | 🟡 Standby | Awaiting implementation tasks |
| **Allegro** | 🟢 Active | Tempo-and-dispatch, Gitea bridge |
These are the signals that matter most:
- Hermes gateway reachable
- local inference surface responding
- heartbeat ticks continuing
- Gitea reachable
- review queue not backing up
- session export / DPO freshness not lagging
- Twitter archive pipeline checkpoint advancing
---
## Current Team Shape
## 📊 System Health
### Direction and Review
### VPS Fleet Status
- Timmy: sovereignty, architecture, release judgment
- Allegro: dispatch, queue hygiene, Gitea bridge
| Host | IP | Role | Status |
|------|-----|------|--------|
| Allegro | 143.198.27.163 | Tempo-and-Dispatch | 🟢 Online |
| Ezra | TBD | Archivist/Research | ⚪ Not deployed |
| Bezalel | TBD | Artificer/Builder | ⚪ Not deployed |
### Research and Memory
### Services
- Perplexity: research triage, integration evaluation
- Ezra: archival memory, RCA, onboarding doctrine
- KimiClaw: long-context reading and synthesis
| Service | Status | Notes |
|---------|--------|-------|
| Gitea | 🟢 Running | 19 open issues |
| Hermes | 🟡 Configured | Awaiting model setup |
| Overnight Loop | 🔴 Stopped | Issue #72 reported |
| Uni-Wizard | 🟢 Ready | PR created |
### Execution
---
- Codex Agent: workflow hardening, cleanup, migration verification
- Groq: fast bounded implementation
- Manus: moderate-scope follow-through
- Claude: hard refactors and deep implementation
- Gemini: frontier architecture and long-range design
- Grok: adversarial review and edge cases
## 🔄 Recent Activity
## Recommended Checks
### Last 24 Hours
### Start of Day
1. **Uni-Wizard v4 Completed** — Four-pass architecture evolution
2. **PR Created** — feature/uni-wizard-v4-production
3. **Allegro Lane Narrowed** — Focused on Gitea/Hermes bridge
4. **Issue #72 Reported** — Overnight loop not running
1. Open the review queue and unassigned queue.
2. Check `pipeline-freshness.sh`.
3. Check the latest heartbeat tick.
4. Check whether archive checkpoints and DPO artifacts advanced.
### Pending Actions
### Before Merging
1. Deploy Ezra VPS (archivist/research)
2. Deploy Bezalel VPS (artificer/builder)
3. Start overnight loop
4. Configure Syncthing mesh
5. Implement caching layer (#103)
1. Confirm the PR is aligned with Heartbeat, Harness, or Portal.
2. Confirm verification is real, not implied.
3. Confirm the change does not silently cross repo boundaries.
4. Confirm the change does not revive deprecated loop-era behavior.
---
### End of Day
## 🎯 Recommendations
1. Check for duplicate issues and duplicate PR momentum.
2. Check whether Timmy is carrying routine queue work that Allegro should own.
3. Check whether builders were given work inside their real lanes.
### Immediate (Next 24h)
## Anti-Patterns
1. **Review Uni-Wizard v4 PR** — Ready for merge
2. **Start Overnight Loop** — If operational approval given
3. **Deploy Ezra VPS** — For research tasks
Avoid:
- treating archived dashboard-era issues as the live roadmap
- using stale docs that assume `uni-wizard` is still the center
- routing work by habit instead of by current lane
- letting open loops multiply faster than they are reviewed
### Short-term (This Week)
## Success Condition
1. Implement caching layer (#103) — High impact
2. Build backend registry (#95) — Enables routing
3. Create telemetry dashboard (#91) — Visibility
### Medium-term (This Month)
1. Complete Grand Timmy epic (#94)
2. Dissolve wizard identities (#99)
3. Deploy Evennia world shell (#83, #84)
---
## 📈 Metrics
| Metric | Current | Target |
|--------|---------|--------|
| Issues Open | 19 | < 10 |
| PRs Open | 1 | — |
| VPS Online | 1/3 | 3/3 |
| Loop Cycles | 0 | 100/day |
---
*Dashboard updated: March 30, 2026*
*Next update: March 31, 2026*
The system is healthy when:
- work is routed cleanly
- review is keeping pace
- private learning loops are producing artifacts
- Timmy is spending time on sovereignty and judgment rather than queue untangling

255
docs/QUICK_REFERENCE.md
View File

@@ -1,220 +1,89 @@
# Uni-Wizard v4 — Quick Reference
# Timmy Workflow Quick Reference
## Installation
Updated: April 4, 2026
## What Lives Where
- `~/.timmy`: Timmy's workspace, lived data, heartbeat, archive artifacts
- `~/.timmy/timmy-config`: Timmy's identity and orchestration sidecar repo
- `~/.hermes`: Hermes harness, sessions, config overlay, helper scripts
## Most Useful Commands
### Workflow Status
```bash
# Run setup script
sudo ./scripts/setup-uni-wizard.sh
# Or manual install
cd uni-wizard/v4
pip install -e .
~/.hermes/bin/ops-panel.sh
~/.hermes/bin/ops-gitea.sh
~/.hermes/bin/timmy-dashboard
```
## Basic Usage
```python
from uni_wizard import Harness, House, Mode
# Create harness
harness = Harness(house=House.TIMMY, mode=Mode.INTELLIGENT)
# Execute tool
result = harness.execute("git_status", repo_path="/path/to/repo")
# Check prediction
print(f"Predicted success: {result.provenance.prediction:.0%}")
# Get result
if result.success:
print(result.data)
else:
print(f"Error: {result.error}")
```
## Command Line
### Workflow Helpers
```bash
# Simple execution
uni-wizard execute git_status --repo-path /path
# With specific house
uni-wizard execute git_status --house ezra --mode intelligent
# Batch execution
uni-wizard batch tasks.json
# Check health
uni-wizard health
# View stats
uni-wizard stats
source ~/.hermes/bin/ops-helpers.sh
ops-help
ops-review-queue
ops-unassigned all
ops-queue codex-agent all
```
## Houses
| House | Role | Best For |
|-------|------|----------|
| `House.TIMMY` | Sovereign | Final decisions, critical ops |
| `House.EZRA` | Archivist | Reading, analysis, documentation |
| `House.BEZALEL` | Artificer | Building, testing, implementation |
| `House.ALLEGRO` | Dispatch | Routing, connectivity, tempo |
## Modes
| Mode | Use When | Features |
|------|----------|----------|
| `Mode.SIMPLE` | Scripts, quick tasks | Direct execution, no overhead |
| `Mode.INTELLIGENT` | Production work | Predictions, learning, adaptation |
| `Mode.SOVEREIGN` | Critical decisions | Full provenance, approval gates |
## Common Tasks
### Check System Status
```python
result = harness.execute("system_info")
print(result.data)
```
### Git Operations
```python
# Status
result = harness.execute("git_status", repo_path="/path")
# Log
result = harness.execute("git_log", repo_path="/path", max_count=10)
# Pull
result = harness.execute("git_pull", repo_path="/path")
```
### Health Check
```python
result = harness.execute("health_check")
print(f"Status: {result.data['status']}")
```
### Batch Operations
```python
tasks = [
{"tool": "git_status", "params": {"repo_path": "/path1"}},
{"tool": "git_status", "params": {"repo_path": "/path2"}},
{"tool": "system_info", "params": {}}
]
results = harness.execute_batch(tasks)
```
## Service Management
### Pipeline Freshness
```bash
# Start services
sudo systemctl start uni-wizard
sudo systemctl start timmy-health
sudo systemctl start timmy-task-router
# Check status
sudo systemctl status uni-wizard
# View logs
sudo journalctl -u uni-wizard -f
tail -f /opt/timmy/logs/uni-wizard.log
# Restart
sudo systemctl restart uni-wizard
~/.hermes/bin/pipeline-freshness.sh
```
## Troubleshooting
### Archive Pipeline
### Service Won't Start
```bash
# Check logs
journalctl -u uni-wizard -n 50
# Verify config
cat /opt/timmy/config/uni-wizard.yaml
# Test manually
python -m uni_wizard health
python3 - <<'PY'
import json, sys
sys.path.insert(0, '/Users/apayne/.timmy/timmy-config')
from tasks import _archive_pipeline_health_impl
print(json.dumps(_archive_pipeline_health_impl(), indent=2))
PY
```
### No Predictions
- Check pattern database exists: `ls /opt/timmy/data/patterns.db`
- Verify learning is enabled in config
- Run a few tasks to build patterns
### Gitea Integration Failing
- Verify API token in config
- Check Gitea URL is accessible
- Test: `curl http://143.198.27.163:3000/api/v1/version`
## Configuration
Location: `/opt/timmy/config/uni-wizard.yaml`
```yaml
house: timmy
mode: intelligent
enable_learning: true
pattern_db: /opt/timmy/data/patterns.db
log_level: INFO
gitea:
url: http://143.198.27.163:3000
token: YOUR_TOKEN_HERE
poll_interval: 300
hermes:
stream_enabled: true
db_path: /root/.hermes/state.db
```bash
python3 - <<'PY'
import json, sys
sys.path.insert(0, '/Users/apayne/.timmy/timmy-config')
from tasks import _know_thy_father_impl
print(json.dumps(_know_thy_father_impl(), indent=2))
PY
```
## API Reference
### Manual Dispatch Prompt
### Harness Methods
```python
# Execute single tool
harness.execute(tool_name, **params) -> ExecutionResult
# Execute async
await harness.execute_async(tool_name, **params) -> ExecutionResult
# Execute batch
harness.execute_batch(tasks) -> List[ExecutionResult]
# Get prediction
harness.predict(tool_name, params) -> Prediction
# Get stats
harness.get_stats() -> Dict
# Get patterns
harness.get_patterns() -> Dict
```bash
~/.hermes/bin/agent-dispatch.sh groq 542 Timmy_Foundation/the-nexus
```
### ExecutionResult Fields
## Best Files to Check
```python
result.success # bool
result.data # Any
result.error # Optional[str]
result.provenance # Provenance
result.suggestions # List[str]
```
### Operational State
### Provenance Fields
- `~/.timmy/heartbeat/last_tick.json`
- `~/.hermes/model_health.json`
- `~/.timmy/twitter-archive/checkpoint.json`
- `~/.timmy/twitter-archive/metrics/progress.json`
```python
provenance.house # str
provenance.tool # str
provenance.mode # str
provenance.prediction # float
provenance.execution_time_ms # float
provenance.input_hash # str
provenance.output_hash # str
```
### Archive Feedback
---
- `~/.timmy/twitter-archive/notes/`
- `~/.timmy/twitter-archive/knowledge/profile.json`
- `~/.timmy/twitter-archive/training/dpo/`
*For full documentation, see ARCHITECTURE.md*
### Review and Queue
- Gitea PR queue
- Gitea unassigned issues
- Timmy/Allegro assigned review queue
## Rules of Thumb
- If it changes identity or orchestration, review it carefully in `timmy-config`.
- If it changes lived outputs or training inputs, it probably belongs in `timmy-home`.
- If it only “sounds right” but is not proven by runtime state, it is not verified.
- If a change is major, package it as a PR for Timmy review.

152
docs/SCORECARD.md
View File

@@ -1,125 +1,71 @@
# Scorecard Generator Documentation
# Workflow Scorecard
## Overview
Updated: April 4, 2026
The Scorecard Generator analyzes overnight loop JSONL data and produces comprehensive reports with statistics, trends, and recommendations.
The old overnight `uni-wizard` scorecard is no longer the primary operational metric.
The current scorecard should measure whether Timmy's real workflow is healthy.
## Usage
## What To Score
### Basic Usage
### Queue Health
```bash
# Generate scorecard from default input directory
python uni-wizard/scripts/generate_scorecard.py
- unassigned issue count
- PRs waiting on Timmy or Allegro review
- overloaded assignees
- duplicate issue / duplicate PR pressure
# Specify custom input/output directories
python uni-wizard/scripts/generate_scorecard.py \
--input ~/shared/overnight-loop \
--output ~/timmy/reports
```
### Runtime Health
### Cron Setup
- Hermes gateway reachable
- local provider responding
- latest heartbeat tick present
- model health reporting accurately
```bash
# Generate scorecard every morning at 6 AM
0 6 * * * /root/timmy/venv/bin/python /root/timmy/uni-wizard/scripts/generate_scorecard.py
```
### Learning Loop Health
## Input Format
- archive checkpoint advancing
- notes and knowledge artifacts being emitted
- DPO files growing
- freshness lag between sessions and exports
JSONL files in `~/shared/overnight-loop/*.jsonl`:
## Suggested Daily Questions
```json
{"task": "read-soul", "status": "pass", "duration_s": 19.7, "timestamp": "2026-03-29T21:54:12Z"}
{"task": "check-health", "status": "fail", "duration_s": 5.2, "error": "timeout", "timestamp": "2026-03-29T22:15:33Z"}
```
1. Did review keep pace with execution today?
2. Did any builder receive work outside their lane?
3. Did Timmy spend time on judgment rather than routine queue cleanup?
4. Did the private learning pipeline produce usable artifacts?
5. Did any stale doc, helper, or default try to pull the system back into old habits?
Fields:
- `task`: Task identifier
- `status`: "pass" or "fail"
- `duration_s`: Execution time in seconds
- `timestamp`: ISO 8601 timestamp
- `error`: Error message (for failed tasks)
## Useful Inputs
## Output
- `~/.timmy/heartbeat/ticks_YYYYMMDD.jsonl`
- `~/.timmy/metrics/local_YYYYMMDD.jsonl`
- `~/.timmy/twitter-archive/checkpoint.json`
- `~/.timmy/twitter-archive/metrics/progress.json`
- Gitea open PR queue
- Gitea unassigned issue queue
### JSON Report
## Suggested Ratings
`~/timmy/reports/scorecard_YYYYMMDD.json`:
### Queue Discipline
```json
{
"generated_at": "2026-03-30T06:00:00Z",
"summary": {
"total_tasks": 100,
"passed": 95,
"failed": 5,
"pass_rate": 95.0,
"duration_stats": {
"avg": 12.5,
"median": 10.2,
"p95": 45.0,
"min": 1.2,
"max": 120.5
}
},
"by_task": {...},
"by_hour": {...},
"errors": {...},
"recommendations": [...]
}
```
- Strong: review and dispatch are keeping up, little duplicate churn
- Mixed: queue moves, but ambiguity or duplication is increasing
- Weak: review is backlogged or agents are being misrouted
### Markdown Report
### Runtime Reliability
`~/timmy/reports/scorecard_YYYYMMDD.md`:
- Strong: heartbeat, Hermes, and provider surfaces all healthy
- Mixed: intermittent downtime or weak health signals
- Weak: major surfaces untrusted or stale
- Executive summary with pass/fail counts
- Duration statistics (avg, median, p95)
- Per-task breakdown with pass rates
- Hourly timeline showing performance trends
- Error analysis with frequency counts
- Actionable recommendations
### Learning Throughput
## Report Interpretation
- Strong: checkpoint advances, DPO output accumulates, eval gates are visible
- Mixed: some artifacts land, but freshness or checkpointing lags
- Weak: sessions occur without export, or learning artifacts stall
### Pass Rate Thresholds
## The Goal
| Pass Rate | Status | Action |
|-----------|--------|--------|
| 95%+ | ✅ Excellent | Continue current operations |
| 85-94% | ⚠️ Good | Monitor for degradation |
| 70-84% | ⚠️ Fair | Review failing tasks |
| <70% | ❌ Poor | Immediate investigation required |
### Duration Guidelines
| Duration | Assessment |
|----------|------------|
| <5s | Fast |
| 5-15s | Normal |
| 15-30s | Slow |
| >30s | Very slow - consider optimization |
## Troubleshooting
### No JSONL files found
```bash
# Check input directory
ls -la ~/shared/overnight-loop/
# Ensure Syncthing is syncing
systemctl status syncthing@root
```
### Malformed lines
The generator skips malformed lines with a warning. Check the JSONL files for syntax errors.
### Empty reports
If no data exists, verify:
1. Overnight loop is running and writing JSONL
2. File permissions allow reading
3. Input path is correct
The point of the scorecard is not to admire activity.
The point is to tell whether the system is becoming more reviewable, more sovereign, and more capable of learning from lived work.