feat: improvement proposal generator — ROI-based waste analysis (closes #168) #187

Rockachopa · 2026-04-15T14:47:32Z

Rockachopa commented

2026-04-15 14:47:32 +00:00

What changed

Added scripts/improvement_proposals.py — analyzes fleet session data to identify waste patterns and generates concrete improvement proposals with ROI estimates.

Waste Detection

Detects 4 pattern types:

Repeated errors — same error across 3+ sessions (normalizes paths/hashes for grouping)
Slow tools — tools averaging >5000ms latency across 5+ calls
Failed retries — tools called 3+ times consecutively (suggests retry loops)
Manual processes — sessions with 10+ tool calls but ≤3 unique tools (automatable)

Proposal Generation

Each proposal includes:

Title, problem statement, proposed solution
Weekly/monthly hours saved estimate
Implementation time estimate (with 1.5x overhead multiplier)
ROI break-even in weeks
Priority (critical/high/medium/low based on ROI)
Supporting evidence

Usage

# Markdown output (default)
python3 scripts/improvement_proposals.py --input analytics.json

# JSON output for programmatic consumption
python3 scripts/improvement_proposals.py --input analytics.json --format json

# Write to file with custom threshold
python3 scripts/improvement_proposals.py --input analytics.json --output proposals.md --threshold 5.0

Input Format

{"sessions": [
  {"session_id": "abc123", "repo": "hermes-agent", "duration_minutes": 30,
   "errors": [{"message": "push timeout"}],
   "tool_calls": [{"tool": "git_push", "latency_ms": 8000}]}
]}

Tests

15 tests covering: empty input, clean sessions, repeated error detection (incl. threshold), slow tool detection, failed retry detection, manual process detection, proposal generation, ROI validation, sorting, markdown/JSON output, error normalization, CLI integration.

All existing tests pass (8 + 15 = 23 total).

Closes #168

## What changed Added `scripts/improvement_proposals.py` — analyzes fleet session data to identify waste patterns and generates concrete improvement proposals with ROI estimates. ### Waste Detection Detects 4 pattern types: - **Repeated errors** — same error across 3+ sessions (normalizes paths/hashes for grouping) - **Slow tools** — tools averaging >5000ms latency across 5+ calls - **Failed retries** — tools called 3+ times consecutively (suggests retry loops) - **Manual processes** — sessions with 10+ tool calls but ≤3 unique tools (automatable) ### Proposal Generation Each proposal includes: - Title, problem statement, proposed solution - Weekly/monthly hours saved estimate - Implementation time estimate (with 1.5x overhead multiplier) - ROI break-even in weeks - Priority (critical/high/medium/low based on ROI) - Supporting evidence ### Usage ```bash # Markdown output (default) python3 scripts/improvement_proposals.py --input analytics.json # JSON output for programmatic consumption python3 scripts/improvement_proposals.py --input analytics.json --format json # Write to file with custom threshold python3 scripts/improvement_proposals.py --input analytics.json --output proposals.md --threshold 5.0 ``` ### Input Format ```json {"sessions": [ {"session_id": "abc123", "repo": "hermes-agent", "duration_minutes": 30, "errors": [{"message": "push timeout"}], "tool_calls": [{"tool": "git_push", "latency_ms": 8000}]} ]} ``` ### Tests 15 tests covering: empty input, clean sessions, repeated error detection (incl. threshold), slow tool detection, failed retry detection, manual process detection, proposal generation, ROI validation, sorting, markdown/JSON output, error normalization, CLI integration. All existing tests pass (8 + 15 = 23 total). Closes #168

Rockachopa added 2 commits 2026-04-15 14:47:33 +00:00

feat: add improvement proposal generator (#168 ) 1d47665dd4

test: add improvement proposal generator tests (#168 ) f1175df79d

Timmy approved these changes 2026-04-15 16:33:15 +00:00

Timmy left a comment

LGTM. Improvement proposal generator with ROI-based waste analysis. Good pattern detection for repeated errors, manual processes, slow tools, and failed retries. ROI calculation (weeks to break even) is a useful decision metric. Approved.