126 lines
2.8 KiB
Markdown
126 lines
2.8 KiB
Markdown
|
# Scorecard Generator Documentation
|
||
|
|
|
||
|
|
## Overview
|
||
|
|
|
||
|
|
The Scorecard Generator analyzes overnight loop JSONL data and produces comprehensive reports with statistics, trends, and recommendations.
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
### Basic Usage
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Generate scorecard from default input directory
|
||
|
|
python uni-wizard/scripts/generate_scorecard.py
|
||
|
|
|
||
|
|
# Specify custom input/output directories
|
||
|
|
python uni-wizard/scripts/generate_scorecard.py \
|
||
|
|
--input ~/shared/overnight-loop \
|
||
|
|
--output ~/timmy/reports
|
||
|
|
```
|
||
|
|
|
||
|
|
### Cron Setup
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Generate scorecard every morning at 6 AM
|
||
|
|
0 6 * * * /root/timmy/venv/bin/python /root/timmy/uni-wizard/scripts/generate_scorecard.py
|
||
|
|
```
|
||
|
|
|
||
|
|
## Input Format
|
||
|
|
|
||
|
|
JSONL files in `~/shared/overnight-loop/*.jsonl`:
|
||
|
|
|
||
|
|
```json
|
||
|
|
{"task": "read-soul", "status": "pass", "duration_s": 19.7, "timestamp": "2026-03-29T21:54:12Z"}
|
||
|
|
{"task": "check-health", "status": "fail", "duration_s": 5.2, "error": "timeout", "timestamp": "2026-03-29T22:15:33Z"}
|
||
|
|
```
|
||
|
|
|
||
|
|
Fields:
|
||
|
|
- `task`: Task identifier
|
||
|
|
- `status`: "pass" or "fail"
|
||
|
|
- `duration_s`: Execution time in seconds
|
||
|
|
- `timestamp`: ISO 8601 timestamp
|
||
|
|
- `error`: Error message (for failed tasks)
|
||
|
|
|
||
|
|
## Output
|
||
|
|
|
||
|
|
### JSON Report
|
||
|
|
|
||
|
|
`~/timmy/reports/scorecard_YYYYMMDD.json`:
|
||
|
|
|
||
|
|
```json
|
||
|
|
{
|
||
|
|
"generated_at": "2026-03-30T06:00:00Z",
|
||
|
|
"summary": {
|
||
|
|
"total_tasks": 100,
|
||
|
|
"passed": 95,
|
||
|
|
"failed": 5,
|
||
|
|
"pass_rate": 95.0,
|
||
|
|
"duration_stats": {
|
||
|
|
"avg": 12.5,
|
||
|
|
"median": 10.2,
|
||
|
|
"p95": 45.0,
|
||
|
|
"min": 1.2,
|
||
|
|
"max": 120.5
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"by_task": {...},
|
||
|
|
"by_hour": {...},
|
||
|
|
"errors": {...},
|
||
|
|
"recommendations": [...]
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
### Markdown Report
|
||
|
|
|
||
|
|
`~/timmy/reports/scorecard_YYYYMMDD.md`:
|
||
|
|
|
||
|
|
- Executive summary with pass/fail counts
|
||
|
|
- Duration statistics (avg, median, p95)
|
||
|
|
- Per-task breakdown with pass rates
|
||
|
|
- Hourly timeline showing performance trends
|
||
|
|
- Error analysis with frequency counts
|
||
|
|
- Actionable recommendations
|
||
|
|
|
||
|
|
## Report Interpretation
|
||
|
|
|
||
|
|
### Pass Rate Thresholds
|
||
|
|
|
||
|
|
| Pass Rate | Status | Action |
|
||
|
|
|-----------|--------|--------|
|
||
|
|
| 95%+ | ✅ Excellent | Continue current operations |
|
||
|
|
| 85-94% | ⚠️ Good | Monitor for degradation |
|
||
|
|
| 70-84% | ⚠️ Fair | Review failing tasks |
|
||
|
|
| <70% | ❌ Poor | Immediate investigation required |
|
||
|
|
|
||
|
|
### Duration Guidelines
|
||
|
|
|
||
|
|
| Duration | Assessment |
|
||
|
|
|----------|------------|
|
||
|
|
| <5s | Fast |
|
||
|
|
| 5-15s | Normal |
|
||
|
|
| 15-30s | Slow |
|
||
|
|
| >30s | Very slow - consider optimization |
|
||
|
|
|
||
|
|
## Troubleshooting
|
||
|
|
|
||
|
|
### No JSONL files found
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Check input directory
|
||
|
|
ls -la ~/shared/overnight-loop/
|
||
|
|
|
||
|
|
# Ensure Syncthing is syncing
|
||
|
|
systemctl status syncthing@root
|
||
|
|
```
|
||
|
|
|
||
|
|
### Malformed lines
|
||
|
|
|
||
|
|
The generator skips malformed lines with a warning. Check the JSONL files for syntax errors.
|
||
|
|
|
||
|
|
### Empty reports
|
||
|
|
|
||
|
|
If no data exists, verify:
|
||
|
|
1. Overnight loop is running and writing JSONL
|
||
|
|
2. File permissions allow reading
|
||
|
|
3. Input path is correct
|