51 lines
1.4 KiB
Markdown
51 lines
1.4 KiB
Markdown
|
# Nightly Pipeline Scheduler
|
||
|
|
|
||
|
|
Auto-starts batch pipelines when inference is available.
|
||
|
|
|
||
|
|
## What It Does
|
||
|
|
|
||
|
|
1. Checks inference provider health (OpenRouter, Ollama, RunPod)
|
||
|
|
2. Checks if it's off-peak hours (configurable, default: after 6PM)
|
||
|
|
3. Checks interactive session load (don't fight with live users)
|
||
|
|
4. Checks daily token budget (configurable limit)
|
||
|
|
5. Starts the highest-priority incomplete pipeline
|
||
|
|
|
||
|
|
## Pipeline Priority Order
|
||
|
|
|
||
|
|
| Priority | Pipeline | Deps | Max Tokens |
|
||
|
|
|----------|----------|------|------------|
|
||
|
|
| 1 | playground-factory | none | 100,000 |
|
||
|
|
| 2 | training-factory | none | 150,000 |
|
||
|
|
| 3 | knowledge-mine | training-factory running | 80,000 |
|
||
|
|
| 4 | adversary | knowledge-mine running | 50,000 |
|
||
|
|
| 5 | codebase-genome | none | 120,000 |
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Normal run (used by cron)
|
||
|
|
./scripts/nightly-pipeline-scheduler.sh
|
||
|
|
|
||
|
|
# Dry run (show what would start)
|
||
|
|
./scripts/nightly-pipeline-scheduler.sh --dry-run
|
||
|
|
|
||
|
|
# Status report
|
||
|
|
./scripts/nightly-pipeline-scheduler.sh --status
|
||
|
|
|
||
|
|
# Force start during peak hours
|
||
|
|
./scripts/nightly-pipeline-scheduler.sh --force
|
||
|
|
```
|
||
|
|
|
||
|
|
## Configuration
|
||
|
|
|
||
|
|
Set via environment variables:
|
||
|
|
- `PIPELINE_TOKEN_LIMIT`: Daily token budget (default: 500,000)
|
||
|
|
- `PIPELINE_PEAK_START`: Peak hours start (default: 9)
|
||
|
|
- `PIPELINE_PEAK_END`: Peak hours end (default: 18)
|
||
|
|
- `HERMES_HOME`: Hermes home directory (default: ~/.hermes)
|
||
|
|
|
||
|
|
## Cron
|
||
|
|
|
||
|
|
Runs every 30 minutes. Off-peak only (unless --force).
|
||
|
|
See `cron/pipeline-scheduler.yml`.
|