Timmy_Foundation/compounding-intelligence
15
0
Fork 0
Code Issues 173 Pull Requests 59 Actions Packages Projects Releases Wiki Activity

[Retro] Build batch harvester for parallel processing #18

New Issue
Open
opened 2026-04-14 15:15:24 +00:00 by Timmy · 0 comments
Timmy commented 2026-04-14 15:15:24 +00:00
Owner
Copy Link

Epic: #5 (Retroactive Harvest)

Task

Process the session backlog using mimo-swarm workers in parallel.

Architecture

sampler.py → priority queue → N workers (mimo-swarm) → harvester.py → knowledge store

Interface

python3 batch-harvest.py --workers 3 --limit 500
python3 batch-harvest.py --resume  # Resume from last checkpoint

Requirements

  • Checkpoint after each session (don't re-harvest on restart)
  • Rate-limit to avoid overwhelming local inference
  • Track: processed, failed, facts extracted, time elapsed
  • Deduplicate across parallel workers

Acceptance Criteria

  • Processes 500 sessions in <2 hours using 3 workers
  • Checkpoint survives restart
  • No duplicate facts from parallel workers
  • Progress report every 50 sessions
## Epic: #5 (Retroactive Harvest) ### Task Process the session backlog using mimo-swarm workers in parallel. ### Architecture ``` sampler.py → priority queue → N workers (mimo-swarm) → harvester.py → knowledge store ``` ### Interface ```bash python3 batch-harvest.py --workers 3 --limit 500 python3 batch-harvest.py --resume # Resume from last checkpoint ``` ### Requirements - Checkpoint after each session (don't re-harvest on restart) - Rate-limit to avoid overwhelming local inference - Track: processed, failed, facts extracted, time elapsed - Deduplicate across parallel workers ### Acceptance Criteria - [ ] Processes 500 sessions in <2 hours using 3 workers - [ ] Checkpoint survives restart - [ ] No duplicate facts from parallel workers - [ ] Progress report every 50 sessions
Timmy added the retroactivepipelinemilestone:4 labels 2026-04-14 15:15:24 +00:00
hermes was assigned by Rockachopa 2026-04-15 01:50:36 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acceptance-criteria
batch-pipeline
Token masterplan batch pipeline
 bootstrapper
Pre-session context injection
 epic
Epic-level issue
 harvester
Session knowledge extraction
 measurer
Compounding metrics
 milestone:1
Milestone 1: Foundation
 milestone:2
Milestone 2: Integration
 milestone:3
Milestone 3: Measurement
 milestone:4
Milestone 4: Retroactive
 pipeline
Pipeline/integration work
 pipeline
priority:high
priority:medium
retroactive
Processing existing sessions
 throughput-10x
throughput-10x label
 token-masterplan
token-masterplan label
No Label milestone:4 pipeline retroactive
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Rockachopa Timmy allegro antigravity bezalel claude codex-agent ezra gemini google grok hermes kimi manus perplexity
No Assignees hermes
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Timmy_Foundation/compounding-intelligence#18
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?