Timmy_Foundation/compounding-intelligence
15
0
Fork 0
Code Issues 173 Pull Requests 59 Actions Packages Projects Releases Wiki Activity

feat: Knowledge deduplication — content hash + semantic similarity #196

New Issue
Closed
opened 2026-04-15 15:18:03 +00:00 by Rockachopa · 0 comments
Rockachopa commented 2026-04-15 15:18:03 +00:00
Owner
Copy Link

Epic: #136 (Knowledge Pipeline v2)

Task

Deduplicate harvested knowledge to avoid training on duplicates.

Dedup Strategy

  1. Exact dedup: content hash (SHA256 of normalized text)
  2. Semantic dedup: embedding similarity > 0.95 = duplicate
  3. Near-dup: same topic, different phrasing → merge

Deliverables

  • Dedup module: compounding-intelligence/dedup.py
  • Content hashing for exact duplicates
  • Semantic similarity using local embeddings (no cloud)
  • Merge strategy: keep highest quality version
  • Test: inject 20 duplicates, verify they're removed

Labels: dedup, quality, priority:high

## Epic: #136 (Knowledge Pipeline v2) ### Task Deduplicate harvested knowledge to avoid training on duplicates. ### Dedup Strategy 1. **Exact dedup**: content hash (SHA256 of normalized text) 2. **Semantic dedup**: embedding similarity > 0.95 = duplicate 3. **Near-dup**: same topic, different phrasing → merge ### Deliverables - [ ] Dedup module: `compounding-intelligence/dedup.py` - [ ] Content hashing for exact duplicates - [ ] Semantic similarity using local embeddings (no cloud) - [ ] Merge strategy: keep highest quality version - [ ] Test: inject 20 duplicates, verify they're removed ### Labels: dedup, quality, priority:high
hermes was assigned by Rockachopa 2026-04-15 16:23:34 +00:00
hermes was unassigned by Rockachopa 2026-04-17 05:06:19 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acceptance-criteria
batch-pipeline
Token masterplan batch pipeline
 bootstrapper
Pre-session context injection
 epic
Epic-level issue
 harvester
Session knowledge extraction
 measurer
Compounding metrics
 milestone:1
Milestone 1: Foundation
 milestone:2
Milestone 2: Integration
 milestone:3
Milestone 3: Measurement
 milestone:4
Milestone 4: Retroactive
 pipeline
Pipeline/integration work
 pipeline
priority:high
priority:medium
retroactive
Processing existing sessions
 throughput-10x
throughput-10x label
 token-masterplan
token-masterplan label
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Rockachopa Timmy allegro antigravity bezalel claude codex-agent ezra gemini google grok hermes kimi manus perplexity
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Timmy_Foundation/compounding-intelligence#196
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?