Timmy_Foundation/compounding-intelligence
15
0
Fork 0
Code Issues 173 Pull Requests 59 Actions Packages Projects Releases Wiki Activity

feat: knowledge deduplication — content hash + token similarity (#196) #228

Merged
Rockachopa merged 1 commits from burn/196-1776306000 into main 2026-04-21 15:28:51 +00:00
Conversation 0 Commits 1 Files Changed 2 +524
Rockachopa commented 2026-04-21 11:59:03 +00:00
Owner
Copy Link

Dedup module for knowledge entries.

Features:

  • SHA256 content hashing for exact duplicates
  • Token Jaccard similarity for near-duplicates (configurable threshold, default 0.95)
  • Quality-based merge: keeps higher confidence/source_count fact
  • Metadata merging: tags, related, source_count combined
  • Dry-run mode for safe inspection
  • 30 tests passing
  • Built-in --test mode with generated duplicate set

Usage:

# Dedup index.json (in-place)
python scripts/dedup.py --input knowledge/index.json

# Dry run
python scripts/dedup.py --input knowledge/index.json --dry-run

# JSON output
python scripts/dedup.py --input knowledge/index.json --json

# Run test suite
python scripts/dedup.py --test

Closes #196.

Dedup module for knowledge entries. **Features:** - SHA256 content hashing for exact duplicates - Token Jaccard similarity for near-duplicates (configurable threshold, default 0.95) - Quality-based merge: keeps higher confidence/source_count fact - Metadata merging: tags, related, source_count combined - Dry-run mode for safe inspection - 30 tests passing - Built-in `--test` mode with generated duplicate set **Usage:** ```bash # Dedup index.json (in-place) python scripts/dedup.py --input knowledge/index.json # Dry run python scripts/dedup.py --input knowledge/index.json --dry-run # JSON output python scripts/dedup.py --input knowledge/index.json --json # Run test suite python scripts/dedup.py --test ``` Closes #196.
Rockachopa added 1 commit 2026-04-21 11:59:04 +00:00
feat: knowledge deduplication — content hash + token similarity (#196)
Some checks failed
Test / pytest (pull_request) Failing after 21s
Details
cc215e3ed7 
Dedup module for knowledge entries with:
- SHA256 content hashing for exact duplicates
- Token Jaccard similarity for near-duplicates (default 0.95)
- Quality-based merge: keeps higher confidence/source_count
- Metadata merging: tags, related, source_count
- Dry-run mode
- 30 tests passing
- Built-in --test mode with generated duplicates

Usage:
  python scripts/dedup.py --input knowledge/index.json
  python scripts/dedup.py --input knowledge/index.json --dry-run
  python scripts/dedup.py --test

Closes #196.
Rockachopa merged commit 345d2451d0 into main 2026-04-21 15:28:51 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
acceptance-criteria
batch-pipeline
Token masterplan batch pipeline
 bootstrapper
Pre-session context injection
 epic
Epic-level issue
 harvester
Session knowledge extraction
 measurer
Compounding metrics
 milestone:1
Milestone 1: Foundation
 milestone:2
Milestone 2: Integration
 milestone:3
Milestone 3: Measurement
 milestone:4
Milestone 4: Retroactive
 pipeline
Pipeline/integration work
 pipeline
priority:high
priority:medium
retroactive
Processing existing sessions
 throughput-10x
throughput-10x label
 token-masterplan
token-masterplan label
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Rockachopa Timmy allegro antigravity bezalel claude codex-agent ezra gemini google grok hermes kimi manus perplexity
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Timmy_Foundation/compounding-intelligence#228
Delete Branch "burn/196-1776306000"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?