47 lines
1.8 KiB
Markdown
47 lines
1.8 KiB
Markdown
# raglite — inverted index + BM25 retrieval (stdlib only)
|
||
|
||
Minimal passage retrieval for agents: index a file or directory of markdown/text,
|
||
rank with BM25, get JSON hits. No dependencies, no network, deterministic.
|
||
|
||
## Files
|
||
|
||
- `raglite.py` — library + CLI
|
||
- `test_raglite.py` — self-contained test suite (own fixtures, own temp dirs)
|
||
|
||
## Library
|
||
|
||
```python
|
||
import raglite
|
||
idx = raglite.build_index(["./docs"]) # file or directory; recursive
|
||
raglite.save_index(idx, "index.json")
|
||
idx = raglite.load_index("index.json")
|
||
hits = raglite.search(idx, "bm25 ranking", k=3)
|
||
# -> [{"id": "docs/bm25.md::0", "score": 3.63..., "text": "..."}]
|
||
```
|
||
|
||
A passage is a blank-line-delimited block, addressed `<path>::<paragraph-index>`.
|
||
The index is plain JSON (`postings`, `lengths`, `texts`, `n_docs`, `avgdl`, `k1`, `b`).
|
||
|
||
## CLI
|
||
|
||
```
|
||
python3 raglite.py index ./docs --out index.json
|
||
python3 raglite.py query index.json "bm25 ranking" -k 3
|
||
python3 raglite.py query index.json "bm25 ranking" -k 3 --json
|
||
```
|
||
|
||
## Evidence (measured, 2026-10-22)
|
||
|
||
`python3 test_raglite.py` → **25/25 checks pass, exit 0**.
|
||
|
||
Checks cover: tokenizer punctuation/lowercasing, passage splitting, passage count
|
||
(4 docs × 2 paragraphs = 8), top-hit correctness for three queries, unknown-term →
|
||
no hits, positive and descending scores, a negative control (codeforge must not rank
|
||
for a kibana query), JSON round-trip identity, exact BM25 constants on a 2-doc corpus
|
||
(compared to values recomputed independently in `bm25_probe.py`), tf ordering, and the
|
||
CLI end-to-end via subprocess (exit codes, passage count, valid JSON, top hit).
|
||
|
||
Real-corpus smoke test: `raglite.py index /root/troupe-standup/wizard-work/null`
|
||
→ **139 passages**; query `"json patch apply"` → top hit `jsonpatch/README.md::0`;
|
||
query `"scheduler cron"` → top hit `schedkit/README.md::0`.
|