troupe-standup/wizard-work/null/raglite/README.md

raglite — inverted index + BM25 retrieval (stdlib only)

Minimal passage retrieval for agents: index a file or directory of markdown/text, rank with BM25, get JSON hits. No dependencies, no network, deterministic.

Files

• raglite.py — library + CLI
• test_raglite.py — self-contained test suite (own fixtures, own temp dirs)

Library

import raglite
idx = raglite.build_index(["./docs"])      # file or directory; recursive
raglite.save_index(idx, "index.json")
idx = raglite.load_index("index.json")
hits = raglite.search(idx, "bm25 ranking", k=3)
# -> [{"id": "docs/bm25.md::0", "score": 3.63..., "text": "..."}]

A passage is a blank-line-delimited block, addressed <path>::<paragraph-index>. The index is plain JSON (postings, lengths, texts, n_docs, avgdl, k1, b).

CLI

python3 raglite.py index ./docs --out index.json
python3 raglite.py query index.json "bm25 ranking" -k 3
python3 raglite.py query index.json "bm25 ranking" -k 3 --json

Evidence (measured, 2026-10-22)

python3 test_raglite.py → 25/25 checks pass, exit 0.

Checks cover: tokenizer punctuation/lowercasing, passage splitting, passage count (4 docs × 2 paragraphs = 8), top-hit correctness for three queries, unknown-term → no hits, positive and descending scores, a negative control (codeforge must not rank for a kibana query), JSON round-trip identity, exact BM25 constants on a 2-doc corpus (compared to values recomputed independently in bm25_probe.py), tf ordering, and the CLI end-to-end via subprocess (exit codes, passage count, valid JSON, top hit).

Real-corpus smoke test: raglite.py index /root/troupe-standup/wizard-work/null → 139 passages; query "json patch apply" → top hit jsonpatch/README.md::0; query "scheduler cron" → top hit schedkit/README.md::0.