Timmy_Foundation/hermes-agent
15
0
Fork 0
Code Issues 222 Pull Requests 98 Actions 139 Packages Projects Releases Wiki Activity

Implement Reader-Guided Reranking — Bridge R@5 vs E2E Gap (+10-20 Accuracy) #666

New Issue
Closed
opened 2026-04-14 19:34:56 +00:00 by Rockachopa · 0 comments
Rockachopa commented 2026-04-14 19:34:56 +00:00
Owner
Copy Link

Research Source

Issue #660: R@5 vs End-to-End Accuracy Gap
Finding: Reader-guided reranking achieves +10-20 top-1 accuracy gains

Problem

Retrieval succeeds (98.4% R@5) but answering fails (17% E2E). The LLM can't use what it retrieves. Reader-guided reranking fixes this by using the LLM's own predictions to rerank passages.

Solution

Implement RIDER (Reader-Guided Passage Reranking):

  1. Retrieve top-K passages (existing vector/FTS5 search)
  2. Have LLM predict answer from each passage individually
  3. Rerank passages by LLM confidence in its predictions
  4. Return top-N reranked passages for final answer generation

Why It Works

  • Aligns retrieval with what the LLM can actually use
  • No training required — uses reader's predictions as signal
  • Achieves 48.3 EM on Natural Questions with only 1,024 tokens

Acceptance Criteria

  • RIDER class with rerank(passages, query) method
  • LLM prediction from each passage (batch inference)
  • Confidence-based reranking
  • Integration with session_search
  • Benchmark: measure E2E accuracy before/after
  • Target: +10-20 top-1 accuracy improvement
  • Tests: test_reader_guided_reranking.py
  • Documentation: docs/reader-guided-reranking.md

Implementation Steps

  1. Create agent/rider.py with RIDER class
  2. Implement batch LLM prediction from passages
  3. Implement confidence scoring and reranking
  4. Wire into session_search pipeline
  5. Benchmark on LongMemEval subset
  6. Measure E2E accuracy improvement
  7. Write tests
  8. Document

Success Metric

+10-20 top-1 accuracy improvement. E2E accuracy increases measurably.

Effort: 3 days

## Research Source Issue #660: R@5 vs End-to-End Accuracy Gap Finding: Reader-guided reranking achieves +10-20 top-1 accuracy gains ## Problem Retrieval succeeds (98.4% R@5) but answering fails (17% E2E). The LLM can't use what it retrieves. Reader-guided reranking fixes this by using the LLM's own predictions to rerank passages. ## Solution Implement RIDER (Reader-Guided Passage Reranking): 1. Retrieve top-K passages (existing vector/FTS5 search) 2. Have LLM predict answer from each passage individually 3. Rerank passages by LLM confidence in its predictions 4. Return top-N reranked passages for final answer generation ## Why It Works - Aligns retrieval with what the LLM can actually use - No training required — uses reader's predictions as signal - Achieves 48.3 EM on Natural Questions with only 1,024 tokens ## Acceptance Criteria - [ ] RIDER class with rerank(passages, query) method - [ ] LLM prediction from each passage (batch inference) - [ ] Confidence-based reranking - [ ] Integration with session_search - [ ] Benchmark: measure E2E accuracy before/after - [ ] Target: +10-20 top-1 accuracy improvement - [ ] Tests: test_reader_guided_reranking.py - [ ] Documentation: docs/reader-guided-reranking.md ## Implementation Steps 1. Create agent/rider.py with RIDER class 2. Implement batch LLM prediction from passages 3. Implement confidence scoring and reranking 4. Wire into session_search pipeline 5. Benchmark on LongMemEval subset 6. Measure E2E accuracy improvement 7. Write tests 8. Document ## Success Metric +10-20 top-1 accuracy improvement. E2E accuracy increases measurably. ## Effort: 3 days
Rockachopa added the p0-critical label 2026-04-14 19:34:56 +00:00
Timmy was assigned by Rockachopa 2026-04-14 19:34:56 +00:00
Timmy closed this issue 2026-04-15 11:58:04 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CI
Continuous integration, runners, workflow issues
 QA
Quality assurance, testing, production audit
 assigned-claw-code
Queued for Code Claw (qwen/openrouter)
 assigned-kimi
Task assigned to KimiClaw for processing
 audit-finding
batch-pipeline
batch-pipeline label
 blocked
Blocked by external dependency or merge conflict
 bug
claw-code-done
Code Claw completed this task
 claw-code-in-progress
Code Claw is actively working
 critical
duplicate
Duplicate of another issue
 epic
Epic - large feature with multiple sub-tasks
 epic:memory
Unified Memory Architecture epic
 gaming
Gaming agent capabilities
 infra
Infrastructure, VPS, DNS, deployment
 kimi-done
KimiClaw has completed this task
 kimi-in-progress
KimiClaw is actively working on this
 mcp
MCP (Model Context Protocol) tools & servers
 morrowind
Morrowind Agent gameplay & MCP integration
 needs-review
PR or issue requires reviewer sign-off before merge/close
 ops
p0-critical
p1-important
phase:1-activate
Phase 1: Activate existing systems
 phase:2-operational
Phase 2: Operational memory bridge
 phase:3-episodic
Phase 3: Proactive episodic recall
 phase:4-hygiene
Phase 4: Memory hygiene and quality
 phase:5-integration
Phase 5: Integration and testing
 poka-yoke
research
security
Security hardening, vulnerability fixes
 shared-infra
Shared infrastructure across projects (Playground + LPM)
 stale
No activity, pending triage or closure
 throughput-10x
throughput-10x label
 token-masterplan
token-masterplan label
 tool-evaluation
velocity-engine
Auto-generated by velocity engine
 wont-fix
Closed as intentionally not fixing — explicit descope
No Label p0-critical
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Rockachopa Timmy allegro antigravity bezalel claude claw-code codex-agent ezra gemini google grok hermes kimi manus perplexity sonnet
No Assignees Timmy
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Timmy_Foundation/hermes-agent#666
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?