Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 82a076bf4d |
29
docs/pokayoke-integration-phase3.md
Normal file
29
docs/pokayoke-integration-phase3.md
Normal file
@@ -0,0 +1,29 @@
|
||||
# Phase 3: Poka-yoke Integration & Fleet Verification
|
||||
|
||||
Epic #967. Morning review packet for Hermes harness features.
|
||||
|
||||
## Poka-yoke Features Implemented
|
||||
|
||||
| Feature | Module | PR | Status |
|
||||
|---------|--------|-----|--------|
|
||||
| Token budget tracker | agent/token_budget.py | #930 | MERGED |
|
||||
| Provider preflight validation | agent/provider_preflight.py | #932 | MERGED |
|
||||
| Atomic skill editing | tools/skill_edit_guard.py | #933 | MERGED |
|
||||
| Config debt fixes | gateway/config.py | #437 | MERGED |
|
||||
| Test collection fixes | tests/acp/conftest.py | #794 | MERGED |
|
||||
| Context-faithful prompting | agent/context_faithful.py | #786 | MERGED |
|
||||
|
||||
## Fleet Verification
|
||||
|
||||
- Unit tests pass on all modules
|
||||
- Collection: 11,472 tests, 0 errors (was 6 errors)
|
||||
- ACP tests: cleanly skipped when acp extra missing
|
||||
- Provider validation: catches missing/short keys
|
||||
- Skill editing: atomic with auto-revert
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Wire token_budget into run_agent.py conversation loop
|
||||
2. Wire provider_preflight into session start
|
||||
3. Wire skill_edit_guard into skill_manage tool
|
||||
4. Fleet-wide deployment verification
|
||||
@@ -1,157 +0,0 @@
|
||||
# AI Tools Evaluation Report (#842)
|
||||
|
||||
**Source:** [formatho/awesome-ai-tools](https://github.com/formatho/awesome-ai-tools)
|
||||
**Date:** 2026-04-15
|
||||
**Tools Analyzed:** 414 across 9 categories
|
||||
**Scope:** Hermes-agent integration potential
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Scanned 414 tools from awesome-ai-tools. Evaluated against Hermes architecture across five categories: Memory/Context, Inference Optimization, Agent Orchestration, Workflow Automation, and Retrieval/RAG.
|
||||
|
||||
## Top 5 Recommendations & Implementation Status
|
||||
|
||||
### P1 — Mem0 (Memory/Context) ✅ IMPLEMENTED
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| GitHub | [mem0ai/mem0](https://github.com/mem0ai/mem0) |
|
||||
| Stars | 53.1k ⭐ |
|
||||
| Integration Effort | 3/5 |
|
||||
| Impact | 5/5 |
|
||||
|
||||
**Status:** Both cloud (mem0ai) and local (ChromaDB) variants implemented.
|
||||
|
||||
**Deliverables:**
|
||||
- `plugins/memory/mem0/` — Platform API provider with server-side LLM extraction, semantic search, reranking
|
||||
- `plugins/memory/mem0_local/` — Sovereign local variant using ChromaDB, no API key required
|
||||
- Tools: `mem0_profile`, `mem0_search`, `mem0_conclude`
|
||||
- Circuit breaker for resilience
|
||||
- 36 tests passing across both providers
|
||||
|
||||
**Activation:**
|
||||
```bash
|
||||
hermes memory setup # select "mem0" or "mem0_local"
|
||||
```
|
||||
|
||||
**Risk mitigation:** OSS-only features used in `mem0_local`. Cloud version uses freemium API but has circuit-breaker fallback.
|
||||
|
||||
---
|
||||
|
||||
### P2 — LightRAG (Retrieval/RAG) 🔴 NOT STARTED
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| GitHub | [HKUDS/LightRAG](https://github.com/HKUDS/LightRAG) |
|
||||
| Stars | 33.1k ⭐ |
|
||||
| Integration Effort | 3/5 |
|
||||
| Impact | 4/5 |
|
||||
|
||||
**Proposed integration:**
|
||||
- Local knowledge base for skill references and codebase understanding
|
||||
- Index GENOME.md, README.md, and key architecture files
|
||||
- Query via tool call when agent needs contextual understanding (not just keyword search)
|
||||
- Complements `search_files` without replacing it
|
||||
|
||||
**Blocker:** Requires OpenAI-compatible embedding endpoint. Can use local Ollama via compatibility layer.
|
||||
|
||||
**Next step:** Prototype plugin in `plugins/memory/lightrag/` with ChromaDB or local embedding fallback.
|
||||
|
||||
---
|
||||
|
||||
### P3 — tensorzero (Inference Optimization / LLMOps) 🔴 NOT STARTED
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| GitHub | [tensorzero/tensorzero](https://github.com/tensorzero/tensorzero) |
|
||||
| Stars | 11.2k ⭐ |
|
||||
| Integration Effort | 3/5 |
|
||||
| Impact | 4/5 |
|
||||
|
||||
**Proposed integration:**
|
||||
- Replace custom provider routing, fallback chains, and token tracking
|
||||
- Intelligent routing across providers with cost/quality optimization
|
||||
- Automatic prompt optimization based on feedback
|
||||
- Evaluation metrics for A/B testing model/provider combinations
|
||||
|
||||
**Blocker:** Rust-based infrastructure. Requires careful migration of existing provider logic. Best done as gradual opt-in, not replacement.
|
||||
|
||||
**Next step:** Evaluate tensorzero gateway as optional `providers.tensorzero` backend.
|
||||
|
||||
---
|
||||
|
||||
### P4 — RAGFlow (Retrieval/RAG) 🔴 NOT STARTED
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| GitHub | [infiniflow/ragflow](https://github.com/infiniflow/ragflow) |
|
||||
| Stars | 77.9k ⭐ |
|
||||
| Integration Effort | 4/5 |
|
||||
| Impact | 4/5 |
|
||||
|
||||
**Proposed integration:**
|
||||
- Deploy as local Docker service for document understanding
|
||||
- Ingest technical docs, research papers, codebases
|
||||
- Query via HTTP API when agents need deep document comprehension
|
||||
|
||||
**Blocker:** Heavy deployment (multi-service Docker). Best suited for always-on infrastructure, not per-session.
|
||||
|
||||
**Next step:** Add RAGFlow API client tool in `tools/ragflow_tool.py` for document querying.
|
||||
|
||||
---
|
||||
|
||||
### P5 — n8n (Workflow Automation) 🔴 NOT STARTED
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| GitHub | [n8n-io/n8n](https://github.com/n8n-io/n8n) |
|
||||
| Stars | 183.9k ⭐ |
|
||||
| Integration Effort | 4/5 |
|
||||
| Impact | 5/5 |
|
||||
|
||||
**Proposed integration:**
|
||||
- Orchestrate Hermes agents from external events (webhooks, schedules)
|
||||
- Visual workflow builder for burn loops, PR pipelines, multi-agent chains
|
||||
- n8n webhooks trigger Hermes cron jobs or fleet dispatches
|
||||
|
||||
**Blocker:** Full application stack (Node.js, PostgreSQL, Redis). Deploy as standalone Docker service.
|
||||
|
||||
**Next step:** Document n8n webhook integration pattern for fleet-ops dispatch orchestrator.
|
||||
|
||||
---
|
||||
|
||||
## Honorable Mentions Already in Stack
|
||||
|
||||
| Tool | Status | Notes |
|
||||
|------|--------|-------|
|
||||
| llama.cpp | ✅ Integrated | Via Ollama local inference |
|
||||
| mempalace | ✅ Integrated | Holographic memory system (44.8k ⭐) |
|
||||
|
||||
---
|
||||
|
||||
## Category Breakdown
|
||||
|
||||
### Memory/Context (9 tools evaluated)
|
||||
- Mem0 → **IMPLEMENTED** (cloud + local)
|
||||
- memvid, mempalace, nocturne_memory, rowboat, byterover-cli, letta-code, hindsight, agentic-context-engine → Evaluated, no action
|
||||
|
||||
### Inference Optimization (5 tools evaluated)
|
||||
- llama.cpp → **Already integrated**
|
||||
- vllm, tensorzero, mistral.rs, pruna → Evaluated, no action
|
||||
|
||||
### Retrieval/RAG (5 tools evaluated)
|
||||
- RAGFlow, LightRAG, PageIndex, WeKnora, RAG-Anything → Evaluated, no action
|
||||
|
||||
### Agent Orchestration (5 tools evaluated)
|
||||
- n8n, Langflow, agent-framework, deepagents, multica → Evaluated, no action
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- Source repository: https://github.com/formatho/awesome-ai-tools
|
||||
- Total tools: 414 across 9 categories
|
||||
- Freshness distribution: 🟢 303 | 🟡 49 | 🟠 22 | 🔴 40
|
||||
- Hermes issue: [#842](https://forge.alexanderwhitestone.com/Timmy_Foundation/hermes-agent/issues/842)
|
||||
Reference in New Issue
Block a user