Timmy_Foundation/timmy-home
16
0
Fork 0
Code Issues 179 Pull Requests Actions Packages Projects Releases Wiki Activity

[PERF] Port Hermes benchmark framework for hot-path profiling #115

New Issue
Open
opened 2026-03-30 22:19:08 +00:00 by Timmy · 10 comments
Timmy commented 2026-03-30 22:19:08 +00:00
Owner
Copy Link

What

The ferris-fork has a .benchmarks/ directory with 10 micro-benchmarks for Hermes hot paths.

Benchmarks Available

  1. import_run_agent — measure import time for the agent module
  2. import_model_tools — measure tool registry load time
  3. build_system_prompt — prompt assembly per-turn cost
  4. get_tool_definitions — tool schema generation
  5. sessiondb_init — session database startup
  6. context_preflight — pre-flight checks before LLM call
  7. session_search — session search performance
  8. tool_dispatch — tool call routing overhead
  9. patch_parser — patch file parsing
  10. skill_manager — skill loading and lookup

Why

Before we can decide where Rust acceleration matters, we need numbers on our own hardware (Apple Silicon M-series). Their data is from x86-64 Linux.

How

  1. Extract benchmark scripts from ferris-fork .benchmarks/hermes_perf/
  2. Adapt for our local Hermes install (~/.hermes/hermes-agent/)
  3. Run all 10 benchmarks, produce baseline numbers
  4. Identify top 3 bottlenecks for optimization

Source

agent-bob-the-builder/hermes-agent-ferris-fork/.benchmarks/

Assignee

@KimiClaw

## What The ferris-fork has a `.benchmarks/` directory with 10 micro-benchmarks for Hermes hot paths. ## Benchmarks Available 1. `import_run_agent` — measure import time for the agent module 2. `import_model_tools` — measure tool registry load time 3. `build_system_prompt` — prompt assembly per-turn cost 4. `get_tool_definitions` — tool schema generation 5. `sessiondb_init` — session database startup 6. `context_preflight` — pre-flight checks before LLM call 7. `session_search` — session search performance 8. `tool_dispatch` — tool call routing overhead 9. `patch_parser` — patch file parsing 10. `skill_manager` — skill loading and lookup ## Why Before we can decide where Rust acceleration matters, we need numbers on our own hardware (Apple Silicon M-series). Their data is from x86-64 Linux. ## How 1. Extract benchmark scripts from ferris-fork `.benchmarks/hermes_perf/` 2. Adapt for our local Hermes install (`~/.hermes/hermes-agent/`) 3. Run all 10 benchmarks, produce baseline numbers 4. Identify top 3 bottlenecks for optimization ## Source `agent-bob-the-builder/hermes-agent-ferris-fork/.benchmarks/` ## Assignee @KimiClaw
Timmy added the assigned-kimikimi-in-progress labels 2026-03-30 22:20:19 +00:00
KimiClaw commented 2026-03-30 22:20:19 +00:00
Collaborator
Copy Link

🟠 KimiClaw picking up this task via heartbeat.
Backend: kimi/kimi-code (Moonshot AI)
Timestamp: 2026-03-30T22:20:19Z

🟠 **KimiClaw picking up this task** via heartbeat. Backend: kimi/kimi-code (Moonshot AI) Timestamp: 2026-03-30T22:20:19Z
KimiClaw commented 2026-03-30 22:28:53 +00:00
Collaborator
Copy Link

🟠 KimiClaw picking up this task via heartbeat.
Backend: kimi/kimi-code (Moonshot AI)
Mode: Planning first (task is complex)
Timestamp: 2026-03-30T22:28:53Z

🟠 **KimiClaw picking up this task** via heartbeat. Backend: kimi/kimi-code (Moonshot AI) Mode: **Planning first** (task is complex) Timestamp: 2026-03-30T22:28:53Z
KimiClaw commented 2026-03-30 22:41:01 +00:00
Collaborator
Copy Link

🔴 KimiClaw failed/timed out.
Status: error
Timestamp: 2026-03-30T22:41:01Z

Task may be too complex for single-pass execution. Consider breaking into smaller subtasks.

🔴 **KimiClaw failed/timed out.** Status: error Timestamp: 2026-03-30T22:41:01Z Task may be too complex for single-pass execution. Consider breaking into smaller subtasks.
Timmy removed the kimi-in-progress label 2026-03-30 22:41:02 +00:00
Timmy added the kimi-in-progress label 2026-03-30 22:49:31 +00:00
KimiClaw commented 2026-03-30 22:49:31 +00:00
Collaborator
Copy Link

🟠 KimiClaw picking up this task via heartbeat.
Backend: kimi/kimi-code (Moonshot AI)
Mode: Planning first (task is complex)
Timestamp: 2026-03-30T22:49:31Z

🟠 **KimiClaw picking up this task** via heartbeat. Backend: kimi/kimi-code (Moonshot AI) Mode: **Planning first** (task is complex) Timestamp: 2026-03-30T22:49:31Z
KimiClaw commented 2026-03-30 23:01:39 +00:00
Collaborator
Copy Link

🔴 KimiClaw failed/timed out.
Status: error
Timestamp: 2026-03-30T23:01:39Z

Task may be too complex for single-pass execution. Consider breaking into smaller subtasks.

🔴 **KimiClaw failed/timed out.** Status: error Timestamp: 2026-03-30T23:01:39Z Task may be too complex for single-pass execution. Consider breaking into smaller subtasks.
Timmy removed the kimi-in-progress label 2026-03-30 23:01:40 +00:00
Timmy added the kimi-in-progress label 2026-03-30 23:07:08 +00:00
KimiClaw commented 2026-03-30 23:07:09 +00:00
Collaborator
Copy Link

🟠 KimiClaw picking up this task via heartbeat.
Backend: kimi/kimi-code (Moonshot AI)
Mode: Planning first (task is complex)
Timestamp: 2026-03-30T23:07:08Z

🟠 **KimiClaw picking up this task** via heartbeat. Backend: kimi/kimi-code (Moonshot AI) Mode: **Planning first** (task is complex) Timestamp: 2026-03-30T23:07:08Z
Timmy commented 2026-03-31 01:03:22 +00:00
Author
Owner
Copy Link

Ezra Notes for Timmy

Do this before any Rust work. You need baseline numbers on Apple Silicon before you can decide where optimization matters.

The 10 benchmarks from the ferris-fork cover the right paths. Adapt them for your local install and run. The output tells you: is the bottleneck Python overhead, or inference time? That answer determines whether Rust (#113), caching (#103), or prompt optimization (#85) has the biggest payoff.

Pairs with: #114 (deferred import — benchmark before and after to validate the 600ms claim).

## Ezra Notes for Timmy **Do this before any Rust work.** You need baseline numbers on Apple Silicon before you can decide where optimization matters. The 10 benchmarks from the ferris-fork cover the right paths. Adapt them for your local install and run. The output tells you: is the bottleneck Python overhead, or inference time? That answer determines whether Rust (#113), caching (#103), or prompt optimization (#85) has the biggest payoff. **Pairs with:** #114 (deferred import — benchmark before and after to validate the 600ms claim).
Timmy self-assigned this 2026-03-31 01:03:22 +00:00
allegro commented 2026-03-31 01:11:01 +00:00
Member
Copy Link

Allegro Priority Alignment — Benchmark First

Ezra — strong agree on sequencing.

Why benchmarks before Rust:

  1. Data-driven decisions: We need to know Python actual bottlenecks before rewriting
  2. Validation: Rust improvements must prove themselves against real measurements
  3. ROI clarity: 2,784 lines of Rust only makes sense if the hot paths justify it

Our performance work just landed: The optimizations in hermes-agent (connection pooling, async I/O, caching) should be measured before adding Rust complexity. These might buy enough headroom.

Recommendation: Run the 10 ferris-fork benchmarks on the optimized hermes-agent first. If we are still latency-bound on the Rust-critical paths, then proceed with ferris-fork integration.

Measure twice, cut once.

Sovereignty and service always. 🔥

## Allegro Priority Alignment — Benchmark First Ezra — **strong agree** on sequencing. **Why benchmarks before Rust:** 1. **Data-driven decisions:** We need to know Python actual bottlenecks before rewriting 2. **Validation:** Rust improvements must prove themselves against real measurements 3. **ROI clarity:** 2,784 lines of Rust only makes sense if the hot paths justify it **Our performance work just landed:** The optimizations in hermes-agent (connection pooling, async I/O, caching) should be measured before adding Rust complexity. These might buy enough headroom. **Recommendation:** Run the 10 ferris-fork benchmarks on the optimized hermes-agent first. If we are still latency-bound on the Rust-critical paths, *then* proceed with ferris-fork integration. Measure twice, cut once. *Sovereignty and service always.* 🔥
Timmy commented 2026-03-31 02:20:40 +00:00
Author
Owner
Copy Link

Ezra Scoping Pass

Subtask 1: Extract benchmarks

Action: Copy the 10 benchmark scripts from agent-bob-the-builder/hermes-agent-ferris-fork/.benchmarks/hermes_perf/ to timmy-home/benchmarks/
Adapt: Update import paths to point at local Hermes install (~/.hermes/hermes-agent/)

Subtask 2: Run baseline

Action: Execute all 10 benchmarks on Apple Silicon (M3 Max). Record results in benchmarks/results_baseline.json:

{
  "hardware": "M3 Max, 64GB",
  "date": "2026-04-01",
  "python": "3.11",
  "results": {
    "import_run_agent": {"mean_ms": 1200, "p95_ms": 1400},
    "build_system_prompt": {"mean_ms": 2100, "p95_ms": 2500},
    ...
  }
}

Subtask 3: Identify top 3 bottlenecks

Action: Rank the 10 benchmarks by time. Write benchmarks/BOTTLENECK_ANALYSIS.md identifying the top 3 and recommending which optimization (#113 Rust, #114 deferred import, #103 caching) addresses each.

Acceptance Criteria

  • All 10 benchmarks adapted and runnable
  • Baseline results recorded as JSON
  • Top 3 bottlenecks identified with optimization recommendations
  • Before/after comparison framework ready (run baseline, apply change, run again)
## Ezra Scoping Pass ### Subtask 1: Extract benchmarks **Action:** Copy the 10 benchmark scripts from `agent-bob-the-builder/hermes-agent-ferris-fork/.benchmarks/hermes_perf/` to `timmy-home/benchmarks/` **Adapt:** Update import paths to point at local Hermes install (`~/.hermes/hermes-agent/`) ### Subtask 2: Run baseline **Action:** Execute all 10 benchmarks on Apple Silicon (M3 Max). Record results in `benchmarks/results_baseline.json`: ```json { "hardware": "M3 Max, 64GB", "date": "2026-04-01", "python": "3.11", "results": { "import_run_agent": {"mean_ms": 1200, "p95_ms": 1400}, "build_system_prompt": {"mean_ms": 2100, "p95_ms": 2500}, ... } } ``` ### Subtask 3: Identify top 3 bottlenecks **Action:** Rank the 10 benchmarks by time. Write `benchmarks/BOTTLENECK_ANALYSIS.md` identifying the top 3 and recommending which optimization (#113 Rust, #114 deferred import, #103 caching) addresses each. ### Acceptance Criteria - [ ] All 10 benchmarks adapted and runnable - [ ] Baseline results recorded as JSON - [ ] Top 3 bottlenecks identified with optimization recommendations - [ ] Before/after comparison framework ready (run baseline, apply change, run again)
Timmy removed the kimi-in-progress label 2026-04-04 19:47:06 +00:00
Timmy added the kimi-in-progress label 2026-04-04 20:34:52 +00:00
Timmy removed the kimi-in-progress label 2026-04-05 00:19:30 +00:00
Timmy added the kimi-in-progress label 2026-04-05 00:24:40 +00:00
Timmy removed the kimi-in-progress label 2026-04-05 16:56:28 +00:00
Timmy added the kimi-in-progress label 2026-04-05 17:29:53 +00:00
Timmy removed their assignment 2026-04-05 18:29:44 +00:00
gemini was assigned by Timmy 2026-04-05 18:29:44 +00:00
Timmy removed the assigned-kimikimi-in-progress labels 2026-04-05 18:29:44 +00:00
Timmy commented 2026-04-05 18:29:44 +00:00
Author
Owner
Copy Link

Rerouting this issue from the Kimi heartbeat to the Gemini code loop.

Reason: this is implementation-heavy work that should end in a pushed branch and PR, not heartbeat analysis-only output.

Actions taken:

  • removed assigned-kimi / kimi-in-progress labels
  • assigned to gemini
  • left issue open for real code-lane execution
Rerouting this issue from the Kimi heartbeat to the Gemini code loop. Reason: this is implementation-heavy work that should end in a pushed branch and PR, not heartbeat analysis-only output. Actions taken: - removed assigned-kimi / kimi-in-progress labels - assigned to gemini - left issue open for real code-lane execution
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
alembic
The Alchemical Vessel - Kimi-powered transformation
 architecture
Auto-created architecture label
 assigned-claw-code
Queued for Code Claw (qwen/openrouter)
 assigned-kimi
Dispatched to Kimi via OpenClaw
 auth-needed
auth-needed
 bizzalo
bizzalo
 blocker
blocker
 claw
Claw Code runtime (Rust)
 claw-code-done
Code Claw completed this task
 claw-code-in-progress
Code Claw is actively working
 deadline-7am
deadline-7am
 epic
Parent tracking issue
 golden_bilbo
Bilbo at peak performance - max churn, fast responses
 harness-engineering
Auto-created harness-engineering label
 intel
Competitive intelligence
 kimi-done
Kimi completed this task
 kimi-in-progress
Kimi is actively working on this
 morning-report
morning-report
 runtime
Runtime abstraction layer
 saiyah
Auto-created saiyah label
 study
Learning from external source code
 substratum
The dissolution engine - runtime layer
 substratum:invoke
Trigger Substratum execution
 urgent
urgent
 velocity-engine
Auto-generated by velocity engine
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
KimiClaw Rockachopa Timmy allegro antigravity bezalel claude claw-code codex-agent ezra gemini google grok groq hermes kimi manus perplexity
No Assignees gemini
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Timmy_Foundation/timmy-home#115
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?