Rockachopa/Timmy-time-dashboard
1
0
Fork 2
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

Performance: eliminate redundant LLM calls in the request path #24

New Issue
Closed
opened 2026-03-14 13:33:34 +00:00 by Rockachopa · 0 comments
Rockachopa commented 2026-03-14 13:33:34 +00:00
Owner
Copy Link

What

A single user message can trigger 2-4 separate LLM inference calls before producing a response:

  1. Routing call — TimmyOrchestrator.orchestrate() asks Helm to classify the request (now replaced by YAML patterns, but old code may still be in the path)
  2. Main inference — the actual response generation
  3. Fact extraction — session.py:_extract_facts() runs on every message
  4. Agentic loop — if triggered, each step is a separate inference call with the full plan re-serialized

On a local model where each call takes 30-60 seconds, this is devastating.

Current flow (worst case)

User message
  → _extract_facts() [regex, no LLM — OK]
  → agent.arun() [LLM call #1]
    → model decides to call plan_and_execute
      → planning prompt [LLM call #2]
      → step 1 execution [LLM call #3]
      → step 2 execution [LLM call #4]
      → summary [LLM call #5]

What to do

  1. Audit the hot path — trace what happens from session.chat() through to response. Count LLM calls.
  2. Kill unnecessary calls — fact extraction is regex-only (good), but verify nothing else sneaks in
  3. Agentic loop optimization — don't re-serialize the full plan + all previous results into every step prompt. Use a sliding window.
  4. Context budget — the system prompt + memory context + conversation history can eat 50%+ of context window before the user's message even arrives. Measure and cap it.
  5. Cache system prompts — get_system_prompt() is called fresh each time. Cache it.

Files

  • src/timmy/session.py — hot path entry point
  • src/timmy/agent.py — agent creation, prompt assembly
  • src/timmy/agentic_loop.py — multi-step execution
  • src/timmy/prompts.py — system prompt generation

Estimated effort: 2-3 hours

## What A single user message can trigger 2-4 separate LLM inference calls before producing a response: 1. **Routing call** — `TimmyOrchestrator.orchestrate()` asks Helm to classify the request (now replaced by YAML patterns, but old code may still be in the path) 2. **Main inference** — the actual response generation 3. **Fact extraction** — `session.py:_extract_facts()` runs on every message 4. **Agentic loop** — if triggered, each step is a separate inference call with the full plan re-serialized On a local model where each call takes 30-60 seconds, this is devastating. ## Current flow (worst case) ``` User message → _extract_facts() [regex, no LLM — OK] → agent.arun() [LLM call #1] → model decides to call plan_and_execute → planning prompt [LLM call #2] → step 1 execution [LLM call #3] → step 2 execution [LLM call #4] → summary [LLM call #5] ``` ## What to do 1. **Audit the hot path** — trace what happens from `session.chat()` through to response. Count LLM calls. 2. **Kill unnecessary calls** — fact extraction is regex-only (good), but verify nothing else sneaks in 3. **Agentic loop optimization** — don't re-serialize the full plan + all previous results into every step prompt. Use a sliding window. 4. **Context budget** — the system prompt + memory context + conversation history can eat 50%+ of context window before the user's message even arrives. Measure and cap it. 5. **Cache system prompts** — `get_system_prompt()` is called fresh each time. Cache it. ## Files - `src/timmy/session.py` — hot path entry point - `src/timmy/agent.py` — agent creation, prompt assembly - `src/timmy/agentic_loop.py` — multi-step execution - `src/timmy/prompts.py` — system prompt generation ## Estimated effort: 2-3 hours
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
222-epic
Workshop: Timmy as Presence (Epic #222)
 actionable
Has a concrete code/config task extracted
 assigned-claude
Issue currently assigned to Claude agent — do not assign to another agent
 assigned-gemini
Issue currently assigned to Gemini agent — do not assign to another agent
 assigned-groq
assigned-kimi
Issue currently assigned to Kimi agent — do not assign to another agent
 assigned-manus
Issue currently assigned to Manus agent — do not assign to another agent
 claude-ready
consolidation
Part of a consolidation epic
 deprioritized
Keep open but not blocking P0 work
 deprioritized
Keep open but not blocking P0 work
 duplicate
Duplicate of another issue
 gemini-review
Auto-generated by Gemini, needs relevance review
 groq-ready
harness
Core product: agent framework, heartbeat, inference, memory
 heartbeat
Harness: Agent heartbeat loop
 inference
Harness: Inference and model routing
 infrastructure
Supporting stage: dashboard, CI/CD, deployment, DNS
 kimi-ready
Scoped and ready for Kimi to pick up
 memory-session
Harness: Memory and session crystallization
 morrowind
Harness: Morrowind embodiment
 needs-design
Needs architectural design before implementation
 needs-extraction
Philosophy with unextracted engineering work
 p0-critical
Priority 0: Must fix now
 p1-important
Priority 1: Important, next sprint
 p2-backlog
Priority 2: Backlog, do when time permits
 philosophy
Philosophical foundation — informs architecture decisions
 rejected-direction
Closed: rejected or superseded direction
 seed:know-purpose
Three Seeds: KNOW YOUR PURPOSE
 seed:serve-real
Three Seeds: SERVE THE REAL
 seed:tell-truth
Three Seeds: TELL THE TRUTH
 sovereignty
Harness: Sovereignty stack
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Timmy claude grok groq kimi manus perplexity Rockachopa
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Rockachopa/Timmy-time-dashboard#24
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?