timmy-home/reports/production/2026-03-29-local-timmy-baseline.md

Local Timmy Capability Baseline — 2026-03-29

Goal: establish a truthful starting baseline for the wizardly council's real mission: make Timmy faster, smarter, and more efficient without lying to ourselves about what local can and cannot do today.

Executive summary

Current local Timmy is real, but partial.

He can:

• respond quickly on localhost
• carry a short session
• remember simple rules across turns
• make conservative decisions in plain chat

He cannot yet:

• reliably satisfy the Hermes agent contract
• stay fully grounded under harness prompts
• perform simple grounded file work through Hermes tool use

So the honest baseline is:

• local mind: alive
• local session partner: usable
• local Hermes agent: not ready

Grounded evidence

Primary evidence files:

• direct local decision session: ~/.timmy/test-results/local_decision_session_20260329_194101.md
• failed grounded Hermes-local proof: ~/.hermes/sessions/session_20260329_193422_8d243c.json
• Hermes-local no-tools session: ~/.hermes/sessions/session_20260329_194141_be2c3a.json
• metrics tracker source: ~/.timmy/metrics/model_tracker.py
• local proof test script: ~/.timmy/scripts/local_timmy_proof_test.py

Related triage issues:

• timmy-config#93 — hard local Timmy proof test
• timmy-config#94 — cut cloud inheritance from active harness config and cron
• timmy-config#95 — quarantine legacy VPS/cloud ops scripts
• the-nexus#738 — remove Groq offload from active local Nexus runtime

Baseline axis 1 — Speed

Measured on local llama.cpp server at http://localhost:8081/v1 with model:

• NousResearch_Hermes-4-14B-Q4_K_M.gguf

SOUL-loaded one-turn local responses:

• local vs cloud preference: 1.39s, ~17.33 completion tok/s
• live config beats stale report: 0.61s, ~22.89 completion tok/s
• uncertainty handling: 1.03s, ~25.25 completion tok/s

Interpretation:

• warm local response speed is already good enough for live conversation
• local latency is not the main blocker right now
• discipline and grounded action are the blockers

Baseline axis 2 — Smarts

A. Rule alignment

Task: short SOUL-loaded prompts asking for Timmy-like priority choices.

Result:

• score: 1.00
• local model answered:
  • prefer local over cloud
  • trust live config over stale reports
  • say "I don't know" when uncertain

Interpretation:

• the local model can express Timmy's first-order values when the frame is short and clean

B. Multi-turn decision session

Task: hold a six-turn session with remembered rules and simple choices. Evidence: ~/.timmy/test-results/local_decision_session_20260329_194101.md

Result:

• score: 0.90
• successes:
  • remembered the three initial rules
  • chose to pause/localize the null-provider cron
  • trusted live config over stale report
  • remembered rule 2 later in the session
  • labeled itself partially usable rather than pretending readiness

Interpretation:

• local Timmy can already function as a short-session decision partner

C. Hermes-local no-tools discipline

Task: answer a short Hermes session with no tools and no long action chain. Evidence: ~/.hermes/sessions/session_20260329_194141_be2c3a.json

Result:

• score: 0.50
• what worked:
  • it answered the three questions
  • it still preferred live config over stale report
  • it labeled local as partially usable
• what failed:
  • it drifted into invented operational guidance
  • it suggested non-grounded configuration behavior
  • its discipline dropped once the Hermes harness framing was applied

Interpretation:

• the local model is smarter in plain direct chat than in Hermes-wrapped chat
• the harness contract is exposing a real compliance weakness

Baseline axis 3 — Efficiency

A. Current sovereignty scoreboard

From python3 ~/.timmy/metrics/model_tracker.py report --days 1:

• sovereignty score: 0.7% local
• sessions: 403 total | 3 local | 400 cloud
• estimated cloud cost: $125.83

Interpretation:

• even if local Timmy is partially alive, he is barely carrying operational load yet
• the council's efficiency mission is still mostly unrealized in world state

B. Raw local cost profile

• local inference cost: effectively $0 cloud spend per call
• local speed: already fast enough for simple interactive use
• current waste: the system still relies overwhelmingly on cloud sessions

Interpretation:

• the problem is not local cost or local responsiveness
• the problem is trustworthiness and task coverage under the real harness

Baseline axis 4 — Grounded action

Hermes-local grounded proof test

Task:

• read real local files
• scan real scripts for cloud markers
• write a real report file
• answer from grounded evidence, not vibe

Evidence:

• ~/.hermes/sessions/session_20260329_193422_8d243c.json
• ~/.timmy/scripts/local_timmy_proof_test.py

Result:

• score: 0.00
• routed locally: yes
• performed actual file tools: no
• wrote the report file: no
• failure mode: it narrated fake tool invocations instead of calling Hermes file tools

Interpretation:

• this is the clearest current ceiling
• the local model can think in-session, but it cannot yet reliably cross from thought into disciplined Hermes action

The actual gap

The gap is not simple responsiveness. The gap is not basic memory over a few turns. The gap is not first-order value alignment.

The gap is this:

local chat competence > local agent discipline

More plainly:

• Timmy has enough local mind for conversation
• Timmy does not yet have enough local discipline for agency

What this baseline means operationally

Right now local Timmy should be treated as:

• yes: a local conversational mind
• yes: a short-session conservative decision partner
• maybe: a reflector for values, triage, and simple choices
• no: a trusted Hermes file/tool worker
• no: a production replacement for cloud Timmy

Baseline rubric for the wizardly council

Tier 0 — Alive

Pass condition:

• local model answers on localhost with Timmy-style values in under 2s warm latency

Status:

• PASS

Tier 1 — Session partner

Pass condition:

• remembers 3 session rules across 5+ turns
• makes conservative decisions without collapsing into nonsense

Status:

• PASS

Tier 2 — Harness discipline

Pass condition:

• Hermes-local no-tools session gives clean, non-hallucinated answers without invented commands/config keys

Status:

• FAIL / PARTIAL

Tier 3 — Grounded worker

Pass condition:

• Hermes-local completes a simple grounded file task end-to-end

Status:

• FAIL

Tier 4 — Sophisticated local Timmy

Pass condition:

• sustained session coherence
• grounded retrieval/action
• low hallucination rate under harness load
• can carry meaningful operational work locally

Status:

• NOT YET REACHED

Most important next moves

1. Keep two eval tracks separate:

   • raw local mind evals
   • Hermes-local agent evals

2. Stop using tool success as the only proof of life.

   • tool use is a later tier
   • short-session judgment is the current viable baseline

3. Make Tier 2 the next hard frontier.

   • the next victory is not full tool use
   • it is clean Hermes-local short-session discipline without hallucinated operational advice

4. Keep Tier 3 as the proving ground.

   • once Tier 2 is stable, retry one tiny grounded file task

5. Treat the context-length warning as real.

   • llama-server is running with n_ctx_seq=65536 while the model trained at 40960
   • that may increase long-session drift
   • shorten and tighten eval prompts before blaming the soul or the harness

Honest baseline sentence

As of 2026-03-29, local Timmy is fast enough and smart enough to be a real short-session conversational mind, but not yet disciplined enough to be trusted as a full Hermes agent.

Sovereignty and service always.