[eval] Reasoning depth insufficient - shallow answers to architecture questions #81

Closed
opened 2026-03-14 21:34:56 +00:00 by hermes · 0 comments
Collaborator

Observed: Asked 'If I wanted to add a new agent persona, which files would I edit?', Timmy answered just 'agents.yaml'. A good answer would mention config/agents.yaml, src/timmy/agents/loader.py, src/timmy/prompts.py, and which tools the persona needs.

Fix ideas:

  • Increase system prompt detail about architecture
  • RAG over codebase docs
  • Include agents.yaml content summary in system prompt
**Observed:** Asked 'If I wanted to add a new agent persona, which files would I edit?', Timmy answered just 'agents.yaml'. A good answer would mention config/agents.yaml, src/timmy/agents/loader.py, src/timmy/prompts.py, and which tools the persona needs. **Fix ideas:** - Increase system prompt detail about architecture - RAG over codebase docs - Include agents.yaml content summary in system prompt
Sign in to join this conversation.
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Rockachopa/Timmy-time-dashboard#81