hermes-agent/docs/research/warm-session-provisioning.md

Warm Session Provisioning: Pre-Proficient Agent Sessions

Research Document
Issue: #327
Date: April 2026
Status: Research & Prototype

Executive Summary

Empirical analysis reveals a counterintuitive finding: marathon sessions (100+ messages) exhibit lower per-tool error rates (5.7%) than mid-length sessions (9.0% at 51-100 messages). This suggests agents improve with experience within a session, learning user patterns and establishing successful tool-call conventions.

This research explores whether we can pre-seed sessions with proficiency patterns, effectively creating "warm" sessions that start at marathon-level reliability.

Key Findings from Empirical Audit

1. Session Length vs. Error Rate

• 0-50 messages: 7.2% error rate
• 51-100 messages: 9.0% error rate (peak)
• 100+ messages: 5.7% error rate (lowest)

2. Hypothesis: Context Richness Drives Proficiency

Marathon sessions develop:

• User-specific patterns: How the user phrases requests
• Tool-call conventions: Successful argument formats
• Error recovery patterns: How to handle failures
• Context anchoring: Shared reference points

3. Research Questions

1. What specific context elements drive proficiency?
2. Can we extract and transfer these patterns?
3. Does compression preserve proficiency or reset it?
4. What's the minimum viable warm-up sequence?

Technical Approach

Phase 1: Analysis (Current)

Analyze existing marathon sessions to identify proficiency markers:

• Tool-call success patterns
• User interaction conventions
• Error recovery sequences
• Context window utilization

Phase 2: Template Extraction

Extract successful patterns from marathon sessions:

# Conceptual template structure
session_template = {
    "tool_patterns": [
        {"tool": "terminal", "success_pattern": "..."},
        {"tool": "file_operations", "conventions": "..."}
    ],
    "user_patterns": {
        "request_style": "direct",
        "feedback_style": "terse"
    },
    "recovery_patterns": [
        {"error": "FileNotFound", "recovery": "..."}
    ]
}

Phase 3: Warm Session Creation

Implement session seeding mechanism:

1. Run 10-20 diverse tasks to build context
2. Compress the session (preserving proficiency markers)
3. Save as warm template
4. New sessions start from template

Phase 4: A/B Testing

Compare warm vs. cold sessions:

• Same tasks, different starting conditions
• Measure: error rate, time to completion, user satisfaction
• Statistical significance testing

Implementation Plan

1. Session Profiling System

• Add telemetry to track proficiency markers
• Identify what makes marathon sessions successful
• Extract transferable patterns

2. Template Management

• Save/load session templates
• Version control for templates
• Template validation

3. Warm Session Bootstrapping

• Inject template context at session start
• Preserve tool-call conventions
• Maintain user pattern awareness

4. Research Infrastructure

• A/B testing framework
• Statistical analysis tools
• Visualization of results

Expected Outcomes

Short-term (Weeks 1-2)

• Session profiling system operational
• Initial pattern extraction from 5-10 marathon sessions
• Basic template storage

Medium-term (Weeks 3-4)

• Warm session prototype
• Initial A/B test results
• Paper outline

Long-term (Month 2+)

• Production-ready warm session provisioning
• Published research paper
• Open-source template sharing

Paper-Worthy Contributions

1. Empirical finding: Session proficiency increases with length
2. Novel approach: Pre-seeding sessions with proficiency patterns
3. Production system: Warm session provisioning infrastructure
4. Open dataset: Session proficiency markers and templates

Risks and Mitigations

Technical Risks

• Compression resets proficiency: Test different compression strategies
• Patterns don't transfer: Start with similar task domains
• Template bloat: Keep templates minimal, focus on high-impact patterns

Research Risks

• Statistical insignificance: Run sufficient A/B test iterations
• Confounding variables: Control for task difficulty, user expertise
• Publication bias: Report all results, including negative findings

Next Steps

1. ✅ Create research framework (this document)
2. ⏳ Implement session profiling telemetry
3. ⏳ Analyze 5-10 marathon sessions
4. ⏳ Extract proficiency patterns
5. ⏳ Build template storage system
6. ⏳ Implement warm session bootstrapping
7. ⏳ Run A/B tests
8. ⏳ Write paper

References

• Empirical Audit 2026-04-12, Finding 4
• Session compression research (trajectory_compressor.py)
• Tool-call error analysis (existing telemetry)