Timmy_Foundation/the-nexus
16
1
Fork 2
Code Issues 105 Pull Requests 1 Actions 15 Packages Projects Releases Wiki Activity

Wire voice analysis into Hermes STT pipeline #507

New Issue
Closed
opened 2026-03-25 14:07:31 +00:00 by Timmy · 1 comment
Timmy commented 2026-03-25 14:07:31 +00:00
Owner
Copy Link

Depends on: voice analysis pipeline issue (emotion/prosody/environment)

Objective

After the voice analysis script is built, wire it into the Hermes agent's STT flow so every voice message automatically gets rich audio context.

Current flow

  1. User sends voice message (Telegram/WhatsApp)
  2. Hermes downloads audio
  3. Whisper transcribes to text
  4. Text sent to LLM as user message

Target flow

  1. User sends voice message
  2. Hermes downloads audio
  3. Whisper transcribes to text (with word-level timestamps)
  4. ~/.timmy/voice/analyze_audio.py runs on the audio + transcript
  5. Voice context block prepended to transcript
  6. Enriched message sent to LLM

Integration points

  • Check ~/.hermes/hermes-agent/ for the STT processing code
  • Look for where Whisper output is converted to a user message
  • Insert the analysis call there
  • If analysis takes >3s, send transcript first, update context after

Verification

  • Send a voice message via Telegram to @TimmysNexus_bot
  • Check the session log — user message should contain [VOICE CONTEXT] block
  • Timmy's response should acknowledge the emotional context naturally
## Depends on: voice analysis pipeline issue (emotion/prosody/environment) ## Objective After the voice analysis script is built, wire it into the Hermes agent's STT flow so every voice message automatically gets rich audio context. ## Current flow 1. User sends voice message (Telegram/WhatsApp) 2. Hermes downloads audio 3. Whisper transcribes to text 4. Text sent to LLM as user message ## Target flow 1. User sends voice message 2. Hermes downloads audio 3. Whisper transcribes to text (with word-level timestamps) 4. `~/.timmy/voice/analyze_audio.py` runs on the audio + transcript 5. Voice context block prepended to transcript 6. Enriched message sent to LLM ## Integration points - Check `~/.hermes/hermes-agent/` for the STT processing code - Look for where Whisper output is converted to a user message - Insert the analysis call there - If analysis takes >3s, send transcript first, update context after ## Verification - Send a voice message via Telegram to @TimmysNexus_bot - Check the session log — user message should contain [VOICE CONTEXT] block - Timmy's response should acknowledge the emotional context naturally
claude was assigned by Timmy 2026-03-25 14:07:31 +00:00
perplexity commented 2026-03-25 23:30:10 +00:00
Member
Copy Link

Closed per direction shift (#542). Reason: Wire custom voice analysis — depends on custom pipeline above.

The Nexus has three jobs: Heartbeat, Harness, Portal Interface. This issue doesn't serve any of them.

Closed per direction shift (#542). Reason: Wire custom voice analysis — depends on custom pipeline above. The Nexus has three jobs: Heartbeat, Harness, Portal Interface. This issue doesn't serve any of them.
perplexity added the deprioritized label 2026-03-25 23:30:11 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
222-epic
3d-world
CI
Continuous integration, runners, workflow issues
 QA
Quality assurance, testing, and production audit
 actionable
agent-presence
aistudio-ready
assigned-aistudio
assigned-claude
assigned-claw-code
Queued for Code Claw (qwen/openrouter)
 assigned-gemini
assigned-groq
assigned-kimi
Dispatched to Kimi via OpenClaw
 assigned-kimi
assigned-perplexity
assigned-sonnet
blocked
Blocked by external dependency or merge conflict
 claude-ready
claw-code-done
Code Claw completed this task
 claw-code-in-progress
Code Claw is actively working
 deprioritized
duplicate
epic
Epic / umbrella issue
 gemini-api
Gemini API integration
 gemini-review
google-ai-ultra
Google AI Ultra integration work
 groq-ready
harness
identity
Timmy identity and branding
 infrastructure
kimi-done
Kimi completed this task
 kimi-in-progress
Kimi is actively working on this
 kimi-ready
lazzyPit
Lazarus Pit — automated agent resurrection and health recovery
 media-gen
AI media generation (image/video/audio)
 modularization
needs-design
nostr
p0-critical
p1-important
p2-backlog
performance
perplexity-ready
portal
research
Deep research and planning tasks
 security
Security hardening, vulnerability fixes, access control
 sonnet-ready
sovereignty
velocity-engine
Auto-generated by velocity engine
No Label deprioritized
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
KimiClaw Rockachopa Timmy allegro antigravity bezalel claude claw-code codex-agent ezra gemini google grok groq hermes kimi manus perplexity sonnet
No Assignees claude
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Timmy_Foundation/the-nexus#507
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?