- Auto-TTS: voice messages get spoken response (audio first, then text)
- STT: Groq Whisper fallback when VOICE_TOOLS_OPENAI_KEY not set
- Futuristic UI: glassmorphism, centered container, purple theme, glow effects
- Voice bubble: custom waveform player with seek and progress
- Invisible TTS playback via play_tts() method (no audio file in chat)
- Add hermes-web toolset with full tool access
- Register Platform.WEB in toolset/config maps
- Update docs for voice conversation feature
Detect all network interfaces instead of relying on UDP trick which
returns VPN IP. Prefers 192.168.x.x/10.x.x.x over VPN ranges.
Shows all available IPs in console output.
Type /remote-control from any platform (Telegram, Discord, etc.) to
instantly start the web UI without restarting the gateway.
- Auto-generates access token if not provided
- Shows URL + token in response
- Optional: /remote-control [port] [token]
- Reports status if already running
- Added to /help command list
New platform adapter that serves a full-featured chat interface via HTTP.
Enables access from any device on the network (phone, tablet, desktop).
Features:
- aiohttp server with WebSocket real-time messaging
- Token-based authentication
- Markdown rendering (marked.js) + code highlighting (highlight.js)
- Voice recording via MediaRecorder API + STT transcription
- Image, voice, and document display
- Typing indicator + message editing (streaming support)
- Mobile responsive dark theme
- Auto-reconnect on disconnect
- Media file cleanup (24h TTL)
Config: WEB_UI_ENABLED=true, WEB_UI_PORT=8765, WEB_UI_TOKEN=<token>
No new dependencies — uses aiohttp already in [messaging] extra.
Cover CLI voice mode, Telegram/Discord auto voice reply, and Discord
voice channel support. Include setup guide with bot permissions, OAuth2
invite URL, privileged intents, system dependencies, and Python packages.
Update discord.md voice messages section with correct STT key reference.
Phase 2 of voice channel support: bot listens to users speaking in VC,
transcribes speech via Groq Whisper, and processes through the agent pipeline.
- Add VoiceReceiver class for RTP packet capture, NaCl/DAVE decryption, Opus decode
- Add silence detection and per-user PCM buffering
- Wire voice input callback from adapter to GatewayRunner
- Fix adapter dict key: use Platform.DISCORD enum instead of string
- Fix guild_id extraction for synthetic voice events via SimpleNamespace raw_message
- Pause/resume receiver during TTS playback to prevent echo
- Send Discord voice messages with flags=8192 and waveform metadata
so they render as native voice bubbles instead of file attachments
- Use .mp3 output path for TTS so edge-tts opus conversion works
correctly (edge always outputs mp3, convert was skipped for .ogg)
- Use actual file_path from TTS result after potential opus conversion
- Register /voice as Discord slash command with mode choices
- Fix _send_voice_reply to handle adapters that don't accept metadata
parameter (Discord) by inspecting the method signature at runtime
- /voice on: reply with voice when user sends voice messages
- /voice tts: reply with voice to all messages
- /voice off: disable, text-only replies
- /voice status: show current mode
- Per-chat state persisted to gateway_voice_mode.json
- Dedup: skips auto-reply if agent already called text_to_speech tool
- drop_pending_updates=True to ignore stale Telegram messages on restart
- 25 tests covering command handler, reply logic, and edge cases
The counter was incremented in start/stop/cancel but never read
anywhere in the codebase. The race condition it was meant to guard
against is practically impossible with the persistent stream design.
- Keep InputStream alive across recordings to avoid CoreAudio hang on
repeated open/close cycles on macOS. New _ensure_stream() creates the
stream once; start()/stop()/cancel() only toggle frame collection.
- Add _close_stream_with_timeout() with daemon thread to prevent
stream.stop()/close() from blocking indefinitely.
- Add generation counter to detect stale stream-open completions after
cancel or restart.
- Run recorder.cancel() in background thread from Ctrl+C handler to
keep the event loop responsive.
- Add shutdown() method called on /voice off to release audio resources.
- Fix silence timer reset during active speech: use dip tolerance for
_resume_start tracker so natural speech pauses (< 0.3s) don't prevent
the silence timer from being reset.
- Update tests to match persistent stream behavior.
AudioRecorder now auto-stops after 15 seconds if no speech is detected
(_has_spoken remains False). In quiet environments where ambient RMS
never exceeds the silence threshold (200), the recording would wait
indefinitely. The new _max_wait parameter fires the silence callback
after the timeout, triggering the normal "No speech detected" flow.
- Set max_retries=0 on the STT OpenAI client. The SDK default (2) honors
Groq's retry-after header (often 53s), blocking the thread for up to
~106s on rate limits. Voice STT should fail fast, not retry silently.
- Stop continuous recording mode after 3 consecutive no-speech cycles to
prevent infinite restart loops when nobody is talking.
- Set OpenAI client timeout=30s in transcribe_audio() — default 600s
blocks _voice_processing for 10 min if Groq/OpenAI stalls
- Move _voice_start_recording in _voice_stop_and_transcribe finally
block to a daemon thread (same pattern as Ctrl+B handler and
process_loop)
- Add _should_exit guard at top of _voice_start_recording so all 4
call sites respect shutdown without individual checks
- Replace sd.wait() with a poll loop + sd.stop() in play_beep().
sd.wait() calls Event.wait() without timeout — hangs forever if the
audio device stalls. Poll with a 2s ceiling and force-stop instead.
- Wrap _on_silence callback in try-except so exceptions are logged
instead of silently lost in the daemon thread. Prevents recording
state from becoming inconsistent on unexpected errors.
- process_loop's continuous mode restart called _voice_start_recording()
directly, blocking the loop if play_beep/sd.wait hangs — queued user
input would stall silently. Dispatch to daemon thread like Ctrl+B handler.
- Replace print() with _cprint() in _handle_voice_command for consistency
with the rest of the voice mode code.
The handle_voice_record key binding runs in prompt_toolkit's event-loop
thread. When silence auto-stopped recording, _voice_recording was False
but recorder.stop() still held AudioRecorder._lock. A concurrent Ctrl+B
press entered the START path and blocked on that lock, freezing all
keyboard input.
Three changes:
- Set _voice_processing atomically with _voice_recording=False in
_voice_stop_and_transcribe to close the race window
- Add _voice_processing guard in the START path to prevent starting
while stop/transcribe is still running
- Dispatch _voice_start_recording to a daemon thread so play_beep
(sd.wait) and AudioRecorder.start (lock acquire) never block the
event loop
browser_tool.py registered SIGINT/SIGTERM handlers that called sys.exit()
at module import time. When a signal arrived during a lock acquisition
(e.g. AudioRecorder._lock in voice mode), SystemExit was raised inside
prompt_toolkit's async event loop, corrupting coroutine state and making
the process unkillable (required SIGKILL).
atexit handler already ensures browser sessions are cleaned up on any
normal exit path, so the signal handlers were redundant and harmful.
- edge_tts NameError: _generate_edge_tts now calls _import_edge_tts()
instead of referencing bare module name (tts_tool.py)
- TTS thread leak: chat() finally block sends sentinel to text_queue,
sets stop_event, and joins tts_thread on exception paths (cli.py)
- output_stream leak: moved close() into finally block so audio device
is released even on exception (tts_tool.py)
- Ctrl+C continuous mode: cancel handler now resets _voice_continuous
to prevent auto-restart after user cancels recording (cli.py)
- _disable_voice_mode: now calls stop_playback() and sets
_voice_tts_done so TTS stops when voice mode is turned off (cli.py)
- _show_voice_status: reads record key from config instead of
hardcoding Ctrl+B (cli.py)
Bug A: Replace stale _HAS_ELEVENLABS/_HAS_AUDIO boolean imports with
lazy import function calls (_import_elevenlabs, _import_sounddevice).
The old constants no longer exist in tts_tool -- the try/except
silently swallowed the ImportError, leaving streaming TTS dead.
Bug B: Use user message prefix instead of modifying system prompt for
voice mode instruction. Changing ephemeral_system_prompt mid-session
invalidates the prompt cache. Now the concise-response hint is
prepended to the user_message passed to run_conversation while
conversation_history keeps the original text.
Minor: Add force parameter to _vprint so critical error messages
(max retries, non-retryable errors, API failures) are always shown
even during streaming TTS playback.
Tests: 15 new tests in test_voice_cli_integration.py covering all
three fixes -- lazy import activation, message prefix behavior,
history cleanliness, system prompt stability, and AST verification
that all critical _vprint calls use force=True.
- AudioRecorder.start() now catches InputStream errors gracefully
with a clear error message about microphone availability
- Fix config key mismatch: cli.py was reading "push_to_talk_key"
but config.py defines "record_key" -- now consistent
- Add format conversion from config format ("ctrl+b") to
prompt_toolkit format ("c-b")
1. Fully lazy imports: sounddevice, numpy, elevenlabs, edge_tts, and
openai are never imported at module level. Each is imported only when
the feature is explicitly activated, preventing crashes in headless
environments (SSH, Docker, WSL, no PortAudio).
2. No core agent loop changes: streaming TTS path extracted from
_interruptible_api_call() into separate _streaming_api_call() method.
The original method is restored to its upstream form.
3. Configurable key binding: push-to-talk key changed from Ctrl+R
(conflicts with readline reverse-search) to Ctrl+B by default.
Configurable via voice.push_to_talk_key in config.yaml.
4. Environment detection: new detect_audio_environment() function checks
for SSH, Docker, WSL, and missing audio devices before enabling voice
mode. Auto-disables with clear warnings in incompatible environments.
5. Graceful degradation: every audio touchpoint (sd.play, sd.InputStream,
sd.OutputStream) wrapped in try/except with ImportError/OSError
handling. Failures produce warnings, not crashes.
- Fix Gemini streaming tool call merge bug: multiple tool calls with same
index but different IDs are now parsed as separate calls instead of
concatenating names (e.g. ha_call_serviceha_call_service)
- Handle partial results in voice mode: show error and stop continuous
mode when agent returns partial/failed results with empty response
- Fix error display during streaming TTS: error messages are shown in
full response box even when streaming box was already opened
- Add duplicate sentence filter in TTS: skip near-duplicate sentences
from LLM repetition
- Fix fake HA server state mutation: turn_on/turn_off/set_temperature
correctly update entity states; temperature sensor simulates change
when thermostat is adjusted
- Add _vprint() helper to suppress log output when stream_callback is active
- Expand Whisper hallucination filter with multi-language phrases and regex pattern for repetitive text
- Stop continuous voice mode when agent returns a failed result (e.g. 429 rate limit)
- Atomic check-and-set for _voice_recording flag with _voice_lock
- Guard _voice_stop_and_transcribe against concurrent invocation
- Remove premature flag clearing from Ctrl+R handler
- Clean up temp WAV files in finally block (_play_via_tempfile)
- Use buffer-level regex for <think> block filtering (handles chunked tags)
- Prevent /voice on prompt accumulation on repeated calls
- Include Groq in STT key error message
Move screen output from stream_callback to display_callback called by
TTS consumer thread. Text now appears sentence-by-sentence in sync with
audio instead of streaming ahead at LLM speed. Removes quiet_mode hack.
sounddevice raises OSError (not ImportError) when the PortAudio C
library is missing. This broke test collection on CI runners that
have the Python package installed but lack the native library.
Stream audio to speaker as the agent generates tokens instead of
waiting for the full response. First sentence plays within ~1-2s
of agent starting to respond.
- run_agent: add stream_callback to run_conversation/chat, streaming
path in _interruptible_api_call accumulates chunks into mock
ChatCompletion while forwarding content deltas to callback
- tts_tool: add stream_tts_to_speaker() with sentence buffering,
think block filtering, markdown stripping, ElevenLabs pcm_24000
streaming to sounddevice OutputStream
- cli: wire up streaming TTS pipeline in chat(), detect elevenlabs
provider + sounddevice availability, skip batch TTS when streaming
is active, signal stop on interrupt
Falls back to batch TTS for Edge/OpenAI providers or when
elevenlabs/sounddevice are not available. Zero impact on non-voice
mode (callback defaults to None).
- Track submitted state locally instead of using racy qsize() check
- Allow Ctrl+R to stop recording even while agent is running
- Add double-start guard to prevent concurrent recording attempts
- Audio cues: beep on record start (880Hz), double beep on stop (660Hz)
- Silence detection: auto-stop recording after 3s of silence (RMS-based)
- Continuous mode: auto-restart recording after agent responds
- Ctrl+R starts continuous mode, Ctrl+R during recording exits it
- Waits for TTS to finish before restarting to avoid recording speaker
- Tests: 7 new tests for beep generation and silence detection
The test was failing because GROQ_API_KEY leaked from the environment.
Now both VOICE_TOOLS_OPENAI_KEY and GROQ_API_KEY are removed to
properly test the "no STT key" scenario.
- Change record key from c-@ to c-r (Ctrl+R) for macOS compatibility
- Add missing tempfile and time imports that caused silent TTS crash
- Use MP3 output for CLI TTS playback (afplay doesn't handle OGG well)
- Strip markdown formatting from text before sending to TTS
- Remove duplicate transcript echo in voice pipeline
- Add multi-provider STT support (OpenAI > Groq fallback) in transcription_tools
- Auto-correct model selection when provider doesn't support the configured model
- Change voice record key from Ctrl+Space to Ctrl+R (macOS compatibility)
- Fix duplicate transcript echo in voice pipeline
- Add GROQ_API_KEY to .env.example
