Salvage of #3399 by @binhnt92 with true agent interruption added on top.
When a streaming /v1/chat/completions client disconnects mid-stream, the agent is now interrupted via agent.interrupt() so it stops making LLM API calls, and the asyncio task wrapper is cancelled.
Closes#3399.
python-telegram-bot's BadRequest inherits from NetworkError, so the
send() retry loop was catching 'Message thread not found' as a transient
network error and retrying 3 times before silently failing. This killed
all tool progress messages, streaming responses, and typing indicators
when the incoming message carried an invalid message_thread_id.
Now detect BadRequest inside the NetworkError handler:
- 'thread not found' + thread_id set → clear thread_id and retry once
(message still reaches the chat, just without topic threading)
- Other BadRequest errors → raise immediately (permanent, don't retry)
- True NetworkError → retry as before (transient)
252 silent failures in gateway.log traced to this on 2026-03-26.
5 new tests for thread fallback, non-thread BadRequest, no-thread sends,
network retry, and multi-chunk fallback.
* feat(telegram): auto-discover fallback IPs via DoH when api.telegram.org is unreachable
On some networks (university, corporate), api.telegram.org resolves to a
valid Telegram IP that is unreachable due to routing/firewall rules. A
different IP in the same Telegram-owned 149.154.160.0/20 block works fine.
This adds automatic fallback IP discovery at connect time:
1. Query Google and Cloudflare DNS-over-HTTPS for api.telegram.org A records
2. Exclude the system-DNS IP (the unreachable one), use the rest as fallbacks
3. If DoH is also blocked, fall back to a seed list (149.154.167.220)
4. TelegramFallbackTransport tries primary first, sticks to whichever works
No configuration needed — works automatically. TELEGRAM_FALLBACK_IPS env var
still available as manual override. Zero impact on healthy networks (primary
path succeeds on first attempt, fallback never exercised).
No new dependencies (uses httpx already in deps + stdlib socket).
* fix: share transport instance and downgrade seed fallback log to info
- Use single TelegramFallbackTransport shared between request and
get_updates_request so sticky IP is shared across polling and API calls
- Keep separate HTTPXRequest instances (different timeout settings)
- Downgrade "using seed fallback IPs" from warning to info to avoid
noisy logs on healthy networks
* fix: add telegram.request mock and discovery fixture to remaining test files
The original PR missed test_dm_topics.py and
test_telegram_network_reconnect.py — both need the telegram.request
mock module. The reconnect test also needs _no_auto_discovery since
_handle_polling_network_error calls connect() which now invokes
discover_fallback_ips().
---------
Co-authored-by: Mohan Qiao <Gavin-Qiao@users.noreply.github.com>
* fix(gateway): add media download retry to Mattermost, Slack, and base cache
Media downloads on Mattermost and Slack fail permanently on transient
errors (timeouts, 429 rate limits, 5xx server errors). Telegram and
WhatsApp already have retry logic, but these platforms had single-attempt
downloads with hardcoded 30s timeouts.
Changes:
- base.py cache_image_from_url: add retry with exponential backoff
(covers Signal and any platform using the shared cache helper)
- mattermost.py _send_media_url: retry on 429/5xx/timeout (3 attempts)
- slack.py _download_slack_file: retry on timeout/5xx (3 attempts)
- slack.py _download_slack_file_bytes: same retry pattern
* test: add tests for media download retry
---------
Co-authored-by: dieutx <dangtc94@gmail.com>
When user messages have empty content (e.g., Discord @mention-only
messages, unrecognized attachments), the Anthropic API rejects the
request with 'user messages must have non-empty content'.
Changes:
- anthropic_adapter.py: Add empty content validation for user messages
(string and list formats), matching the existing pattern for assistant
and tool messages. Empty content gets '(empty message)' placeholder.
- discord.py: Defense-in-depth check at gateway layer to catch empty
messages before they enter session history.
- Add 4 regression tests covering empty string, whitespace-only,
empty list, and empty text block scenarios.
Fixes#3143
Co-authored-by: Bartok9 <bartok9@users.noreply.github.com>
signal-cli sends SSE comment lines (':') as keepalives every ~15s. The
SSE listener only counted 'data:' lines as activity, so the health
monitor reported false idle warnings every 2 minutes during quiet
periods. Recognize ':' lines as valid activity per the SSE spec.
Salvaged from PR #2938 by ticketclosed-wontfix.
The API server adapter was creating agents without specifying
enabled_toolsets, causing ALL tools to load — including clarify,
send_message, and text_to_speech which don't work without interactive
callbacks or gateway dispatch.
Changes:
- toolsets.py: Add hermes-api-server toolset (core tools minus clarify,
send_message, text_to_speech)
- api_server.py: Resolve toolsets from config.yaml platform_toolsets
via _get_platform_tools() — same path as all other gateway platforms.
Falls back to hermes-api-server default when no override configured.
- tools_config.py: Add api_server to PLATFORMS dict so users can
customize via 'hermes tools' or platform_toolsets.api_server in
config.yaml
- 12 tests covering toolset definition, config resolution, and
user override
Reported by thatwolfieguy on Discord.
When send() fails due to a network error (ConnectError, ReadTimeout, etc.),
the failure was silently logged and the user received no feedback — appearing
as a hang. In one reported case, a user waited 1+ hour for a response that
had already been generated but failed to deliver (#2910).
Adds _send_with_retry() to BasePlatformAdapter:
- Transient errors: retry up to 2x with exponential backoff + jitter
- On exhaustion: send delivery-failure notice so user knows to retry
- Permanent errors: fall back to plain-text version (preserves existing behavior)
- SendResult.retryable flag for platform-specific transient errors
All adapters benefit automatically via BasePlatformAdapter inheritance.
Cherry-picked from PR #3108 by Mibayy.
Co-authored-by: Mibayy <mibayy@users.noreply.github.com>
When the homeserver returns an error response, matrix-nio parses it
as a SyncError return value rather than raising an exception. The sync
loop only had backoff in the except handler, so SyncError caused a
tight retry loop (~489 req/s) flooding logs and hammering the
homeserver. Check the return value and sleep 5s before retry.
Cherry-picked from PR #2937 by ticketclosed-wontfix.
Co-authored-by: ticketclosed-wontfix <ticketclosed-wontfix@users.noreply.github.com>
After a Telegram 502, _handle_polling_network_error calls updater.stop()
then start_polling(). If start_polling() also raises, the old code logged
a warning and returned — but the comment 'The next network error will
trigger another attempt' was wrong. The updater loop is dead after stop(),
so no further error callbacks ever fire. The gateway stays alive but
permanently deaf to messages.
Fix: when start_polling() fails in the except branch, schedule a new
_handle_polling_network_error task to continue the exponential backoff
retry chain. The task is tracked in _background_tasks (preventing GC).
Guarded by has_fatal_error to avoid spurious retries during shutdown.
Closes#3173.
Salvaged from PR #3177 by Mibayy.
Python's asyncio event loop holds only weak references to tasks.
Without a strong reference, the garbage collector can destroy a task
while it's awaiting I/O — silently dropping messages. Python 3.12+
made this more aggressive.
Audit of all gateway platform adapters found 6 untracked create_task
calls across 6 files:
Per-message tasks (tracked via _background_tasks set from base class):
- gateway/platforms/webhook.py: handle_message task
- gateway/platforms/sms.py: handle_message task
- gateway/platforms/signal.py: SSE response aclose task
Long-running infrastructure tasks (stored in named instance vars):
- gateway/platforms/slack.py: Socket Mode handler (_socket_mode_task)
- gateway/platforms/discord.py: bot client (_bot_task)
- gateway/platforms/whatsapp.py: message poll loop (_poll_task, 2 sites)
All other adapters (telegram, mattermost, matrix, email, homeassistant,
dingtalk) already tracked their tasks correctly.
Salvaged from PR #3160 by memosr — expanded from 1 file to 6.
Add timeout=30 to all bare ClientSession, IMAP4_SSL, smtplib.SMTP, and
ws_connect calls that previously had no timeout, preventing indefinite
hangs when an external server is slow or unresponsive.
Adapters hardened:
- HomeAssistant: REST + WS session creation, ws_connect handshake
- Email: all IMAP4_SSL (x2) and smtplib.SMTP (x3) calls
- Mattermost: session creation, _api_get, _api_post, _upload_file (60s)
- SMS: session creation in connect() + fallback session in send()
Salvaged from PRs #3161, #3168, #3170 (memosr) and #3201 (binhnt92).
SMS fallback ClientSession on send() also patched (missed in #3201).
Co-authored-by: memosr <memosr@users.noreply.github.com>
Co-authored-by: nguyen binh <binhnt92@users.noreply.github.com>
Salvages PR #3005 by web3blind. Cherry-picked onto current main with functional skill binding and docs added.
- DM topic creation via createForumTopic (Bot API 9.4, Feb 2026)
- Config-driven topics with thread_id persistence across restarts
- Session isolation via existing build_session_key thread_id support
- auto_skill field on MessageEvent for topic-skill bindings
- Gateway auto-loads bound skill on new sessions (same as /skill commands)
- Docs: full Private Chat Topics section in Telegram messaging guide
- 20 tests (17 original + 3 for auto_skill)
Closes#2598
Co-authored-by: web3blind <web3blind@users.noreply.github.com>
Two fixes for a race where Discord's typing indicator lingers after the
agent finishes:
1. _keep_typing (root cause): after outer stop_typing() clears the task
dict, _keep_typing wakes from its 2s sleep and calls send_typing()
again, recreating an orphaned loop. Add a finally block so _keep_typing
always calls stop_typing() on exit, cleaning up any loop it recreated.
2. _process_message_background (safety net): add stop_typing() after
cancelling the typing task, catching any platform-level persistent
typing tasks that slipped through.
Combines fixes from PR #2945 by catbusconductor (root cause in
_keep_typing) and PR #2832 by subrih (safety net in
_process_message_background).
* fix(run_agent): ensure _fire_first_delta() is called for tool generation events
Added calls to _fire_first_delta() in the AIAgent class to improve the handling of tool generation events, ensuring timely notifications during the processing of function calls and tool usage.
* fix(run_agent): improve timeout handling for chat completions
Enhanced the timeout configuration for chat completions in the AIAgent class by introducing customizable connection, read, and write timeouts using environment variables. This ensures more robust handling of API requests during streaming operations.
* fix(run_agent): reduce default stream read timeout for chat completions
Updated the default stream read timeout from 120 seconds to 60 seconds in the AIAgent class, enhancing the timeout configuration for chat completions. This change aims to improve responsiveness during streaming operations.
* fix(run_agent): enhance streaming error handling and retry logic
Improved the error handling and retry mechanism for streaming requests in the AIAgent class. Introduced a configurable maximum number of stream retries and refined the handling of transient network errors, allowing for retries with fresh connections. Non-transient errors now trigger a fallback to non-streaming only when appropriate, ensuring better resilience during API interactions.
* fix(api_server): streaming breaks when agent makes tool calls
The agent fires stream_delta_callback(None) to signal the CLI display
to close its response box before tool execution begins. The API server's
_on_delta callback was forwarding this None directly into the SSE queue,
where the SSE writer treats it as end-of-stream and terminates the HTTP
response prematurely.
After tool calls complete, the agent streams the final answer through
the same callback, but the SSE response was already closed — so Open
WebUI (and similar frontends) never received the actual answer.
Fix: filter out None in _on_delta so the SSE stream stays open. The SSE
loop already detects completion via agent_task.done(), which handles
stream termination correctly without needing the None sentinel.
Reported by Rohit Paul on X.
Add downloadMediaMessage() calls for documents, audio/voice notes, and
video in bridge.js — previously only images were downloaded, leaving all
other file types inaccessible to the agent.
Handle local file paths from the bridge for DOCUMENT, VOICE, and VIDEO
types in whatsapp.py with proper MIME detection. Inject text content
inline for readable files (.txt, .md, .csv, .json, etc.).
Follow-up fixes applied during salvage:
- Remove unused cache_document_from_bytes import
- Add 100KB size cap on text injection (matches Telegram/Discord/Slack)
- Align injection format with other platforms
Cherry-picked from PR #2818. Also fixes#2856 (bugs 1 & 2).
PR #2865 by ayberkesn fixed the same voice note issue.
Co-authored-by: noestelar <hola@noeali.com>
Add reply_to_mode setting (off/first/all) to control whether Telegram
replies quote/thread to the user's original message.
- 'off': Never thread replies (no quote bubble)
- 'first': Only first chunk threads to user's message (default, preserves existing behavior)
- 'all': All chunks in multi-part replies thread to user's message
Configurable via:
- reply_to_mode in platform config (gateway config YAML)
- TELEGRAM_REPLY_TO_MODE env var
Based on PR #855 by raulvidis.
Cherry-picked from PR #2583 by Glucksberg.
The MEDIA: regex used \S+ which truncated paths at the first space.
Added a space-aware alternative anchored to known media extensions.
Also updated extract_local_files to allow spaces in path segments.
Follow-up fix: changed \s to [^\S\n] in the space-matching group
so the regex doesn't greedily match across newlines (broke multi-line
MEDIA: tags).
Cherry-picked from PR #2575 by ticketclosed-wontfix.
Filters out Discord system messages (thread renames, pins, member joins,
boosts) that were being treated as regular user messages.
Follow-up fix: also allow MessageType.reply (value 19) — the original
filter only allowed MessageType.default, which would silently drop all
reply-based interactions.
Added pytest.importorskip for discord dependency in tests.
Three fixes for the Matrix adapter:
1. Remove RoomMessageMedia callback registration — RoomMessageImage
inherits from it, causing images to be processed twice.
2. Add event ID deduplication to both text and media handlers.
nio can fire the same event more than once; bounded deque+set
tracks the last 1000 events.
3. Cache images locally via Matrix client download. MXC URLs require
authentication, so the vision pipeline couldn't access them.
Images are now downloaded via the authenticated client and saved
to the local cache (same pattern as Telegram/Discord).
Cherry-picked from PR #2353 by williamtwomey.
Co-authored-by: williamtwomey <williamtwomey@users.noreply.github.com>
Closes#2476
The polling error callback previously only handled Conflict errors
(409 from multiple getUpdates callers). All other errors, including
NetworkError and TimedOut that python-telegram-bot raises when the
host loses connectivity (Mac sleep, WiFi switch, VPN reconnect),
were logged and silently discarded. The bot would stop responding
until manually restarted.
Fix:
- Add _looks_like_network_error() to classify transient connectivity
errors (NetworkError, TimedOut, OSError, ConnectionError).
- Add _handle_polling_network_error() with exponential back-off
reconnect: retries up to 10 times with delays 5s, 10s, 20s, 40s,
60s (capped). On exhaustion, marks the adapter retryable-fatal so
launchd/systemd can restart the gateway process.
- Refactor _polling_error_callback() to route network errors to the
new handler before falling through to a generic error log.
- Track _polling_network_error_count (reset on successful reconnect)
independently from _polling_conflict_count.
- Download and cache .pdf, .docx, .xlsx, .pptx attachments locally
instead of passing expiring CDN URLs to the agent
- Inject .txt and .md content (≤100 KB) into event.text so the agent
sees file content without needing to fetch the URL
- Add 20 MB size guard and SUPPORTED_DOCUMENT_TYPES allowlist
- Fix: unsupported types (.zip etc.) no longer get MessageType.DOCUMENT
- Add 9 unit tests in test_discord_document_handling.py
Mirrors the Slack implementation from PR #784. Discord CDN URLs are
publicly accessible so no auth header is needed (unlike Slack).
Co-authored-by: Dilee <uzmpsk.dilekakbas@gmail.com>
The /v1/responses endpoint used an in-memory OrderedDict that lost
all conversation state on gateway restart. Replace with SQLite-backed
storage at ~/.hermes/response_store.db.
- Responses and conversation name mappings survive restarts
- Same LRU eviction behavior (configurable max_size)
- WAL mode for concurrent read performance
- Falls back to in-memory SQLite if disk path unavailable
- Conversation name→response_id mapping moved into the store
Based on PR #2427 by @oxngon (core feature extracted, reformatting
and unrelated changes dropped).
Discord's TYPING_START gateway event is unreliable for bot DMs. This
adds a background typing loop that hits POST /channels/{id}/typing
every 8 seconds (indicator lasts ~10s) until the response is sent.
- send_typing() starts a per-channel background loop (idempotent)
- stop_typing() cancels it (called after _run_agent returns)
- Base adapter gets stop_typing() as a no-op default
- Per-channel tracking via _typing_tasks dict prevents duplicates
imap.uid('search') can return data=[] when the mailbox is empty or
has no matching messages. Accessing data[0] without checking len first
raises IndexError: list index out of range.
Fixed at both call sites in gateway/platforms/email.py:
- Line 233 (connect): ALL search on startup
- Line 298 (fetch): UNSEEN search in the polling loop
Closes#2137
Cherry-picked from PR #2017 by @simpolism. Fixes#2011.
Discord slash commands in threads were missing thread_id in the
SessionSource, causing them to route to the parent channel session.
Commands like /usage and /reset returned wrong data or affected the
wrong session.
Detects discord.Thread channels in _build_slash_event and sets
chat_type='thread' with thread_id. Two tests added.
Five improvements to the /api/jobs endpoints:
1. Startup availability check — cron module imported once at class load,
endpoints return 501 if unavailable (not 500 per-request import error)
2. Input limits — name ≤ 200 chars, prompt ≤ 5000 chars, repeat must be
positive int
3. Update field whitelist — only name/schedule/prompt/deliver/skills/
repeat/enabled pass through to cron.jobs.update_job, preventing
arbitrary key injection
4. Deduplicated validation — _check_job_id and _check_jobs_available
helpers replace repeated boilerplate
5. 32 new tests covering all endpoints, validation, auth, and
cron-unavailable cases
CRUD + actions for cron jobs on the existing API server (port 8642):
GET /api/jobs — list jobs
POST /api/jobs — create job
GET /api/jobs/{id} — get job
PATCH /api/jobs/{id} — update job
DELETE /api/jobs/{id} — delete job
POST /api/jobs/{id}/pause — pause job
POST /api/jobs/{id}/resume — resume job
POST /api/jobs/{id}/run — trigger immediate run
All endpoints use existing API_SERVER_KEY auth. Job ID format
validated (12 hex chars). Logic ported from PR #2111 by nock4,
adapted from FastAPI to aiohttp on the existing API server.
The Mattermost adapter now only responds to messages in channels and
groups when the bot is @-mentioned. DMs are always processed without
filtering.
Detection checks both the bot's @username and user ID in the message
text, providing a reliable fallback when the structured mentions field
is unavailable.
Fixes#2174
Cherry-picked from PR #2365 by @xerpert.
Three bugs preventing Signal image attachments from being processed:
1. signal-cli getAttachment RPC expects 'id', not 'attachmentId'
2. signal-cli daemon returns dict {"data": "base64..."} not raw base64
3. MessageType.IMAGE doesn't exist — correct enum is MessageType.PHOTO
The MarkdownV2 format_message conversion left unescaped ( ) { }
in edge cases where placeholder processing didn't cover them (e.g.
partial link matches, URLs with parens). This caused Telegram to
reject the message with 'character ( is reserved and must be escaped'
and fall back to plain text — losing all formatting.
Added a safety-net pass (step 12) after placeholder restoration that
escapes any remaining bare ( ) { } outside code blocks and valid
MarkdownV2 link syntax.
Fixes#1803. send_image_file, send_document, and send_video were missing
message_thread_id forwarding, causing them to fail in Telegram forum/supergroups
where thread_id is required. send_voice already handled this correctly. Adds
metadata parameter + message_thread_id to all three methods, and adds tests
covering the thread_id forwarding path.
Bare strings like "image", "audio", "document" were appended to
media_types, but downstream run.py checks mtype.startswith("image/")
and mtype.startswith("audio/"), which never matched. This caused all
Mattermost file attachments to be silently dropped from vision/STT
processing. Use the actual MIME type from file_info instead.
A single Telegram 409 Conflict from getUpdates permanently killed
Telegram polling with no recovery possible (retryable=False on
first occurrence). This is too aggressive for production use with
process supervisors.
Transient 409s are expected during:
- --replace handoffs where the old long-poll session lingers on
Telegram servers for a few seconds after SIGTERM
- systemd Restart=on-failure respawns that overlap with the dying
instance cleanup
Now _handle_polling_conflict() retries up to 3 times with a
10-second delay between attempts. The 30-second total retry window
lets stale server-side sessions expire. If all retries fail, the
error is still marked as permanently fatal — preserving the original
protection against genuine dual-instance conflicts.
Tests updated: split the single conflict test into two — one verifying
retry on transient conflict, one verifying fatal after exhausted
retries.
Closes#2296
- Convert ~~text~~ to ~text~ (MarkdownV2 strikethrough)
- Protect ||text|| from pipe escaping (MarkdownV2 spoiler)
- Preserve > at line start as blockquote instead of escaping it
- Update _strip_mdv2() to strip ~strikethrough~ and ||spoiler|| markers
- Add tests covering new formatting paths and edge cases
Salvaged from PR #2162 by @Zindar. Reply prefix changes excluded (already
on main via #1756 configurable prefix).
Bridge improvements (bridge.js):
- Download incoming images to ~/.hermes/image_cache/ via downloadMediaMessage
so the agent can actually see user-sent photos
- Add getMessage callback required for Baileys 7.x E2EE session
re-establishment (without it, some messages arrive as null)
- Build LID→phone reverse map for allowlist resolution (WhatsApp LID format)
- Add placeholder body for media without caption: [image received]
- Bind express to 127.0.0.1 instead of 0.0.0.0 for security
- Use 127.0.0.1 consistently throughout (more reliable than localhost)
Adapter improvements (whatsapp.py):
- Detect and reuse already-running bridge (only if status=connected)
- Handle local file paths from bridge-cached images in _build_message_event
- Don't kill external bridges on disconnect
- Use 127.0.0.1 throughout for consistency with bridge binding
Fix vs original PR: bridge reuse now checks status=connected, not just
HTTP 200. A disconnected bridge gets restarted instead of reused.
Co-authored-by: Zindar <zindar@users.noreply.github.com>
Support Signal 'Note to Self' messages in single-number setups where
signal-cli is linked as a secondary device on the user's own account.
syncMessage.sentMessage envelopes addressed to the bot's own account
are now promoted to dataMessage for normal processing, while other
sync events (read receipts, typing, etc.) are still filtered.
Echo-back prevention mirrors the WhatsApp bridge pattern:
- Track timestamps of recently sent messages (bounded set of 50)
- When a Note to Self sync arrives, check if its timestamp matches
a recent outbound — skip if so (agent echo-back)
- Only process sync messages that are genuinely user-initiated
Based on PR #2115 by @Stonelinks with added echo-back protection.
Previously, if an error occurred during response processing in
_process_message_background (e.g. during extract_media, send, or
any uncaught exception from the handler), the error was only logged
to server console and the user was left with radio silence — typing
indicator stops but no message arrives.
Now the outer except block attempts to send the error type and detail
(truncated to 300 chars) to the user's chat, matching the format
already used by the inner handler in gateway/run.py.
Co-authored-by: Test <test@test.com>
Fixes#1842
The MessageEvent dataclass expects 'reply_to_message_id' but the Matrix
connector was passing 'reply_to'. This caused replies to fail with:
MessageEvent.__init__() got an unexpected keyword argument 'reply_to'
Changed the parameter name to match the dataclass definition.
