Add bounded inference queue, cancellation, and overload fallback #64

Open
rockachopa wants to merge 3 commits from timmy/17-bounded-inference-queue into main

3 Commits

Author SHA1 Message Date
9118b69057 feat: graceful shutdown, disconnect-driven cancellation, and queue acceptance coverage
All checks were successful
Quality gates / quality (pull_request) Successful in 1m58s
- server.mjs tracks in-flight chats: browser disconnects abort the
  queued turn (releasing its slot), and SIGTERM/SIGINT cancels every
  in-flight turn so no Hermes subprocess is orphaned.
- vision-service forwards a per-request AbortSignal combined with the
  provider deadline via AbortSignal.any.
- acceptance suites cover overload fallback, urgent bypass under
  saturation, disconnect cleanup, recovery, and shutdown orphan checks.
- zero-call invariant tests prove urgent text never reaches Hermes or
  the vision provider under load.
2026-08-22 23:01:20 +00:00
5dcaaae4d6 feat: queue agent turns with cancellation and orphan-free timeout kills
Chat turns now flow through the bounded inference queue: FIFO with
configurable depth (TIMMY_AGENT_MAX_QUEUE_DEPTH) and concurrency
(TIMMY_AGENT_MAX_CONCURRENT_TURNS), sanitized overload/timeout
fallbacks, browser-disconnect cancellation via AbortSignal, and child
process SIGTERM/SIGKILL teardown so no Hermes subprocess survives a
cancelled turn.
2026-08-22 22:04:17 +00:00
38a59c8ee9 feat: bounded inference queue with deterministic cancellation
Shared primitive for Timmy #17: concurrency slots, FIFO max-depth
overload rejection with sanitized manual-fallback copy, queue-deadline
expiry, per-request AbortSignal delivery, immediate slot release on
external cancel.
2026-08-22 21:39:55 +00:00