Timmy_Foundation/hermes-agent
15
0
Fork 0
Code Issues 113 Pull Requests 98 Actions 14 Packages Projects Releases Wiki Activity

BUG: Cron ticker thread not starting in gateway — 30 jobs stuck #342

New Issue
Closed
opened 2026-04-13 06:49:09 +00:00 by Timmy · 0 comments
Timmy commented 2026-04-13 06:49:09 +00:00
Owner
Copy Link

Root Cause

The gateway process (PID 83608, started 2026-04-13 02:02AM) is alive but the cron ticker thread never started.

Evidence:

  • Gateway log shows "Cron ticker started (interval=60s)" only on April 10 (two old entries)
  • No "Cron ticker started" message for the current gateway session (April 13)
  • Gateway log has 0 tick mentions in the last 200 lines
  • _start_cron_ticker is defined at line 8718 of gateway/run.py and called at line 8924
  • The thread is created but never starts executing cron_tick()

Impact:

  • 30+ cron jobs are scheduled but never execute
  • mimo-swarm-dispatcher, mimo-swarm-worker-2, mimo-swarm-worker-3 stuck with stale errors
  • All burn loops, nightwatch jobs, and swarm jobs affected
  • Only jobs that work are those triggered manually via hermes cron run <id>

Diagnosis Steps

  1. hermes cron list shows jobs with Next run in the past — scheduler not ticking
  2. grep "Cron ticker" ~/.hermes/logs/gateway.log shows only April 10 entries
  3. hermes cron tick works manually — the scheduler code is fine, the thread just isn't running
  4. Gateway PID is alive (kill -0 83608 succeeds) but no cron activity

Hypothesis

The gateway's _start_cron_ticker thread is created at line 8924 via threading.Thread(target=_start_cron_ticker, ...). The thread may be:

  • Blocked waiting for adapters/loop that aren't provided
  • Crashing silently on startup (exception caught by outer try/except)
  • Not started because the gateway was launched with --replace which may skip the ticker

Acceptance Criteria

  1. Gateway log shows "Cron ticker started" on every gateway restart
  2. Cron jobs fire within 60 seconds of gateway start
  3. Manual hermes cron tick is not required for jobs to execute
  4. All 4 mimo-swarm jobs (dispatcher + 3 workers) run successfully on schedule
  5. No stale "interpreter shutdown" errors after gateway replacement

Files

  • gateway/run.py — _start_cron_ticker (line 8718), thread creation (line 8924)
  • cron/scheduler.py — tick() function
  • Gateway log: ~/.hermes/logs/gateway.log
## Root Cause The gateway process (PID 83608, started 2026-04-13 02:02AM) is alive but the cron ticker thread never started. Evidence: - Gateway log shows "Cron ticker started (interval=60s)" only on April 10 (two old entries) - No "Cron ticker started" message for the current gateway session (April 13) - Gateway log has 0 tick mentions in the last 200 lines - `_start_cron_ticker` is defined at line 8718 of gateway/run.py and called at line 8924 - The thread is created but never starts executing `cron_tick()` Impact: - 30+ cron jobs are scheduled but never execute - mimo-swarm-dispatcher, mimo-swarm-worker-2, mimo-swarm-worker-3 stuck with stale errors - All burn loops, nightwatch jobs, and swarm jobs affected - Only jobs that work are those triggered manually via `hermes cron run <id>` ## Diagnosis Steps 1. `hermes cron list` shows jobs with `Next run` in the past — scheduler not ticking 2. `grep "Cron ticker" ~/.hermes/logs/gateway.log` shows only April 10 entries 3. `hermes cron tick` works manually — the scheduler code is fine, the thread just isn't running 4. Gateway PID is alive (`kill -0 83608` succeeds) but no cron activity ## Hypothesis The gateway's `_start_cron_ticker` thread is created at line 8924 via `threading.Thread(target=_start_cron_ticker, ...)`. The thread may be: - Blocked waiting for adapters/loop that aren't provided - Crashing silently on startup (exception caught by outer try/except) - Not started because the gateway was launched with `--replace` which may skip the ticker ## Acceptance Criteria 1. Gateway log shows "Cron ticker started" on every gateway restart 2. Cron jobs fire within 60 seconds of gateway start 3. Manual `hermes cron tick` is not required for jobs to execute 4. All 4 mimo-swarm jobs (dispatcher + 3 workers) run successfully on schedule 5. No stale "interpreter shutdown" errors after gateway replacement ## Files - `gateway/run.py` — `_start_cron_ticker` (line 8718), thread creation (line 8924) - `cron/scheduler.py` — `tick()` function - Gateway log: `~/.hermes/logs/gateway.log`
claude self-assigned this 2026-04-13 06:53:37 +00:00
Timmy closed this issue 2026-04-13 07:15:31 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CI
Continuous integration, runners, workflow issues
 QA
Quality assurance, testing, production audit
 assigned-claw-code
Queued for Code Claw (qwen/openrouter)
 assigned-kimi
Task assigned to KimiClaw for processing
 blocked
Blocked by external dependency or merge conflict
 claw-code-done
Code Claw completed this task
 claw-code-in-progress
Code Claw is actively working
 duplicate
Duplicate of another issue
 epic
Epic - large feature with multiple sub-tasks
 epic:memory
Unified Memory Architecture epic
 gaming
Gaming agent capabilities
 infra
Infrastructure, VPS, DNS, deployment
 kimi-done
KimiClaw has completed this task
 kimi-in-progress
KimiClaw is actively working on this
 mcp
MCP (Model Context Protocol) tools & servers
 morrowind
Morrowind Agent gameplay & MCP integration
 needs-review
PR or issue requires reviewer sign-off before merge/close
 p0-critical
p1-important
phase:1-activate
Phase 1: Activate existing systems
 phase:2-operational
Phase 2: Operational memory bridge
 phase:3-episodic
Phase 3: Proactive episodic recall
 phase:4-hygiene
Phase 4: Memory hygiene and quality
 phase:5-integration
Phase 5: Integration and testing
 poka-yoke
research
security
Security hardening, vulnerability fixes
 stale
No activity, pending triage or closure
 tool-evaluation
velocity-engine
Auto-generated by velocity engine
 wont-fix
Closed as intentionally not fixing — explicit descope
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Rockachopa Timmy allegro antigravity bezalel claude claw-code codex-agent ezra gemini google grok hermes kimi manus perplexity sonnet
No Assignees claude
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Timmy_Foundation/hermes-agent#342
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?