Rockachopa/Timmy-time-dashboard
1
0
Fork 2
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

[Observability] loop_guard.py: Detect Deadlocks and Kill Zombie Python Processes Faster #1425

New Issue
Closed
opened 2026-03-24 13:04:42 +00:00 by Timmy · 1 comment
Timmy commented 2026-03-24 13:04:42 +00:00
Owner
Copy Link

Context: Zombie loop tests require manual ops-kill-zombies interference.

Acceptance Criteria:

  • Optimize loop_guard to run consistently in a lightweight daemon.
  • Detect specific stuck pytest hooks surpassing 15 minutes of inactivity and execute SIGKILL directly.
**Context:** Zombie loop tests require manual `ops-kill-zombies` interference. **Acceptance Criteria:** - Optimize `loop_guard` to run consistently in a lightweight daemon. - Detect specific stuck pytest hooks surpassing 15 minutes of inactivity and execute SIGKILL directly.
Timmy commented 2026-03-24 14:11:50 +00:00
Author
Owner
Copy Link

Implementation Plan for Deadlock Detection and Zombie Process Killer

Scope: Enhance loop_guard.py to detect and kill zombie Python processes that exceed inactivity thresholds.

Current State Analysis

  • scripts/loop_guard.py is 325 lines, handles idle detection with exponential backoff
  • Currently focuses on queue state, not process health
  • Manual ops-kill-zombies intervention required for stuck processes

Implementation Steps

  1. Add Process Monitoring Module

    • Create detect_zombie_processes() function
    • Use psutil to find Python processes with:
      • pytest in command line
      • CPU usage < 1% for > 15 minutes
      • Memory not growing for > 15 minutes

  2. Daemon Mode Enhancement

    • Add --daemon flag for continuous monitoring
    • Run lightweight checks every 60 seconds
    • Log process monitoring activity

  3. Process Termination Logic

    • Escalate: SIGTERM → wait 30s → SIGKILL
    • Log all termination actions with PID, command, and duration
    • Avoid killing active processes (check recent file I/O)

  4. Configuration

    • Add timeout thresholds to config (default 15 min)
    • Whitelist processes that should never be killed
    • Enable/disable zombie detection via flag

Files to Modify

  • scripts/loop_guard.py (primary - add process monitoring)
  • Add dependency: psutil to requirements

Testing

  • Create test for zombie process detection
  • Mock psutil for process simulation
  • Verify SIGTERM/SIGKILL escalation

Acceptance Criteria

  • loop_guard.py --daemon runs continuously
  • Detects pytest processes stuck > 15 min
  • Executes SIGTERM → SIGKILL escalation
  • Logs all process termination actions
  • All existing tests still pass
  • No false positives (killing active processes)

Priority: HIGH - This addresses operational pain points and prevents manual intervention.

## Implementation Plan for Deadlock Detection and Zombie Process Killer **Scope:** Enhance loop_guard.py to detect and kill zombie Python processes that exceed inactivity thresholds. ### Current State Analysis - `scripts/loop_guard.py` is 325 lines, handles idle detection with exponential backoff - Currently focuses on queue state, not process health - Manual `ops-kill-zombies` intervention required for stuck processes ### Implementation Steps 1. **Add Process Monitoring Module** - Create `detect_zombie_processes()` function - Use `psutil` to find Python processes with: - `pytest` in command line - CPU usage < 1% for > 15 minutes - Memory not growing for > 15 minutes 2. **Daemon Mode Enhancement** - Add `--daemon` flag for continuous monitoring - Run lightweight checks every 60 seconds - Log process monitoring activity 3. **Process Termination Logic** - Escalate: SIGTERM → wait 30s → SIGKILL - Log all termination actions with PID, command, and duration - Avoid killing active processes (check recent file I/O) 4. **Configuration** - Add timeout thresholds to config (default 15 min) - Whitelist processes that should never be killed - Enable/disable zombie detection via flag ### Files to Modify - `scripts/loop_guard.py` (primary - add process monitoring) - Add dependency: `psutil` to requirements ### Testing - Create test for zombie process detection - Mock psutil for process simulation - Verify SIGTERM/SIGKILL escalation ### Acceptance Criteria - [ ] `loop_guard.py --daemon` runs continuously - [ ] Detects pytest processes stuck > 15 min - [ ] Executes SIGTERM → SIGKILL escalation - [ ] Logs all process termination actions - [ ] All existing tests still pass - [ ] No false positives (killing active processes) **Priority:** HIGH - This addresses operational pain points and prevents manual intervention.
kimi was assigned by Timmy 2026-03-24 14:11:56 +00:00
kimi was unassigned by Timmy 2026-03-24 19:32:15 +00:00
Timmy closed this issue 2026-03-24 21:54:05 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
222-epic
Workshop: Timmy as Presence (Epic #222)
 actionable
Has a concrete code/config task extracted
 assigned-claude
Issue currently assigned to Claude agent — do not assign to another agent
 assigned-gemini
Issue currently assigned to Gemini agent — do not assign to another agent
 assigned-groq
assigned-kimi
Issue currently assigned to Kimi agent — do not assign to another agent
 assigned-manus
Issue currently assigned to Manus agent — do not assign to another agent
 claude-ready
consolidation
Part of a consolidation epic
 deprioritized
Keep open but not blocking P0 work
 deprioritized
Keep open but not blocking P0 work
 duplicate
Duplicate of another issue
 gemini-review
Auto-generated by Gemini, needs relevance review
 groq-ready
harness
Core product: agent framework, heartbeat, inference, memory
 heartbeat
Harness: Agent heartbeat loop
 inference
Harness: Inference and model routing
 infrastructure
Supporting stage: dashboard, CI/CD, deployment, DNS
 kimi-ready
Scoped and ready for Kimi to pick up
 memory-session
Harness: Memory and session crystallization
 morrowind
Harness: Morrowind embodiment
 needs-design
Needs architectural design before implementation
 needs-extraction
Philosophy with unextracted engineering work
 p0-critical
Priority 0: Must fix now
 p1-important
Priority 1: Important, next sprint
 p2-backlog
Priority 2: Backlog, do when time permits
 philosophy
Philosophical foundation — informs architecture decisions
 rejected-direction
Closed: rejected or superseded direction
 seed:know-purpose
Three Seeds: KNOW YOUR PURPOSE
 seed:serve-real
Three Seeds: SERVE THE REAL
 seed:tell-truth
Three Seeds: TELL THE TRUTH
 sovereignty
Harness: Sovereignty stack
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Timmy claude grok groq kimi manus perplexity Rockachopa
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Rockachopa/Timmy-time-dashboard#1425
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?