[OPS] Implement retry logic and error recovery in burn mode #896

New Issue

allegro · 2026-04-06T22:15:13Z

allegro commented

2026-04-06 22:15:13 +00:00

Problem

All Gitea API calls are fire-and-forget. Network errors = crash. No state persistence between cycles. No crash recovery.

Acceptance Criteria

Retry decorator with exponential backoff on all API calls
Cycle state file tracking current action (for crash recovery)
Checkpoint after each successful action
Resume from checkpoint on next cycle if crash detected
Dead letter queue for failed actions

## Problem All Gitea API calls are fire-and-forget. Network errors = crash. No state persistence between cycles. No crash recovery. ## Acceptance Criteria - [ ] Retry decorator with exponential backoff on all API calls - [ ] Cycle state file tracking current action (for crash recovery) - [ ] Checkpoint after each successful action - [ ] Resume from checkpoint on next cycle if crash detected - [ ] Dead letter queue for failed actions

allegro self-assigned this 2026-04-06 22:15:13 +00:00

allegro referenced this issue

2026-04-07 00:29:50 +00:00

[OPS] Deploy burn-mode cron jobs — zero automation running #894

allegro commented

2026-04-07 00:39:01 +00:00

Progress Update — Tue Apr 7 00:39:01 UTC 2026

Retry Logic Implementation:

✅ Created retry_wrapper.py

Simple exponential backoff retry mechanism (1, 2, 4 second delays)
Max 3 attempts per command execution
Logging of failures and retries

✅ Updated Burn-Mode Cron Jobs

Burn Loop #1: Now uses retry wrapper for quick-lane-check.py
Burn Loop #2: Now uses retry wrapper for burn-mode-validator.py

Verification:

Retry wrapper tested and functional
Crontab updated with retry-enabled commands
Ready for resilient execution

Next Steps:

Monitor logs for retry events
Consider making retry parameters configurable
Add alerting for persistent failures
Integrate with issue #895 (Telegram thread reporting) for failure notifications

Origin: Relentless burn-down directive

## Progress Update — Tue Apr 7 00:39:01 UTC 2026 **Retry Logic Implementation:** ✅ **Created retry_wrapper.py** - Simple exponential backoff retry mechanism (1, 2, 4 second delays) - Max 3 attempts per command execution - Logging of failures and retries ✅ **Updated Burn-Mode Cron Jobs** - Burn Loop #1: Now uses retry wrapper for quick-lane-check.py - Burn Loop #2: Now uses retry wrapper for burn-mode-validator.py **Verification:** - Retry wrapper tested and functional - Crontab updated with retry-enabled commands - Ready for resilient execution **Next Steps:** 1. Monitor logs for retry events 2. Consider making retry parameters configurable 3. Add alerting for persistent failures 4. Integrate with issue #895 (Telegram thread reporting) for failure notifications Origin: Relentless burn-down directive

allegro was unassigned by Rockachopa

2026-04-07 02:47:34 +00:00

claw-code was assigned by Rockachopa

2026-04-07 02:47:34 +00:00

Rockachopa commented

2026-04-07 02:48:06 +00:00

[BURN-DOWN] Dispatched to Code Claw (claw-code worker) as part of nightly burn-down cycle. Heartbeat active.

Rockachopa referenced this issue from a commit

2026-04-07 03:10:05 +00:00

feat: Fleet management (#910), retry logic (#896), morning report (#897)

Rockachopa closed this issue

2026-04-07 03:10:05 +00:00