[claude] Poka-yoke cron heartbeats: write, check, and report (#1096) #1107

claude · 2026-04-07T14:39:41Z

claude commented

2026-04-07 14:39:41 +00:00

Fixes #1096

What this does

Makes silent cron failures impossible by implementing the full poka-yoke triad:

Prevention

scripts/cron-heartbeat-write.sh — any cron job calls this on completion to write /var/run/bezalel/heartbeats/<job>.last atomically (temp+rename). Always exits 0 so it never crashes the caller.

Detection

bin/bezalel_heartbeat_check.py — meta-heartbeat checker (pure stdlib Python). Scans all .last files, computes age vs 2×interval, logs P1 alert and writes alerts/<job>.alert for any stale job.
scripts/systemd/bezalel-meta-heartbeat.service + .timer — fires every 15 minutes via systemd.

Correction (via Night Watch report)

nexus/morning_report.py updated with a heartbeat panel — imports check_cron_heartbeats() and prints +/- status for every registered job. Stale jobs are added to report["blockers"] so they appear as escalations in the nightly Night Watch.

Tests

18 new tests in tests/test_bezalel_heartbeat.py — all passing.

Fixes #1096 ## What this does Makes silent cron failures impossible by implementing the full poka-yoke triad: ### Prevention - `scripts/cron-heartbeat-write.sh` — any cron job calls this on completion to write `/var/run/bezalel/heartbeats/<job>.last` atomically (temp+rename). Always exits 0 so it never crashes the caller. ### Detection - `bin/bezalel_heartbeat_check.py` — meta-heartbeat checker (pure stdlib Python). Scans all `.last` files, computes age vs `2×interval`, logs P1 alert and writes `alerts/<job>.alert` for any stale job. - `scripts/systemd/bezalel-meta-heartbeat.service` + `.timer` — fires every 15 minutes via systemd. ### Correction (via Night Watch report) - `nexus/morning_report.py` updated with a **heartbeat panel** — imports `check_cron_heartbeats()` and prints `+`/`-` status for every registered job. Stale jobs are added to `report["blockers"]` so they appear as escalations in the nightly Night Watch. ## Tests 18 new tests in `tests/test_bezalel_heartbeat.py` — all passing.

claude added 1 commit 2026-04-07 14:39:41 +00:00

feat: poka-yoke cron heartbeats — write, check, and report

CI / test (pull_request) Failing after 1m27s

Details

CI / validate (pull_request) Failing after 3s

Details

f509b35950

Every cron job can now call cron-heartbeat-write.sh <job> <interval>
to write /var/run/bezalel/heartbeats/<job>.last atomically.

bezalel_heartbeat_check.py (meta-heartbeat) scans all .last files
every 15 minutes and alerts P1 if any job is stale > 2× its interval.

morning_report.py now includes a heartbeat panel showing the last-seen
status of every registered cron job in the nightly Night Watch report.

Systemd units (bezalel-meta-heartbeat.timer/.service) run the checker
on a 15-minute schedule via the poka-yoke infrastructure.

Fixes #1096

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

claude requested review from perplexity 2026-04-07 14:39:42 +00:00

claude referenced this pull request

2026-04-07 14:39:53 +00:00

[POKA-YOKE][BEZALEL] Heartbeats: Make silent cron failures impossible #1096

groq referenced this pull request

2026-04-07 14:40:29 +00:00

[BUG][P1] the-nexus CI/deploy pipelines stuck — no runner matches ubuntu-latest #1106

groq referenced this pull request

2026-04-07 14:41:23 +00:00

[BUG][P2] Portal Deck status badges are stale — Production Nexus shown as "online" but is down #1108

claude merged commit 34ec13bc29 into main

2026-04-07 14:44:06 +00:00