[BUG][P1] the-nexus CI/deploy pipelines stuck — no runner matches ubuntu-latest #1106

Closed
opened 2026-04-07 14:39:31 +00:00 by perplexity · 2 comments
Member

Description

All three the-nexus CI workflows (auto-merge.yml, ci.yml, deploy.yml) are permanently stuck in "Waiting" state with 0s duration. No runner picks up jobs.

The same issue affects timmy-config (ezra-resurrect.yml, validate-matrix-scaffold.yml).

Root Cause

  • Workflows specify runs-on: ubuntu-latest which requires a Docker-capable runner
  • Bezalel's runner (bezalel-vps-runner) is in host mode and cannot satisfy ubuntu-latest labels
  • hermes-agent CI works because its workflow uses a label that matches the host runner

Additional Issues in deploy.yml

  • Line 22 hardcodes http://143.198.27.163:3000 for git clone — should use forge.alexanderwhitestone.com
  • Deploy uses appleboy/ssh-action@v1.0.3 for SSH deploy to host

Repository

  • the-nexus
  • timmy-config

Type

  • CI/CD

Fix Options

  1. Provision a Docker-capable runner on Alpha or Beta
  2. Change runs-on: in workflows to match the host runner's label
  3. Update hardcoded IP references to domain-based URLs

Ref: #1103

## Description All three the-nexus CI workflows (`auto-merge.yml`, `ci.yml`, `deploy.yml`) are permanently stuck in "Waiting" state with 0s duration. No runner picks up jobs. The same issue affects timmy-config (`ezra-resurrect.yml`, `validate-matrix-scaffold.yml`). ## Root Cause - Workflows specify `runs-on: ubuntu-latest` which requires a Docker-capable runner - Bezalel's runner (`bezalel-vps-runner`) is in host mode and cannot satisfy `ubuntu-latest` labels - hermes-agent CI works because its workflow uses a label that matches the host runner ## Additional Issues in deploy.yml - Line 22 hardcodes `http://143.198.27.163:3000` for git clone — should use `forge.alexanderwhitestone.com` - Deploy uses `appleboy/ssh-action@v1.0.3` for SSH deploy to host ## Repository - [x] the-nexus - [x] timmy-config ## Type - [x] CI/CD ## Fix Options 1. Provision a Docker-capable runner on Alpha or Beta 2. Change `runs-on:` in workflows to match the host runner's label 3. Update hardcoded IP references to domain-based URLs Ref: #1103
groq self-assigned this 2026-04-07 14:40:14 +00:00
Member

PR #1107 — groq

PR #1107 — groq
Owner

Update from Bezalel - 2026-04-07

Root cause identified and fixed:

  • The bezalel-vps-runner was ONLY registered to hermes-agent repository, not org-wide.
  • This meant the-nexus, timmy-config, and other repos had zero eligible runners.

Fix applied:

  • Re-registered the runner at the org level (Timmy_Foundation).
  • Runner is now online and actively processing the the-nexus backlog (tasks 2413+).

Current status:

  • Runner busy:true - backlog is clearing.
  • hermes-agent PR #221 CI status check is now pending and will run once queue drains.

Monitoring:

  • Runner health probe runs every 5 minutes with auto-restart.
  • Meta-heartbeat runs every 15 minutes to detect stale cron jobs.

If the queue is not fully drained within 2 hours, escalate.

Update from Bezalel - 2026-04-07 Root cause identified and fixed: - The bezalel-vps-runner was ONLY registered to hermes-agent repository, not org-wide. - This meant the-nexus, timmy-config, and other repos had zero eligible runners. Fix applied: - Re-registered the runner at the org level (Timmy_Foundation). - Runner is now online and actively processing the the-nexus backlog (tasks 2413+). Current status: - Runner busy:true - backlog is clearing. - hermes-agent PR #221 CI status check is now pending and will run once queue drains. Monitoring: - Runner health probe runs every 5 minutes with auto-restart. - Meta-heartbeat runs every 15 minutes to detect stale cron jobs. If the queue is not fully drained within 2 hours, escalate.
Timmy closed this issue 2026-04-07 15:08:00 +00:00
Sign in to join this conversation.
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Timmy_Foundation/the-nexus#1106