Files
the-nexus/bin/__pycache__/nexus_watchdog.cpython-312.pyc

276 lines
28 KiB
Plaintext
Raw Normal View History

<EFBFBD>
ܒ<>i<EFBFBD>]<00><01> <00>dZddlmZddlZddlZddlZddlZddlZddlZddl Z ddl
Z
ddl Z ddl m Z mZddlmZddlmZmZmZmZ ddlmZdZej6ej8d
d <0B> <0C>ej:d <0A>ZdZdZ ejB<00>dz dz Z"dZ#dZ$dZ%dZ&ejNjQdd<17>Z)ejNjQdd<19>Z*ejNjQdd<1B>Z+dZ,dZ-e Gd<1E>d<1F><00>Z.e Gd <20>d!<21><00>Z/ee fd1d"<22>Z0d2d#<23>Z1e"e#f d3d$<24>Z2d2d%<25>Z3e%e&f d4d&<26>Z4d5d6d'<27>Z5d7d(<28>Z6d8d)<29>Z7d9d*<2A>Z8d:d+<2B>Z9ee e"e#f d;d,<2C>Z:d<d=d-<2D>Z;d>d.<2E>Z<d/<2F>Z=e>d0k(re=<3D>yy#e$rd ZY<00><01><wxYw)?u<>
Nexus Watchdog — The Eye That Never Sleeps
Monitors the health of the Nexus consciousness loop and WebSocket
gateway, raising Gitea issues when components go dark.
The nexus was dead for hours after a syntax error crippled
nexus_think.py. Nobody knew. The gateway kept running, but the
consciousness loop — the only part that matters — was silent.
This watchdog ensures that never happens again.
HOW IT WORKS
============
1. Probes the WebSocket gateway (ws://localhost:8765)
→ Can Timmy hear the world?
2. Checks for a running nexus_think.py process
→ Is Timmy's mind awake?
3. Reads the heartbeat file (~/.nexus/heartbeat.json)
→ When did Timmy last think?
4. If any check fails, opens a Gitea issue (or updates an existing one)
with the exact failure mode, timestamp, and diagnostic info.
5. If all checks pass after a previous failure, closes the issue
with a recovery note.
USAGE
=====
# One-shot check (good for cron)
python bin/nexus_watchdog.py
# Continuous monitoring (every 60s)
python bin/nexus_watchdog.py --watch --interval 60
# Dry-run (print diagnostics, don't touch Gitea)
python bin/nexus_watchdog.py --dry-run
# Crontab entry (every 5 minutes)
*/5 * * * * cd /path/to/the-nexus && python bin/nexus_watchdog.py
HEARTBEAT PROTOCOL
==================
The consciousness loop (nexus_think.py) writes a heartbeat file
after each think cycle:
~/.nexus/heartbeat.json
{
"pid": 12345,
"timestamp": 1711843200.0,
"cycle": 42,
"model": "timmy:v0.1-q4",
"status": "thinking"
}
If the heartbeat is older than --stale-threshold seconds, the
mind is considered dead even if the process is still running
(e.g., hung on a blocking call).
KIMI HEARTBEAT
==============
The Kimi triage pipeline writes a cron heartbeat file after each run:
/var/run/bezalel/heartbeats/kimi-heartbeat.last
(fallback: ~/.bezalel/heartbeats/kimi-heartbeat.last)
{
"job": "kimi-heartbeat",
"timestamp": 1711843200.0,
"interval_seconds": 900,
"pid": 12345,
"status": "ok"
}
If the heartbeat is stale (>2x declared interval), the watchdog reports
a Kimi Heartbeat failure alongside the other checks.
ZERO DEPENDENCIES
=================
Pure stdlib. No pip installs. Same machine as the nexus.
<EFBFBD>)<01> annotationsN)<02> dataclass<73>field)<01>Path)<04>Any<6E>Dict<63>List<73>Optional)<01>write_cron_heartbeatTFz)%(asctime)s %(levelname)-7s %(message)sz%Y-%m-%d %H:%M:%S)<03>level<65>format<61>datefmtznexus.watchdog<6F> localhosti="z.nexuszheartbeat.json<6F>,<00><zkimi-heartbeatg@<40> GITEA_URLz%https://forge.alexanderwhitestone.com<6F> GITEA_TOKEN<45><00>
NEXUS_REPOzTimmy_Foundation/the-nexus<75>watchdogz
[watchdog]c<01>L<00>eZdZUdZded<ded<ded<ee<06><07>Zded <y
) <0B> CheckResultz Result of a single health check.<2E>str<74>name<6D>bool<6F>healthy<68>message)<01>default_factoryzDict[str, Any]<5D>detailsN)<08>__name__<5F>
__module__<EFBFBD> __qualname__<5F>__doc__<5F>__annotations__r<00>dictr<00><00><00>bin/nexus_watchdog.pyrr<00>s#<00><00>*<2A>
<0A>I<EFBFBD> <11>M<EFBFBD> <10>L<EFBFBD>#<23>D<EFBFBD>9<>G<EFBFBD>^<5E>9r'rc<01>T<00>eZdZUdZded<ded<dZded<d <09>Zed d
<EFBFBD><04>Zdd <0B>Z y )<0F> HealthReportz(Aggregate health report from all checks.<2E>float<61> timestamp<6D>List[CheckResult]<5D>checksTr<00>overall_healthyc<01>F<00>td<01>|jD<00><00>|_y)Nc3<01>4K<00>|]}|j<00><01><00>y<00>w<01>N)r<00><02>.0<EFBFBD>cs r(<00> <genexpr>z-HealthReport.__post_init__.<locals>.<genexpr><3E>s<00><00><><00>"B<><11>1<EFBFBD>9<EFBFBD>9<EFBFBD>"B<><42><00>)<03>allr.r/)<01>selfs r(<00> __post_init__zHealthReport.__post_init__<5F>s<00><00>"<22>"B<>d<EFBFBD>k<EFBFBD>k<EFBFBD>"B<>B<><04>r'c<01>Z<00>|jD<00>cgc]}|jr<01>|<01><02>c}Scc}wr2)r.r)r9r5s r(<00> failed_checkszHealthReport.failed_checks<6B>s <00><00><1F>;<3B>;<3B>8<>a<EFBFBD>a<EFBFBD>i<EFBFBD>i<EFBFBD><01>8<>8<><38>8s<00>(<04>(c
<01>^<00>tjdtj|j<00><00>}|jrdnd}d|<01><00>d|<02><00>dddg}|j
D]A}|j rd nd
}|jd |j<00>d |<05>d |j<00>d <0A><07><00>C|jr<>|jd<06>|jd<0E>|jD]<5D>}|jd|j<00>d<10><03>|jd<11>|j|j<00>|jr0|jtj|jd<12><13><00>|jd<11><00><>|jd<06>|jd|<01>d<15><03>dj|<03>S)zFormat as a Gitea issue body.z%Y-%m-%d %H:%M:%S UTCu🟢 ALL SYSTEMS OPERATIONALu🔴 FAILURES DETECTEDu## Nexus Health Report — z **Status:** rz| Check | Status | Details |z|:------|:------:|:--------|<7C><>❌z| z | z |z### Failure Diagnosticsz
**z:**z```<60><00><01>indentz%*Generated by `nexus_watchdog.py` at <20>*<2A>
)<0F>time<6D>strftime<6D>gmtimer,r/r.r<00>appendrrr<r<00>json<6F>dumps<70>join)r9<00>ts<74>status<75>linesr5<00>icons r(<00> to_markdownzHealthReport.to_markdown<77>ss<00><00> <11>]<5D>]<5D>2<>D<EFBFBD>K<EFBFBD>K<EFBFBD><04><0E><0E>4O<34> P<><02>37<33>3G<33>3G<33>/<2F>Me<4D><06>*<2A>"<22><14> .<2E><1A>6<EFBFBD>(<28> #<23> <0E> *<2A> *<2A> 
<EFBFBD><05><16><1B><1B> A<01>A<EFBFBD><1D>I<EFBFBD>I<EFBFBD>5<EFBFBD>5<EFBFBD>D<EFBFBD> <11>L<EFBFBD>L<EFBFBD>2<EFBFBD>a<EFBFBD>f<EFBFBD>f<EFBFBD>X<EFBFBD>S<EFBFBD><14><06>c<EFBFBD>!<21>)<29>)<29><1B>B<EFBFBD>?<3F> @<40> A<01> <10> <1D> <1D> <11>L<EFBFBD>L<EFBFBD><12> <1C> <11>L<EFBFBD>L<EFBFBD>2<> 3<><19>'<27>'<27> %<25><01><15> <0C> <0C>t<EFBFBD>A<EFBFBD>F<EFBFBD>F<EFBFBD>8<EFBFBD>3<EFBFBD>/<2F>0<><15> <0C> <0C>s<EFBFBD>$<24><15> <0C> <0C>Q<EFBFBD>Y<EFBFBD>Y<EFBFBD>'<27><14>9<EFBFBD>9<EFBFBD><19>L<EFBFBD>L<EFBFBD><14><1A><1A>A<EFBFBD>I<EFBFBD>I<EFBFBD>a<EFBFBD>!@<40>A<><15> <0C> <0C>s<EFBFBD>$<24>  %<25> <0E> <0C> <0C>R<EFBFBD><18> <0A> <0C> <0C><<3C>R<EFBFBD>D<EFBFBD><01>B<>C<><13>y<EFBFBD>y<EFBFBD><15><1F>r'N)<02>returnr-)rQr)
r r!r"r#r$r/r:<00>propertyr<rPr&r'r(r*r*<00>s:<00><00>2<><14><14> <1D><1D> <20>O<EFBFBD>T<EFBFBD> <20>C<01><0E>9<><0E>9<> r'r*c <01><><00> tjtjtj<00>}|jd<01>|j ||f<02>}|j <00>|dk(rt ddd|<00>d|<01><00><04><07>St ddd |<00>d|<01>d
|<03>d <0B>|||d <0C><03> <0A>S#t$r)}t ddd|<04><00>||t|<04>d<0F><03> <0A>cYd}~Sd}~wwxYw)z<>Check if the WebSocket gateway is accepting connections.
Uses a raw TCP socket probe (not a full WebSocket handshake) to avoid
depending on the websockets library. If TCP connects, the gateway
process is alive and listening.
<20>rzWebSocket GatewayTz Listening on <20>:<3A>rrrFzConnection refused on z (errno=<3D>))<03>host<73>port<72>errno<6E>rrrrzProbe failed: )rXrY<00>errorN) <09>socket<65>AF_INET<45> SOCK_STREAM<41>
settimeout<EFBFBD>
connect_ex<EFBFBD>closer<00> Exceptionr)rXrY<00>sock<63>result<6C>es r(<00>check_ws_gatewayrg<00>s<><00><00>
<EFBFBD><15>}<7D>}<7D>V<EFBFBD>^<5E>^<5E>V<EFBFBD>-?<3F>-?<3F>@<40><04> <0C><0F><0F><01><1A><15><1F><1F>$<24><04><1C>.<2E><06> <0C>
<EFBFBD>
<EFBFBD> <0C> <11>Q<EFBFBD>;<3B><1E>(<28><1C>'<27><04>v<EFBFBD>Q<EFBFBD>t<EFBFBD>f<EFBFBD>5<><0E> <0E> <1F>(<28><1D>0<><14><06>a<EFBFBD><04>v<EFBFBD>X<EFBFBD>f<EFBFBD>X<EFBFBD>Q<EFBFBD>O<>!%<25>t<EFBFBD>f<EFBFBD>E<> <0E> <0E><> <15>
<EFBFBD><1A>$<24><19>$<24>Q<EFBFBD>C<EFBFBD>(<28>!<21>4<EFBFBD>#<23>a<EFBFBD>&<26>A<> 
<EFBFBD>
<EFBFBD><EFBFBD>
<EFBFBD>s$<00>A>B<00>B<00> C<03>'C <03>C<03> Cc
<01><><00> tjgd<01>ddd<03><04>}|jdk(r<>|jj <00>j d<06>D<00>cgc]#}|j <00>s<01>|j <00><00><02>%}}t tj<00><00>}|D<00>cgc]
}||k7s<01> |<01><02> }}|r$tdddd j|<02><00>d
<EFBFBD>d |i<01> <0C>Stdd dd|ji<01> <0C>Scc}wcc}w#t$rtddd<10><11>cYSt$r'}tdd d|<04><00>dt |<04>i<01> <0C>cYd}~Sd}~wwxYw)z<>Check if nexus_think.py is running as a process.
Uses `pgrep -f` to find processes matching the script name.
This catches both `python nexus_think.py` and `python -m nexus.nexus_think`.
)<03>pgrepz-f<> nexus_thinkTrT)<03>capture_output<75>text<78>timeoutrrDzConsciousness LoopzRunning (PID: <20>, rW<00>pidsr[Fu6nexus_think.py is not running — Timmy's mind is dark<72>pgrep_returncodez+pgrep not available, skipping process checkrVzProcess check failed: r\N) <0A>
subprocess<EFBFBD>run<75>
returncode<EFBFBD>stdout<75>strip<69>splitr<00>os<6F>getpidrrK<00>FileNotFoundErrorrc)re<00>pro<00>own_pidrfs r(<00>check_mind_processr|<00>sO<00><00> '
<EFBFBD><1B><1E><1E> *<2A><1F>d<EFBFBD>A<EFBFBD>
<EFBFBD><06>
<12> <1C> <1C><01> !<21>'-<2D>}<7D>}<7D>':<3A>':<3A>'<<3C>'B<>'B<>4<EFBFBD>'H<>V<>!<21>A<EFBFBD>G<EFBFBD>G<EFBFBD>I<EFBFBD>A<EFBFBD>G<EFBFBD>G<EFBFBD>I<EFBFBD>V<>D<EFBFBD>V<><19>"<22>)<29>)<29>+<2B>&<26>G<EFBFBD>#<23>4<>!<21>q<EFBFBD>G<EFBFBD>|<7C>A<EFBFBD>4<>D<EFBFBD>4<><13>"<22>-<2D> <20>,<2C>T<EFBFBD>Y<EFBFBD>Y<EFBFBD>t<EFBFBD>_<EFBFBD>,=<3D>Q<EFBFBD>?<3F>#<23>T<EFBFBD>N<EFBFBD> <12><12><1B>%<25><19>L<>'<27><16>):<3A>):<3A>;<3B> 
<EFBFBD>
<EFBFBD><EFBFBD>W<01><>5<><35> <1D>
<EFBFBD><1A>%<25><18>A<>
<EFBFBD>
<EFBFBD>
<15>
<EFBFBD><1A>%<25><19>,<2C>Q<EFBFBD>C<EFBFBD>0<><1C>c<EFBFBD>!<21>f<EFBFBD>%<25> 
<EFBFBD>
<EFBFBD><EFBFBD>
<EFBFBD>sT<00>AC><00>C4<04>-C4<04>?#C><00>"
C9<04>-C9<04>1'C><00>C><00>4
C><00>>E<03>E<03>E<03>;E<03>Ec<01><><00>|j<00>stddd|<00>d<04>dt|<00>i<01><06>S tj|j <00><00>}|jd
d <0B>}tj<00>|z
}|jd d <0A>}|jdd<0F>}|jdd<0F>}||kDr'tdddt|<05><00>d|<01>d|<06>d|<07>d|<08><00>
|<02><06>Stddd|<06>dt|<05><00>d|<07><00>|<02><06>S#tj tf$r1}tddd|<03><00>t|<00>t|<03>d<08><02><06>cYd }~Sd }~wwxYw)z<>Check if the heartbeat file exists and is recent.
The consciousness loop should write this file after each think
cycle. If it's missing or stale, the mind has stopped thinking
even if the process is technically alive.
<20> HeartbeatF<74>No heartbeat file at u — mind has never reported<65>pathr[<00>Heartbeat file corrupt: <20>r<>r\Nr,r<00>cycle<6C>?<3F>model<65>unknownrMuStale heartbeat — last pulse zs ago (threshold: z s). Cycle #z, model=<3D> , status=TuAlive — cycle #rnz s ago, model=) <0B>existsrrrI<00>loads<64> read_text<78>JSONDecodeError<6F>OSError<6F>getrE<00>int) r<><00>stale_threshold<6C>datarfr,<00>ager<65>r<>rMs r(<00>check_heartbeatr<74>ss<00><00> <10>;<3B>;<3B>=<3D><1A><1C><19>+<2B>D<EFBFBD>6<EFBFBD>1M<31>N<><1B>S<EFBFBD><14>Y<EFBFBD>'<27> 
<EFBFBD>
<EFBFBD>
<EFBFBD><13>z<EFBFBD>z<EFBFBD>$<24>.<2E>.<2E>*<2A>+<2B><04><15><08><08><1B>a<EFBFBD>(<28>I<EFBFBD>
<0E>)<29>)<29>+<2B> <09>
!<21>C<EFBFBD> <10>H<EFBFBD>H<EFBFBD>W<EFBFBD>c<EFBFBD> "<22>E<EFBFBD> <10>H<EFBFBD>H<EFBFBD>W<EFBFBD>i<EFBFBD> (<28>E<EFBFBD> <11>X<EFBFBD>X<EFBFBD>h<EFBFBD> <09> *<2A>F<EFBFBD>
<EFBFBD>_<EFBFBD><1C><1A><1C><19>1<>#<23>c<EFBFBD>(<28><1A><<1F>.<2E>/<2F>0<1A><1F><17><08><15><07>y<EFBFBD><16><08>B<01><19> 
<EFBFBD>
<EFBFBD> <17> <18><14>#<23>E<EFBFBD>7<EFBFBD>"<22>S<EFBFBD><13>X<EFBFBD>J<EFBFBD>m<EFBFBD>E<EFBFBD>7<EFBFBD>K<><14>  <06><06><>5 <11> <20> <20>'<27> *<2A>
<EFBFBD><1A><1C><19>.<2E>q<EFBFBD>c<EFBFBD>2<> <20><14>Y<EFBFBD><13>Q<EFBFBD><16>8<> 
<EFBFBD>
<EFBFBD><EFBFBD>
<EFBFBD>s<00>#C?<00>?E <03>&E<03>>E <03>E c <01><00>tt<00>jjdz dz }|j<00>st ddd<05><06>S |j <00>}t |t|<00>d<07>t dddt|<01><00>d <09><03><06>S#t$rq}t dd
d |j<00>d |j<00><00>t|<00>|j|j|jxsd j<00>d<0E><04><0F>cYd}~Sd}~wwxYw)a Verify nexus_think.py can be parsed by Python.
This catches the exact failure mode that killed the nexus: a syntax
error introduced by a bad commit. Python's compile() is a fast,
zero-import check that catches SyntaxErrors before they hit runtime.
<20>nexusznexus_think.pyz Syntax HealthTz3nexus_think.py not found at expected path, skippingrV<00>execz!nexus_think.py compiles cleanly (z bytes)FzSyntaxError at line z: r)<04>file<6C>line<6E>offsetrlr[N)r<00>__file__<5F>parentr<74>rr<><00>compiler<00>len<65> SyntaxError<6F>lineno<6E>msgr<67>rlru)<03> script_path<74>sourcerfs r(<00>check_syntax_healthr<68>Ls<><00><00><17>x<EFBFBD>.<2E>'<27>'<27>.<2E>.<2E><17>8<>;K<>K<>K<EFBFBD> <16> <1D> <1D> <1F><1A> <20><18>I<>
<EFBFBD>
<EFBFBD> 
<EFBFBD><1C>&<26>&<26>(<28><06><0F><06><03>K<EFBFBD>(<28>&<26>1<><1A> <20><18>7<><03>F<EFBFBD> <0B>}<7D>G<EFBFBD>L<>
<EFBFBD>
<EFBFBD><EFBFBD>
<17> 
<EFBFBD><1A> <20><19>*<2A>1<EFBFBD>8<EFBFBD>8<EFBFBD>*<2A>B<EFBFBD>q<EFBFBD>u<EFBFBD>u<EFBFBD>g<EFBFBD>><3E><1B>K<EFBFBD>(<28><19><08><08><1B>(<28>(<28><1A><16><16><1C>2<EFBFBD>,<2C>,<2C>.<2E> <0E>

<EFBFBD>
<EFBFBD><EFBFBD> 
<EFBFBD>s <00> AB
<00>
D<03>A&C?<03>9D<03>?Dc<01><><00>td<01>}tj<00>dz dz }tjj d<04>}|r t|<04>}nK|j <00>r|}n8|j <00>r|}n%t ddddt|<02>t|<03>gi<01> <09>S||<00>d
<EFBFBD>z }|j <00>st ddd |<06>d <0C>d t|<06>i<01> <09>S tj|j<00><00>}t|j dd<12><00>} t|j dd<12><00>}
|j dd<15>} tj<00>| z
} |
dkrd}
||
z} | | kD}| dkrt| <0C><00>d<18>n"t| dz<00><00>dt| dzdz<00><00>d<1B>}|
dkrt|
<EFBFBD><00>d<18>n"t|
dz<00><00>dt|
dzdz<00><00>d<1B>}|r't ddd|<0F>d|<01>d|<10>dt| <0A><00>d | <0B><00>
|<07> <09>St dd!d"|<0F>d#|<10>d$| <0B>d%<25>|<07> <09>S#tjtf$r1}t ddd|<08><00>t|<06>t|<08>d<0F><02> <09>cYd}~Sd}~wwxYw)&a<>Check if the Kimi Heartbeat cron job is alive.
Reads the ``<job>.last`` file from the standard Bezalel heartbeat
directory (``/var/run/bezalel/heartbeats/`` or fallback
``~/.bezalel/heartbeats/``). The file is written atomically by the
cron_heartbeat module after each successful triage pipeline run.
A job is stale when:
``time.time() - timestamp > stale_multiplier * interval_seconds``
(same rule used by ``check_cron_heartbeats.py``).
z/var/run/bezalel/heartbeatsz.bezalel<65>
heartbeats<EFBFBD>BEZALEL_HEARTBEAT_DIRzKimi HeartbeatFuAHeartbeat directory not found — no triage pipeline deployed yet<65>searchedr[z.lastru, — Kimi triage pipeline has never reportedr<64>r<>r<>Nr,r<00>interval_secondsrMr<>ii<00>szh r<00>mz Silent for z (threshold: zx z = z s). Status: TuAlive — last beat z ago (interval r<>rW)r<00>homerw<00>environr<6E>r<>rrrIr<>r<>r<>r<>r+r<>rE)<11>job<6F>stale_multiplier<65>primary<72>fallback<63>env_dir<69>hb_dir<69>hb_filer<65>rfr,<00>interval<61>
raw_statusr<EFBFBD><00> threshold<6C>is_stale<6C>age_str<74> interval_strs r(<00>check_kimi_heartbeatr<74>qs<><00><00> <13>0<>1<>G<EFBFBD><13>y<EFBFBD>y<EFBFBD>{<7B>Z<EFBFBD>'<27>,<2C>6<>H<EFBFBD><10>j<EFBFBD>j<EFBFBD>n<EFBFBD>n<EFBFBD>4<>5<>G<EFBFBD><0E><15>g<EFBFBD><1D><06> <10><1E><1E> <19><18><06> <11><1F><1F> <1A><19><06><1A>!<21><19>W<><1F>#<23>g<EFBFBD>,<2C><03>H<EFBFBD> <0A>!><3E>?<3F> 
<EFBFBD>
<EFBFBD><15>#<23><15>e<EFBFBD>}<7D>$<24>G<EFBFBD> <12>><3E>><3E> <1B><1A>!<21><19>+<2B>G<EFBFBD>9<EFBFBD>4`<60>a<><1B>S<EFBFBD><17>\<5C>*<2A> 
<EFBFBD>
<EFBFBD>
<EFBFBD><13>z<EFBFBD>z<EFBFBD>'<27>+<2B>+<2B>-<2D>.<2E><04><16>d<EFBFBD>h<EFBFBD>h<EFBFBD>{<7B>A<EFBFBD>.<2E>/<2F>I<EFBFBD><12>4<EFBFBD>8<EFBFBD>8<EFBFBD>.<2E><01>2<>3<>H<EFBFBD><15><18><18>(<28>I<EFBFBD>.<2E>J<EFBFBD>
<0E>)<29>)<29>+<2B> <09>
!<21>C<EFBFBD><0F>1<EFBFBD>}<7D><17><08> <20>8<EFBFBD>+<2B>I<EFBFBD><12>Y<EFBFBD><EFBFBD>H<EFBFBD> #<23>d<EFBFBD>
<EFBFBD><13>S<EFBFBD><18>
<EFBFBD>!<21>n<EFBFBD>3<EFBFBD>s<EFBFBD>d<EFBFBD>{<7B>3C<33>2D<32>B<EFBFBD>s<EFBFBD>C<EFBFBD>RV<52>J<EFBFBD>[]<5D>K]<5D>G^<5E>F_<46>_`<60>0a<30>G<EFBFBD>*2<>T<EFBFBD>/<2F>c<EFBFBD>(<28>m<EFBFBD>_<EFBFBD>A<EFBFBD>&<26>#<23>h<EFBFBD>RV<52>FV<46>BW<42>AX<41>XZ<58>[^<5E>`h<>ko<6B>`o<>tv<74>_v<5F>[w<>Zx<5A>xy<78>?z<>L<EFBFBD><0F><1A>!<21><19><1D>g<EFBFBD>Y<EFBFBD>'<1F>/<2F>0<><02><<3C>.<2E><03>C<EFBFBD> <09>N<EFBFBD>CS<43>T<1B>%<25>,<2C>(<28><19> 
<EFBFBD>
<EFBFBD> <17> <1D><14>&<26>w<EFBFBD>i<EFBFBD><EFBFBD>|<7C>n<EFBFBD>I<EFBFBD>V`<60>Ua<55>ab<61>c<><14>  <06><06><>G <11> <20> <20>'<27> *<2A>
<EFBFBD><1A>!<21><19>.<2E>q<EFBFBD>c<EFBFBD>2<> <20><17>\<5C>C<EFBFBD><01>F<EFBFBD>;<3B> 
<EFBFBD>
<EFBFBD><EFBFBD>
<EFBFBD>s<00>#H<00>I <03>/&I<03>I <03>I c<01>b<00>ddl}ddl}tjd<03><00>d|<01><00>}|r#t j
|<02>j <00>nd}|jj|||<00><05>}tr|jddt<00><00><02>|jdd <09>|jd
d <09> |jj|d <0B> <0C>5}|j<00>j<00>}|j<00>rt j|<08>nicddd<02>S#1swYyxYw#|j j"$rJ} t$j'd | j(| j<00>j<00>dd<00>Yd} ~ yd} ~ wt*$r } t$j'd| <09>Yd} ~ yd} ~ wwxYw)z<Make a Gitea API request. Returns parsed JSON or empty dict.rN<>/z/api/v1)r<><00>method<6F> Authorizationztoken z Content-Typezapplication/json<6F>Accept<70>)rmz Gitea %d: %s<><73>zGitea request failed: %s)<16>urllib.request<73> urllib.errorr<00>rstriprIrJ<00>encode<64>request<73>Requestr<00>
add_header<EFBFBD>urlopen<65>read<61>decoderur<>r\<00> HTTPError<6F>logger<65>warning<6E>coderc)
r<EFBFBD>r<>r<><00>urllib<69>url<72>body<64>req<65>resp<73>rawrfs
r(<00>_gitea_requestr<74><00>sM<00><00><19><17> <16> <1D> <1D>c<EFBFBD> "<22> #<23>7<EFBFBD>4<EFBFBD>&<26>
1<EFBFBD>C<EFBFBD>(,<2C>4<EFBFBD>:<3A>:<3A>d<EFBFBD> <1B> "<22> "<22> $<24>$<24>D<EFBFBD>
<10>.<2E>.<2E>
<20>
<20><13>4<EFBFBD><06>
<20>
?<3F>C<EFBFBD><12> <0B><0E><0E><EFBFBD>&<26><1B> <0A>(><3E>?<3F><07>N<EFBFBD>N<EFBFBD>><3E>#5<>6<><07>N<EFBFBD>N<EFBFBD>8<EFBFBD>/<2F>0<> <14> <13>^<5E>^<5E> #<23> #<23>C<EFBFBD><12> #<23> 4<> :<3A><04><16>)<29>)<29>+<2B>$<24>$<24>&<26>C<EFBFBD>&)<29>i<EFBFBD>i<EFBFBD>k<EFBFBD>4<EFBFBD>:<3A>:<3A>c<EFBFBD>?<3F>r<EFBFBD> :<3A> :<3A> :<3A><> <12><<3C><<3C> !<21> !<21><14><0E><0E><0E>~<7E>q<EFBFBD>v<EFBFBD>v<EFBFBD>q<EFBFBD>v<EFBFBD>v<EFBFBD>x<EFBFBD><EFBFBD><EFBFBD>/@<40><14>#<23>/F<>G<><13><> <14><14><0E><0E><0E>1<>1<EFBFBD>5<><13><><14>sD<00>,D$<00> AD<03> D$<00>D!<07>D$<00>!D$<00>$F.<03>=AF<03> F.<03>F)<03>)F.c<01><><00>tddt<00>d<03><03>}|rt|t<00>sy|D]-}|j dd<06>}|j t <00>s<01>+|cSy)z-Find an existing open watchdog issue, if any.<2E>GET<45>/repos/z'/issues?state=open&type=issues&limit=20N<30>titler)r<><00>
GITEA_REPO<EFBFBD>
isinstance<EFBFBD>listr<74><00>
startswith<EFBFBD>WATCHDOG_TITLE_PREFIX)<03>issues<65>issuer<65>s r(<00>find_open_watchdog_issuer<65><00>se<00><00> <1B> <0A>
<11>*<2A><1C>D<>E<><06>F<EFBFBD> <12><1A>F<EFBFBD>D<EFBFBD>1<><13><17><19><05><15> <09> <09>'<27>2<EFBFBD>&<26><05> <10> <1B> <1B>1<> 2<><18>L<EFBFBD><19> r'c<01><><00>|j}djd<02>|D<00><00>}t<00>d|<02><00>}tddt<00>d<06>||j <00>dgd<08><03> <09>S)
z*Create a Gitea issue for a health failure.rnc3<01>4K<00>|]}|j<00><01><00>y<00>wr2)rr3s r(r6z%create_alert_issue.<locals>.<genexpr><3E>s<00><00><><00>2<>a<EFBFBD>1<EFBFBD>6<EFBFBD>6<EFBFBD>2<>r7z Nexus health failure: <20>POSTr<54>z/issues<65>Timmy)r<>r<><00> assignees<65>r<>)r<rKr<>r<>r<>rP)<04>report<72>failed<65>
componentsr<EFBFBD>s r(<00>create_alert_issuer<65><00>sh<00><00> <13> !<21> !<21>F<EFBFBD><15><19><19>2<>6<EFBFBD>2<>2<>J<EFBFBD>$<24>%<25>%<<3C>Z<EFBFBD>L<EFBFBD> I<>E<EFBFBD> <19><0E>
<11>*<2A><1C>W<EFBFBD>%<25><1A><1A>&<26>&<26>(<28>!<21><19>
<EFBFBD> <06>r'c<01>T<00>tddt<00>d|<00>d<04>d|j<00>i<01><06>S)z>Add a comment to an existing watchdog issue with new findings.r<>r<><00>/issues/<2F> /commentsr<73>r<><00>r<>r<>rP<00><02> issue_numberr<72>s r(<00>update_alert_issuer<65>s5<00><00> <19><0E>
<11>*<2A><1C>X<EFBFBD>l<EFBFBD>^<5E>9<EFBFBD>=<3D><14>f<EFBFBD>(<28>(<28>*<2A> +<2B> <06>r'c<01><><00>tddt<00>d|<00>d<04>dd|j<00>zdzi<01><08>td dt<00>d|<00><00>d
d i<01><08>y ) z/Close a watchdog issue when health is restored.r<>r<>r<>r<>r<>u## 🟢 Recovery Confirmed
u(
*Closing — all systems operational.*r<><00>PATCH<43>state<74>closedNr<4E>r<>s r(<00>close_alert_issuer<65> sh<00><00><12><0E>
<11>*<2A><1C>X<EFBFBD>l<EFBFBD>^<5E>9<EFBFBD>=<3D><14> ,<2C><14> <20> <20>"<22> #<23>:<3A> ;<3B> <0B><06><13><0F>
<11>*<2A><1C>X<EFBFBD>l<EFBFBD>^<5E>4<><15>x<EFBFBD> <20>r'c<01><><00>t||<01>t<00>t||<03>t<00>t <00>g}t t j <00>|<04><01>S)z6Run all health checks and return the aggregate report.)r,r.)rgr|r<>r<>r<>r*rE)<05>ws_host<73>ws_port<72>heartbeat_pathr<68>r.s r(<00>run_health_checksr<73>!sF<00><00> <19><17>'<27>*<2A><1A><1C><17><0E><0F>8<><1B><1D><1C><1E> <06>F<EFBFBD> <18>$<24>)<29>)<29>+<2B>f<EFBFBD> =<3D>=r'c<01><><00>|r*tjd|jrd<02>yd<03>ytstj d<05>yt <00>}|jr,|r)tjd|d<00>t |d|<00>yy|r)tjd|d<00>t|d|<00>yt|<00>}|r,|jd<07>rtjd |d<00>yyy)
z=Create, update, or close Gitea issues based on health status.u DRY RUN — would %s Gitea issuerbz create/updateNu,GITEA_TOKEN not set — cannot create issuesu%Health restored — closing issue #%d<>numberu&Still unhealthy — updating issue #%dzCreated alert issue #%d)
r<EFBFBD><00>infor/rr<>r<>r<>r<>r<>r<>)r<><00>dry_run<75>existingres r(<00>alert_on_failurer2s<><00><00><0E><0E> <0B> <0B>6<> &<26> 6<> 6<>W<EFBFBD> M<01><0E>=L<01> M<01><0E> <16><0E><0E><0E>E<>F<><0E>'<27>)<29>H<EFBFBD> <0A><1D><1D> <13> <12>K<EFBFBD>K<EFBFBD>?<3F><18>(<28>AS<41> T<> <1D>h<EFBFBD>x<EFBFBD>0<>&<26> 9<> <14> <14> <12>K<EFBFBD>K<EFBFBD>@<40>(<28>8<EFBFBD>BT<42> U<> <1E>x<EFBFBD><08>1<>6<EFBFBD> :<3A>'<27><06>/<2F>F<EFBFBD><15>&<26>*<2A>*<2A>X<EFBFBD>.<2E><16> <0B> <0B>5<>v<EFBFBD>h<EFBFBD>7G<37>H<>/<2F>vr'c<01><><00>t|j|jt|j<00>|j
<00><01>}|j D]k}|jrtjntj}|jrdnd}tj|d||j|j<00><00>m|jst!||j"<00><05>n#|j"st!||j"<00><05>t$r t'dd<07><08>|jS|jS#t($rY|jSwxYw) z4Run one health check cycle. Returns True if healthy.<2E>r<>r<>r<>r<>r>r?z %s %s: %s)r<00>nexus_watchdogr)r<>)r<>r<>r<>rr<>r<>r.r<00>logging<6E>INFO<46>ERRORr<52><00>logrrr/rr<00>_HAS_CRON_HEARTBEAT<41>_write_cron_heartbeatrc)<05>argsr<73><00>checkr rOs r(<00>run_oncerMs<00><00> <1E><14> <0C> <0C><14> <0C> <0C><1B>D<EFBFBD>/<2F>/<2F>0<><1C>,<2C>,<2C> <06>F<EFBFBD><18><1D><1D>H<01><05> %<25> <0A> <0A><07> <0C> <0C>7<EFBFBD>=<3D>=<3D><05><1D> <0A> <0A>u<EFBFBD>5<EFBFBD><04><0E>
<EFBFBD>
<EFBFBD>5<EFBFBD>+<2B>t<EFBFBD>U<EFBFBD>Z<EFBFBD>Z<EFBFBD><15><1D><1D>G<>H<01>
<12> !<21> !<21><18><16><14><1C><1C>6<> <11>\<5C>\<5C><18><16><14><1C><1C>6<>
<1B> <11> !<21>"2<>S<EFBFBD> I<> <12> !<21> !<21>!<21>6<EFBFBD> !<21> !<21>!<21><><19> <11> <10> <11> !<21> !<21>!<21> <11>s<00> D.<00>. E<03>Ec<01><><00><07>tjd<01><02>}|jdtd<04><05>|jdtt
d<07><08>|jd t t<00>d
<EFBFBD><05>|jd ttd <0C><08>|jd dd<0F><10>|jdttd<12><08>|jddd<14><10>|jdddd<17><18>|jdtdt<00>d<1B><03><05>|jdttdt<00>d<1B><03><08>|j<00>}|jr<>tj!d|j"<00>d<1F><07>fd <20>}t%j$t$j&|<02>t%j$t$j(|<02><00>rBt+|<01>t-|j"<00>D]}<03>snt/j0d!<21><00><00>r<01>Ayyt+|<01>}|j2r<>t5|j6|j8t;|j<<00>|j><00>"<22>}tAtCjD|jF|jH|jJD<00>cgc]1}|jL|jN|jP|jRd#<23><04><02>3c}d$<24>d%<25>&<26><00>tUjV|rd'nd!<21>ycc}w)(Nu5Nexus Watchdog — monitors consciousness loop health)<01> descriptionz --ws-hostz+WebSocket gateway host (default: localhost))<02>default<6C>helpz --ws-portz&WebSocket gateway port (default: 8765))<03>typerrz--heartbeat-pathzPath to heartbeat filez--stale-thresholdz;Seconds before heartbeat is considered stale (default: 300)z--watch<63>
store_truez$Run continuously instead of one-shot)<02>actionrz
--intervalz2Seconds between checks in watch mode (default: 60)z --dry-runz/Print diagnostics without creating Gitea issuesz--json<6F> output_jsonz9Output results as JSON (for integration with other tools))r<00>destrz
--kimi-jobz"Kimi heartbeat job name (default: rWz--kimi-stale-multiplierz.Kimi heartbeat staleness multiplier (default: z4Watchdog starting in continuous mode (interval: %ds)Tc<01>6<00><01>d<01>tjd|<00>y)NFz!Received signal %d, shutting down)r<>r<>)<03>signum<75>frame<6D>_runnings <20>r(<00>_handle_sigtermzmain.<locals>._handle_sigterm<72>s<00><><00><1C>H<EFBFBD> <12>K<EFBFBD>K<EFBFBD>;<3B>V<EFBFBD> Dr'<00>rr[)rr,r.r@rAr),<2C>argparse<73>ArgumentParser<65> add_argument<6E>DEFAULT_WS_HOSTr<54><00>DEFAULT_WS_PORTr<00>DEFAULT_HEARTBEAT_PATH<54>DEFAULT_STALE_THRESHOLD<4C>DEFAULT_INTERVAL<41>KIMI_HEARTBEAT_JOBr+<00>KIMI_HEARTBEAT_STALE_MULTIPLIER<45>
parse_args<EFBFBD>watchr<68>r<>r<><00>signal<61>SIGTERM<52>SIGINTr<00>rangerE<00>sleeprr<>r<>r<>rr<>r<><00>printrIrJr/r,r.rrrr<00>sys<79>exit)<08>parserr r<00>_rr<>r5rs @r(<00>mainr4ms<><00><><00> <15> $<24> $<24>K<><06>F<EFBFBD> <0B><17><17><13>_<EFBFBD> :<3A><18><06> <0B><17><17><13>#<23><EFBFBD> 5<><18><06> <0B><17><17><1A>C<EFBFBD>(><3E>$?<3F> %<25><18><06> <0B><17><17><1B>#<23>/F<> J<><18><06> <0B><17><17><11>,<2C> 3<><18><06> <0B><17><17><14>3<EFBFBD>(8<> A<><18><06> <0B><17><17><13>L<EFBFBD> ><3E><18><06> <0B><17><17><10><1C>M<EFBFBD> H<><18><06> <0B><17><17><14>0<>1<>2D<32>1E<31>Q<EFBFBD> G<><18><06> <0B><17><17>!<21><05>7V<37>=<3D>>]<5D>=^<5E>^_<> `<60><18><06>
<12> <1C> <1C> <1E>D<EFBFBD> <0B>z<EFBFBD>z<EFBFBD><0E> <0B> <0B>J<>D<EFBFBD>M<EFBFBD>M<EFBFBD>Z<><17><08> E<01>
<0F> <0A> <0A>f<EFBFBD>n<EFBFBD>n<EFBFBD>o<EFBFBD>6<><0E> <0A> <0A>f<EFBFBD>m<EFBFBD>m<EFBFBD>_<EFBFBD>5<><16> <14>T<EFBFBD>N<EFBFBD><1A>4<EFBFBD>=<3D>=<3D>)<29> <1E><01><1F><19><14>
<EFBFBD>
<EFBFBD>1<EFBFBD> <0A> <1E><17><1B>4<EFBFBD>.<2E><07> <0F> <1B> <1B>&<26><1C> <0C> <0C><1C> <0C> <0C>#<23>D<EFBFBD>$7<>$7<>8<> $<24> 4<> 4<> <0E>F<EFBFBD> <12>$<24>*<2A>*<2A>!<21>1<>1<>#<23>-<2D>-<2D>$<24>]<5D>]<5D><12><1A><1F>V<EFBFBD>V<EFBFBD><01> <09> <09> !<21> <09> <09>a<EFBFBD>i<EFBFBD>i<EFBFBD>A<01><12><0E><18><19> <1A> <0C><08><08>g<EFBFBD><11>1<EFBFBD>%<25><>s<00> 6K(<10>__main__)rXrrYr<>rQr)rQr)r<>rr<>r<>rQr)r<>rr<>r+rQrr2)r<>rr<>rr<><00>Optional[dict]rQr)rQr6)r<>r*rQr6)r<>r<>r<>r*rQr6)r<>r<>r<>r*rQ<00>None)
r<EFBFBD>rr<>r<>r<>rr<>r<>rQr*)F)r<>r*rrrQr7)r zargparse.NamespacerQr)?r#<00>
__future__rrrIrrwr*r]rqr0rE<00> dataclassesrr<00>pathlibr<00>typingrrr r
<00>nexus.cron_heartbeatr r r
<00> ImportError<6F> basicConfigr<00> getLoggerr<72>r!r"r<>r#r$r%r&r'r<>r<>rrr<><00>WATCHDOG_LABELr<4C>rr*rgr|r<>r<>r<>r<>r<>r<>r<>r<>r<>rrr4r r&r'r(<00><module>rAs;<00><01>Q<04>f#<23><0F> <0B><0E> <09> <0A> <0A><11>
<EFBFBD> <0B>(<28><18>,<2C>,<2C>
 <20>R<><1E><17><14><07><13><13>
<11>,<2C>,<2C> 6<> <1F><02>
<1B><17> <1A> <1A>+<2B> ,<2C><06><1E><0F><16><0F>"<22><14><19><19><1B>x<EFBFBD>/<2F>2B<32>B<><16><1D><17><15><10>&<26><12>"%<25><1F> <0E>J<EFBFBD>J<EFBFBD>N<EFBFBD>N<EFBFBD>;<3B>(O<> P<> <09><10>j<EFBFBD>j<EFBFBD>n<EFBFBD>n<EFBFBD>]<5D>B<EFBFBD>/<2F> <0B> <0F>Z<EFBFBD>Z<EFBFBD>^<5E>^<5E>L<EFBFBD>*F<> G<>
<EFBFBD><1B><0E>$<24><15>
 <0B>:<3A>:<3A> <0B>:<3A> <0B>+ <20>+ <20> <0B>+ <20>`"1<>o<EFBFBD> 
<EFBFBD>F-
<EFBFBD>b(<28>2<>3<06>
<0E>3<06><18>3<06><11>3<06>l"
<EFBFBD>L"<22>=<3D>T<06> <0C>T<06><1B>T<06><11>T<06>r<14>2 <10> <06>"<06><06>*#<23>"<22>1<>2<> ><3E> <10>><3E> <10>><3E><19>><3E><19> ><3E>
<12> ><3E>"I<01>6"<22>@U&<26>p <0C>z<EFBFBD><19><08>F<EFBFBD><1A><>w<13> <20><1F><17> <20>s<00>F<00>F <03> F