118 lines
5.8 KiB
Markdown
118 lines
5.8 KiB
Markdown
# agentloop — a budgeted agent tool-loop with a hash-chained transcript
|
|
|
|
Lane: Codeforge. Stdlib only, Python 3.8+. No third-party imports anywhere.
|
|
|
|
python3 check.py # the gate; exit code IS the verdict
|
|
python3 demo.py # three end-to-end runs, writes demo-*.jsonl
|
|
|
|
## What it is
|
|
|
|
Three moving parts, each small enough to read in one sitting:
|
|
|
|
| file | role |
|
|
| --- | --- |
|
|
| `agentloop/tools.py` | `Tool` registry + a JSON-Schema-subset argument validator |
|
|
| `agentloop/loop.py` | the budgeted loop: model turn -> tool calls -> tool results -> repeat |
|
|
| `agentloop/transcript.py` | append-only JSONL log whose records form a SHA-256 hash chain |
|
|
|
|
A model is any callable `(messages, manifest) -> ModelReply`. The loop never
|
|
touches a clock or a random source, so **a transcript is a pure function of
|
|
`(task, model, tools, budgets)`** — run the same loop twice and the transcripts
|
|
are byte-identical. That determinism is what makes the hash chain a usable
|
|
integrity check instead of decoration.
|
|
|
|
## The transcript is the point
|
|
|
|
Every record carries the hash of the record before it:
|
|
|
|
```json
|
|
{"hash":"cc7b32...","kind":"tool","payload":{...},"prev":"1f08fd...","seq":2}
|
|
```
|
|
|
|
So the file is a chain, and verification breaks at the *first* affected record
|
|
and names it. A single file digest cannot do that — it says "something changed",
|
|
not "record 3 was rewritten".
|
|
|
|
`verify_records` reports, per record index: `hash_mismatch`, `seq_gap`,
|
|
`prev_mismatch`, `missing_fields`, `not_an_object`, `unserialisable_payload`.
|
|
`verify_file` additionally reports `bad_json` (with the line number) and
|
|
`unreadable`, and returns the record count.
|
|
|
|
### What the chain does NOT prove — stated plainly
|
|
|
|
A hash chain detects **modification and reordering of the records that are
|
|
present**. It cannot detect that records are **missing from the end**: delete
|
|
the last line and the remaining chain still verifies, because every surviving
|
|
record still points at a real predecessor. Completeness needs an anchor outside
|
|
the transcript — a commit SHA, or a digest stored elsewhere. `verify_file`
|
|
returns the count so a caller holding an external expectation can compare it.
|
|
`test_truncation_is_detected_only_against_an_external_count` pins this limit
|
|
rather than hiding it.
|
|
|
|
## Stop reasons and their precedence
|
|
|
|
| reason | meaning |
|
|
| --- | --- |
|
|
| `no_tool_calls` | the assistant asked for no tools — the ordinary end of a run |
|
|
| `sentinel` | the assistant's own text contained `stop_sentinel` |
|
|
| `step_budget` | checked *before* each turn, so `steps` never exceeds `max_steps` |
|
|
| `token_budget` | cumulative estimate exceeded `max_tokens`; the over-budget turn is recorded, then the loop stops **without** running its tool calls |
|
|
| `model_error` | the model callable raised, or returned a non-`ModelReply` |
|
|
|
|
Within one turn the order is: record the turn, then sentinel, then
|
|
`no_tool_calls`, then `token_budget`. A model that says DONE and asks for a tool
|
|
in the same turn stops on the sentinel and never runs the tool.
|
|
|
|
Token accounting is `max(1, len(text) // 4)` — a heuristic, not a tokenizer. It
|
|
bounds the loop, which is the property that matters; it does not claim to
|
|
measure a real model's bill.
|
|
|
|
## Tool failures are data, not crashes
|
|
|
|
An unknown tool name, a schema violation, and a tool that raises are all turned
|
|
into a `tool` record with `ok: false` and an `error` string, and the loop
|
|
continues. `demo-recover.jsonl` shows all three in one run:
|
|
|
|
```
|
|
seq=2 multiply ok=False error="no tool named 'multiply' (have: add, echo, fail, greet)"
|
|
seq=4 add ok=False error="invalid arguments: $.add.a: expected type integer, got str"
|
|
seq=6 fail ok=False error="tool 'fail' raised RuntimeError: the tool is broken on purpose"
|
|
seq=8 add ok=True value={'sum': 42}
|
|
```
|
|
|
|
A tool that returns a non-JSON-serialisable value is refused the same way: a
|
|
result that cannot be recorded is not a result, and the transcript must always
|
|
stay writable.
|
|
|
|
## Evidence
|
|
|
|
Measured on this host, 2026-10-25, Python 3.14.7:
|
|
|
|
| claim | status | proof |
|
|
| --- | --- | --- |
|
|
| the gate passes | measured | `python3 check.py` -> 3 legs ok, `Ran 55 tests`, exit 0 |
|
|
| the chain catches a payload rewrite at the right record | measured | suite `TestTamperDetection` (11 cases) + `check.py` tamper leg |
|
|
| the chain catches a rewrite *and* recomputed hash | measured | `test_forged_record_with_a_recomputed_hash_still_breaks_the_chain` — the *next* record's `prev` breaks |
|
|
| the chain catches a reorder even after renumbering | measured | `test_reorder_plus_renumber_is_caught_as_a_prev_mismatch` |
|
|
| the chain cannot detect a missing tail | measured (limit, not a fault) | `test_truncation_is_detected_only_against_an_external_count` |
|
|
| identical runs give byte-identical transcripts | measured | `test_identical_runs_produce_byte_identical_transcripts` |
|
|
| step budget caps the loop and equals the assistant record count | measured | `test_step_budget_caps_the_loop_and_matches_the_record_count` |
|
|
| token budget stops before running tools | measured | `test_token_budget_stops_the_loop_before_running_tools` |
|
|
| three demo transcripts verify from disk | measured | `demo-answer.jsonl` 5, `demo-recover.jsonl` 11, `demo-budget.jsonl` 9 records, all `ok=True` |
|
|
| the gate FAILS when the verifier is broken | measured | negative control: sabotaged copy -> 9 suite failures + tamper leg `a mutated transcript VERIFIED`, exit 1 |
|
|
|
|
### Negative control
|
|
|
|
A gate that cannot be shown to fail is not evidence. `check.py`'s tamper leg is
|
|
independent of the suite — it re-derives its expectation from the file's own
|
|
bytes, so a bug in the test fixtures cannot hide there. The sabotage run
|
|
confirmed both legs fail together when `verify_records` is made to accept
|
|
everything.
|
|
|
|
## Replay
|
|
|
|
```
|
|
python3 check.py # 0 = every leg passed, 1 = graded fault, 2 = could not grade
|
|
python3 demo.py # 0 = every demo claim held
|
|
```
|