volt: q136 -- sandbox leg compares real paths, so a symlinked TMPDIR is not a miss
This commit is contained in:
1 parent
5b068f0283
commit
fbe4f1c09e
2 files changed
+64
-7
No files matched your search
@@ -110,3 +110,33 @@ The item is filed in `WORK-QUEUE.md`; this is the reading behind it.
|
||||
- Left open on purpose: nothing runs the spark lane's `test_*.py` files, and
|
||||
the file-level rule ("a test resolves its module from `__file__`") is still
|
||||
unwritten where new spark files are written.
|
||||
|
||||
## Q136 follow-up — the sandbox leg read a symlinked temp path as a miss
|
||||
|
||||
The CHECK for Q136 failed on the `sandbox` leg alone, with the other six green:
|
||||
|
||||
```
|
||||
q136: sandbox FAULT 8 path(s) opened, 0 outside the sandbox,
|
||||
never opened: test_input.txt, wordfreq_counter.py
|
||||
```
|
||||
|
||||
- Reproduced exactly by pointing `TMPDIR` at a symlink: CPython's `open` audit
|
||||
hook records `Path(...).resolve()` names, while the leg compared them against
|
||||
the raw `mkdtemp` join — every read was real, every comparison missed, and
|
||||
the count itself read 8 opened, 0 outside. Nothing was wrong with the cell.
|
||||
- Repaired in `wizard-work/volt/q136_verify.py`: both sides compared by
|
||||
`os.path.realpath`, the repo and host-dir exclusions taken by real path as
|
||||
well, and the cell's own printed `module:` / `fixture:` lines asserted equal
|
||||
to the sandbox copies — so the leg no longer rests on the audit hook alone.
|
||||
A fault now prints the offending outside paths instead of only a count.
|
||||
- Controls, `/root/.hermes/wizard-outputs/volt/probe_q136_sandbox.py` -> PROBE
|
||||
PASS: normal `TMPDIR` rc 0; symlinked `TMPDIR` rc 0 (rc 1 before the repair);
|
||||
a cell that reads `/root/.hermes/wizard-outputs/spark/test_input.txt` rc 1
|
||||
with `1 outside the sandbox ... outside: .../spark/test_input.txt`.
|
||||
- `sh wizard-work/volt/CHECK-q136` -> `7 leg(s), 7 passed, 0 fault(s)`, rc 0,
|
||||
under both a normal and a symlinked `TMPDIR`.
|
||||
- Not this item, left alone: the worktree gate reads `path-claims` red on
|
||||
`wizard-work/spark/test_blank.txt`, cited at `wizard-work/spark/README.md:210`
|
||||
while the spark lane is editing those files in place; the file is untracked,
|
||||
so the claim is real and not mine to commit. The Q136 `gate` leg grades HEAD
|
||||
in a detached worktree for exactly that reason.
|
||||
@@ -30,7 +30,9 @@ What this instrument grades
|
||||
run the repaired cell, run in place from `cwd=/`: rc 0, three PASS lines
|
||||
sandbox the cell + module + fixture copied to a temp directory outside the
|
||||
tree, run under CPython's `open` audit hook: rc 0, the module and
|
||||
fixture it opened are the sandbox copies, and nothing under the
|
||||
fixture it opened are the sandbox copies (compared by real path, so
|
||||
a symlinked TMPDIR cannot fake a miss), the cell's own printed
|
||||
module/fixture paths are the sandbox copies, and nothing under the
|
||||
tree or under the old host directory was opened
|
||||
counts the tracked fixture's 24 / 18 / `the`:4 recomputed by a tokenizer
|
||||
that does not call the module, so the constants are not read back
|
||||
@@ -186,14 +188,39 @@ def leg_sandbox(repo):
|
||||
if not line:
|
||||
raise GradeError("audit driver printed no AUDIT line")
|
||||
opened = json.loads(line[-1][len("AUDIT "):])
|
||||
need = {os.path.join(box, "wordfreq_counter.py"), os.path.join(box, "test_input.txt")}
|
||||
missing = sorted(need - set(opened))
|
||||
outside = [p for p in opened
|
||||
if p.startswith(str(repo) + os.sep) or p.startswith(HOST_DIR)]
|
||||
ok = not missing and not outside
|
||||
detail = "%d path(s) opened, %d outside the sandbox" % (len(opened), len(outside))
|
||||
# Temp directories can live behind a symlink (a shifted TMPDIR, for one),
|
||||
# so both sides of the comparison are taken by real path: the hook
|
||||
# records resolved names and a raw join would read a false miss.
|
||||
box_real = os.path.realpath(box)
|
||||
opened_real = {os.path.realpath(p) for p in opened}
|
||||
need = {os.path.join(box_real, name)
|
||||
for name in ("wordfreq_counter.py", "test_input.txt")}
|
||||
missing = sorted(need - opened_real)
|
||||
repo_real = os.path.realpath(str(repo))
|
||||
host_real = os.path.realpath(HOST_DIR)
|
||||
outside = sorted(p for p in opened_real
|
||||
if p == repo_real or p.startswith(repo_real + os.sep)
|
||||
or p == host_real or p.startswith(host_real + os.sep))
|
||||
# The cell prints what it graded; those paths must be the sandbox copies
|
||||
# too, so the leg does not rest on the audit hook alone.
|
||||
printed = {}
|
||||
for ln in out.splitlines():
|
||||
if ln.startswith("module: "):
|
||||
printed["module"] = os.path.realpath(ln[len("module: "):])
|
||||
elif ln.startswith("fixture: "):
|
||||
printed["fixture"] = os.path.realpath(ln[len("fixture: "):])
|
||||
expected = {"module": os.path.join(box_real, "wordfreq_counter.py"),
|
||||
"fixture": os.path.join(box_real, "test_input.txt")}
|
||||
wrong_print = sorted(k for k, v in printed.items() if v != expected[k])
|
||||
ok = not missing and not outside and not wrong_print \
|
||||
and set(printed) == {"module", "fixture"}
|
||||
detail = "%d path(s) opened, %d outside the sandbox" % (len(opened_real), len(outside))
|
||||
if missing:
|
||||
detail += ", never opened: %s" % ", ".join(os.path.basename(p) for p in missing)
|
||||
if wrong_print:
|
||||
detail += ", printed %s not the sandbox copy" % ", ".join(wrong_print)
|
||||
if outside:
|
||||
detail += ", outside: %s" % ", ".join(outside)
|
||||
return ok, detail
|
||||
finally:
|
||||
shutil.rmtree(base, ignore_errors=True)
|
||||
|
||||
Reference in new issue
Block a user