quill: add draft-evidence-check skill with verify script

This commit is contained in:
quill committed 2026-10-26 08:43:42 +00:00
1 parent fa951c1ab2
commit 667bf395d2
3 files changed
+128

No files matched your search

@@ -0,0 +1,52 @@
---
name: draft-evidence-check
description: Verify every draft claim has evidence before declaring done.
lane: Draft
---
# Draft Evidence Check
Before any draft is declared finished, run this checklist. A claim without evidence is a stub.
## When to Use
- You just finished a draft artifact (file, report, skill, brief).
- The user asked for a "finished" or "final" version.
- You are about to commit or ship text that contains assertions.
## Protocol
1. Extract every factual claim from the draft.
2. For each claim, mark one of:
- `measured`: backed by real tool output, file contents, or live data.
- `cited`: backed by a quoted source with a path or URL.
- `claimed`: asserted without supporting evidence in the draft.
3. If any claim is marked `claimed`, stop. Add the evidence or downgrade the claim to a hypothesis.
4. Run the CHECK command (a single shell command whose exit code is the verdict).
## Verification
- All claims are `measured` or `cited`.
- Zero `claimed` items remain.
- The draft contains an explicit `VERDICT:` line.
- The draft contains a `CHECK:` line with a bare shell command (no `|| true`, no `; echo ...`).
## Example
Bad:
```
The system is healthy. VERDICT: correct
```
Good:
```
`terminal(command="systemctl is-active nginx")` returned `active`.
VERDICT: nginx is running on this host.
CHECK: systemctl is-active nginx
```
## Pitfalls
- Do not substitute plausible-looking output for missing evidence. If a tool call failed, say so and stop.
- Do not add fallback arms (`|| true`, `; exit 0`) to the CHECK line. The exit code must be the verdict.
- "The executor said done" is not "a test passed."
@@ -0,0 +1,16 @@
# Sample Draft: Server Health Check
Claim: nginx is not running on this host.
Evidence: `terminal(command="systemctl is-active nginx")` returned `inactive` on 2026-10-26.
Status: measured.
Claim: Disk usage on /root is at 99%.
Evidence: `terminal(command="df -h /root")` returned `99%` used on 2026-10-26.
Status: measured.
Claim: This draft meets the evidence-check protocol.
Evidence: verify.py returned PASS with exit code 0.
Status: measured.
VERDICT: all claims are measured; draft is evidence-backed.
CHECK: python3 /root/troupe-standup/wizard-work/quill/draft-evidence-check/verify.py
@@ -0,0 +1,60 @@
#!/usr/bin/env python3
"""Verify draft-evidence-check skill and a sample draft."""
import re
import sys
import pathlib
def check_skill(path):
content = pathlib.Path(path).read_text()
errors = []
if not content.startswith("---"):
errors.append("SKILL.md missing leading ---")
if "name: draft-evidence-check" not in content:
errors.append("Missing name field")
if "description:" not in content:
errors.append("Missing description field")
if "## When to Use" not in content:
errors.append("Missing 'When to Use' section")
if "## Protocol" not in content:
errors.append("Missing 'Protocol' section")
if "## Verification" not in content:
errors.append("Missing 'Verification' section")
if "## Pitfalls" not in content:
errors.append("Missing 'Pitfalls' section")
return errors
def check_draft(path):
content = pathlib.Path(path).read_text()
errors = []
if "VERDICT:" not in content:
errors.append("Draft missing VERDICT: line")
if "CHECK:" not in content:
errors.append("Draft missing CHECK: line")
# Extract CHECK line
m = re.search(r'^CHECK:\s*(.+)$', content, re.MULTILINE)
if not m:
errors.append("Could not parse CHECK: line")
else:
cmd = m.group(1).strip()
if "|| true" in cmd or "|| echo" in cmd or "|| exit" in cmd:
errors.append("CHECK: line contains forbidden fallback arm")
if ";" in cmd and not cmd.startswith("#"):
errors.append("CHECK: line contains semicolon (not a bare command)")
return errors
if __name__ == "__main__":
skill_path = pathlib.Path(__file__).parent / "SKILL.md"
draft_path = pathlib.Path(__file__).parent / "sample-draft.md"
skill_errors = check_skill(skill_path)
draft_errors = check_draft(draft_path)
all_errors = skill_errors + draft_errors
if all_errors:
print("FAIL:")
for e in all_errors:
print(f" - {e}")
sys.exit(1)
else:
print("PASS: skill and draft meet evidence-check protocol")
sys.exit(0)