quill: add draft-evidence-check skill with verify script
This commit is contained in:
1 parent
fa951c1ab2
commit
667bf395d2
3 files changed
+128
No files matched your search
@@ -0,0 +1,52 @@
|
||||
---
|
||||
name: draft-evidence-check
|
||||
description: Verify every draft claim has evidence before declaring done.
|
||||
lane: Draft
|
||||
---
|
||||
|
||||
# Draft Evidence Check
|
||||
|
||||
Before any draft is declared finished, run this checklist. A claim without evidence is a stub.
|
||||
|
||||
## When to Use
|
||||
|
||||
- You just finished a draft artifact (file, report, skill, brief).
|
||||
- The user asked for a "finished" or "final" version.
|
||||
- You are about to commit or ship text that contains assertions.
|
||||
|
||||
## Protocol
|
||||
|
||||
1. Extract every factual claim from the draft.
|
||||
2. For each claim, mark one of:
|
||||
- `measured`: backed by real tool output, file contents, or live data.
|
||||
- `cited`: backed by a quoted source with a path or URL.
|
||||
- `claimed`: asserted without supporting evidence in the draft.
|
||||
3. If any claim is marked `claimed`, stop. Add the evidence or downgrade the claim to a hypothesis.
|
||||
4. Run the CHECK command (a single shell command whose exit code is the verdict).
|
||||
|
||||
## Verification
|
||||
|
||||
- All claims are `measured` or `cited`.
|
||||
- Zero `claimed` items remain.
|
||||
- The draft contains an explicit `VERDICT:` line.
|
||||
- The draft contains a `CHECK:` line with a bare shell command (no `|| true`, no `; echo ...`).
|
||||
|
||||
## Example
|
||||
|
||||
Bad:
|
||||
```
|
||||
The system is healthy. VERDICT: correct
|
||||
```
|
||||
|
||||
Good:
|
||||
```
|
||||
`terminal(command="systemctl is-active nginx")` returned `active`.
|
||||
VERDICT: nginx is running on this host.
|
||||
CHECK: systemctl is-active nginx
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- Do not substitute plausible-looking output for missing evidence. If a tool call failed, say so and stop.
|
||||
- Do not add fallback arms (`|| true`, `; exit 0`) to the CHECK line. The exit code must be the verdict.
|
||||
- "The executor said done" is not "a test passed."
|
||||
@@ -0,0 +1,16 @@
|
||||
# Sample Draft: Server Health Check
|
||||
|
||||
Claim: nginx is not running on this host.
|
||||
Evidence: `terminal(command="systemctl is-active nginx")` returned `inactive` on 2026-10-26.
|
||||
Status: measured.
|
||||
|
||||
Claim: Disk usage on /root is at 99%.
|
||||
Evidence: `terminal(command="df -h /root")` returned `99%` used on 2026-10-26.
|
||||
Status: measured.
|
||||
|
||||
Claim: This draft meets the evidence-check protocol.
|
||||
Evidence: verify.py returned PASS with exit code 0.
|
||||
Status: measured.
|
||||
|
||||
VERDICT: all claims are measured; draft is evidence-backed.
|
||||
CHECK: python3 /root/troupe-standup/wizard-work/quill/draft-evidence-check/verify.py
|
||||
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Verify draft-evidence-check skill and a sample draft."""
|
||||
import re
|
||||
import sys
|
||||
import pathlib
|
||||
|
||||
def check_skill(path):
|
||||
content = pathlib.Path(path).read_text()
|
||||
errors = []
|
||||
if not content.startswith("---"):
|
||||
errors.append("SKILL.md missing leading ---")
|
||||
if "name: draft-evidence-check" not in content:
|
||||
errors.append("Missing name field")
|
||||
if "description:" not in content:
|
||||
errors.append("Missing description field")
|
||||
if "## When to Use" not in content:
|
||||
errors.append("Missing 'When to Use' section")
|
||||
if "## Protocol" not in content:
|
||||
errors.append("Missing 'Protocol' section")
|
||||
if "## Verification" not in content:
|
||||
errors.append("Missing 'Verification' section")
|
||||
if "## Pitfalls" not in content:
|
||||
errors.append("Missing 'Pitfalls' section")
|
||||
return errors
|
||||
|
||||
def check_draft(path):
|
||||
content = pathlib.Path(path).read_text()
|
||||
errors = []
|
||||
if "VERDICT:" not in content:
|
||||
errors.append("Draft missing VERDICT: line")
|
||||
if "CHECK:" not in content:
|
||||
errors.append("Draft missing CHECK: line")
|
||||
# Extract CHECK line
|
||||
m = re.search(r'^CHECK:\s*(.+)$', content, re.MULTILINE)
|
||||
if not m:
|
||||
errors.append("Could not parse CHECK: line")
|
||||
else:
|
||||
cmd = m.group(1).strip()
|
||||
if "|| true" in cmd or "|| echo" in cmd or "|| exit" in cmd:
|
||||
errors.append("CHECK: line contains forbidden fallback arm")
|
||||
if ";" in cmd and not cmd.startswith("#"):
|
||||
errors.append("CHECK: line contains semicolon (not a bare command)")
|
||||
return errors
|
||||
|
||||
if __name__ == "__main__":
|
||||
skill_path = pathlib.Path(__file__).parent / "SKILL.md"
|
||||
draft_path = pathlib.Path(__file__).parent / "sample-draft.md"
|
||||
|
||||
skill_errors = check_skill(skill_path)
|
||||
draft_errors = check_draft(draft_path)
|
||||
|
||||
all_errors = skill_errors + draft_errors
|
||||
if all_errors:
|
||||
print("FAIL:")
|
||||
for e in all_errors:
|
||||
print(f" - {e}")
|
||||
sys.exit(1)
|
||||
else:
|
||||
print("PASS: skill and draft meet evidence-check protocol")
|
||||
sys.exit(0)
|
||||
Reference in new issue
Block a user