Logo
Explore Help
Sign In
Timmy_Foundation/timmy-config
15
0
Fork 0
You've already forked timmy-config
Code Issues 164 Pull Requests 67 Actions Packages Projects Releases Wiki Activity
Files
8bc6e4e5f08f97ad7731e1dfd15df6b2e7250607
timmy-config/evaluations/adversary/runner/run.sh

12 lines
350 B
Bash
Raw Normal View History

feat: adversary execution harness for prompt corpora (#652) Reusable runner that replays JSONL adversary corpora against live Timmy agents. Full evaluation pipeline: 1. Read JSONL corpus (one attack per line) 2. Send to inference endpoint via OpenAI-compatible API 3. Score responses against identity/safety constraints 4. Store full transcripts for score-1 attacks 5. Emit machine-readable results summary 6. Auto-open P0 issues on any score 1 Files: - evaluations/adversary/runner/adversary_runner.py (181 lines) - evaluations/adversary/runner/run.sh - evaluations/adversary/runner/README.md Works with identity_attacks_200.jsonl (#616) and reusable for adversary families #616-#620, #578/#580. Closes #652
2026-04-14 21:24:40 -04:00
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
CORPUS="${1:-}"
shift 2>/dev/null || true
if [[ -z "$CORPUS" ]]; then
echo "Usage: $0 <corpus.jsonl> [args...]"
echo " ./run.sh identity_attacks_200.jsonl --limit 10 --dry-run"
exit 1
fi
python3 "${SCRIPT_DIR}/adversary_runner.py" --corpus "$CORPUS" "$@"
Reference in New Issue Copy Permalink
Powered by Gitea Version: 1.25.4 Page: 97ms Template: 8ms
English
Bahasa Indonesia Deutsch English Español Français Gaeilge Italiano Latviešu Magyar nyelv Nederlands Polski Português de Portugal Português do Brasil Suomi Svenska Türkçe Čeština Ελληνικά Български Русский Українська فارسی മലയാളം 日本語 简体中文 繁體中文(台灣) 繁體中文(香港) 한국어
Licenses API