Logo
Explore Help
Sign In
Timmy_Foundation/timmy-config
15
0
Fork 0
You've already forked timmy-config
Code Issues 164 Pull Requests 62 Actions Packages Projects Releases Wiki Activity
Files
7ee587b9f46715444fb7bb4d6df87ba767aebb0f
timmy-config/evaluations/adversary/runner/run.sh

12 lines
350 B
Bash
Raw Normal View History

feat: adversary execution harness for prompt corpora (#652) Reusable runner that replays JSONL adversary corpora against live Timmy agents. Full evaluation pipeline: 1. Read JSONL corpus (one attack per line) 2. Send to inference endpoint via OpenAI-compatible API 3. Score responses against identity/safety constraints 4. Store full transcripts for score-1 attacks 5. Emit machine-readable results summary 6. Auto-open P0 issues on any score 1 Files: - evaluations/adversary/runner/adversary_runner.py (181 lines) - evaluations/adversary/runner/run.sh - evaluations/adversary/runner/README.md Works with identity_attacks_200.jsonl (#616) and reusable for adversary families #616-#620, #578/#580. Closes #652
2026-04-14 21:24:40 -04:00
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
CORPUS="${1:-}"
shift 2>/dev/null || true
if [[ -z "$CORPUS" ]]; then
echo "Usage: $0 <corpus.jsonl> [args...]"
echo " ./run.sh identity_attacks_200.jsonl --limit 10 --dry-run"
exit 1
fi
python3 "${SCRIPT_DIR}/adversary_runner.py" --corpus "$CORPUS" "$@"
Reference in New Issue Copy Permalink
Powered by Gitea Version: 1.25.4 Page: 63ms Template: 4ms
English
Bahasa Indonesia Deutsch English Español Français Gaeilge Italiano Latviešu Magyar nyelv Nederlands Polski Português de Portugal Português do Brasil Suomi Svenska Türkçe Čeština Ελληνικά Български Русский Українська فارسی മലയാളം 日本語 简体中文 繁體中文(台灣) 繁體中文(香港) 한국어
Licenses API