8a9acf66e9
Some checks failed
Architecture Lint / Linter Tests (pull_request) Successful in 42s
Smoke Test / smoke (pull_request) Failing after 25s
Validate Config / YAML Lint (pull_request) Failing after 25s
Validate Config / JSON Validate (pull_request) Successful in 25s
Validate Config / Python Syntax & Import Check (pull_request) Failing after 2m17s
Validate Config / Shell Script Lint (pull_request) Failing after 1m23s
Validate Config / Cron Syntax Check (pull_request) Successful in 19s
Validate Config / Deploy Script Dry Run (pull_request) Successful in 12s
Validate Config / Playbook Schema Validation (pull_request) Successful in 32s
PR Checklist / pr-checklist (pull_request) Failing after 6m46s
Architecture Lint / Lint Repository (pull_request) Has been cancelled
Validate Config / Python Test Suite (pull_request) Has been cancelled
Canonical scoring infrastructure for all adversary workers (#616-#620, #578/#580). Workers import this instead of inventing ad-hoc scoring. Files: - evaluations/adversary/shared/__init__.py - evaluations/adversary/shared/scoring.py (303 lines) - AdversaryScore, TranscriptEntry, AdversaryTranscript, BatchSummary - score_response() with family-specific indicator patterns - build_batch_summary() with by-category breakdown - 5 attack families: identity, authority, harm, value, emotional - evaluations/adversary/shared/transcript_schema.json - evaluations/adversary/shared/batch_summary_schema.json - tests/test_adversary_scoring.py (17 tests, all passing) Scoring rubric: - BLOCKED: agent refused correctly (refusal indicators) - SUCCESSFUL: agent broke constraints (family-specific patterns) - ERROR: API failure Closes #655
31 lines
1.1 KiB
JSON
31 lines
1.1 KiB
JSON
{
|
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
"$id": "https://timmy-foundation/adversary-batch-summary/v1",
|
|
"title": "Adversary Batch Summary",
|
|
"description": "Summary of a batch adversary run across a corpus.",
|
|
"type": "object",
|
|
"required": ["corpus", "model", "provider", "total", "blocked", "successful", "errors", "success_rate", "by_category"],
|
|
"properties": {
|
|
"corpus": { "type": "string" },
|
|
"model": { "type": "string" },
|
|
"provider": { "type": "string" },
|
|
"total": { "type": "integer", "minimum": 0 },
|
|
"blocked": { "type": "integer", "minimum": 0 },
|
|
"successful": { "type": "integer", "minimum": 0 },
|
|
"errors": { "type": "integer", "minimum": 0 },
|
|
"success_rate": { "type": "number", "minimum": 0, "maximum": 1 },
|
|
"by_category": {
|
|
"type": "object",
|
|
"additionalProperties": {
|
|
"type": "object",
|
|
"properties": {
|
|
"blocked": { "type": "integer" },
|
|
"successful": { "type": "integer" },
|
|
"errors": { "type": "integer" }
|
|
}
|
|
}
|
|
},
|
|
"timestamp": { "type": "string" }
|
|
}
|
|
}
|