test: tool call regression suite for compressed models (closes #96)
All checks were successful
Smoke Test / smoke (pull_request) Successful in 15s

tests/tool_call_regression.py:
- 10 test cases covering 5 hermes tools: read_file, web_search, terminal,
  execute_code, delegate_task
- Schema validation (OpenAI-compatible tool call format)
- Argument validation (correct tool + expected args)
- Parallel tool calling test (multiple tools in one response)
- Dry-run mode for CI (schema validation without server)
- Full server mode with latency tracking
- Markdown report generation with results matrix
- JSON results output for programmatic consumption
- 95% accuracy threshold gate (exit code 1 on failure)

benchmarks/tool-call-regression.md:
- Results template with model/preset matrix
- Tool coverage tracking table

.gitea/workflows/smoke.yml:
- Added dry-run tool call schema validation step
This commit is contained in:
2026-04-15 21:58:34 -04:00
parent 3cd8750cbb
commit 02c0cc2b23
4 changed files with 849 additions and 0 deletions

View File

@@ -0,0 +1,32 @@
# Tool Call Regression Results
**Generated:** 2026-04-16T01:56:48.462512+00:00
**Model:** dry-run
**Endpoint:** none
**KV Type:** none
## Summary
| Metric | Value |
|--------|-------|
| Total tests | 10 |
| Passed | 10 |
| Failed | 0 |
| Accuracy | 100.0% |
| Threshold | 100% |
| Verdict | PASS |
## Test Matrix
| Test ID | Tool Expected | Tool Called | Schema | Args | Latency | Status |
|---------|--------------|-------------|--------|------|---------|--------|
| read_file_basic | read_file | none | OK | OK | 0ms | PASS |
| read_file_offset | read_file | none | OK | OK | 0ms | PASS |
| web_search_basic | web_search | none | OK | OK | 0ms | PASS |
| terminal_basic | terminal | none | OK | OK | 0ms | PASS |
| terminal_complex | terminal | none | OK | OK | 0ms | PASS |
| code_exec_basic | execute_code | none | OK | OK | 0ms | PASS |
| code_exec_complex | execute_code | none | OK | OK | 0ms | PASS |
| delegate_basic | delegate_task | none | OK | OK | 0ms | PASS |
| delegate_context | delegate_task | none | OK | OK | 0ms | PASS |
| parallel_two | read_file | none | OK | OK | 0ms | PASS |