PRIMA
02c0cc2b23
All checks were successful
Smoke Test / smoke (pull_request) Successful in 15s
tests/tool_call_regression.py: - 10 test cases covering 5 hermes tools: read_file, web_search, terminal, execute_code, delegate_task - Schema validation (OpenAI-compatible tool call format) - Argument validation (correct tool + expected args) - Parallel tool calling test (multiple tools in one response) - Dry-run mode for CI (schema validation without server) - Full server mode with latency tracking - Markdown report generation with results matrix - JSON results output for programmatic consumption - 95% accuracy threshold gate (exit code 1 on failure) benchmarks/tool-call-regression.md: - Results template with model/preset matrix - Tool coverage tracking table .gitea/workflows/smoke.yml: - Added dry-run tool call schema validation step
1.1 KiB
1.1 KiB
Tool Call Regression Results
Generated: 2026-04-16T01:56:48.462512+00:00 Model: dry-run Endpoint: none KV Type: none
Summary
| Metric | Value |
|---|---|
| Total tests | 10 |
| Passed | 10 |
| Failed | 0 |
| Accuracy | 100.0% |
| Threshold | 100% |
| Verdict | PASS |
Test Matrix
| Test ID | Tool Expected | Tool Called | Schema | Args | Latency | Status |
|---|---|---|---|---|---|---|
| read_file_basic | read_file | none | OK | OK | 0ms | PASS |
| read_file_offset | read_file | none | OK | OK | 0ms | PASS |
| web_search_basic | web_search | none | OK | OK | 0ms | PASS |
| terminal_basic | terminal | none | OK | OK | 0ms | PASS |
| terminal_complex | terminal | none | OK | OK | 0ms | PASS |
| code_exec_basic | execute_code | none | OK | OK | 0ms | PASS |
| code_exec_complex | execute_code | none | OK | OK | 0ms | PASS |
| delegate_basic | delegate_task | none | OK | OK | 0ms | PASS |
| delegate_context | delegate_task | none | OK | OK | 0ms | PASS |
| parallel_two | read_file | none | OK | OK | 0ms | PASS |