Timmy eed87e454e test: Benchmark Gemma 4 vision accuracy vs current approach (#817) ...

Vision benchmark suite comparing Gemma 4 (google/gemma-4-27b-it) vs
current Gemini 3 Flash Preview (google/gemini-3-flash-preview).

Metrics:
- OCR accuracy (character + word overlap)
- Description completeness (keyword coverage)
- Structural quality (length, sentences, numbers)
- Latency (ms per image)
- Token usage
- Consistency across runs

Features:
- 24 diverse test images (screenshots, diagrams, photos, charts)
- Category-specific evaluation prompts
- Automated verdict with composite scoring
- JSON + markdown report output
- 28 unit tests passing

Usage:
  python benchmarks/vision_benchmark.py --images benchmarks/test_images.json
  python benchmarks/vision_benchmark.py --url https://example.com/img.png
  python benchmarks/vision_benchmark.py --generate-dataset

Closes #817.

2026-04-15 23:02:02 -04:00

..

acp

fix(acp): declare session load and resume capabilities in initialize response (#6985)

2026-04-10 03:45:36 -07:00

agent

feat: add tests/agent/test_privacy_filter.py from PR #397

2026-04-16 01:35:21 +00:00

cli

fix: resolve CI test failures — add missing functions, fix stale tests (#9483)

2026-04-14 01:43:45 -07:00

cron

fix(cron): include job_id in delivery and guide models on removal workflow (#10242)

2026-04-15 03:46:58 -07:00

e2e

refactor: extract shared helpers to deduplicate repeated code patterns (#7917)

2026-04-11 13:59:52 -07:00

environments/benchmarks

fix(security): consolidated security hardening — SSRF, timing attack, tar traversal, credential leakage (#5944)

2026-04-07 17:28:37 -07:00

fakes

fix: streaming tool call parsing, error handling, and fake HA state mutation

2026-03-14 14:27:20 +03:00

gateway

fix(gateway): fix matrix lingering typing indicator

2026-04-15 04:16:16 -07:00

hermes_cli

fix(cli): restore messaging toolset for gateway platforms

2026-04-14 23:13:35 -07:00

honcho_plugin

feat(honcho): add opt-in initOnSessionStart for tools mode and respect explicit peerName (#6995)

2026-04-11 00:43:27 -07:00

integration

refactor: remove dead code — 1,784 lines across 77 files (#9180)

2026-04-13 16:32:04 -07:00

plugins

feat: sort tool search results by score and add corresponding unit test

2026-04-14 10:49:35 -07:00

run_agent

fix: strip reasoning item IDs from Responses API input when store=False (#10217)

2026-04-15 03:19:43 -07:00

skills

fix(migration): don't auto-archive OpenClaw source directory

2026-04-12 00:33:54 -07:00

tools

feat: add tests/tools/test_confirmation_daemon.py from PR #397

2026-04-16 01:35:24 +00:00

__init__.py

A bit of restructuring for simplicity and organization

2025-10-01 23:29:25 +00:00

conftest.py

fix(tests): fix several failing/flaky tests on main (#6777)

2026-04-09 13:17:06 -07:00

run_interrupt_test.py

fix: thread safety for concurrent subagent delegation (#1672)

2026-03-17 02:53:33 -07:00

test_batch_runner_checkpoint.py

fix: sanitize chat payloads and provider precedence

2026-03-13 23:59:12 -07:00

test_cli_file_drop.py

fix(gateway): reject file paths in get_command() + file-drop tests (#7356)

2026-04-10 13:06:02 -07:00

test_cli_skin_integration.py

fix: CLI/UX batch — ChatConsole errors, curses scroll, skin-aware banner, git state banner (#5974)

2026-04-07 17:59:42 -07:00

test_ctx_halving_fix.py

fix(tests): fix 78 CI test failures and remove dead test (#9036)

2026-04-13 10:50:24 -07:00

test_empty_model_fallback.py

fix: fall back to provider's default model when model config is empty (#8303)

2026-04-12 03:53:30 -07:00

test_evidence_store.py

feat: add OSS Security Forensics skill (Skills Hub) (#1482)

2026-03-15 21:59:53 -07:00

test_hermes_constants.py

fix(gateway): harden Docker/container gateway pathway

2026-04-12 16:36:11 -07:00

test_hermes_logging.py

fix(tests): fix 78 CI test failures and remove dead test (#9036)

2026-04-13 10:50:24 -07:00

test_hermes_state.py

fix(state): orphan children instead of cascade-deleting in prune/delete (#6513)

2026-04-09 02:41:56 -07:00

test_honcho_client_config.py

feat(memory): pluggable memory provider interface with profile isolation, review fixes, and honcho CLI restoration (#4623)

2026-04-02 15:33:51 -07:00

test_ipv4_preference.py

feat: add network.force_ipv4 config to fix IPv6 timeout issues (#8196)

2026-04-11 23:12:11 -07:00

test_mcp_serve.py

feat: add MCP server mode — hermes mcp serve (#3795)

2026-03-29 15:47:19 -07:00

test_minisweagent_path.py

chore: remove all remaining mini-swe-agent references

2026-03-24 08:19:23 -07:00

test_model_picker_scroll.py

fix: CLI/UX batch — ChatConsole errors, curses scroll, skin-aware banner, git state banner (#5974)

2026-04-07 17:59:42 -07:00

test_model_tools_async_bridge.py

fix: use per-thread persistent event loops in worker threads

2026-03-20 15:41:06 -04:00

test_model_tools.py

feat(plugins): let pre_tool_call hooks block tool execution

2026-04-13 22:01:49 -07:00

test_ollama_num_ctx.py

fix: provider/model resolution — salvage 4 PRs + MiniMax aux URL fix (#5983)

2026-04-07 22:23:28 -07:00

test_packaging_metadata.py

chore: prepare Hermes for Homebrew packaging (#4099)

2026-03-30 17:34:43 -07:00

test_plugin_skills.py

feat(plugins): namespaced skill registration for plugin skill bundles

2026-04-14 10:42:58 -07:00

test_project_metadata.py

refactor(matrix): swap matrix-nio for mautrix-python dependency

2026-04-10 21:15:59 -07:00

test_retry_utils.py

feat(agent): add jittered retry backoff

2026-04-08 00:41:36 -07:00

test_sql_injection.py

fix(security): eliminate SQL string formatting in execute() calls

2026-03-19 15:16:35 +01:00

test_subprocess_home_isolation.py

fix: per-profile subprocess HOME isolation (#4426) (#7357)

2026-04-10 13:37:45 -07:00

test_timezone.py

fix: remove 115 verified dead code symbols across 46 production files

2026-04-10 03:44:43 -07:00

test_toolset_distributions.py

test: add unit tests for 8 modules (batch 2)

2026-02-26 13:54:20 +03:00

test_toolsets.py

fix(mcp): make server aliases explicit

2026-04-14 17:19:20 -07:00

test_trajectory_compressor_async.py

fix(tests): fix 78 CI test failures and remove dead test (#9036)

2026-04-13 10:50:24 -07:00

test_trajectory_compressor.py

fix: load credentials from HERMES_HOME .env in trajectory_compressor

2026-04-14 10:24:19 -07:00

test_utils_truthy_values.py

Gate tool-gateway behind an env var, so it's not in users' faces until we're ready. Even if users enable it, it'll be blocked server-side for now, until we unlock for non-admin users on tool-gateway.

2026-03-30 13:28:10 +09:00

test_vision_benchmark.py

test: Benchmark Gemma 4 vision accuracy vs current approach (#817)

2026-04-15 23:02:02 -04:00