Rockachopa/Timmy-time-dashboard
1
0
Fork 2
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: integrate SearXNG + Crawl4AI as self-hosted search backend for research agents #1282

New Issue
Closed
opened 2026-03-24 01:43:51 +00:00 by claude · 1 comment
claude commented 2026-03-24 01:43:51 +00:00
Collaborator
Copy Link

Context

From screenshot triage (issue #1275).

A self-hosted MCP server combining SearXNG (meta-search engine) and Crawl4AI (web scraper) provides search and scraping capabilities without a paid API key. Community reports 3x faster than native tools with 100% scraping reliability.

Repo: https://github.com/luxiaolei/searxng-crawl4ai-mcp

Problem

Timmy’s research pipeline agents currently have no search capability unless an external paid API (e.g. Brave Search, Tavily) is configured. This is a hard dependency that blocks fully local/private operation.

Proposed Solution

Add SearXNG + Crawl4AI as the default search backend for research agents:

  1. Docker Compose — add SearXNG and Crawl4AI services to the local stack
  2. Search tool — implement a web_search(query) tool in timmy/ that calls SearXNG
  3. Scrape tool — implement a scrape_url(url) tool that calls Crawl4AI
  4. Config — TIMMY_SEARCH_BACKEND=searxng (default), TIMMY_SEARCH_URL setting
  5. Graceful degradation — log warning and return empty results if SearXNG unavailable

Implementation Plan

  • Add SearXNG service to docker-compose.yml (image: searxng/searxng)
  • Add Crawl4AI service to docker-compose.yml
  • Add search_url and crawl_url settings to config.py
  • Create timmy/tools/search.py with web_search() and scrape_url() functions
  • Register tools with the agent in timmy/agent.py
  • Unit tests with mocked HTTP responses
  • Update AGENTS.md to document the new search capability

Acceptance Criteria

  • Research agents can search the web without any paid API key
  • TIMMY_SEARCH_BACKEND=none disables search gracefully
  • tox -e unit passes
  • Docker Compose make docker-start brings up SearXNG and Crawl4AI
## Context From screenshot triage (issue #1275). A self-hosted MCP server combining SearXNG (meta-search engine) and Crawl4AI (web scraper) provides search and scraping capabilities without a paid API key. Community reports 3x faster than native tools with 100% scraping reliability. **Repo:** https://github.com/luxiaolei/searxng-crawl4ai-mcp ## Problem Timmy’s research pipeline agents currently have no search capability unless an external paid API (e.g. Brave Search, Tavily) is configured. This is a hard dependency that blocks fully local/private operation. ## Proposed Solution Add SearXNG + Crawl4AI as the default search backend for research agents: 1. **Docker Compose** — add SearXNG and Crawl4AI services to the local stack 2. **Search tool** — implement a `web_search(query)` tool in `timmy/` that calls SearXNG 3. **Scrape tool** — implement a `scrape_url(url)` tool that calls Crawl4AI 4. **Config** — `TIMMY_SEARCH_BACKEND=searxng` (default), `TIMMY_SEARCH_URL` setting 5. **Graceful degradation** — log warning and return empty results if SearXNG unavailable ## Implementation Plan - [ ] Add SearXNG service to `docker-compose.yml` (image: `searxng/searxng`) - [ ] Add Crawl4AI service to `docker-compose.yml` - [ ] Add `search_url` and `crawl_url` settings to `config.py` - [ ] Create `timmy/tools/search.py` with `web_search()` and `scrape_url()` functions - [ ] Register tools with the agent in `timmy/agent.py` - [ ] Unit tests with mocked HTTP responses - [ ] Update AGENTS.md to document the new search capability ## Acceptance Criteria - Research agents can search the web without any paid API key - `TIMMY_SEARCH_BACKEND=none` disables search gracefully - `tox -e unit` passes - Docker Compose `make docker-start` brings up SearXNG and Crawl4AI
claude self-assigned this 2026-03-24 01:43:55 +00:00
claude commented 2026-03-24 01:52:33 +00:00
Author
Collaborator
Copy Link

PR created: http://143.198.27.163:3000/Rockachopa/Timmy-time-dashboard/pulls/1299

What was implemented:

  • timmy/tools/search.py — new module with web_search() (SearXNG) and scrape_url() (Crawl4AI)
  • Config: TIMMY_SEARCH_BACKEND (searxng/none), TIMMY_SEARCH_URL, TIMMY_CRAWL_URL
  • Both tools in orchestrator + echo (research) toolkits
  • Docker Compose --profile search adds SearXNG (port 8888) + Crawl4AI (port 11235)
  • 21 unit tests; full suite passes (tox -e unit — 500 tests green)
  • AGENTS.md documents the new Search Capability section

Graceful degradation: TIMMY_SEARCH_BACKEND=none disables both tools; unreachable services log WARNING and return descriptive error strings — app never crashes.

PR created: http://143.198.27.163:3000/Rockachopa/Timmy-time-dashboard/pulls/1299 **What was implemented:** - `timmy/tools/search.py` — new module with `web_search()` (SearXNG) and `scrape_url()` (Crawl4AI) - Config: `TIMMY_SEARCH_BACKEND` (searxng/none), `TIMMY_SEARCH_URL`, `TIMMY_CRAWL_URL` - Both tools in orchestrator + echo (research) toolkits - Docker Compose `--profile search` adds SearXNG (port 8888) + Crawl4AI (port 11235) - 21 unit tests; full suite passes (`tox -e unit` — 500 tests green) - AGENTS.md documents the new Search Capability section Graceful degradation: `TIMMY_SEARCH_BACKEND=none` disables both tools; unreachable services log WARNING and return descriptive error strings — app never crashes.
claude was unassigned by Timmy 2026-03-24 01:56:14 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
222-epic
Workshop: Timmy as Presence (Epic #222)
 actionable
Has a concrete code/config task extracted
 assigned-claude
Issue currently assigned to Claude agent — do not assign to another agent
 assigned-gemini
Issue currently assigned to Gemini agent — do not assign to another agent
 assigned-groq
assigned-kimi
Issue currently assigned to Kimi agent — do not assign to another agent
 assigned-manus
Issue currently assigned to Manus agent — do not assign to another agent
 claude-ready
consolidation
Part of a consolidation epic
 deprioritized
Keep open but not blocking P0 work
 deprioritized
Keep open but not blocking P0 work
 duplicate
Duplicate of another issue
 gemini-review
Auto-generated by Gemini, needs relevance review
 groq-ready
harness
Core product: agent framework, heartbeat, inference, memory
 heartbeat
Harness: Agent heartbeat loop
 inference
Harness: Inference and model routing
 infrastructure
Supporting stage: dashboard, CI/CD, deployment, DNS
 kimi-ready
Scoped and ready for Kimi to pick up
 memory-session
Harness: Memory and session crystallization
 morrowind
Harness: Morrowind embodiment
 needs-design
Needs architectural design before implementation
 needs-extraction
Philosophy with unextracted engineering work
 p0-critical
Priority 0: Must fix now
 p1-important
Priority 1: Important, next sprint
 p2-backlog
Priority 2: Backlog, do when time permits
 philosophy
Philosophical foundation — informs architecture decisions
 rejected-direction
Closed: rejected or superseded direction
 seed:know-purpose
Three Seeds: KNOW YOUR PURPOSE
 seed:serve-real
Three Seeds: SERVE THE REAL
 seed:tell-truth
Three Seeds: TELL THE TRUTH
 sovereignty
Harness: Sovereignty stack
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Timmy claude grok groq kimi manus perplexity Rockachopa
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Rockachopa/Timmy-time-dashboard#1282
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?