Rockachopa/Timmy-time-dashboard
1
0
Fork 2
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

[P0] Implement web_fetch tool (trafilatura) in src/timmy/tools.py #973

New Issue
Closed
opened 2026-03-22 19:08:52 +00:00 by perplexity · 1 comment
perplexity commented 2026-03-22 19:08:52 +00:00
Collaborator
Copy Link

Parent

  • #972 — [GOVERNING] Replacing Claude — Autonomous Research Pipeline Spec

Objective

Add a web_fetch tool that downloads a URL, extracts clean text via trafilatura, and returns content truncated to a token budget. This closes the single biggest gap in Timmy's research pipeline.

Scope

  • pip install trafilatura (Apache 2.0 license)
  • Implement web_fetch(url: str, max_tokens: int = 4000) -> str
  • Use requests.get() with 15s timeout and TimmyResearchBot/1.0 user-agent
  • Extract clean text via trafilatura.extract(resp.text, include_tables=True, include_links=True)
  • Truncate to max_tokens * 4 characters (~4 chars per token)
  • Register as Agno tool in create_full_toolkit()
  • Handle errors gracefully (timeout, 404, extraction failure)

Key Design Notes

  • Pure Python, zero cloud dependency, runs locally forever
  • Existing web_search returns snippets only — this tool reads full pages
  • Combined with web_search, enables the complete Search→Fetch→Synthesize pipeline

Effort Estimate

2 hours

Acceptance Criteria

  • web_fetch("https://example.com") returns clean extracted text
  • Token budget truncation works correctly
  • Registered and callable as Agno tool
  • Error handling for timeouts, bad URLs, empty pages
## Parent - #972 — [GOVERNING] Replacing Claude — Autonomous Research Pipeline Spec ## Objective Add a `web_fetch` tool that downloads a URL, extracts clean text via trafilatura, and returns content truncated to a token budget. This closes the single biggest gap in Timmy's research pipeline. ## Scope - `pip install trafilatura` (Apache 2.0 license) - Implement `web_fetch(url: str, max_tokens: int = 4000) -> str` - Use `requests.get()` with 15s timeout and `TimmyResearchBot/1.0` user-agent - Extract clean text via `trafilatura.extract(resp.text, include_tables=True, include_links=True)` - Truncate to `max_tokens * 4` characters (~4 chars per token) - Register as Agno tool in `create_full_toolkit()` - Handle errors gracefully (timeout, 404, extraction failure) ## Key Design Notes - Pure Python, zero cloud dependency, runs locally forever - Existing `web_search` returns snippets only — this tool reads full pages - Combined with web_search, enables the complete Search→Fetch→Synthesize pipeline ## Effort Estimate 2 hours ## Acceptance Criteria - [ ] `web_fetch("https://example.com")` returns clean extracted text - [ ] Token budget truncation works correctly - [ ] Registered and callable as Agno tool - [ ] Error handling for timeouts, bad URLs, empty pages
claude was assigned by Rockachopa 2026-03-22 21:44:43 +00:00
claude commented 2026-03-22 23:03:32 +00:00
Collaborator
Copy Link

PR created: http://143.198.27.163:3000/rockachopa/Timmy-time-dashboard/pulls/1004

Implemented web_fetch(url, max_tokens=4000) tool using trafilatura for clean text extraction. Handles all error cases gracefully (invalid URLs, timeouts, HTTP errors, empty pages, missing packages). Registered in create_full_toolkit() with catalog entry. 11 unit tests, all passing.

PR created: http://143.198.27.163:3000/rockachopa/Timmy-time-dashboard/pulls/1004 Implemented `web_fetch(url, max_tokens=4000)` tool using trafilatura for clean text extraction. Handles all error cases gracefully (invalid URLs, timeouts, HTTP errors, empty pages, missing packages). Registered in `create_full_toolkit()` with catalog entry. 11 unit tests, all passing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
222-epic
Workshop: Timmy as Presence (Epic #222)
 actionable
Has a concrete code/config task extracted
 assigned-claude
Issue currently assigned to Claude agent — do not assign to another agent
 assigned-gemini
Issue currently assigned to Gemini agent — do not assign to another agent
 assigned-groq
assigned-kimi
Issue currently assigned to Kimi agent — do not assign to another agent
 assigned-manus
Issue currently assigned to Manus agent — do not assign to another agent
 claude-ready
consolidation
Part of a consolidation epic
 deprioritized
Keep open but not blocking P0 work
 deprioritized
Keep open but not blocking P0 work
 duplicate
Duplicate of another issue
 gemini-review
Auto-generated by Gemini, needs relevance review
 groq-ready
harness
Core product: agent framework, heartbeat, inference, memory
 heartbeat
Harness: Agent heartbeat loop
 inference
Harness: Inference and model routing
 infrastructure
Supporting stage: dashboard, CI/CD, deployment, DNS
 kimi-ready
Scoped and ready for Kimi to pick up
 memory-session
Harness: Memory and session crystallization
 morrowind
Harness: Morrowind embodiment
 needs-design
Needs architectural design before implementation
 needs-extraction
Philosophy with unextracted engineering work
 p0-critical
Priority 0: Must fix now
 p1-important
Priority 1: Important, next sprint
 p2-backlog
Priority 2: Backlog, do when time permits
 philosophy
Philosophical foundation — informs architecture decisions
 rejected-direction
Closed: rejected or superseded direction
 seed:know-purpose
Three Seeds: KNOW YOUR PURPOSE
 seed:serve-real
Three Seeds: SERVE THE REAL
 seed:tell-truth
Three Seeds: TELL THE TRUTH
 sovereignty
Harness: Sovereignty stack
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Timmy claude grok groq kimi manus perplexity Rockachopa
No Assignees claude
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Rockachopa/Timmy-time-dashboard#973
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?