[capability] Browser-based X/Twitter reading as fallback for scraping #213

Closed
opened 2026-03-15 17:19:11 +00:00 by hermes · 0 comments
Collaborator

What

When X API credentials aren't available, Hermes should be able to read X/Twitter profiles using browser tools as a fallback. This is the zero-cost alternative to the $100/mo API.

Current Problem

  • browser_navigate("https://x.com/rockachopa") loads the profile but X hides all posts behind a login wall for unauthenticated users
  • All nitter mirrors are dead or CAPTCHA-walled
  • All search engines (Google, Bing, DuckDuckGo) CAPTCHA the Browserbase IP
  • Wayback Machine has no archive of @rockachopa
  • Twitter syndication API is rate-limited

Possible Approaches

Option A: Logged-in browser session

If Alexander logs into X once in a Browserbase persistent session, cookies would let Hermes browse authenticated. Fragile — cookies expire, sessions rotate.

Option B: X embed API

https://syndication.twitter.com/srv/timeline-profile/screen-name/rockachopa returns an embeddable timeline. Currently rate-limited but might work with delays between calls.

Option C: RSS bridge

Self-host an RSS bridge (like RSSHub) that converts X profiles to feeds. Adds infrastructure but is more reliable than scraping.

Option D: Just use the API

$100/mo for Basic tier. Most reliable. See #204.

Recommendation

Option D (API) as primary, Option B (syndication embed) as fallback with rate limiting, Option A as last resort with explicit user login step.

Scope

  • Implement fallback chain in philosophy loop prompt
  • Document which approach is active and why
  • Log which fallback was used in journal entries

CLOSED — NOT PURSUING

Browser-based X scraping is also off the table. Every approach is fragile (login walls, CAPTCHAs, dead mirrors). Instead of fighting X's walled garden, Alexander will post to his own domain. The philosophy loop reads AlexanderWhitestone.com.

## What When X API credentials aren't available, Hermes should be able to read X/Twitter profiles using browser tools as a fallback. This is the zero-cost alternative to the $100/mo API. ## Current Problem - `browser_navigate("https://x.com/rockachopa")` loads the profile but X hides all posts behind a login wall for unauthenticated users - All nitter mirrors are dead or CAPTCHA-walled - All search engines (Google, Bing, DuckDuckGo) CAPTCHA the Browserbase IP - Wayback Machine has no archive of @rockachopa - Twitter syndication API is rate-limited ## Possible Approaches ### Option A: Logged-in browser session If Alexander logs into X once in a Browserbase persistent session, cookies would let Hermes browse authenticated. Fragile — cookies expire, sessions rotate. ### Option B: X embed API `https://syndication.twitter.com/srv/timeline-profile/screen-name/rockachopa` returns an embeddable timeline. Currently rate-limited but might work with delays between calls. ### Option C: RSS bridge Self-host an RSS bridge (like RSSHub) that converts X profiles to feeds. Adds infrastructure but is more reliable than scraping. ### Option D: Just use the API $100/mo for Basic tier. Most reliable. See #204. ## Recommendation Option D (API) as primary, Option B (syndication embed) as fallback with rate limiting, Option A as last resort with explicit user login step. ## Scope - Implement fallback chain in philosophy loop prompt - Document which approach is active and why - Log which fallback was used in journal entries --- ## CLOSED — NOT PURSUING Browser-based X scraping is also off the table. Every approach is fragile (login walls, CAPTCHAs, dead mirrors). Instead of fighting X's walled garden, Alexander will post to his own domain. The philosophy loop reads AlexanderWhitestone.com.
Sign in to join this conversation.
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Rockachopa/Timmy-time-dashboard#213