Back to News
Advertisement

One Wikipedia page costs your AI agent 68,000 tokens

aarhamislam5766 about 6 hours ago 6 comments

ES version is available. Content is displayed in original English for accuracy.

i use claude code daily and measured what pages cost it while doing research. an average wikipedia article, for instance, is 68,240 tokens of raw html (tiktoken); nike's homepage is 353,000.

claude code's built-in webfetch handles the easy case well. it summarizes wikipedia to about 950 tokens and clears cloudflare on some sites like indeed and ticketmaster. but, and there's always a but, on js-rendered and some anti-bot pages it returns nothing.

quotes.toscrape.com/js gives "no quotes found"; nike.com gives a 403. your agent then dumps the raw html back into context and still fails. (note: i have also had cases where i read through the chat at the end and saw that it failed and just pulled from either training data or stale caches from other sources)

so i worked on building an open-source stealth browser (recompiled chromium) that runs as an mcp for claude code, cursor, and claude desktop. essentially all i have to change form my end is add the mcp, and it returns the cleaned up tokens while also beating detection: the js quotes come back in 285 tokens, nike in about 700 instead of a 403.

there is still stuff i am actively working on: there's no residential egress yet, and it won't beat kasada-style walls. it's for agents, qa, and research.

repo and the reproducible benchmark: https://github.com/tiliondev/fortress/tree/main/mcp i'm the author and here for feedback.

Advertisement

⚡ Community Insights

Discussion Sentiment

100% Positive

Analyzed from 124 words in the discussion.

Trending Topics

#jina#bot#reader#pages#rendered#anti#stripping#markdown#trafilatura#before

Discussion (6 Comments)Read Original on HackerNews

chonghaojuabout 3 hours ago
stripping to markdown with Jina Reader or Trafilatura before passing to the agent cuts that 68k down to ~3-5k for most Wikipedia pages, and handles the JS-rendered case too.
armanluthra_22 minutes ago
+1. i think the the additional value prop is the bypassing blockers. ideally op should just use jina on top of whatever they are building.
mkbknabout 1 hour ago
And how much water?
arthurcolleabout 6 hours ago
one of the early jina.ai products was/is reader api, and they trained ReaderLM for this purpose. definitely a good idea to check out existing implementations
m463about 4 hours ago
> on js-rendered and some anti-bot pages it returns nothing

yeah, and my (human piloted) browser gets blocked by so many websites now. I routinely get "Sorry." when trying to log into hn.

sigh.

dlcarrierabout 4 hours ago
Hacker News is the one page that hasn't assumed I'm a bot, yet.

At this rate, we'll need to use bots to browse the web for us, because they're the only way to get through the anti-bot filters.