Gentoo bugzilla closed due AI bot scraper overload

hhapposai about 3 hours ago 30 commentsRead Article on social.treehouse.systems

DE version is available. Content is displayed in original English for accuracy.

⚡ Community Insights

Discussion Sentiment

75% Positive

Analyzed from 986 words in the discussion.

Discussion (30 Comments)Read Original on HackerNews

mrweasel•about 2 hours ago

While we're dealing with the same issue at work, I sometimes still wonder exactly who these scrapers are. OpenAI, Google, Anthropic and others are normally fairly well behaved (Minus Anthropic attempting to hide behind a browser-for-hire company). Mostly you can get IP range and user-agents for the large players, while it problems mostly stem from bots pretending to be Chrome.

Our largest offenders seems to be mostly limited to South-East Asia, so probably mostly Chinese AI projects, but that's speculation. I also don't recall ever seeing Grok IP ranges or a specific Grok UA, but that doesn't mean that they're hiding, perhaps they're just not interested.

marginalia_nu•about 1 hour ago

Most of the nonsense comes through botnets/residential proxies, so it's very hard to say.

shevy-java•about 1 hour ago

Well, some websites claimed China is behind it, which could make sense (I would not know either way). At the same time, though, I kind of doubt your carte blanche here for all those companies. Why would you think none of them are responsible for the AI slop spam?

> Our largest offenders seems to be mostly limited to South-East Asia, so probably mostly Chinese AI projects, but that's speculation

Ok. So you also don't know. Well, I don't know either, but I don't make a speculation by claiming x, y, and z companies to be exempt. In my book they are all responsible.

coldbrewed•33 minutes ago

We see constant abuse from the Tencent ASN/associated ACE ASN, and I've memorized the china169 backbone asn as AS4837 because of thr nonstop crawlers splattered across their network ranges. It's not possible to ID the operator running the crawlers running from these networks but there's a clear signal of the origin of some of these entities.

littlecranky67•about 1 hour ago

It is time for micropayments integrated in the browser. Pay 5 cents to access each bug report. Not fun, it shouldn't be like this, but better than not having a bugtracker at all.

kangalioo•about 1 hour ago

0.05 cents even. The beautiful thing is, minuscule amounts of micropayments are already enough to fix the incentives

comboy•30 minutes ago

Not sure it works, isn't it already like that if you have the cursed javascript-only pages? It's basically proof of work to access their content (and if you are scrapping en masse you likely also need some LLM getting involved)

vlyan•13 minutes ago

and do you happen to know a zero friction payment system that works internationally like the Internet itself does? because I promise you that unless you already have a captive audience, even a minuscule amount of friction to access your service will cost you an overwhelming percentage of organic human visitors.

tarpitt•5 minutes ago

Something like Chaum's blind-signature based ecash, or GNU Taler would be ideal in terms of efficiency, but it's still centralized in distribution. Freenet currently uses something like this.

Another option I was thinking of would be a pretty inflationary (or demurrage) cryptocurrency in which you have some sort of RandomX or other CPU-bound PoW. A web server could act as a mining pool and use mining shares interchangibly with micropayments.

You could do this mining-share method with Monero right now, it's just that you have higher transaction size in Monero and no real analogue to Bitcoin's LN-based microtransactions. Also you would want the cryptocurrency to be more inflationary (or demurrage-based) to promote usage.

Monero's FCMP++ lays some groundwork for payment channels, but it still lacks the nessisary timelocks. Also there was DLSAG which could have enabled payment channels I think, but it's no longer relevant. I also insist that you would need to change the tokenomics to favor greater inflation (maybe you could make coinbase scale linearly with hashrate?), otherwise the miner reward would be economicially insufficient.

pduggishetti•32 minutes ago

I think it's gonna kill off the internet not just for bots but to whole class of lower income countries, especially for younger learners

csomar•about 1 hour ago

How many pages does the average software developer visit everyday? 1.000? Price at 0.001 per load and it’ll be completely impractical for crawlers but super cheap for the average connected human on earth.

pduggishetti•33 minutes ago

That would gate the internet to whole lower income countries

bluefirebrand•16 minutes ago

Aren't many of those countries basically being de-facto blocked by Cloudflare filtering out spam already anyways?

Genuine question, I'm not up to date on how Cloudflare operates right now

jpcom•18 minutes ago

What are they scraping the gentoo bugzilla for? I'm confused. Unless you're actively using Gentoo why would this be a resource? Very confusing. Also you'd think we'd have LLM BitTorrent by now, where if they want to scrape something we get a DHT hash for the content and share it with one another, rather than melt servers with the millionth request of the day.

ComputerPerson•about 2 hours ago

There are definitely patterns you can use against the scrapers. This maintainer just didn't have time for it, which is understandable.

We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time.

Most scrapers are relatively honest in some way shape or form.

cogman10•about 1 hour ago

One thing that surpised me about gentoo is just how low budget it is as an operation. They are doing everything with a $12k budget. [1]

[1] https://www.gentoo.org/news/2026/01/05/new-year.html

sporp•22 minutes ago

How did you implement this? My site's getting hammered, any tips would be appreciated

Symbiote•about 1 hour ago

> Most scrapers are relatively honest in some way shape or form.

Did you miss a "dis" in there?

c-fe•about 2 hours ago

going on a bit of a tangent here - some discussion there seems to be imply that AI bots mostly use IPv4s, which makes sense to me given that they are probably bots hosted by some cloud. Whereas IPv6 may be more organic traffic from e.g. mobile users (not in this case probably)... which made me think if at some point IPv6 may at some point win against IPv4 just because its more organic traffic (i.e. not from big tech cloud), leading to pages blocking IPv4? Just speculation on my side

Retr0id•about 2 hours ago

The hardest-to-mitigate bot traffic tends to come from residential proxies, and most residential connections are still v4-only.

However, it is also easy to get large numbers of v6 addresses cheaply.

neuroticnews25•about 2 hours ago

This is weird, every VPS I've rented in the last 8 years had IPv6, and some didn't even had IPv4.

TingPing•about 1 hour ago

If a problematic bot is on any mainstream VPS it will just get shut down.

Symbiote•about 1 hour ago

How would I shut down a bot on Digital Ocean making over a million requests a day?

jiffygist•about 1 hour ago

So where are bugs reported now?

nubinetwork•8 minutes ago

That's the thing, they aren't.

shevy-java•about 1 hour ago

AI skynet is winning. It is stealing time from humans, thus forcing down their activity, as can be seen here.

A headline would be great if those AI companies would close down. I hold them all responsible for this.

lilbigdoot•42 minutes ago

Plot twist: the AI wins not through terminators but a kafka nightmare

ece•about 1 hour ago

Every website that wants to, can just charge for dumps. Anubis or even Cloudflare can handle the rest I think.

Symbiote•about 1 hour ago

Scrapers will ignore the dumps.

You can link to them in 10 million HTTP 429 responses, they will still ignore them.

calvinmorrison•about 2 hours ago

The year is 2026, somehow peoples basic web apps are not able to keep up with scrapers. Scrapers are not new. My side load is like .01 even with 10x the traffic of last year. It's called "serving static content", "caching" and many other things that are not new concepts.

Then you've got good old cloudflare which is free to use

sethops1•36 minutes ago

Are you volunteering to setup and host that extra complexity for the Gentoo project?