An update on residential proxies and the scraper situation

cchmaynard about 4 hours ago 54 commentsRead Article on lwn.net

DE version is available. Content is displayed in original English for accuracy.

⚡ Community Insights

Discussion Sentiment

67% Positive

Analyzed from 1495 words in the discussion.

Discussion (54 Comments)Read Original on HackerNews

mips_avatar•about 3 hours ago

I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs.

I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.

nobodywillobsrv•about 1 hour ago

Feels like it would be a good time for freenet and the like to catch on.

jay_kyburz•about 2 hours ago

I agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape.

You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.

jay_kyburz•about 2 hours ago

How about a website header with a link to a static zip that contains the whole website in one hit. The Zip could be hosted on some big public sever. Perhaps even mirrored locally for each nation.

mips_avatar•about 2 hours ago

that's hard to do with rendered content, oftentimes the result depends on a backend service. Maybe you should make the service it's running public but that might be a line most aren't willing to cross.

sixtyj•about 3 hours ago

The issue with scrapping is the intensity and volume of bots.

I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.

Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.

ccgreg•about 2 hours ago

A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.

sumedh•about 1 hour ago

Cloudflare can verify certain bots when they come from known ip addresses. So if your site is using cloudflare it can let CCBot if it has done the verification.

everfrustrated•about 3 hours ago

I wonder how much of this is traffic caused by peoples agents using web tools causing searches and fetches rather than general trawls of the internet.

corbet•about 3 hours ago

Very little of it. When you see a million IPs systematically working their way through your URL space, it's pretty clear that there's a central control node behind it all.

everfrustrated•about 2 hours ago

Your earlier article suggests you aren't using a CDN. Might be well worth looking into - not for any bot detection so much as just having a good old fashioned cache in front of you.

noxvilleza•about 1 hour ago

Most well-known/large agentic web tools I've seen are actually super honest about who they are -- even when they write out scripts they're very keen to identify themselves using user-agents. Most of the time those tools are fine - it's the ones that happen to have a random choice of the 5 most common Chrome/Firefox user-agents making sequential scrapes but cycling through IPs on African and South American residential IPs that are the problem!

dawnerd•about 2 hours ago

I've seen some logs where a bunch of random ips were hitting a client's search endpoint feeding what looked like user questions to it. Of course none of them returned anything useful but it was causing a lot of strain and even causing the site to go down (gotta love wordpress's stock search).

I'm guessing the training companies are taking real/synthesized user queries and trying to distill what they can from site searches.

Bratmon•about 3 hours ago

Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.

thomasahle•about 1 hour ago

TIL:

> Many providers build their proxy pools by partnering with device owners who agree to share their bandwidth, while others use embedded SDKs in free apps or VPNs.

WTF. That's just botnets.

Source: https://www.fbi.gov/investigate/cyber/alerts/2026/evading-re...

BoorishBears•about 2 hours ago

Thank god for residential proxies.

Highly unethical but the way the internet is going they're the last anti-hero of a somewhat open internet

zuzululu•about 1 hour ago

i know a few very large startups that used it to fake their way into an exit

unethical yes but really raises the question as to what we see is real or not

morkalork•about 1 hour ago

Money is real. DAU that don't pay subscriptions, or don't lead to paid conversions on hosted ads, are worthless.

dang•about 1 hour ago

One article mentioned in the OP was discussed here:

Disrupting the largest residential proxy network - https://news.ycombinator.com/item?id=46802748 - Jan 2026 (221 comments)

fragmede•about 1 hour ago

How does HN fare with scraper load? Is it just CDN and pay the extra bandwidth bill for anon hit requests?

arjie•about 1 hour ago

What a pity. Mostly I just want personal archives of things so that I can search them much faster than commercial solutions and the like.

tingletech•about 3 hours ago

The comments are not showing up for me now, but when they were still showing for anonymous users, there was a link to https://commoncrawl.org. I've been sort of worried about letting agents hit websites, I wonder if a fetch_url agent tool could be made to look in common crawl first before hitting the web for it?

colinsane•about 1 hour ago

just their smallest dataset looks to be 6 TB _compressed_. not a thing you can really ship as part of the agent. but if somebody made a fetch_url tool that sharded that across all users of it, i'd give it a try. could probably just layer that on top of bittorrent or IPFS or something.

andai•about 1 hour ago

>There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.

I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses in a few weeks.

Then they'd switch to IPv6, and... well, are we using IPv6 for anything important?

Like we need it for IoT, but do you want random IoT devices talking to your web server? (IPv4 handled mobile phones just fine not that long ago, right?)

cyanydeez•about 4 hours ago

mmm, in many cases these residential proxies are media boxes, and they consent as much as anyone else consents to what amazon, or google or facebook does; it's buried somewhere in the recesses of the TOS.

The question is more about why the US and others can't properly enforce the bullshit all this amounts to.

SR2Z•about 1 hour ago

Because this isn't clearly against the law, nor should it be. If websites want to ban based on IP address lots of innocent users get caught in the cross-fire.

I'm not sure what the solution would look like - maybe Cloudflare's payment required for requests beyond a certain limit? But I think that the world needs user freedoms now more than ever.

mschuster91•about 1 hour ago

> The question is more about why the US and others can't properly enforce the bullshit all this amounts to.

It would cost too much money, either for police to raid all the physical shops and ebay sellers selling dodgy IPTV boxes, or for ISPs to hire enough competent support staff to monitor and respond to abuse@ email addresses and follow through.

bell-cot•about 3 hours ago

"He who has the gold makes the rules" is older than the pyramids.

eduction•about 2 hours ago

Can BitTorrent’s architecture contribute anything useful here?

I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.

fragmede•about 1 hour ago

Yes but it's getting bit owners to use it is there problem. There's already the common crawl repository to start with but it isn't being used.

zb3•about 1 hour ago

> widespread scraping of web sites in search of training data for large language models and related projects

This is a good thing, thanks to this we have powerful open source LLMs.

> This activity overwhelms sites with traffic.

When LLMs get good enough, we won't need those sites anymore :)

[not satire, this is what I think, without self-censorship]

atomic128•about 3 hours ago

There is a large community of people that poison scrapers.

The poison gets better every day, and the community is continuously growing. Poison Fountain, alone, transmits hundreds of gigabytes of poison per day, which goes into scrapers, git repositories on every hosting platform, social media, etc.

Part of the poisoning community on Reddit, for example: https://www.reddit.com/r/PoisonFountain/comments/1uocaii/a_n...

dang•about 2 hours ago

I've banned this account because we don't allow single-purpose accounts on HN, and your account has been doing that for quite some time now.

We ban such accounts regardless of what the single purpose happens to be. Pre-existing agendas are not what HN is for and destroy the curious conversation that it is supposed to be for.

https://news.ycombinator.com/newsguidelines.html

dredmorbius•about 1 hour ago

Seriously, dang?

10 comments (excluding subsequent in-thread replies) over four months, always in contexts in which either the topic of LLM scraping or Poison Fountain itself has already been mentioned.

This strikes me as contextually informational, and is no different from other project representatives appearing in threads discussing their own subjects or posts. Such as, say, Jon Corbet (@corbet), of LWN, whose activity on HN shows a similar pattern and roughly equivalent frequency.

I hope it goes without saying I'm not suggesting corbet's handle be banned, anything but.

atomic128's comments are predictable, but apposite, informative, non-disruptive, and address an increasingly urgent issue. Whether or not it's an effective mitigation is of course another discussion, but it seems plausible at first blush.

As dang should well know but others may not, I often contact mods directly for HN issues, including numerous "one-note flute" alerts. atomic128's account should be un-banned, though perhaps they might communicate with HN's mods over what would be a more acceptable mode of interaction.

user-•about 1 hour ago

Just curious dang, did you warn them before banning?

Im not against the ban perse (single purpose accounts are bad), just curious if they had a chance to change their contribution style.

nekusar•about 2 hours ago

This is a strong positive sign that poison fountain works.

I wasn't aware of this project. Thanks for the heads up.

logancbrown•about 3 hours ago

People think this is causing issues for data collection for LLMs, but in reality it's not and there are several very trivial mechanisms to employ in data collection to bypass the "poison data" issue. The internet landscape was already poisoned with fake data, fringe conspiracies, and text before this Poison Fountain initiative.

zuzululu•about 1 hour ago

exactly i took a look at that subreddit and doesnt look like theres any professionals just bunch of anti-AI users who thinks they are smarter

its very easy to detect and bypass poison type of tools largely because of the fact that there are far more outlets for truthful info so unless you can get everyone to buy in (with real legal liabilities) its not effective

also its possible to poison the poisoners with a certain pill that would have very real consequences for those maintaining whatever github repo/communities

tiahura•about 2 hours ago

Again, why do we allow China on the Internet?

Backbone operators should not be allowed to knowingly maintain connections to networks that allow connections from China or Russia.