HI version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
57% Positive
Analyzed from 1579 words in the discussion.
Trending Topics
#anubis#scraping#scrapers#more#web#don#proof#cost#challenge#where
Discussion (53 Comments)Read Original on HackerNews
Specifically, bad actors were ignoring robots.txt and rotating IPs to make blocking difficult. Anubis serves to make new connections more expensive, so that people will reuse a connection/cookie, and then falls back to normal means of preventing bad actors.
(That mass scraping is the result of AI training companies, yes, but it’s the mass scraping that’s the problem, not the LLMs.)
How big of a deterrent is a millionth of a cent per page going to be? A non-existent one. Just the bandwidth from a residential proxy will cost the scraper orders of magnitude more.
Proof of work just isn't a viable counter-abuse challenge, even for something as low-yield as scraping. (It might be economically viable as a counter against some types of DDoS attacks, since they're even lower yield. But in practice it just moves the attack surface to the proof of work validation service.)
Anubis works to the extent it does because in counter-abuse security by obscurity tends to be the best security of them all. Right now Anubis isn't popular enough for many scrapers to try working around, so they don't. But they will pour resources into working around Turnstile and Recapcha. In this regime, the exact challenge is irrelevant. Anubis would be just as effective if the challenge was running a javascript function to add two numbers instead of a proof of work.
Proof of work is just a uniquely dismal basis for a counter-abuse challenge.
In practice, this would seem to be false, according to the admins of said sites.
I do this at scale and it took me very little time to set up and almost no resistance. So anyone who thinks that this is difficult or you're preventing people from doing this at scale, you're wrong.
You do the proof of work, you get the content. Not all web scrapers are willing to do the work, which reduces the strain put on web servers.
It's by no means a perfect system. It's goals in part prevent it from doing so. It tries to not be too annoying for humans, to not block real users, and not be privacy invasive.
The little anime girl is pretty off putting. I bounce when I see it.
It’s a good strategy; personal websites and blogs can display the anime girl without fear, and companies that care about their image end up paying. Win-win.
For instance: The act of searching the Arch Linux wiki produces a picture of the anime girl.
(Should I just not use Arch professionally?)
Wrong, anubis stops mass crawling of web pages, by requiring a proof of work, which makes accessing these websites more expensive.
Anubis was not built to stop individual users with llms.
If it becomes widespread (as it has been doing), they will. Anubis' strategy only works while it remains a niche approach only adopted by a small number of sites.
The Anubis cookies are bound to particular IP address. The scrapers are often using a large set of IP addresses, so they'll be paying a far higher cost than this suggests.
One goal is to reduce excessive scraping, usually for monetary or performance reasons. This is the goal you mention, and is motivated by a desire keeping the thing working at all.
Another goal is to stop bots from ingesting the content, carte blanche. This goal is motivated by a desire to dictate how bots (and by extension, people) may use the information that is otherwise freely-available on the web.
These are not the same goals.
> Anubis is a Web AI Firewall Utility that weighs the soul of your connection[1] using one or more challenges in order to protect upstream resources from scraper bots.
* https://anubis.techaro.lol/docs/
> The Weighing of the Heart would take place in Duat (the Underworld), in which the dead were judged by Anubis, using a feather, representing Ma'at, the goddess of truth and justice responsible for maintaining order in the universe. The heart was the seat of the life-spirit (ka). Hearts heavier than the feather of Ma'at were rejected and eaten by Ammit, the Devourer of Souls.
* https://en.wikipedia.org/wiki/Weighing_of_souls#Ancient_Egyp...
That's knowledge usually learnt in primary school.
In any case, it's not the type of lifetime common-knowledge which warrants your scornful response.
Perhaps where you reside, I’m unsure why you would believe it to be universal.
There can't be a truly open web if the guys with all the resources in the world have the incentive to absolutely crush you by draining all of your resources.
They won't crush you out of malice, but by accident, like an ant.
Not a fan of either solution, but the sites simply going offline does not promote an "open web" either.
I have set it up on a few sites being relentlessly hammered by clearly idiotic bot traffic, and it drops bot the traffic levels from insane to manageable. I don't even care if the traffic is AI or bots, if the bot has gone to the same effort as the author has the bot may even just access the site in a responsible manner and that's fine.