Understanding the recent DDoS attack against Read the Docs

ddavidfischer about 3 hours ago 17 commentsRead Article on about.readthedocs.com

FR version is available. Content is displayed in original English for accuracy.

⚡ Community Insights

Discussion Sentiment

71% Positive

Analyzed from 623 words in the discussion.

Discussion (17 Comments)Read Original on HackerNews

fn-mote•5 minutes ago

There’s an assumption that turning on Cloudflare’s “under attack” mode would mitigate the attack.

Given how adaptive the rest of the attack was, I would be very curious to find out how it would approach that obstacle.

gopher_space•6 minutes ago

My naive take on a Cloudflare perspective wants to combine "three times is enemy action" with toddler-speed block dropping and manual clearance. What's the money reason this problem isn't handled at the ISP level?

tescreal•about 2 hours ago

I'm curious if anybody could speculate who would be attacking a documentation silo, and to what end?

davidfischer•about 1 hour ago

I'm the author of the blog. I don't know. Internally, we were half joking that we were going to get ransom notice, but we never did.

The only thing that sort of correlates with this attack is that before it started, we began rolling out some slightly more aggressive rate limits one by one. This was mostly because anytime any new "company" thinks they're going to catchup with Claude/OpenAI, they scrape us very aggressively (and they're not respectful about it). My guess is that the attackers behind this attack were already probing us (they were) and they thought the window of opportunity might be closing.

tescreal•about 1 hour ago

Good to know. I use your site (with a manual transmission user-agent) often, and it's fantastic. Thanks for your work and the writeup!

davidfischer•about 1 hour ago

I've never seen the phrase "manual transmission user-agent". Using your own browser yourself is the new stick shift. Love it.

SoftTalker•about 1 hour ago

Just curious, if you're tolerant of scraping, do you make an archive of all your content available so that scraping is unnecessary, and if so do the scrapers prefer that?

davidfischer•about 1 hour ago

It's terabytes of content and other than we're the host not really related to each other. However, for most projects, it's possible to download a zip file of all the HTML docs for that project. We have a lower rate limit to pull these, but a scraper can pull thousands of docs at once. We only host a few hundred thousand projects so pulling a zip of the latest docs for all of them could be done in a day or two at a very reasonable rate.

It's also possible to request the docs already processed into markdown[1]. Lastly, basically all of the docs come from Git. A smart scraper could just clone a project's repo.

[1] https://docs.readthedocs.com/platform/stable/reference/markd...

SoftTalker•about 2 hours ago

Could be testing in preparation for attacking something more critical?

lanyard-textile•about 1 hour ago

Maybe just for the pleasure of doing it, too.

RobRivera•30 minutes ago

Could have been a live-fire exercise by a nation state.

kkapelon•about 1 hour ago

Either testing for something bigger OR demonstrating their power to a 3rd party with minimal real disruption

bennett_dev•about 1 hour ago

Interesting that the Under Attack Mode wasn’t used at all here. I understand not wanting to break APIs but I feel temporarily challenging non-API usage could have at least helped without impacting users too much?

davidfischer•about 1 hour ago

I talked about that directly in the post. We didn't want to just challenge everyone. We use JS challenges but we try to use them sparingly. The rest of the ops team and I were fighting to stay up but it never got so bad that it was a choice between complete outage and using the Under Attack mode.

Onavo•44 minutes ago

A more interesting question is, what exactly do the attackers gain from hitting read the docs? Most of their docs hosting is static/easily CDN cached. Unlike database bound sites, you would need a lot more traffic to overload pure/mostly static hosting. Maybe it's a malicious AI lab looking to deny their competitors training data? As far as infosec profiling goes, this is probably the oddest case I have heard of.

I am thinking it's probably an AI lab that misconfigured their data scraper (made it too agentic) and it ended up looking like a DDoS.

The new generation of scrapers are all agentic and self healing. (As an example see YC's https://parse.bot)

davidfischer•34 minutes ago

Author here. This was not a misconfigured data scraper. We see those every week[1]. This attack wasn't scraping useful content. It was almost entirely 404s and 302s and pulled virtually zero real docs. It specifically looked for URLs not served by the CDN and when it found a pattern, did millions of variations of it. Whether built by an AI or not, it was designed to cause outages and financial damage from autoscaling. However, as others have suggested, we may have been a test run for a real target.

[1] https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse...

braiamp•33 minutes ago

> Most of their docs hosting is static/easily CDN cached

The article says

> and it purposefully attacked areas that bypassed caching

So that doesn't work. Also, it seems that they were trying to cause financial harm, not to take down the infrastructure but to make it costly for the org itself. That's smart.