Back to News
Advertisement
Advertisement

⚑ Community Insights

Discussion Sentiment

27% Positive

Analyzed from 4350 words in the discussion.

Trending Topics

#dnssec#https#denic#dns#com#domains#down#domain#single#maxchernoff

Discussion (215 Comments)Read Original on HackerNews

krystofbeβ€’about 4 hours ago
Looks like a DNSSEC issue, not a nameserver outage. Validating resolvers SERVFAIL on every .de name with EDE:

RRSIG with malformed signature found for a0d5d1p51kijsevll74k523htmq406bk.de/nsec3 (keytag=33834) dig +cd amazon.de @8.8.8.8 works, dig amazon.de @a.nic.de works. Zone data is intact, DENIC just published an RRSIG over an NSEC3 record that doesn't validate against ZSK 33834. Every validating resolver therefore refuses to answer.

Intermittency fits anycast: some [a-n].nic.de instances still serve the previous (good) signatures, so retries occasionally land on a healthy auth. Per DENIC's FAQ the .de ZSK rotates every 5 weeks via pre-publish, so this smells like a botched rollover.

qazwsxedchacβ€’about 2 hours ago
So a single configuration mistake in a single place wiped out external reachability of a major economy. It happened in the evening local time and should be fixable, modulo cache TTLs, by morning. This will limit the blast radius somewhat.

Still, at this level, brittle infrastructure is a political risk. The internet's famous "routing around damage" isn't quite working here. Should make for an interesting post mortem.

belornβ€’about 1 hour ago
I am reminded of the warning that zonemaster gives about putting your domain name servers on a single AS, as is common practice for many larger providers. A lot of people do not want others to see this as a problem since a single AS is a convenient configuration for routing, but it has the downside of being a single point of failure.

Building redundant infrastructure that can withstand BGP and DNS configuration mistakes are not that simple but it can be done.

pocksuppetβ€’about 2 hours ago
DNS is a centralization risk, yes. Somehow we've decided this is fine. DNSSEC isn't the only issue - your TLD's nameservers could also be offline, or censored in your country.
skywhopperβ€’about 1 hour ago
DNS is barely centralized. Is there an alternative global name lookup system that is less centralized without even worse downsides?
greatgibβ€’about 1 hour ago
Normally it should not have been, with cache and all, but that was the past...

Think about what would happen the day that letsencrypt is borken for whatever reason technical or like having a retarded US leader and being located in the wrong country. Taken into account the push of letsencrypt with major web browsers to restrict certificate validities for short periods like only a few days...

cyberaxβ€’about 1 hour ago
Not really? .com and .net are still up

If Let's Encrypt goes down, half of the Internet will become inaccessible in a week.

Muromecβ€’about 1 hour ago
>So a single configuration mistake in a single place wiped out external reachability of a major economy.

And fuck nothing at all happened as a result.

Our_Benefactorsβ€’32 minutes ago
Prove it? I’m sure many lifespans were lost to stress
lschuellerβ€’about 2 hours ago
I have a bad feeling, that the impact will be quite severe for some services, as monitoring, performance, and security services might get disrupted. and just cleaning up is a big mess.. Worst case, some ot will experience outage and / or damage. But maybe I am just overestimating the severity of this.
walrus01β€’about 2 hours ago
It looks like a failed key replacement during a scheduled maintenance event. Normally this sort of thing is thoroughly tested and has multiple eyes on for detailed review and planning before changes get committed, but obviously something got missed.
the8472β€’about 2 hours ago
fail-closed protocols have introduced some brittleness. A HTTP 1.0 server from 1999 probably still can service visitors today. A HTTPS/TLS 1.0 server from the same year wouldn't.
dlopes7β€’about 2 hours ago
I love how I work with IT for 20 years and don't understand a single acronym here other than DNSSEC
icedchaiβ€’about 2 hours ago
I've been in IT 30+ years, been running DNS, web servers, etc. since at least 1994. I haven't bothered with DNSSEC due to perceived operational complexity. The penalty for a screw up, a total outage, just doesn't seem worth the security it provides.
walrus01β€’about 2 hours ago
To be fair, advanced real world knowledge of public/private key PKIs (x.509 or other), things like root CAs, are a fairly esoteric and very specialized field of study. There's people whose regular day jobs are nothing but doing stuff with PKI infrastructure and their depth of knowledge on many other non-PKI subjects is probably surface level only.
hannobβ€’about 2 hours ago
I know quite a bit about PKI and X.509, and I can tell you that much: the overlap with how DNSSEC works is limited.
hathawshβ€’about 2 hours ago
Is that actually true, though? Even though it's not really my job, I find myself debugging certificates and keys at least once a month, and that's after automating as much as possible with certbot and cloud certificates. PKI always seems to demand attention.
mschuster91β€’about 2 hours ago
It's not made easier by the fact that a lot of cryptography is either very old and arcane or it's one hell of a mess of code that doesn't make sense without reading standards.

I had the misfortune of having to dig deep into constructing ASN.1 payloads by hand [1] because that's the only thing Java speaks, and oh holy hell is this A MESS because OF COURSE there's two ways to encode a bunch of bytes (BIT STRING vs OCTET STRING) and encoding ed25519 keys uses BOTH [2].

And ed25519 is a mess in itself. The more-or-less standard implementation by orlp [3] is almost completely lacking any comments explaining what is going on where and reading the relevant RFCs alone doesn't help, it's probably only understandable by reading a 500 pages math paper.

It's almost as if cryptographers have zero interest in interested random people to join the field.

End of rant.

[1] https://github.com/msmuenchen/meshcore-packets-java/blob/mai...

[2] https://datatracker.ietf.org/doc/html/rfc8410#appendix-A

[3] https://github.com/orlp/ed25519/tree/master

bfleschβ€’about 2 hours ago
Don't worry, that's by design ;)
Aldipowerβ€’about 2 hours ago
Apparently the DENIC team was on a party this evening! Party hard, but not too hard. https://bsky.app/profile/denic.de/post/3ml4r2lvcjg2h
FinnKuhnβ€’about 2 hours ago
A real party killer if I have ever seen one.
SOLAR_FIELDSβ€’about 2 hours ago
At least all of the appropriate people were in a room together when the outage happened
SpaceNoodledβ€’about 2 hours ago
Sounds like poor risk pooling. If that room crashed, we'd have nobody to fix this.
walrus01β€’about 2 hours ago
Interesting "bus problem" to have in a scenario where everyone who is qualified, experienced and trusted enough to commit lives changes (or perform a revert, undo results of a botched maintenance, etc) in an emergency situation is not completely sober.
femtoβ€’about 1 hour ago
Sobriety is just factor to be weighed in an emergency situation. 30 years ago I was at a ski resort with about 50 friends having a drinking competition in the resort's main bar. Late that night two ski lodges collapsed, trapping people inside. Around midnight, soon after the winner was announced, the police entered and asked "who's able to drive a crane truck?" The winner of the competition put his hand up and informed them of how much he had had to drink. Don't care they said, so he drove a crane big enough to lift a building up a single lane 35km mountain road in nighttime ice conditions. (The crane made it, but sadly most of the people in the ski lodges didn't. https://en.wikipedia.org/wiki/1997_Thredbo_landslide )
jamesfinlaysonβ€’1 minute ago
Sounds like Australian police. I remember 15 or so years ago being in a big team assisting the Australian police with something on a remote farm. There were 20 people that needed to be taken back to base and one 10 seater car. Someone asked the police if everyone could get in the car and policeman shrugged and said you can try. So the policeman drove a four wheel drive across farmland with 16 people stuffed into the back.
Muromecβ€’about 1 hour ago
Sounds like Europe, yes.
SEJeffβ€’about 1 hour ago
Just gonna leave this absolute gem from Thomas Ptacek on DNSSEC here:

https://sockpuppet.org/blog/2015/01/15/against-dnssec/

betabyβ€’44 minutes ago
Aged like a milk.
tptacekβ€’37 minutes ago
Oh, yeah, I'm sure feeling chastened right now. You got me.
aboardRat4β€’7 minutes ago
tom1337β€’about 2 hours ago
I have never used DNSSEC and never really bothered implementing it, but do I understand it correctly that we took the decentralized platform DNS was and added a single-point-of-failure certificate layer on top of it which now breaks because the central organisation managing this certificate has an outage taking basically all domains with them?
gucci-on-fleekβ€’about 2 hours ago
> which now breaks because the central organisation managing this certificate has an outage

The ".de" TLD is inherently managed by a single organization, and things wouldn't be much better if its nameservers went down. Some of the records would be cached by downstream resolvers, but not all of them, and not for very long.

> we took the decentralized platform DNS was and added a single-point-of-failure certificate layer on top of it

DNSSEC actually makes DNS more decentralized: without DNSSEC, the only way to guarantee a trustworthy response is to directly ask the authoritative nameservers. But with DNSSEC, you can query third-party caching resolvers and still be able to trust the response because only a legitimate answer will have a valid signature.

Similarly, without DNSSEC, a domain owner needs to absolutely trust its authoritative nameservers, since they can trivially forge trusted results. But with DNSSEC, you don't need to trust your authoritative nameservers nearly as much [0], meaning that you can safely host some of them with third-parties.

[0]: https://news.ycombinator.com/item?id=47409728

tom1337β€’about 2 hours ago
> DNSSEC actually makes DNS more decentralized: without DNSSEC, the only way to guarantee a trustworthy response is to directly ask the authoritative nameservers. But with DNSSEC, you can query third-party caching resolvers and still be able to trust the response because only a legitimate answer will have a valid signature.

but how would one verify the signature if the DNSKEY expired and you cannot fetch a fresh one because the organisation providing those keys is down? As far as I understood the TTL for those keys is different and for DENIC it seems to be 1h [0]. So if they are down for more than an hour and all RRSIG caches expire, DNS zones which have a higher TTL than 1h but use DNSSEC would also be down?

[0] dig RRSIG de. @8.8.8.8

de. 3600 IN RRSIG DNSKEY 8 1 3600 20260519214514 20260505201514 26755 de. [...]

gucci-on-fleekβ€’about 1 hour ago
> but how would one verify the signature if the DNSKEY expired and you cannot fetch a fresh one because the organisation providing those keys is down?

In theory, this shouldn't happen, because if you use the same TTLs for your DNSSEC records and your "regular" records, then if the regular records are present in the cache, the DNSSEC records will be too.

> So if they are down for more than an hour and all RRSIG caches expire, DNS zones which have a higher TTL than 1h but use DNSSEC would also be down?

Yes, but I'd argue that the DNSSEC records should have the same TTLs for exactly this reason. That's how my domain is set up at least:

  $ dig +nocmd +nocomments +nostats +dnssec @any.ca-servers.ca. maxchernoff.ca. DS
  ;maxchernoff.ca.                        IN      DS
  maxchernoff.ca.         86400   IN      DS      62673 15 2 487B95FEFF04265826F037C9DB2E1F14FF9ADBF2C7BE246A2B9F9BFD 481BE928
  maxchernoff.ca.         86400   IN      RRSIG   DS 13 2 86400 20260512131336 20260505104433 46762 ca. ppc9LrWniPWdAI2Xq1g3FrYJGQVYayA5TtgFRkJfqOqNfe6zu/n0gwti IO3c9pOoUpIum5gPB6GLOGbGU+sfhg==
  
  $ dig +nocmd +nocomments +nostats +dnssec @ns.maxchernoff.ca. maxchernoff.ca. DNSKEY
  ;maxchernoff.ca.                        IN      DNSKEY
  maxchernoff.ca.         86400   IN      DNSKEY  257 3 15 DYs9mPDMRx/hQ9R9iGLi1Ysx1eFdhlXeCujY6PqJWeU=
  maxchernoff.ca.         86400   IN      RRSIG   DNSKEY 15 2 86400 20260518072823 20260504055823 62673 maxchernoff.ca. RgPyEvB/kjXIvoidRNF/hfm7utzDs0kxXn4qJL17TUAVYOdbLl0Vd8zt E52bGBBFv2TNEnf9O9LkiT2GBH0jAA==
  
  $ dig +nocmd +nocomments +nostats +dnssec @ns.maxchernoff.ca. maxchernoff.ca. A
  ;maxchernoff.ca.                        IN      A
  maxchernoff.ca.         86400   IN      A       152.53.36.213
  maxchernoff.ca.         86400   IN      RRSIG   A 15 2 86400 20260518072823 20260504055823 62673 maxchernoff.ca. bRfTVHnMjCFRaIh5uc0aT1vD4yh1UZrqOZDRunLbxFI1eth6nNlTiOOC xti7axVoXwB6VAoHOAnW0nL0eeJNDQ==
wahernβ€’about 1 hour ago
DNSSEC doesn't change the degree to which DNS is decentralized. It's always been hierarchical. In the absence of caching, every DNS query starts with a request to the root DNS servers. For foo.com or foo.de, you first need to query the root servers to determine the nameservers responsible for .com and .de. Then you contact the .com or .de servers to ask for the foo.com and foo.de nameservers. All DNSSEC does is add signatures to these responses, and adds public keys so you can authenticate responses the next level down.

A list of root nameserver IP addresses is included with every local recursive DNS resolver. The list changes, albeit slowly, over the years. With DNSSEC, this list also includes public keys of those root servers, which also rotate, slowly.

Medowarβ€’about 2 hours ago
What you see here is decentralisation working. The issue is with the operator of the de TLD, and as such only that TLD is affected. DNS is not decentralised in such a way, that multiple organisations run the infrastructure of a TLD, those are always run by a single entity.(.com and .net are operated by Verisign)

So what the issue is, that the operator has, does not change the impact.

AndroTuxβ€’about 2 hours ago
What if the root (.) certificate breaks?
pocksuppetβ€’about 2 hours ago
Resolvers are free to cache each TLD's keys. There's a finite, well-known list of TLDs and their keys - you can download all the root zone data from IANA: https://www.iana.org/domains/root/files (it's a few megabytes in uncompressed text form)

The world might be a little bit better with more decentralization of the root zone.

tom1337β€’about 1 hour ago
Cloudflare has now disabled DNSSEC validation on their 1.1.1.1 resolver: https://www.cloudflarestatus.com/incidents/vjrk8c8w37lz
tptacekβ€’about 1 hour ago
Welp. I think can call it on DNSSEC now.
amlutoβ€’about 1 hour ago
Hahaha. You wish :-p
tptacekβ€’38 minutes ago
It's a pretty hard argument to work around: WebPKI certificates should go in the DNS, and also the largest DNS providers might at any moment decide not to validate DNSSEC anymore to get through an outage.
cluckindanβ€’about 1 hour ago
If it turns out the DNSSEC issue was caused by threat actors, this downstream effect could very well have been the reason to do it.
amlutoβ€’about 1 hour ago
It is indeed a bit sad that Cloudflare had to turn off DNSSEC completely. But I completely understand that they don't have a production-ready, tested path to override DNSSEC validation for only some domains.
vendemiatβ€’15 minutes ago
Sorry! status message was not clear. DNSSEC validation is temporarily disabled only for .de domains.
siva7β€’about 3 hours ago
Crazy. I can't remember an incident like this ever happened before and it's still not fixed? .de is probably the most important unrestricted domain after .com from an economical perspective. Millions of businesses are "down".
rwmjβ€’about 3 hours ago
I remember when .com went down, in July 1997.

https://archive.nytimes.com/www.nytimes.com/library/cyber/we...

ctippettβ€’about 2 hours ago
> For instance, the name "www.nytimes.com" corresponds to nine different computers that answer requests for The New York Times on the Web, one of which is 199.181.172.242

  $ dig -x 199.181.172.242 +short
  www2.nytimes.com.
Neat.
AndroTuxβ€’about 2 hours ago
DENIC apparently resolved all .de domains to NXDOMAIN in 2010: https://www.theregister.com/2010/05/12/germany_top_level_dom...
lschuellerβ€’about 3 hours ago
It's Germany, pessimistic time estimation + 1/3 and you are in a realistic time frame for the issue being resolved.
warpspinβ€’about 3 hours ago
It's night. Somebody has to fill a form to approve night work first.
daneel_wβ€’about 1 hour ago
And then fax the form to the correct authority, so that the request is Official(tm).
carstenhagβ€’about 2 hours ago
I know that people are joking, but of course we also have (extra paid) on call shifts.
greyhoundβ€’about 2 hours ago
And send it by post for approval, which will take 5-30 business days.
snapetomβ€’about 2 hours ago
Luckily it's not Sunday. Everyone would be out in the country hiking.
Cockbrandβ€’about 2 hours ago
In addition: it's Germany, pessimistic cost estimation + 2000%, and you are in a realistic budget for the issue being resolved.
lschuellerβ€’about 2 hours ago
:D... before tax!
carstenhagβ€’about 2 hours ago
Well it was already very late in the day (21-22?) so the impact was not big I would say
HDBaseTβ€’about 2 hours ago
Germany isn't as big as you think.
trollbridgeβ€’about 2 hours ago
Yeah it's only the third largest economy in the world
Muromecβ€’38 minutes ago
I just checked and the can of Paulaner in the fridge is not affected by the outage so far, thus my trust into German economy remains unshaken.
pocksuppetβ€’about 3 hours ago
I must be early. There's not a single tptacek DNSSEC rant in this thread yet.
tptacekβ€’about 1 hour ago
What would I need to rant about? Sometimes the world does my ranting for me.
apaprockiβ€’about 1 hour ago
Maybe he drank a little too much MalΓΆrt with the DENIC team last night?
aberohamβ€’about 3 hours ago
He’s busy with MathAcademy earning XP-SEC
0123456789ABCDEβ€’about 2 hours ago
doesn't this event speak for itself though?
Avamanderβ€’about 2 hours ago
Kind-of. But there are worse things than outages when it's PKIs we're talking about. DNSSEC is also extremely opaque and unmonitored. Any compromise will not be noticed. Nor will anyone have any recourse against misbehaving roots.

Fun fact, CloudFlare has used the same KSK for zones it serves more than a decade now.

daneel_wβ€’about 1 hour ago
Which is fine. Not because KSK rollover is supposedly complicated, but if you can't manage to keep your private keys and PKI safe in the first place then key rotation is just a security circus trick. But if you do know how to keep them safe, then...
mike-cardwellβ€’about 3 hours ago
Perhaps he's moribund
chromeheartsβ€’about 3 hours ago
I was STRESSING tf out because I wasn't able to connect to my services & apps through my domains like at all .. they only work when using my phone data ? .. thank god it's not my fault this time
Locke80β€’about 3 hours ago
But we're Germans, and we need someone to blame.
lschuellerβ€’about 3 hours ago
Thank god for the german chain of blame: 1. The system 2. The neighbor 3. China
warpspinβ€’about 2 hours ago
You definitely forgot Merkel and Habeck.
AndroTuxβ€’about 3 hours ago
I'm blaming chromehearts anyways
sundiverβ€’about 4 hours ago
Yes, all .de domains down because of DNSSEC failure at Denic https://dnsviz.net/d/de/dnssec/
notpushkinβ€’about 3 hours ago

  {"data":{"error":"Imgur is temporarily over capacity. Please try again later."},"success":false,"status":403}
There is some strange irony to this, I suppose.
yjftsjthsd-hβ€’about 3 hours ago
In my experience, that error is a lie and is what you get if they've IP blocked you. (Easy to hit on a VPN, in particular)
itvisionβ€’about 3 hours ago
A protection against bad networks, including VPN.

It's been like that for over two years now.

ricardo81β€’about 3 hours ago
I get "content not viewable in your region", from the UK. Not an ideal image sharing website nowadays.
bfleschβ€’about 2 hours ago
We should frame it as "all .de domains are ready to be impersonated because everyone will disable DNSSEC".
sunaookamiβ€’about 3 hours ago
https://status.denic.de/ says "Partial Service Disruption" for DNS Nameservice now.

EDIT: it says "Service Disruption" now

port3000β€’about 2 hours ago
Even when every site in the world’s 3rd biggest economy goes down it’s still just a β€˜Partial’ service disruption :D
gruselhausβ€’about 2 hours ago
Whole Germany is offline. DENIC: "Partial Service Disruption". That's one way to phrase it.
MASNeoβ€’about 3 hours ago
At least they have some humor left.

Edit: Now even the humor is gone.

sunaookamiβ€’about 3 hours ago
Can only be topped when the status page is not reachable anymore :D

EDIT: called it...

lschuellerβ€’about 2 hours ago
Or only accessible through a german dns server
niklasrdeβ€’about 2 hours ago
It says "Server Not Found" now
cubefoxβ€’16 minutes ago
"All Systems Operational"
Zopieuxβ€’14 minutes ago
Yes, it's fixed.
Advertisement
kuerbelβ€’about 3 hours ago
I just spent the better half of an hour to debug unbound and the pihole because I thought it's a me problem...

Good news though, if you add domain-insecure: "de" to your unbound config everything works fine

Benderβ€’about 3 hours ago
I don't even enable DNSSEC in Unbound. There just isn't enough adoption yet for me to feel like I am missing out on something, yet.

"Cloudflare Radar data shows 8.11% of domains are signed with DNSSEC, but only 0.47% of queries are validated end-to-end." [1]

Zones I may care about:

- Amazon.com: unsigned

- My banks: unsigned

- Hacker News: unsigned

- Email that I do not host: unsigned

- My power companies billing: unsigned

- I found some! id.me and irs.gov are signed.

[1] - https://technologychecker.io/blog/dnssec-adoption

V__β€’about 2 hours ago
Just before the outage happened I updated multiple client servers. That was a very stressfull hour trying to figure out why nothing works.
victorbjorklundβ€’about 3 hours ago
Same haha
chromeheartsβ€’about 3 hours ago
SAMEEEEE !!!
__michaelgβ€’about 3 hours ago
Finally establishing the concept of Feiertag on the internet. Come back tomorrow.
throw1234567891β€’about 3 hours ago
Internetfreie Dienstage, 21st century variant of Autofreie Sonntage.
9753268996433β€’about 3 hours ago
Using this newfangled thingamabob on a silent holiday will result in the police kicking in your door the next morning.
1vuio0pswjnm7β€’about 3 hours ago
.de TLD is online. DNS working fine

DNSSEC not working

If using an open resolver, i.e., a shared DNS cache, e.g., third party DNS service such as Google, Cloudflare, etc., then it might fail, or it might not. It depends on the third party DNS provider

https://datatracker.ietf.org/meeting/118/materials/slides-11...

basilikumβ€’18 minutes ago
This is the kind of system failure that we need really good and well tested disaster recovery plans for. While not necessary this time, DENIC and any critical infrastructure provider should be able to rebuild their entire infrastructure from scratch in a tolerable amount of time (Rather days than hours in the case of a full rebuild). Importantly the disaster recovery plan has to work without reliance on either the system that is failing, but also on adjacent systems that might have hidden dependencies on the failing system.

I'm really not too close to Denic and know nothing about their internals, but just close enough to have experienced the frustration of someone working for DENIC second hand, but in person, during the disaster. From the very limited information I happened to gather DENIC had some trouble in addressing the issue because, surprise, infrastructure that they need to do so runs on de domains. [1]

I'm convinced there are all kinds of extended cyclic decencies between different centralization points in the net.

If some important backbone of the internet is down for an extended time, this will absolutely cause cascading failures. And thesw central points of failure are only getting worse. I love Let's Encrypt, but if something causes them to hard fail things will go really bad once certificates start to expire.

We need concrete plans to cold start extended parts of the internet. If things go really bad once and communication lines start to fail, we're in for a bad time.

Maybe governments have redundant, ultra resistant, low tech communication lines, war rooms and a list of important people in the industry who they can find and put in these war rooms so they can coordinate the rebuild of infrastructure. But I doubt it.

[^1] I don't know if there is some kind of disaster plan in the drawer at DENIC that would address this. I don't mean to allege anything against DENIC specifically, but broadly speaking about companies and infrastructure providers, I would not be surprised if there was absolutely no plan on what to do if things really go down and how to cold start cyclic dependencies or where they even are.

kaltsturmβ€’about 2 hours ago
Denic will be added to the "Major DNSSEC Outages and Validation Failures" list: https://ianix.com/pub/dnssec-outages.html
nfreisingβ€’about 2 hours ago
They can join the (rather long) list of TLD DNSSEC outages https://ianix.com/pub/dnssec-outages.html
iknowstuffβ€’about 3 hours ago
Kurzgesagt predicted this, Germany is OVER
irundebianβ€’about 3 hours ago
Danke Merkel
mschuster91β€’about 2 hours ago
Not sure if serious or /s
edb_123β€’about 2 hours ago
Things seem to be on their way up now, and https://status.denic.de/ is working again, at least from here.

DENIC's status page currently says "Frankfurt am Main, 5 May 2026 – DENIC eG is currently experiencing a disruption in its DNS service for .de domains. As a result, all DNSSEC-signed .de domains are currently affected in their reachability. The root cause of the disruption has not yet been fully identified. DENIC’s technical teams are working intensively on analysis and on restoring stable operations as quickly as possible.

Zopieuxβ€’about 1 hour ago
That postmortem should be a fun read, can't wait.
retiredβ€’26 minutes ago
We shall transmit the postmortem to you via fax within 25 business days, ja.
elevationβ€’about 3 hours ago
I've considered hard-coding some addresses into firmware as a fallback for a DNS outtage (which is more likely than not just misconfigured local DNS.) Events like this help justify this approach to the unconcerned.
whalesaladβ€’about 3 hours ago
The irony is that DNS is a global and distributed system meant to be resilient. It’s the DNSSEC layer on top in this case causing problems.
cedillaβ€’about 2 hours ago
denic is the single source of truth for zones under .de.

The only problem with DNSSEC here is that it's complex.

Advertisement
yassinizβ€’about 2 hours ago
Shops open normally from 8am to 8pm in Germany. Today we decided to pilot opening hours for .de domains as well
NooneAtAll3β€’10 minutes ago
quad9 seems to be having problems with DNSSEC as well
kangaliooβ€’about 4 hours ago
So glad I found someone mention this. Amazon.de, SPIEGEL.de is down. Highly prominent sites unreachable. I wonder how long this will last and how big of a thing this ends up being once people talk about it :o Feels big to me
moltarβ€’about 4 hours ago
Both examples open for me
irundebianβ€’about 3 hours ago
Some domains work, some not. I assume that working domains are cached.
balou23β€’about 3 hours ago
amazon.de, spiegel.de are down for me, too. heise.de works, but that might've been cached somewhere on my side.
ykβ€’about 3 hours ago
dig manages to dig out ips for heise.de and tagesschau.de but not spiegel.de amazon.de and google.de However, dig @8.8.8.8 has still amazon.de cached, unlike 1.1.1.1 so perhaps Google to the rescue?

[Edit] After playing around with it, google seems to have at least some pages cached. After setting dns to 8.8.8.8 amazon.de and spiegel.de work again, my blog does not.

theanonymousoneβ€’about 3 hours ago
idealo.de, ebay.de, and spiegel.de are down, but amazon.de opens for me.
dwedgeβ€’about 3 hours ago
On a slightly unrelated note, I was setting nameservers for two .de domains a few weeks ago and thought my provider was being crazily strict because they kept getting rejected. Turns out you can't point to a nameserver until that nameserver has a zone for the domain, and you can't use nameservers from two providers unless those two providers are both in the NS records at both ends
whalesaladβ€’about 3 hours ago
Common paint point with DNSSEC. It’s brutal in the domain industry because when you buy a name with DNSSEC enabled it oftentimes can’t be setup to resolve due to these sorts of issues. Typically seller needs to deactivate first.
kaltsturmβ€’about 3 hours ago
merbβ€’about 3 hours ago
Well at least it’s night time which means it’s hopefully resolved in the morning.

Looks like it failed after a maintenance: https://www.namecheap.com/status-updates/planned-denic-de-re...

https://status.denic.de/

gpvosβ€’about 2 hours ago
If so, it still worked for several hours after the maintenance was completed.
yowmamasitaβ€’about 2 hours ago
The same day Kurzgesagt posted their video β€œGermany is over”. Huh. https://youtu.be/n-gYFcVx-8Y
taf2β€’about 1 hour ago
ok i picked a bad day to move from one register to another... i just spent the last hour frantically trying to figure out why the new register screwed us or the old register was screwing us...
kaltsturmβ€’about 2 hours ago
Denic should work out a desaster recovery test - like: https://blog.apnic.net/2022/02/14/disaster-recovery-with-dns...
0x80hβ€’about 2 hours ago
Am I reading this correctly? All .de domains are down? Looking forward to reading the postmortem.
Advertisement
hmilch99β€’about 4 hours ago
https://pastebin.com/2mQUB8xX seems like someone's going to have a lot of fun tonight
g4cg54g54β€’about 2 hours ago
funfact: enabling DNS sec NOW will fix your domain instantly if dnssec was disabled before

-> no idea if that also "heals" anyone who had dnssec on before.

-> no idea if maybe they need to roll back something and then rebreak the new dnssec i made a minute later lol...

nuilβ€’about 4 hours ago
yosaminoβ€’about 3 hours ago
The last time .de I remember .de had a major outage like this was 2010. I would cite some sources but... you know. That was a fun afternoon, though.

I am very happy that it doesn't happen more often.

kaltsturmβ€’about 2 hours ago
from my analysis DENIC resigned the .de zone today (May 5, 2026, ~17:49 UTC). The DNSSEC signature (RRSIG) for the NSEC3 record covering the hash range of nearly all .de TLD is cryptographically broken (malformed).
warpspinβ€’about 4 hours ago
Whole .de TLD seems to go offline right now due to dnssec or missing nic.de nameservers?
fweimerβ€’about 4 hours ago
This works:

    $ unbound-host -t A www.denic.de
    www.denic.de has address 81.91.170.12
This does not:

    $ unbound-host -D -t A www.denic.de
    www.denic.de has address 81.91.170.12
    validation failure <www.denic.de. A IN>: signature crypto failed from 194.246.96.1 for DS denic.de. while building chain of trust
So it does seem DNSSEC-related.

EDIT My explanation was wrong, this is not how keytags work. The published keytag data is consistent:

    de. 3600 IN DNSKEY 256 3 8 AwEAAfRLmzuIXVf7x5A0+U7hke0dS+GEJG0EdPhnOthCCLhy0t0WqLyoXJOhnfsTJ8vQX5fd9qOJc9gyr3SWJZkXAhPm3yPSC7FWWHF70WZTKKM9CekmKdqwMwq6ZCjMSUcecCuSF4Sbt1MRszV7rFmfGVklA1l5UzNbqwD+Dr5vfcLn ;{id = 33834 (zsk), size = 1024b}
    de. 3600 IN DNSKEY 257 3 8 AwEAAbWUSd/QN9Ae543xzdiacY6qbjwtZ21QfmdgxRdm4Z7bjjHWy249uqxCyjjjoS4LDoRDKmj7ElffMKvTWKE1qFKu0p8TUy4wyhX0M+m5FUjvQ3CiZMi+qY7GSHA5B+Zd73cidmnTeb3e8lso6jEsXg05/VZ2AyAqWF6FexEIFxIqiwwLk4UP0BwZ17Ur3q1qx9VSbPMyHgQ9d6nHUN1EEJsTDA2v0vKumsUyp74ZanRZ/bB/6IzpaaZyr5BLF5pSCNdbRNjVmkwYD0993vm79LueyOeibsoHRc16jhALrIJou1PFjdq7YQsYN0KtqRiJtaAfPprDBREpeamPuW/MnW0= ;{id = 26755 (ksk), size = 2048b}
    de. 3600 IN DNSKEY 256 3 8 AwEAAbTe1PJi8EgIudNGb+KRTxBL2aCu5rXkZ+aIe/TC88pwRdrXYeXODp1ihZWFop5CrbWRBLrk/YUPBE8aBc6oJP+58dSkdMLYkjSkmvdvYx+zXnRLWlF2bapxvZxshATJDfGjGbCiWxKEOoyRx3UhICtHC+cUSddsEvzfacUcBb6n ;{id = 32911 (zsk), size = 1024b}
    de. 3600 IN RRSIG DNSKEY 8 1 3600 20260519030655 20260505013655 26755 de. ke56T5GZt/X6zMBAF+ouyCTnAd7RY7MsnDcfa9jyyOwSouRXhvzim/V13JDTMBAnpAHxWQXoruXrAZ6A6re5N+8Pp2utVkAEKTWs0r4UOLNKoZ2+zMwNplKjNNnY5PJIbHfa5myyziLiIsi//qDIgQEACFk+pZcHXrRdqRoXPCL3UtfaXjk3+duDQdlPnYsJys5UshjVpkALSMChW7J0anzr0sG+f9ytstBneymMwFYOUC3NqbejbLPZsXGPZBQKPAoVJuV5q3znopbcqrDFfjI7bmX3QPYNvOaiT1ElBfi2piJVpDzMaMAmm2jCmvrf5VeTOBccMroh8sBtDPsaEg== ;{id = 26755}
The signature on the SOA record still does not verify:

    de. 86400 IN SOA f.nic.de. dns-operations.denic.de. 1778014672 7200 7200 3600000 7200
    de. 86400 IN RRSIG SOA 8 1 86400 20260519205754 20260505192754 33834 de. aZoiAJ+PaHUDVSHNXfV/R26ZK3GpFB7ek2Z46VnZdmPEDaTww+a7PkiQ98W83xohUunXYSvQCMeGYfUre5UT76eBKThdxW2a6ImX9/x/oEzQ9x/69Y/NSeTckOv9m3HCLBOug01op1koiHOIAVEvonOmXEHHqo1P4sR/fNbcVg4= ;{id = 33834}
kaltsturmβ€’about 3 hours ago
not all: https://www.heise.de/ works
edb_123β€’about 2 hours ago
Doesn't work here, at least not anymore. Every single .de domain I have tried doesn't resolve.
warpspinβ€’about 3 hours ago
Probably just a high TTL.
0123456789ABCDEβ€’about 3 hours ago
can confirm, at least another 54k seconds from where i sit
jamietannaβ€’about 3 hours ago
Was wondering why a few of my sites aren't CSSing, as they use https://classless.de
kaltsturmβ€’about 3 hours ago
cache
victorbjorklundβ€’about 3 hours ago
I was just wondering what was up with our .de site.
Oarchβ€’about 2 hours ago
Germany has fallen.
Advertisement
kaltsturmβ€’about 3 hours ago
even their own status page is not reachable: https://status.denic.de/

As fallback they should use their X account: https://x.com/denic_de

dgellowβ€’about 3 hours ago
Seems to be up now?

May 5, 2026 23:28 CEST

May 5, 2026 21:28 UTC

INVESTIGATING

Frankfurt am Main, 5 May 2026 – DENIC eG is currently experiencing a disruption in its DNS service for .de domains. As a result, all DNSSEC-signed .de domains are currently affected in their reachability. The root cause of the disruption has not yet been fully identified. DENIC’s technical teams are working intensively on analysis and on restoring stable operations as quickly as possible. Based on current information, users and operators of .de domains may experience impairments in domain resolution. Further updates will be provided as soon as reliable findings on the cause and recovery are available. DENIC asks all affected parties for their understanding. For further enquiries, DENIC can be contacted via the usual channels.

elchβ€’about 3 hours ago
All .de domains are down for me.
kaltsturmβ€’about 3 hours ago
with firefox: KO with chrome: OK
sunaookamiβ€’about 2 hours ago
niklasrdeβ€’about 2 hours ago
lxgrβ€’about 3 hours ago
Wow, I thought I was somehow unaffected but my resolver must just have cached the sites I'd tried.
tarrudaβ€’about 3 hours ago
Mailbox.org (also from Germany) seems to be experiencing issues too.
jiveturkeyβ€’about 2 hours ago
It’s not DNS

There’s no way it’s DNS

It was DNSSEC

binghatchβ€’about 4 hours ago
Wow… it’s definitely not all .de TLDs, but a lot of prominent ones definitely.
phit_β€’about 4 hours ago
its gonna be all .de domains once caches dry out, anything that still works right now is bound to eventually fail until the underlying issue is resolved
fossddβ€’about 3 hours ago
Any .de domain with DNSSEC
mrngmβ€’about 3 hours ago
Unfortunately, even domains that did not have DNSSEC enabled earlier today are affected.

We observed issues on a non-DNSSEC .de domain at 19:45Z and confirmed around 20:12Z it wasn't just us, but also more high profile domain names.

meineerdeβ€’about 3 hours ago
Any .de domain is affected, regardless of the domain's dnssec deployment status, as long as you use a resolver which validates dnssec.
eliaskgβ€’about 3 hours ago
Amazon is completely down in Germany. Not only on amazon.de, even in the app.
kaltsturmβ€’about 2 hours ago
With chrome it works again
Animuxβ€’about 2 hours ago
Seems to be fixed now.
whalesaladβ€’about 3 hours ago
You can visually see this anomaly in many of CF Radar's charts: https://radar.cloudflare.com/dns/de?dateRange=1d
bfleschβ€’about 2 hours ago
On Monday there was a huge outage affecting several cities quite close to Frankfurt because someone cut major fiber line; today DENIC is having a party and right when everyone is drunk this happens because some post-rotation task cannot be completed.

There are too many coincidences happening.

dark-starβ€’about 3 hours ago
How come I have zero problems with any .de domain I tried accessing in the last half hour?
AndroTuxβ€’about 3 hours ago
maybe your upstream doesn't validate DNSSEC?
dark-starβ€’about 3 hours ago
maybe? I'm using PiHole and 8.8.8.8/1.1.1.1 as upstream, and both options show "DNSSEC" next to their options in settings, so I assumed DNSSEC was enabled (unless I have to enable this somewhere else as well?)
warpspinβ€’about 3 hours ago
That's weird cause 8.8.8.8/1.1.1.1 will already answer with SERVFAIL right now, unless the domain is still in the cache.
pw6hvβ€’about 3 hours ago
cache
Advertisement
jiggawattsβ€’about 3 hours ago
I work with a few people specialised in IT security, and some of them take their jobs too seriously and will "lock down" everything to the point that it becomes a very real risk that they lock out everyone including themselves.

Fundamentally, security is a solution to an availability problem: The desire of the users is for a system to remain available despite external attack.

Systems that become unavailable to everyone fail this requirement.

A door with its keyhole welded shut is not "secure", it's broken.

QuantumNomad_β€’about 3 hours ago
Security is not just a solution to availability. It is also to keep sensitive data (PII, or business secrets, or passwords, or cryptographic private keys, and so on) away from the hands of bad actors.

If I’m unable to use Amazon for 24 hours it doesn’t really matter. If a photo copy of my passport is leaked that’s worries and potential troubles for years.

senkoraβ€’about 3 hours ago
Security = Confidentiality + Integrity + Availability

or alternatively,

Security = (exclude unauth'd reads) + (exclude unauth'd writes) + (include auth'd reads and auth'd writes)

Gotta satisfy all parts in order to have security.

jiggawattsβ€’about 3 hours ago
If you squint at it, you can convert all three to just availability.

    Confidentiality = available to us, but nobody else.

    Integrity = available to us in a pristine condition.
It's a bit reductive, I'll admit, but it can be a useful exercise in the same way that everything in an economy can be reduce to units of either: "human time", "money" or "energy". Roughly speaking they're interchangeable.

E.g.: What's the benefit to you if your data is so confidential that you can't read it either? This is a real problem with some health information systems, where I can't access my own health records! Ditto with many government bureaucracies that keep my records safe and secure from me.

dnnddidiejβ€’about 2 hours ago
That squint loses too much nuance. I don't think of a site data leak as an availiability problem.

Bad UX and bugs are in general not always an availiability problem.

If it hard to get what you want due to bad design but the site is up, the site is still up.

sanbaidengβ€’about 2 hours ago
aiimageupscaler
siginatorβ€’about 3 hours ago
how is that possible?
aweiherβ€’about 2 hours ago
Solar Flares
dnnddidiejβ€’about 2 hours ago
Took more than cloud flares?
pogii123β€’about 3 hours ago
For me bmw.de works but www.bmw.de not
benny_sβ€’about 3 hours ago
bmw.de is down for me too
MikeNotThePopeβ€’about 3 hours ago
Both domains page load for me from Amsterdam. I wonder if there's communication disruption. Undersea cable severed?
dark-starβ€’about 3 hours ago
You mean the big undersea cable between the Netherlands and Germany? ;-)
pogii123β€’about 3 hours ago
$ nslookup bmw.de ~ Server: 8.8.8.8 Address: 8.8.8.8#53

Non-authoritative answer: Name: bmw.de Address: 160.46.226.165

$ nslookup www.bmw.de ~ ;; Got SERVFAIL reply from 8.8.8.8, trying next server Server: 8.8.4.4 Address: 8.8.4.4#53

* server can't find www.bmw.de: SERVFAIL

dark-starβ€’about 3 hours ago
both work for me from inside Germany
ameliusβ€’about 1 hour ago
Maybe related to this? Crazy idea, but nothing surprises me anymore.

https://edition.cnn.com/2026/05/01/politics/us-troop-withdra...