Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved

can you update the status page with this information?

Thanks for the assurance, jaas! Keep up the good work

Real soon now?

> This is a compliance incident

Uh. I don't know if I like the sound of that...

Stopping all issuance is an pretty standard response if a CA thinks what they are issuing might be non-compliant in any way. It's an action we're required to take. It's not necessarily a sign of a more dramatic failure mode or key compromise. That said, the impact is the same for as long as the downtime lasts so it is unfortunate and we're sorry for the disruption.

I don't think the premise behind short lived (six day) certificates being viable is that CA issuance never goes down. Sure, the runway is shorter, but not that short. Most down time is a few hours or less, which is not a problem for six day certificates that should be renewed every three days.

Short lived certificates are optional though, so if it's not worth it to you there are longer lifetime options.

Considering the open source nature of Letsencrypt, I wonder what the barriers/costs would be (theoretically) to a wealthy benefactor who wanted to duplicate its server side infrastructure and a core staffing level of persons, and fund a "parallel" equally trusted, alternative entity with a solid governing board. Same general idea how Acton funded the Signal foundation.

Somewhere that none of the physical infrastructure/hosting environment overlapped with existing Letsencrypt stuff so that the failure of one entity would have zero blast radius affecting the other.

I know there's a long and complicated process to go through to become a trusted root CA and get your CA public cert auto-installed in every OS and browser trust store. Indeed in the early days of letsencrypt I recall their root CA certs were signed by other older root CAs.

I just find it incredible that in 30+ years the industry hasn't adapted one bit to the brittle failure modes of certificates. I did some subcontract work with Verisign to deploy their CA infrastructure back in the early oughties and it felt like a solution was overdue way back then. I was at Google in the teensies when gmail broke due to expired SMTP certs. WAAAY overdue by then. Here we are, a decade later and it's still the same lol.

>pieces of critical Internet infrastructure that just quietly hums away in the background,

And donation supported no less

Wonder what incident that even could have been.

> like the recently announced 6 day certs

Just you wait for the 1 hour and 59 minutes certs! For security!

Letsencrypt is not the only acme authority. ZeroSSL is the other popular one. There are others.

If you're using ACME to handle certificate rotation, can't you just configure multiple providers?

Only if you’re reissuing right before expiration, which is a stupid thing to do. If you have a 47-day cert, best practice is to reissue on day 30, meaning LE would need to be down for more than two weeks before anything went wrong.

If this outage breaks your system, that’s entirely on you, not Let’s Encrypt.

"We have been made aware of a potential incident and are shutting down all issuance" seems to lean towards the latter and not simply a technical issue :(

Just speculating, but I don't think it's unrelated. Discord heavily utilizes Cloudflare, and Cloudflare uses Let's Encrypt for a certificate issuance. If they happened to have a certificate signing dependency in some operational rollout today, I think it could explain it. Certainly the timing is very correlated.

None, unless someone is renewing their certificates only 2 hours before they expire, which is a dumb thing to do.

It's an interesting thought experiment to consider how much of 'the internet' would still find a way to communicate with each other and fix the problem if somebody waved a magic wand and all http and https servers and clients magically disappeared worldwide instantly.

For instance some of the folks who run core BGP at medium to large sized ISPs would revert back to a few legacy IRC channels and find each other to chat and figure out WTF is going on.

"the internet" would still exist, a subset of the application layer stuff that runs on top it wouldn't...