Disruption with Some GitHub Services

Another useful disclaimer: all of these lets the developers push updates to your computer as they wish by default in most setups, these days you might want to decrease that kind of attack surface and just check the status in the official website when "git push" suddenly stop working.

I'm also not sure why you'd share links to software you don't even recommend yourself? Isn't it better to just not share that then? I think others could use search engine/LLM too if they need whatever, but most generally expect things shared in the comments here to actually at least have been looked at by the person sharing them.

You've a tab from 9 days ago open? Maybe you should be closing your tabs more often :p

Good lord, hundreds?!

Uptime reliability is nkt additive, it's multiplicative, sometimes even logarithmic.

Downtime of 99% and 99% is 98.1%. I hope almost all those services are non-prod related.

I used to work in a small independent team of 30 people within a large corp, half of which was dev. We used to run our own gitlab on-prem, our CI/CD was also on-prem. It worked perfectly, never had down time, devops guy could configure them on-demand to our needs. Me (and some other guys) also jumped in times to times to help (mostly just ssh into the servers for health check, disk partition, etc.). Then we grew (the biz team, dev was the same) and some new PMs with fancy Ivy League degrees came in and pushed for on-cloud Bitbucket. Things went to shit pretty fast after that ... Our codebase was only a few hundred thousands lines, there was only like hundreds of commits per day, the servers our git + CI/CD lived on never saturated ...

We should worry less about what everyone else uses, and more about what we use. I'm self-hosting Gitea and thinking of upgrading to Forgejo. What are you using?

My small org has definitely had internal discussions around self-hosting gitlab. We'll see what happens.

probably not. The people that have done it before or willing to do it now is probably a very % of the commit volume, they leaving wouldn't change much, probably not gonna even move the exponential growth needle.

At the time, most email was either local to the host (big mainframe in the basement) or transferred once a day through scheduled dial-up connections during off-peak phone hours

Thanks, this option is good to know.

We're currently on "GitHub Enterprise Cloud" on github.com and are affected by this outage (even though we use self-hosted runners!), but we're not on "GitHub Enterprise Cloud with data residency" on *.ghe.com, which I understand is/may not be affected by this outage?

Do you have any meaningful level of faith in GitHub's ability to deliver on stability? At this point, I have none.

“GitHub Enterprise Cloud with data residency” is hosted on separate infrastructure and dedicated subdomains under *.ghe.com. It’s been around since November 2024z

It’s not the same thing as GitHub Enterprise Cloud hosted on the shared global network on github.com.

https://docs.github.com/en/enterprise-cloud@latest/admin/dat...

That is a different and later product with a confusingly similar name.

Just to be clear, I am on Github Enterprise, and am also experiencing this disruption both privately and publicly on every org and project I have access to.

There is no such thing as "just split" there is 20+ years of legacy decisions and even if the split is relatively clean it is still probably 1 years work for 200 people for maybe a marginal improvement.

The real money is going to go towards, "make this all more reliable".

Depending on the cause of the current issues, that move would likely cause more harm to paid services than good.

Their last postmortem made clear that their challenges are operational. Scale puts pressure on operation, but it's not what blocks them from keeping up.

Doubling the operation doubles the operational challenges.

We are in GHEC right now and GitHub Actions is not working. It's been down every time githubstatus.com says it's down.

Certificate expired over 100 days ago

My point is: they blame it on scale. If that's the story they want us to believe, then they have to explain WHY such a simple fix isn't possible.

There may be non technical reasons for it, even: legal, or PR, but if they want me to believe the "scalability excuse" they need to convince me they're trying.

RDBMS replication and failover is way more difficult and manual than anyone would like. You can't just set up two postgres, tell them they're clustered and have it basically work; at a minimum you have to design the client to somehow know which one is currently the master, or use some sort of proxy (which becomes its own SPOF).

RDBMS integrity basically requires that one master server is responsible for the whole data set and other servers may replicate from it. And it usually doesn't wait for a quorum of replicas, just for one, because the design is to recover from a hardware failure, not a network partition, although that could be fixed at the cost of increased latency.

Possibly vitess from the latest update:

> primary failover briefly improved performance but did not fully mitigate, we've throttled inbound traffic and are investigating upstream Vitess issues