Incident with Github.com

On the contrary, I believe it's been detrimental to the open internet and only serves to control us by the powerful.

I never trusted Github and never will.

This is the way. Stop centralizing everything.

It's a social network with github-like social features and git hosting, but you can still self host your git and have all the features like issues, PRs and so on just like GitHub.

And it's built on ATProto so you can also have your issues PRs and so on hosted on your infra, but everything interconnects nicely with your single account, just like using GitHub.

They even added a vouching system for users, to know who to trust based on who you already trusted.

I suppose people can use git too if they ever figure out how to send emails from a dedicated client.

https://forgejo.org/2023-01-10-answering-forgejo-federation-...

This splintering should happen more often.

But maybe I’m wrong?

I guess they're quite happy with that TBF, it doesn't seem like they're that interested in being a git forge anyway.

https://imgur.com/b0Xj3Z6

Seems presumptuous to assume a root case while the incident is still ongoing, no?

I would gladly pay that fee if it means Github is reliable for paying customers.

If they make it harder to push slop on github, people might treat it as an incentive to generate less slop. Less tokens sold. Bad. Very bad.

The github paying customers are not going to move away.

Plus, it's probably genuinely hard to scale. And they can't throw compute at the problem - all compute is already going to the LLMs.

I still remember that they launched GitHub Actions in 2018 and back then I was thinking: They can't even make the website reliable, surely this will only cause them more problems.

https://news.ycombinator.com/item?id=48086786

We don't care. We don't have to. We're the phone company.

https://youtu.be/iBDUTaKWVdU

I sympathized with them when they said this a handful of months ago, but then I saw this [0] page that shows how it's been shot for years prior (which tracks with my memory).

I feel for the GH engineers that have to deal with this, especially the SREs. I also don't hate the downtime right now, as I'll make a cup of coffee and do something else. I will say though, I did have a hotfix a week or so ago during the Actions outage, which really was a pain.

You're right that getting angry and ranting isn't the right reaction here, but I do no give them the LLM load excuse. I don't give them an out for having awful uptime during work hours for a product we pay quite a bit for as an org.

[0] https://damrnelson.github.io/github-historical-uptime/

Which means they have a pricing problem. Rate limit non-paying users.

The idea that we should be fine with this unreliability is just amazing. It's not a mental health issue to have problems when important infra fails.

Which they encouraged by pushing Copilot down everyone's throat

Is this rage bait? Isn't Google, Amazon, and all the other services in the world similarly impacted by LLM's? Isn't Claude, OpenAI, etc.? Why can they handle the load but not Github?

If they are facing an influx of commits from bots maybe they need to start metering commits from those accounts and charge them money.

This is a self imposed problem.

Github and their parent company are active participants in pushing LLM-driven coding.

However, your "(you don't)" comment is not going to calm down all the people, who, you know, DO have a hotfix to push now, and DO have an angry customer that could not care less for which part of our infrastructure is breaking _their_ workflow.

The only things would calm everyone down is guarantee that Microsoft would be paying for _our_ SLA breach compensation. But they don't. And I don't think anyone is paying their GH bill with a prorata of the number of time the platform was actually available.

(I'm also aware that the wording of the contract probably clearly says that you should not use GitHub for anything critical, that Microsoft is only a small startup in their garage, that you can't credibly expect 90% uptime anyway, and that it's all the fault of LLM slop ! Bad LLM slop ! Also, please buy our LLMs to generate more slop, please.)

They could easily tighten things up in that regard, and make a choice that is right for their main users at the expense of The MS corpo mandate/mission. It is a choice to do otherwise.

I find it hard to believe that github APIs don't have rate limits to handle traffic spikes.

I’m not a Meta fan but it’s interesting that they manage to keep their systems up with an order of magnitude more traffic. GitHub’s uptime is inexcusable.

> Unless you have a emergency hotfix (you don’t)

Oh, we do. Given the sheer number of users, it's almost guaranteed someone is on fire ever time GitHub is down. Statistics is a very charming branch of reality.

This should not be an excuse since they probably can use AI to fix it /partially sarcasm

hey bud your bias is showing

I have little time in my day to spend with my daughter. But I also have responsibilities. And one client has decided to bet on github. It's the only client I've ever worked with who has their code on github. EVERY other client has hosted their own gitforge or used bitbucket.

So now I am not angry because some critical piece of US-american infrastructure is down all the time.

I am angry because instead of spending time with my daughter I have to work on this later, because there are due dates and "Well, fucking GitHub was down" ain't gonna cut it.

My mind immediately went to this classic xkcd: https://xkcd.com/303/

Did you have the conversation "do we need to move from $whatever CI/CD to GitHub Actions?" beforehand? What were the arguments in favour?

Frontier labs and global saas heavyweights run CI through us, plus a crop of open source tooling like Bazel, vLLM, Firecracker, Rails, Ray, Sorbet and Rust.

Shopify, Uber, Airbnb, Canva stresstested the platform ten years ago for scaling issues.

You need to take a little time to design upfront and take advantage of dynamically programmable primitives. It isn't the same ease/convenience as Actions. Bit more available though.

      steps:
          name: Do the thing
           run: nix build .#

tangled.org!

they've had multiple outages in the last week, so i guess they are proving themselves correct.

in practice, giving ssh/sudo to AI (codex) and telling it to setup ephemeral self-hosted runners on your own host VM (qemu), takes about 30 minutes and works extremely well. it bangs out all the shell scripts. moved my own business over to it. zero problems.

you might even find it is kind of neat too. you can customize your VM with software pre-installed. speeds up your builds quite a bit.

[0] https://www.blacksmith.sh/blog/blacksmith-seed-announcement

    /project/product/issues/<number>

    /in-product-messaging/copilot-budget-request-banner
    /in-product-messaging/code-scanning-ai-findings-preview-banner
    /github-copilot/chat
    /_private/browser/stats
    

Or is their point about abandoned projects where the dev never clicks release after merging, or even never merges?

I am not trying to make any excuses for GitHub, however I recall one tweet that made my eyes bug-out:

> Yup, platform activity is surging. There were 1 billion commits in 2025. Now, it's 275 million per week, on pace for 14 billion this year if growth remains linear (spoiler: it won't.)

> GitHub Actions has grown from 500M minutes/week in 2023 to 1B minutes/week in 2025, and now 2.1B minutes so far this week.

https://x.com/kdaigle/status/2040164759836778878

https://xcancel.com/kdaigle/status/2040164759836778878

---

edit: see a counterpoint I posted down-thread.

Imagine the software development lifecycle is a river blocked by several dams- necessary processes that also slow the water’s volume and speed. Writing the code was a very big dam upstream, and successive dams didn’t need to handle a lot of water. With that dam busted wide open, the torrent of code is inundating downstream processes. GitHub happens to host many of them.

Earlier this year the GitHub COO shared a post on Twitter with explosive numbers they were facing. I cannot seem to find it at the moment. (Edit: shared in other replies to this thread.)

Alternative theories include "azure sucks" and "they're vibecoding everything now".

Many people suspect it's the year-long migrating to Azure and all their products instead of the things they have been using.

Whatever the actual increase in productivity may be, the commits, clones, pushes, action runs, etc. are skyrocketing.

I started with the cloud version, then moved to self-hosted because it wasn't a big effort for our team. We also reduced our monthly invoice by about 50% after the move.

For people who want a similar experience today, I’d recommend trying Codeberg (An official instance of Forgejo) though. However, keep in mind they voted in July to ban mostly AI generated repositories. You can read about that here: https://blog.codeberg.org/protecting-our-floss-commons-from-...

We also have a free public Forgejo instance if you want to try it first.

* Forgejo

Those are my favorites. Both you can self-host for your company, including CI/CD.

"Yeah we're unable to offer services, but so is [X BIG CORP] and [Y BIG CORP]. It's an industry outage, nothing we can do!"

Otherwise there's no "we", Git is decentralized. Everyone can use whatever they want.

I suspect even poor uptime for many services is a cosmetic problem for most users.

https://azure.microsoft.com/en-us/updates?filters=%5B%22Azur...

That's not necessarily a bad thing for stability, but long-term I would not be surprised if it gets EOL-ed, or radically shrunk to just a basic Git mirroring service.