Incident with Issues and Webhooks – Resolved

In other words, the set of github core services has expanded because you don't use third party tooling for some of those services anymore.

> Either AI use surged massively in the last quarter

December 2025 is considered by many people to be a major step function in agentic coding (both due to improvements in harnesses and LLMs themselves). I know my coding has forever changed since then.

Before I was basically always hands on the keyboard while working with AI. Now I'm running experiments with multiple agents over the weekend, only periodically checking in if they have any questions or need further instruction.

The last quarter is where I personally first started to see how this was all going to change things (despite having worked on both the research and product side of AI for the last few years).

> I have a feeling they've tuned the models recently to commit more often, which gives the illusion of more work being done.

Agents certainly are committing more often, but I know, at least for these projects, there really is work being done. An example: I had an agent auto-researching a forecast I was working on. This is something I've done manually for over a decade now. The iteration process is tedious and time consuming, and would often take weeks of setting up and ultimately poorly documenting many, many experiments to see what works. Now I can "set it and forget it", and get the same results I would have in hours (with much more surface area covered and much better documentation). Each experiment is a branch (or work-tree) so yes there are a lot of commits happening, but the results are measurably real.

I often think the big divide related to the success with agents is whether or not the quality of ones work can be objectively measured. For those of us doing work that can be measured, the impact of agents is still hard to comprehend.

Many things at once I suspect:

1. Models have got way better, which means you are far more likely to get something working. I know I used to have little 'tool'/'weekend projects' all the time that wouldn't get off the starting blocks before, now it takes a few minutes often to build them, and once I've built them I tend to want to have them saved on github. Quite how useful they turn out to be is another question though...

2. Related, because the models are a lot better I can generate far more code per unit time. On Sonnet last year I'd have to babysit the model and constantly 'steer' it, which meant a lot of the CC time was actually me reviewing it. Now with Opus4.7 it can often just churn away for 10-30minutes and get something reasonable.

3. Most importantly, just the volume of new users to coding agents - loads of new developers shipping far more far frequently.

4. Many users who were not on github, now signing up and pushing code to it. "Vibe coders" basically who don't have SWE experience and their agent tells them git would be a good idea.

Each of these would be a big increase in scale, but combined it is vvv high

I don't think commits per se puts pressure on the infrastructure.

More likely pulls and pushes, and, naturally, the ci minutes they identify as the main issue.

And how many of those actions do uncached downloads instead of building self-contained offline images... Speaking of which, I wonder if GitHub has implemented any HTTP interception for common mirror sites, like used by apt, etc.

Or how many pushes those commits are spread across; oh, neat, big number.

That’s sort of orthogonal. But if GitHub actually invoked an LLM on initial page load, that would be about par for the course, and it would be amusing for GitHub to then complain that they’ve grown so quickly that their systems can’t keep up.

Working in an organization without a mono-repository I've actually found it extremely difficult to keep a tab on PRs and issues across multiple repositories. For a problem that should be resolved by a "For me" page that just lists out all your active incoming and outgoing PRs their multi-page solution involving search filters that often need to be reset feels extremely weak. I've worked on large multi-tenant solutions before and a page where you can "SELECT * FROM everything LIMIT 10" is the absolute last thing you want to give to users.

It is bizarre to me that so much of their tooling defaults to acting across the whole of github data points without guiding the user towards (or even making available as far as I can tell) a way to easily scope requests down outside of a complex search filter.

I can guarantee you one thing - github's problem isn't coming from git.

Considering all the ci/cd pipelines, PR & issue discussions, social media tracking, rich data and else that github hosts if their true issue is the actual meat and potatoes of running git I would be gobsmacked.

What are you referring to when you say it's "fundamentally computationally inefficient"? It's pretty efficient because it's content-addressed, plus optimizations to reduce storage and data transfer with packfiles.

> I think you need to broaden your focus here - I can't really remember any significant downtime before the Microsoft acquisition and the data supports my memories.

This is the opposite of my recollection, actually. I distinctly remember having conversations about Github struggling to scale well before MS was involved, and people claiming that MS had somehow saved Github because it had stabilized and begun adding features again.

> The query performance was fine before they started serving from Azure.

This may be correct though. The Azure migration seems more aligned with the timeline of struggling to scale.

I don't know why this is downvoted. The data backs you up: https://damrnelson.github.io/github-historical-uptime/

I mean, are any of the other forges, which I presume are also seeing logarithmic increase in commits, also failing as hard as Github?

What if I told you most enterprise customers don't even use the cloud offering and aren't impacted by any of this? Companies like Apple use GHES, and honestly thats where most of their revenue comes from, not the free offering.

For all of it's history (up to and including now possibly?) Github was a big Ruby on Rails monolith. [0] Obviously some things run in their own service, but I'm seeing the core github features fall apart which should be the features packed into the big monolith. If load is this much a problem, not being able to only vertically scale the processes that need the extra headroom is a big problem. Scaling horizontally by just throwing more machines at it, or at least cordoning-off some machines as "the ones that people actually pay for" is all I can think of for an application I can only describe as "accidentally working". Urgency is most-definitely high and that pushes decision making towards permanently-temporary patches instead of actual infra/architecture improvements.

[0] https://github.blog/engineering/architecture-optimization/bu...

The GitHub SREs are working for the Copilot company.

I suppose the idiocy of their parent company is their job security.

Your claim is incorrect. Something purely AI generated may not be covered by copyright in the US. That would make it more valuable to MS as you can reuse it as you like.

However, works with significant human input are covered by copyright, and most code does have such input. Human review, and correction is very common. There is a lot of AI generated code out there, and there are no cases challenging the copyright on it.

You also need to look beyond US law. Software is a global business and most software businesses do not want to write software they can only sell in certain countries.

I think this is minimizing the Xbox platform. They are also a massive digital distribution platform where almost every game is a digital download now.

That being said, you are correct. It is absolutely no surprise to me that Actions has the worst uptime.

We aim for 9 5s

I don't think it's an inferiority complex, negativity sells more and carefully understanding things doesn't sell as much

The thing about an SLA is that once you’ve broken it you’ve lost the trust. It doesn’t _really_ matter what the cost is for breaking it, nobody chooses their platform based on the refund they’ll get if they’re down. But they absolutely do choose based on reliability and uptime. The enterprise SLA refund credit will show as a (big) metering blip, but the problem is the people who signed the contracts are going to be speaking to Gitlab now

Make it nerdy enough to scare of agentic coders only. Also, blackjack and hookers are said to be helpful in such circumstances.

Forgejo is a thing. But the headlines lately make it sound like it’s not in great shape either.

What do you need users for?

GH is not a social network

> We should make an alternative git site, but how to acquire users?

Buy ad space on Github's outage page?

codeberg is doing fine

How do you throttle bots? Everyone will stop having commit msgs mentioning LLM agents. then what?

Some people really abuse the f out of the system in a way optimized to take github down. Like they push every minute or for every commit instead of with certain time intervals (e.g. a single push a few times a day for each repo).

I follow some of the accounts that run 24/7 agent sesssion. Their projects are not even that novel for the number of commits that appear on the profile. Many of the commits have the log of beads, claude session etc (no change to the actual code). Some of them are ports of some projects to another language. AI surely will increase the productivity, but the waste and noise that some people are willing to commit ....

It's a nice thought but I think the revealed preference from the history of the internet is that people actually only want centralised services, no matter what they say they want.

People love to clown on the fediverse because of having to choose a server. Which is no different from email. I guess the difference is that their ISP used to give them email.

Not really. It is possible to implement systems that handle a lot more scale than github has. This is proven by systems that exist today.

It might be hard to create such systems using ruby and microslop AI management though

Three eights is more than a month of downtime every year. Today is the three eights.

I'm curious, do people around you use AI? Because in my own workplace, people use lots of AI, and they ship lots of PRs, which correspond to actual features on the roadmap. I've been doing this a long time, and there is a whole lotta stuff shipping. I'm a manager and in the handful of hours I have I'm shipping the equivalent of what I would have as a full-time eng years ago.

> Is everyone just creating giant piles of throwaway slop?

yes, like the crypto or productivity 'industry', where every project exists to manage your other crypto and productivity projects, the dynamic of 'AI programming' seems to be to make software to manage your other two dozen AI things

mysteriously enough, big open source projects that people actually use in the real world aren't seeing their issues closed at any higher rates, almost as if that requires actual maintenance and programming (which Fred Brooks told us this half a century ago)

I’ve seen this behaviour but IMO it’s a fatal design flaw of Actions. It shouldn’t be possible.

And where does one store and version this harness?

:) :) Sure buddy - mind showing me yours ?