GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

Exactly. SQL injection was caused by treating user input as part of the instruction instead of as the pure data that it was intended as. Separating those two fixed it. Prompt injection is unavoidable because the user input is intended as instruction.

They’re the same type of problem as sql injection but there’s not the same ease of solution. There’s also a lot more subtle problems that can come in, but it’s still a decent comparison to help explain things.

Selecting from a menu is one way, but you can be much more broad about what acts can be taken. Give it an email tool and it can spam customers, give it an email tool locked to only being able to reply and you restrict what can go wrong. Limit exfiltration with restrictions similar to xss kinds of vulnerabilities (rendering images can leak data, etc).

"We can't fix it, therefore we just keep using it."

"How is this a Github vulnerability? The researchers are the ones that grant the agent access to private repos and then ask it to answer questions in public repos.. of course this allows extracting private information?"

I think the assumption is that the permissions are scoped to the repository you're currently asking questions on, rather than your private repositories as well.

I can see arguments for both sides.

Agreed. It seems a core issue underlying these prompt injection attacks is a failure to properly scope the agent's permissions. In this case, depending on what exactly the agent is supposed to actually do, this might be defining a separate workflow agent per repo, or a workflow agent with broader repo access but configured to only be triggered by users on an allow list (still compatible with developing in the open, still allows outsiders to open public issues, but takes into account the different trust to be placed in each). And likely many more options when one properly thinks about it.

But that requires:

1. the technical ability for such fine-grained scoping / permissions

2. actually taking the time to think about what you want to achieve with the agent and what the smallest set of permissions / capabilities is for it to achieve it

Regarding 1., I think this will come, we're still in the wild west phase of agent usage. It'll be interesting to see which abstraction(s) will turn out to be the best interface for humans designing agents (minimize friction for finding and defining scope and permissions) and to limit agent capabilities (again finding the best trade-off between level of detail possible for defining capabilities and the ease of use of actually doing it).

Regarding 2., well, that's still the core problem that's always prevented the construction of high quality software, isn't it? Taking the time to properly think it out,and then taking the time to properly implement it. Which goes counter to the "move fast and break things" approach of people throwing agents at everything.

> If you configure GitHub to allow public code or LLM instructions to run in contexts that have access to sensitive things, they will leak; that’s not GitHub’s fault, it’s yours.

Is there a way to segment access per agentic workflow, so that you can have both habe an agentic workflow that has access to sensitive data and one that has only access to public data? Is the default to set the scope to only the current repository? Does Github appropriately inform about the risk of combining an agentic workflow with access to private repository data?

If the answer to any of those questions is "no", then that's a problem.

(Classic GH Workflows are also riddled with priveledge escalation via PR-triggered workflows, but that's another topic.)

This isn’t a normal software bug, it’s not fixable in the same way you can’t fix regular support staff from being tricked.

The answer is you should not allow LLMs access to untrusted input and sensitive data at the same time.

Fix what? They setup LLM with access to private data and ability to read public comments. That's simply misconfiguration.

I'm done. Moving to Forgejo. It's wonderful and everything works better.

Seriously like everything is instant when you click around, and CI with a runner works beautifully. (The documentation for setting up the runner could be a tad clearer but otherwise everything was so painless.)

Microsoft is a publicly traded company. Which investors are causing them to shit up GitHub with AI features nobody wants? In which venues?

Agreed but I think enterprise AI offerings are pretty impressive, investors and consumers aren’t really aware, employees aren’t able to trade

The revenue is there and also impressive, and supplanting consumer and seat based revenue

The market is still shedding SaaS multiples, which I think is accurate, but break out the revenue in those quarterly reports and there is a huge growth story, from real efficiencies

I suspect you are greatly overestimating the average organization's ability to run a Git server themselves and keep it secure, while also overestimating how evil GitHub and LLM's providers are.

You know how it works. There probably were people who didn't want that, but then there is push from business, deadlines, etc.

It must be something to do with Microsoft being the owner now of GitHub

My Lethal Trifecta talk at the Bay Area AI Security Meetup - https://news.ycombinator.com/item?id=44846922 - August 2025 (115 comments)

https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/

Yea agreed. LLM guardrails are either just written prompts as in "Please do not bad stuff :(" or other LLMs verifying that the first LLM didn't so some bs. Both of wich methods do not work sufficiently as time shows again and again.

Funnily enough, nobody expects quality software anymore and errors became tolerable. So thats a win (for someone like me that lost all passion for the industry).