RU version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
46% Positive
Analyzed from 7211 words in the discussion.
Trending Topics
#github#more#prs#don#spam#issue#problem#solution#contributors#slop
Discussion (236 Comments)Read Original on HackerNews
> When requiring approvals only for first-time contributors (the first two settings), a user that has had any commit or pull request merged into the repository will not require approval. A malicious user could meet this requirement by getting a simple typo or other innocuous change accepted by a maintainer, either as part of a pull request they have authored or as part of another user's pull request.
If companies can screw you over and claim it's a mistake, there isn't much a person can do.
It's all about level's of trust, a maintainer going rogue is less likely, a past contributor going rogue more likely but not too much, a stranger with a typo pr merged even more likely but still, a complete stranger least trust worthy.
If you are insecure because someone has had one of their otherwise completely innocent PRs merged into your repo... you are insecure, period.
A similar system would be nice for issues, though I'm not sure what it'd look like if issues are the springboard for contributing PRs.
Not likely to ever happen (as others said), GitHub/MS want to sell CoPilot subscriptions/tokens and LLM-generated PRs are a part of that business model.
/s
The issue here is the core model is broken (misaligned incentives). That's not something you are going to fix with a github "downstream". A token system could help but it's easy to imagine ways that could be gamed, if not implemented well.
the rate of comits/PRs total
The rate of PRs to repos they don’t own
The reject rate of PRs
The number of ban
An estimated “AI” or bot score or status flag
There are a few better attempts at GitHub metrics calculators but I have not seen any that move beyond the paradigm of more vomits is default assumed good. It’s time to foreground quality not just quantity. The GitHub “4 kpis” are entirely action oriented.
This sentence also illustrates the absurdity of this investment model. It imposes a trade-off between building good software, and complying with the investor's metrics. They probably call such metrics evidence-based, but this example shows that they arbitrarily capture some numbers to obscure the lack of meaningful measurements.
1- https://anubis.techaro.lol
You can’t submit a PR because your laptop is too slow? Rent some hash rate from someone, and now you’ve just made a system of paying botnet owners to be able to make a typo fix on a github repo. HashCash was never used in the real world for a reason, it sounds cute but the incentives are so insane as to only work in a vacuum where you assume everyone isn’t cheating.
Sure, but looking at the cost to do it at scale is the wrong metric. I surely can't compete with a career spammer on emails-per-second or even emails-per-dollar, but I also don't need to.
It's more about the expected-value versus the cost. For example, my expected benefit from one email to my family is (while hard to quantify) hopefully much higher than a spammer's expected benefit of one spam email going out, which has a very small chance of leading to any amount of money. Attaching a CPU-churn cost per email is something I can ignore on my desktop, but they have to at least budget for it.
I'd also like to note that the win-condition isn't as extreme as making spam (or other "crimes") truly unprofitable, it just needs to be less profitable than other things the time/resources could be used for.
We really need to solve SPAM itself here, I think there may be a way to do it. I.e., the problem of spam is NtoN scaling connections. The network has never been able to solve that problem (exponential is the hardest). Limiting communication in terms of mesh networking may be the ultimate solution - bots can't get to you because they can't reach you.
What needs to be invented is a bridging protocol - some way to establish "legitimate" lines of communication over a network, while preserving (to some degree) privacy and decentralization. AI can only enter this network by being explicitly added to the channel, and thereby explicitly and easily blocked (and also solving the general SPAM issue once and for all).
Negative score would be reports from other users because of spammy content or not acknowledged issues, with a middle ground of neutral score (+-0) or little positive score to issues or whatever with clear good intention, but couldn't reach a proper merged PR or were not issues (e.g. issue existed but wasn't the correct repo to be addressed, PR was good but needed other stuff to be implemented prior to it, maybe in the long run, etc)
Given any manipulatable scheme, AI will figure out how to manipulate it. For the OP, what happens if a single AI manages to get through to contributor? Then it starts elevating other AIs to contributor, and we're off again. There doesn't have to be a purpose to this. Trolls will troll, and trolls armed with AI bots can devote endless energy to doing so. The more you work to keep them out, the more fun it becomes for them.
I wish I had an answer for that problem. But I don't.
You could probably use some kind of pairwise ranking algorithm (like anything based on the Bradley-Terry model) to rate human vs. AI contributions, but that would take a lot of manual effort. Google is using it to (supposedly) improve their searching algorithms. They give testers two different versions and ask them what's better.
The totality of someone's currency is their reputation.
Of course, now the decision becomes...who is the central currency issuer that creates it?
Then they'll get removed by the humans? Its about cutting down work, not about eliminating the work entirely
The current approach removes about 99% of their overhead it would seem. If they have to do a few manual interventions here and there, that seems like a huge win overall
https://en.wikipedia.org/wiki/Elo_rating_system
https://en.wikipedia.org/wiki/Elo_rating_system
Frontier users: 527,865 Light indexed: 527,865 Ready to queue: 9,083 Fast scores ready: 0 Activity events 24h: 30,266 Fast scores completed 24h: 19,123 Deep jobs completed 24h: 3,043 Fast-score ETA: n/a Deep-hydrate ETA: 69h Stale running jobs: 0 GitHub backpressure jobs: 19,113 High automation signals: 4,608 Medium automation signals: 1,327 Completed jobs: 74,714
Biggest challenge is Github's rate limits. At this pace it will take two more months to have 98% coverage. But after that the maintenance should be quite straight forward.
The Elo rating system doesn't make sense in this context; it's designed around collecting zero sum game results for a given community of players and building a model around it.
There are a lot of political tricks that get used.
What is scary is that one of those kinds of users are malicious state actors. Like North Korea and Russia...
We made "Github contributions" a metric for people applying for dev jobs. So, of course, because devs are the kind of people we are, they started working out how to game that metric.
Some folks decided to start paying bounties on bug fixes, features, etc. Those bounties are fairly trivial by western standards, but are significant for developing countries. This creates a new career for developers; racing to collect the bounties on offer.
LLMs have exacerbated these problems by allowing existing people doing this to do it faster, and also allowing more people to pretend to be software developers and get in on the action.
If we stopped allowing LLM-authored contributions we'd still have too many shitty PRs. It would just be back to pre-LLM levels of "too many".
The answer is to make Github contributions valueless. Stop paying bounties, and stop using them to assess candidates.
And it is not like AI spam would be limited or even primary targetted at bounties.
This [0] is an example, there are many more.
The whole idea that we have to have a "portfolio" of work.
[0] https://talentslab.io/7-strategies-for-a-junior-developer-to...
Cowboy coders got a virtual cowgirl coder and sold it to everyone, hmm, maybe... (respected or not, solo devs don't always have the requisite skills to not be a cowboy, either due to lack of experience or lack of innate skill)
I don't know that I completely buy this narrative, though. There has been a strong, top-down push for this since the "beginning".
Your suggestion would help a bit but I would prefer the opposite: before someone can 'pollute' my pull request space and draw attention from subscribers I would prefer an acceptance step (just like a moderator on a forum) instead of having to archive the PRs.
This is especially important as (AI) spam increases and just because I am away for a few days or weeks I don't want those PRs lurking around.
My usual experience is this:
1. We open an issue that needs to be fixed 2. slop bots create multiple slop PRs 3. slop bots spam comments on the issues, pointing to their slop PRs
The only general methods for preventing this are are restricting PR's (not comments, I believe) to contributors - which is a hassle to maintain, and restricting to older accounts - which doesn't work because the bot accounts are not newly created.
Then we need to perform _way too many_ just to get rid of the slop: - navigate multiple pages and confirmations to ban the account from our org - open each PR manually - close it manually
This takes at least 15 clicks and is made _so much worse_ by how slooooooooow the UI is. Every click takes 2 seconds!!! How can "ban this account and delete everything it ever did" be more than a max of 2 clicks?
What we really need is a "locked down mode" where every interaction (PR, issue, comment) with the repo that isn't from maintainers or specifically whitelisted people goes into a moderation queue. Maintainers can confirm or deny the action using a single click (which does not take 2 fucking seconds to load).
- add "Pull Request requests" that operate like Friend requests. You can't open PRs until you've been whitelisted (temporarily or not) or are proven to be a good OSS citizen (TBD)
- add a "Burn it with fire" action in new PRs that deletes all comments and PRs opened by the user across the repo, as well as blocking the user.
Organizations already sort of have this, but the action does not delete/close PRs.
This is particularly annoying because PRs also show up in the issue and in the issue list as "this issue has 3 PRs that will close it", when it's all. just. spam.
https://github.com/LibreTranslate/LibreTranslate/blob/main/A...
The writing style in their onboarding doc has common AI tells (in the quote: em dashes, “it’s not A, it’s B” sentence).
I can understand that, perhaps they want to fight fire with fire or don’t have time as they already say. Still, it all feels like inadequate half measures to me.
At least bringing up the underlying method (restrict to contributors) has spawned the discussion about how that's probably a bad idea on the security side.
I am no expert in this, it's just something I noticed.
Yes
Also devs: stop giving us real world problems to solve
Try talking more about the meta of coding itself. Get into the developers head by _talking_ to them and understanding how they would approach and attack different problems. You can show them code and ask them what they would do differently / how they would go about implementing X-Y-Z. Just because you can write foobar doesn't mean you understand how to apply algorithms or w/e specific problems [your] team has. It's _far_ better to understand how they would solve a problem over their syntax anyway.
why not use hooks to automatically reject issue comments / PRs etc. from users that didnt go through onboarding, rather than repurposing GH features that aren't really designed for that use (and are hence in danger of being changed someday)?
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
https://github.com/cncf/foundation/blob/659fd32c86dc/dco-gui...
I bet they claimed to be protecting trans people to get that policy changed too.
The xz supply chain attacker hid their real identity, created fakes one and gained recognition over time in order to gain more access and add the backdoor. So TLAs and other bad actors at least are interested in gaining recognition.
https://news.ycombinator.com/item?id=47701394
Imagine you want to get a doctor's opinion, or maybe a couple of opinions. But a zillion AI-amateurs have registered themselves as doctors. How do you separate wheat from the chaff?
Right, but that's not what happened though.
Someone went to the public square, said "Hey, I'm looking for any sort of doctor, and I'll pay you $900 if you tell me your plan and then whatever plan I chose wins" and then they get surprised they get flooded by zillion AI-amateurs.
You don't generate a ton of chaff then try to find the wheat, you ensure your process doesn't generate a ton of chaff in the first place. Offering large monetary rewards for relatively simple work for anyone in the public is bound to generate a ton of chaff...
When the article mentioned email matching, I was concerned that it would break down when a contributor's email address changes. (I have contributed to more than a few projects over the years, using email addresses that no longer exist.)
However, it looks like they're not using the email address recorded in the author's original git commit, but instead a GitHub-generated address whose unique parts are the GitHub user ID and username. That should survive authors changing their email addresses. It would still break down if a contributor loses access to their account and has to create a new one, but that's probably less common.
https://blog.tangled.org/vouching/
Your solution would be great if GitHub would also allow me to whitelist specific users, but unfortunately this still won't block "implementation plans" in comments.
How does the website trigger the CI script? Through GH rest API?
Soon there will be no more AI doomer comments. The bots will take over that job too.
---
I'm working for an open source company, and my God, are 95% of contributions useless.
There are really dumb ones where the bot writes 10 paragraphs about how he implemented the feature, but the entire changeset is adding one line to .gitignore or adding a CLAUDE.md file.
There are even worse ones where the bot submits 3000 lines of code that seemingly works, but you have to spend an hour to figure out why it doesn't work.
The dumb ones are so much better.
imo AI bots have significantly affected OSS and we need better qualitative measures to define success
Seriously. Just ask for a US$10 deposit for the each PR. If the PR is accepted (not even merged, just accepted as "this is a good effort"), give it back. Hell, give double the amount for good effort and you got yourself a cheap way to attract good contributors.
Best case, bots will balk at the payment. Worst case, the funds can be used to hire someone specifically for triage.
Seriously, chill, then think about how you'd implement it. Then think how it'd go wrong. Then think about how to fix those problems. Repeat until you realize there's a better solution or until you solve the problem without making it overly convoluted. More often than not the former is the better option. More often the latter is just a variant of the sunk cost fallacy and your ego. Reality is (un)surprisingly complex and solutions aren't usually trivial
More than likely GitHub would have to maintain their own internal wallet solution for this, which is a big engineering lift. But we're all just having a discussion.
There's two types of lazy, and this is the kind that creates more work, not less
---
I know it's against convention to comment on downvotes, but really? Really? This is controversial? The OP came up with an elegant solution that cleanly solved their problem without subjecting contributors to anything more than a captcha. Then somebody comes along and says "oh, it's so easy, just charge $10". You're going to set up payment infrastructure, incur administrative overhead with human support managing refunds, and deter 99% of actual humans from contributing, and then call that the easy solution that OP is so stupid for not thinking of first? Give me a fucking break. This site really is just Reddit-lite, anyone who thinks about engineering problems seriously would realise this does not stand up as anything beyond a pithy internet solution with three seconds of thought into what actually implementing it would entail.
Polar.sh is already doing things that are a lot more complex in this space.
If you are in a civilized country which allow direct payments (i.e, anything but North American nowadays) and you don't want to deal with Github or any external system, there is always good old "make a M-PESA/SEPA/Pix/UPI transfer to account XYZ")
> the thought put into it as the actual solution by people with a stake in actually solving the problem
Let me flip your argument: think of how much time and thought is poured into problems like this one by people who don't even try to implement a Pfand system beforehand.
...which is not available to maintainers to use in this way.
> there is always good old "make a M-PESA/SEPA/Pix/UPI transfer to account XYZ"
And then lock out anyone who is not from the same country as the maintainer, on a platform that is known for its global reach.
Moreover, you're introducing significant anti-human friction. For privacy-conscious people, it's a complete non-starter; I'm not giving my payment information, not for a $1 transaction, and compromising my anonymity just to make a PR for the benefit of other people. That's a small subset. Then, you have the lazy people. The majority of the population will simply not bother with something if it has friction. Getting out their credit card is one of those things, and it's why products/services that offer free trials or a free tier tend to be overwhelmingly more successful -- people want to see a tangible benefit to themselves before they engage in high-friction processes (where "high-friction" is as little friction as requiring a payment, yes). "Free to play" video games with microtransactions engineer first-time purchases to be cheap ($1 or $5) and have 5x or 10x the value of the normal microtransactions, because that first hurdle of getting somebody to hand over their payment information is by far the biggest.
I'll take the captcha, thanks. And maintainers will too, because they'd rather have the solution that filters bots and keeps humans contributing rather than the one that filters out both humans and bots.
No one, meat or chip, would just set aside $10 "for the opportunity to contribute"
This is "let them eat cake" level of out of touchness.
If I was told that I could make a deposit of $10 to get less stressed maintainers and a faster PR review cycle, I wouldn't even blink. I wouldn't even ask for the money back.
Or teenagers without full access to online banking.
Or the unemployed.
Unless I knew the maintainers personally, this would prevent most of my contributions, which are most often accepted. Maybe it's worth losing out on my small contributions to avoid slop. But things would absolutely be lost this way.
If you don't trust the maintainer, you can always fork a repo and let them merge on their own.
Let's say I'm a maintainer of an open source project on Github/Gitlab. How would you actually implement this deposit-refund loop in practice?
``` # FIrst-time contributors
Due to the increased number of AI bots and low-effort contributions, we are being forced to add some friction for first-time contributors. PRs are closed for anyone not explicitly added to our list of authorized users.
To be accepted in the list, you must do one of the following:
- Show a history of meaningful contributions in projects from related technologies done before Jan 1st, 2023.
- Be vouched by one of the existing contributors in the core team
- (If you have github sponsors/polar/patreon) Be a sponsor for the project for the last 3 months)
- Submit a small payment, which will be held in escrow until your PR is accepted. The following methods are accepted (choose all that apply: paypal, SEPA, Crypto, Venmo, Pix, UPI, M-Pesa, etc) ```
I was also wondering how automated or manual you would envision the review process. I'm guessing your hope would be that the small deposit would stem the flow of submissions enough to make it all possible to review manually again, and you would also manually return all the payments sent to escrow?
Wonder if a dollar would work for now until more people give bots credit cards.
That's called theft. And for what, one banana?
Unfortunately, the issue is that time is not enough of a filter anymore. The time from machines is basically worthless compared to yours, so you need to give something else, and that something else needs to be something that shows you have actual skin in the game.
Well no, they don't need to. As they said they could just do something else instead of contributing (and I know I would too).
Your proposal would just end up killing those open source projects even more than what you are trying to solve.
Currently, more than 10% of all commits in the archestra repo are essentially noise (369 of 3521 commits), accounting for more than half of all commits in the last month (303 of 578 commits).
But maybe (probably) the amount of such commits will go down over time, compared to the growing amounts of AI slop
> While GitHub reports massive metric growth — a substantial part of which is AI-generated — we as an open source project team have to do the heavy lifting of cleaning up AI slop from our repository and come up with esoteric workarounds to keep the level of legitimacy of our open source audience.
AI generated slop!
- Protect the PR submitting feature behind some CAPTCHA
- Give repo owners some way to manage external contributors, instead of forcing them to do hack like this article
Just move to Codeberg, src.hut, or Gitlab even. Serious contributors will go there with you, the lazy people with LLM farming Github karma probably won't.
(Why there is a race for AI commits/PRs to projects is beyond me though...)
The captcha - maybe.
Altman after the verdict: "It's okay to steal a charity"
I strongly prefer the git email model, where it’s often trivial to control the flow of changes proposal. GitHub does not have the same wealth of tools and versatility.
not just this issue — but the entire repo.
contributors like @ethanwater, @developerfred, and @Geetk172 — people actively working on bounties — were getting buried.
two identity fields — author and committer — and they can be different people.
metric growth — a substantial part of