Do newer coding models end up training on the AI slop generated by older models?

ttbharath about 3 hours ago 6 comments

ES version is available. Content is displayed in original English for accuracy.

As old coding models were writing and pushing code to public, does the new coding models train on them because generated code by old coding models were not so good?

⚡ Community Insights

Discussion Sentiment

50% Positive

Analyzed from 147 words in the discussion.

Discussion (6 Comments)Read Original on HackerNews

OsrsNeedsf2P•about 3 hours ago

Yes, that's why good datasets are important. But usually the difference between model generations is architectural improvements

verdverm•about 3 hours ago

late training is where a lot of capability gains come from too

tbharath•about 1 hour ago

Interesting, how does that work?

verdverm•about 1 hour ago

the art of reinforcement learning

verdverm•about 3 hours ago

How much do we assume human written code for training the original models is free of human made slop? We all know we all take shortcuts and have code we are not proud of. If they are I deed averaging machines, is it possible their output is the average of human output?

I'd argue that the way attention works plays into the slop too

bdangubic•about 1 hour ago

30 years of looking at human written code, not great :) in the early years for me (late 90’s) code was just solid. dotcom boom happened and “overnight” everyone was a fucking coder. there are too many of us and average coder is so bad that you would never hire them. not only that, even at 90 percentile is fairly terrible coder. AI has absolutely been training on this garbage human code so it was terrible at coding initially… but it is evolving now on its own