AI is breaking our proxies for expertise

> Understanding the concrete problem mathematicians are upset about can help us better understand the impact of AI on our own fields,

imo, The author of this essay does not understand the concrete problem that mathematicians are upset about. There is an idea that math [1] and coding [2] are human activities whose purpose is to achieve a certain kind of insight or mental clarity of things. The simplest description of this is by Feyman [3]. AI generated proofs short-circuit human understanding and therefore goes against the primary purpose. The declaration is calling this out loudly to reiterate that the purpose of the endaevor is not the generation and rewarding of proofs.

[1] "On proof and progress in math" https://arxiv.org/pdf/math/9404236

[2] "Programming as theory building" https://pages.cs.wisc.edu/~remzi/Naur.pdf

[3] "What I cannot create, I do not understand"

Anastasia Berg and Jon Baskin wrote a piece in August called “On Radical Preservation,” [1] in which they use the example of the shared concept of 'merit' to warn about the dangers of an AI-filled present and future. They anecdotally compare our situation to that of the Crow people, through the testimony of Plenty Coups, who described the fall of the buffalo and the end of the practices, rituals, concepts, and way of life that his people understood as making up what it was to live an excellent life (or not).

Likewise, they point to merit (which I think we consider very much like expertise) as one such fundamental concept in our society, which is very much undermined by unchecked use of LLMs. This coming not just via "cheating," but by the way in which we so quickly are willing to claim, and ourselves believe, that we deserve praise for that which the machine has created. On a wide scale, their use will not just compete with those who may not use the machine, but will destroy even our shared idea of personal merit.

Beyond merit alone, AI might lead generally to our "own cultural values becom[ing] not just decadent or debatable but unintelligible." At the end, Berg and Baskin basically say that hope is not sufficient (hope that the old concepts will be replaced with new ones); the proper attitude is to do everything in our power to preserve them in the present.

[1]: https://thepointmag.com/letter/on-radical-preservation

> I think it’s broadly unclear whether (3) is true: i.e. whether frontier AI models aren’t generating or can’t generate new mathematical ideas. ... I give basically zero credence to the idea that AIs are incapable of this because of some intrinsic feature of how LLMs work.

LLMs are computer programs, so there are math problems which they cannot solve. AKA, ideas which are not possible for them to generate.

The argument for this is that Busy Beaver function is uncomputable. More specifically, some N-state Turing machine requires a proof that it doesn't halt. At some point N is too large and LLM being a computer program, it cannot generate the required proof.

See the Busy Beaver Frontier [1]

This is VERY DIFFERENT from the Halting Problem. In the Halting Problem, we see that no computer can decide whether an arbitrary given input program halts. With the argument above, for a fixed LLM, there is specific math problem which is beyond the capability of proof by the LLM (though other LLMs or humans could perhaps prove it).

Humans are not bound by the argument since we aren't finite computer programs (no proof for this anyways). LLMs which "evolve" over time with input from the natural world also aren't bound by this, since their code is effectively infinite. The argument only applies to a static program with fixed input, no dynamic information sources.

Some people believe in divine inspiration. Maybe you could believe that humans incorporate information from the natural world which LLMs don't have access to. Either of these beliefs would imply that humans have an edge.

[1]: https://www.scottaaronson.com/papers/bb.pdf

“ Like mathematics, software engineers are going to have to rebuild our cultural sense of the kind of work we value.” From the closing is the change I see going in at every job area and level.

My hunch is that this ultimately doubles back to those that excel at story telling and human coordination. As the AI systems “offload” not just production but I believe some initiation of what to build, the “why” and how to rally groups for any appreciably complex work matters more.

I also hope to see a plenty of solo shops succeeding in spaces that used to take entire teams, but (for now) remain convicted human coordination remains a key need for most endeavors.

I disagree that AI has broken the proxies.

Football analogy; AI is the wide receiver and the human is the quarterback. Even if you're the best WR in the game you're still not producing touchdowns unless you have a decent QB.

It's super easy to smell vibe coded projects.

All of these proofs and vibe code are impossible without human work. Call me when GPT whatever writes gcc from scratch

> i.e. whether frontier AI models aren’t generating or can’t generate new mathematical ideas. [...] I give basically zero credence to the idea that AIs are incapable of this because of some intrinsic feature of how LLMs work.

I also believe there are limitations of LLMs, but not necessarily where people think. I won't expect LLMs to be creative solvers until they can tell a novel funny joke with any recognition/consistency.

What if there are math problems that can only be brute forced? How do we know that Navier-Stokes even has a nice and elegant solution?

Not an anti-AI view at all, but right now people is exaggerating a bit on the "AI is solving math". For now they are good at finding holes on theories, proving by negation, not proving for all cases, nor coming up with new theories.

> If you know proposition X is true, it’s easier to figure out why.

This misses the point entirely. If you know proposition X is true or false, you won’t bother spending years trying to prove or disprove it, developing deep understanding and potentially even developing entire new fields of math in the process. (See FLT.)

That’s why it’s so destructive to the discovery process to have an LLM just generate a proof or counterexample without the side effect of generating useful explanatory or generative structures that we can build on.

That said, there are examples like Ramanujan where someone did an info dump of unexplained theorems that people try to mine useful structures from, but that’s not at all the mainstream of mathematical progress.

I'm sure this delights many people in todays age.

Just like how social media has possessed many people with cultivating an outward facing image that often diverges with reality, AI posses people to portray themselves as an artist/developer/musician/etc. without having put in any of the work.

> We might say that generating ideas is the real intellectual work of mathematics.

The article talks about how many new ideas are relatively worthless and the real goal is to find the "concepts that 'carve nature at its joints.'" I think this is the crux of the whole thing and I haven't seen a satisfying discussion of it anywhere.

I mean, FLT is mentioned. Is that an accessible proof to humans? Is it full of these high value, refined concepts or is it more like a bunch of little hacks that at least dozens if not hundreds of people randomly stumbled upon, in an all out attempt to solve one of the most famous math problems?

I'm not totally convinced what value math concepts have beyond "you can use them to solve even more math problems." I really want to believe there is. But if not, it's just a pure benefit to have faster ways to solve them, no?

> people would know you were a skilled engineer. But now projects like that are worthless, because everyone just assumes they’re vibe-coded.

I do not really care what other people think. Build something for the purity of building it for yourself or for a purpose.

Thanks to AI, I feel like I have been writing the best code in my life. Yes, writing, not vibe-coding. I mainly just ask questions and ask for hints and clues. I do not use LLMs to do the fun parts for me.

I am currently working on a game. If and when I ever finish it, I want to be able to say that I wrote every single line by hand. Will it make me better than anyone? No, not at all. I want to do it for myself.

Mathematics is a solved problem.

Slop article