Paul Graham on LLMs 'Thinking'

2) On the other hand I'm also critical of people who assert that "Ackshelly you didn't do any 'work' by putting the crate from this table to that table" or "sweet potatoes are not 'potatoes' at all, so don't call them potatoes".

In physics, words like 'work', 'power', 'heat' have very specific meanings. The 'heat death of the universe' is thought to be near absolute zero Kelvin a.k.a. frigging cold, no heat anywhere. The meaning of 'work' in physics has a certain overlap with what is called 'work' outside of physics (to wit, a number of different things). A typical biologist or taxonomist's view on species is much more nuanced than the ordinary guy-with-some-general-educational-background's view on the subject: the latter will tend to see species as an absolute given which science just has to uncover. Sure scientists can err but ultimately the truth of biological species is often conceived as incontrovertible by the informed layman instead of what it is: a tool for thought, a human measure brought in to help make sense of chaotic data. Darwin already said as much.

Maybe physics should never have chosen terms like 'force', 'work', 'power'. As a German speaker let me remark that especially 'power' is a bit puzzling; German has 'Leistung' instead which I think is a bit clearer. But even when scientists go and create their own terms like 'energy' or 'quantum leap', neither of which existed before being introduced with a specific scientific definition attached to them, the public often goes and appropriates them anyway. So "he's got such a cold energy" ("cold" being associated with lack of thermal energy) and "a massive quantum leap" (i.e. "a sizeable advancement" instead of "the smallest possible change") are colloquially possible.

Look at more squishy terms from squishier sciences like 'language' as used by linguists and things become even more difficult. Sure, "the language of Islamic architecture in early medieval Spain" is a nice and valid book title and no linguists will take exception. But talk about the new research on the "language of birds" in the newspaper and you will ruffle some feathers.

One has to acknowledge that people are challenged with finding words to describe a new phenomenon. Sure, we could stop at 'calculate' or 'compute', as in "ChatGPT computed some falsehoods" instead of saying it 'lied'. But even when I say that "ChatGPT's answer was wrong" doesn't 'answer', alone and by itself, already imply a degree of intentionality that is (or should be, as far we can tell) absent from a process that purely relies on solving one big formula with a pinch of randomness thrown in? Then again, "what is the calculator's answer?" was already used in the 1970s when pocket calculators were all the rage.