ZH version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
56% Positive
Analyzed from 3140 words in the discussion.
Trending Topics
#taste#test#code#unit#more#down#tests#things#tdd#don
Discussion (65 Comments)Read Original on HackerNews
Follow this line of thinking, and the AI-friendly answer is easy: we just have to externalize everything we know, so Claude can implement what I want.
Except that I can't fully externalize myself. Debugging a system takes more resources than running the system. If I could write down everything I know and hand it to a machine, I'd do that, but it impossible.
People aren't books or hashmaps. If you want to build something, you need to use the tools, not teach the tools to use you.
[edit: I'm trying to figure out if there's something to be done about this. Email me if you want to chat -- tr at tern dot sh]
I am more familiar with taste in coding and it can at best be described—that the resulting code is too subtly different from something else in the codebase, that you're masking a different bug, that you're not following what the code tells you. The good part is that while this cannot be unit tested, you can write documentation and code comments about it that tell people what they need to know.
But for taste of the kind described in the article there's not even a definition. The logic ended up being "trust a bunch of opaque weights the most"
I'd say there are "simple" simple things you can do though, like take automated screenshots and detect colours for jarring colourschemes.
Unit test runs, waits for human input before passing or failing, which might seem out of the norm, but we already have QA do manual testing.
I had this experience doing a port from Big Query to Postgres using Opus. I had unit tests to guarantee parity with the original code, and Opus insisted on building this bespoke query builder (e.g. `def _where(very_complicated_params)`) on top of sqlglot.
Even with the original code being straightforward and legible and repeated instructions to match, I had to fight with it to get close.
In the end, I ended up doing things the "old fashion way" where I copied chunks code into Claude proper and gave explicit instructions for each piece.
I clearly had externalized the requirements, and yet that wasn't sufficient. The only way to unit test further would be to use an AST to evaluate the output against metrics I couldn't even encode.
And that's why it's so hard to get a model to reproduce the specific taste of a person or an organization. My taste is different than yours, so if we dump our aggregate preferences into RL, in averages out to nothing interesting.
For the code-writing case, this means you end up reviewing every line of code, looking for places where you'd thumbs-down the code. Not every line of code contains a real decision, though, so it feels like a waste of time.
If I were to ask you - what convention you want to follow for your database columns - camelcase or snakecase? There's no correct global answer. There's no overarching truth that should apply to all databases in existence (even if you'll focus on a certain type of database). Hence the no.
But yes, because in the context of existing system there is a convention. If it's snakecase, you create new tables with snakecase column names.
LLMs will generally follow conventions, but sometimes they will not, because indeed - global truths (or at least, the "last article it read" truths) sometimes win over (I assume)
LLMs are built for scale so they've given up on the kind of online learning / "long term memory" processes that would individualize them.
The LLM is permanently locked to being a really cracked engineer on their first day at your company, looking at your codebase for the first time.
You can scaffold a bit with .md files, but at the moment they lack the ability to do what humans do: go to sleep, encode things from short to long term memory, and wake up the next day with more specific knowledge baked in.
Outside of AI, I run into this issue when taking basic personality tests. A question may be written for a specific reason, which influences the results, but the reason for my answer may be completely unrelated to the reason intended by the person who made the test.
The co-occurence thing is often not a bug of the algorithm but a genuine part of the stochastic landscape that must be solved. Evolution isn't "failing" when sickle cell vulnerability is ported along with malaria resistance; it's just a real tradeoff being made in the current biological landscape.
I'm not so sure. For instance, you can write down what it means for a program to be free of XSS and other injection vulnerabilities. Now, how would you unit test for that property?
Want to follow certain pattern, or convention - define it, ie active record vs repository pattern, stick is as an ADR! You don't know what you want? Look at what Claude produces and then acquire taste, mark this as convetion that future sessions will follow, but stick to *one* convention!
Treat your LLMs as junior developers willing to apply various patterns willy nilly, caring only about fulfilling the ACs of given task and not about the longevity or well being of the system in general. They will not look at bigger picture to check if given pattern applies globally, or even if there are any other patterns.
And maybe that's just our limits with philosophy, modeling, assumptions, whatever. The danger is not realizing when we're in that zone.
(Fwiw I think unfalsifiability is a limit with any system - "you didn't compile in my syntax/semantics" is an gotcha that's actually valid and useful, but nobody can really determine the hard line)
He couldn't articulate why but they trusted his gut and it did collapse.
A lot of software engineering relies on that kind of intuition and on a good team you can integrate it and benefit from it and avoid all manner of floor collapses.
I'd argue that transformers are a pretty good indication that intelligence isn't "encodable" in the way we think it means. Usually, most "model" vocabulary means that we can explain and constrain the "data" from the "rules". Except the mere "data" is trillions of interacting weights.
That may be encoding in a physical sense, but that still doesn't explain the intuition in any legible way to humans.
Cynically, we've been able to encode everything already by just saying everything's a transition in a huge lookup table. Not very informative though.
There is a reason conference talks are always about plain algorithms and data structures.
The only thing that the business seems to care about is top-down UI testing. This is also convenient because you can leave it until the very end after the customer has already seen several prototypes.
I do think TDD makes sense in isolated scopes (prove this specific custom parser works at the edges), but as the general policy for the entire product it's definitely not a viable practice. Much of the time if comes off as an ego trip to see just how cleverly we can mock something so that we can say we technically tested it.
You would have the same problem if you wrote tests like that after the code.
TDD has no opinion about the level at which you wrote your test, it just assumes it's the correct one.
This is the number one biggest misconception about TDD which I keep seeing repeated on hacker news.
https://news.ycombinator.com/item?id=46810793
https://news.ycombinator.com/item?id=45113016
Or to put it differently: a test is an assertion that no matter what, for all time this should never change again. Even if customer requirements change in the future they won't change in such a way as to break this test (this isn't always true, but you should believe it is true).
A test is most valuable when it alerts you to a real problem when it fails. If the test fails but there isn't a real problem (either because customer requirements have changed, or it is flaky) it was needless cost to investigate it. If the test passes that gives some hope of correctness, but you can never be sure it is really correct vs a bug in the test (even if you use TDD and so the test failed when you wrote it that doesn't mean a refactoring since didn't make this an always pass test).
Part of the problem is if I tell you to write sort() or your new toy language's list type you have an intuitive idea of what it should look like and probably will get them right the first time (other than bugs you want the tests so you catch). These should have tiny micro tests. These things also are really easy to use as examples of how to do TDD - which they are, but they are not representative: this type of code is generally in your standard library already and you are not writing it.
Instead you are writing code that isn't well defined with lots of industry experience. It is not clear what the exact interface should be (or more likely it is clear customer requirements will change but you don't know how yet). You have no idea what the best implementation is. You don't know if this will be used in this one place, or if it will become a useful key part that many future projects depend on. You have to make guesses.
Isn't red-green-refactor pretty ingrained in TDD?
Only write code to make a failing test pass; then refactor while making sure the tests still pass?
Then write a test that fails, repeat?
https://arxiv.org/abs/2602.07900
Substitute static typing for TDD in your comment, and it will remain equally valid statement.
Here I am talking about the basic static typing, and maybe some generics use occasionally, but obviously people also go overboard sometimes with type features and that hinders understanding for newcomers to the codebase.
Human: well-scoped argument that does just enough to get the job done with minimal risk.
AI: Extremely clever and correct legal argument that almost any lawyer would have said not to file (at least as written). It tries to burn the world and seriously risks pissing off the judge.
https://www.youtube.com/watch?v=YRXJnKP6Tu0
> QRank is a ranking signal for Wikidata entities. It gets computed by aggregating page view statistics for Wikipedia, Wikitravel, Wikibooks, Wikispecies and other Wikimedia projects
from https://github.com/brawer/wikidata-qrank/blob/main/doc/desig...
This has been my experience, as well, but it’s a really big support. It just needs adult supervision. I can’t understand how vibe-coded apps, actually work.
As far as “taste,” goes, I test my stuff constantly, checking for even minor “friction points,” sometimes, refactoring back to design, in order to resolve issues that many folks would ship. I’m pretty anal, and want my work to be the best experience possible.
I can’t see any LLM coming close to being able to evaluate the user experience, like I can.
But overall I agree, LLMs are currently awful at being beta testers. They miss the most basic stuff that any human would immediately catch as being poor UX, and for all their visual prowess they are terrible at auditing UI.
I wrote about this a few months back. Rick Rubin is famous for this. I do think it is something that can be trained though, it just needs a lot more context. Taste builds over time through lots of unit tests, through lots of content writing, through an accumulation of product decisions. It’s hard to put it in the individual spec, but it can be teased out of 100 project specs. And when you get to that scale the AI starts to do it pretty well.
If you watch his interview on Rick Beato's channel, this myth will fall apart. He plays guitar, had his own punk rock band and his guitar playing is featured on some high-profile records he produced. Also, he has a lot of practical experience with all kinds of studio equipment.
Well, you can package it up, otherwise Rick wouldn't exist.
I do it too because it's a common expression, and a marathon is of course longer than a sprint, but both have in common that properly raced, they are absolutely brutal efforts that leave you without a single additional drop at the end. The effort length and instantaneous power output changes, of course. Maybe "it's a marathon build, not the race" would be more precise at the loss of nearly all its expressive power (but with a lot more pedanticism points) :-p .
Nice project !
but that's what the phrase is meant to convey, right?
Don't run through consumable X (energy/money/etc) like there's no tomorrow - even though there's <some big important milestone> now, we've got dozens more of those that we need to meet, so you're better off getting this one done at 75% than committing 100% to it and failing on all the others.
Working, useful, delightful, in that order. Testing can make things more likely to work, that's it.
I think people have been trying for the written word, with some degree of success (anti-slop skills). I have been trying for visuals, and it's pretty meh. It's easy to get a multimodal LLM to follow a style guide, but a style guide doesn't capture everything that accounts for taste. And anything that is dynamic (not a screenshot test) seems really hard or really expensive.
You absolutely can unit test for taste, just put an agent into loop, and write into prompt what you like. Then do scoring...
Iceland is really bad example, it basically has one populated site (capital) and circular road that goes around the island.