Lies, Damn Lies and Database Benchmarks

The database wars of the late 1990s were full of this kind of stuff. Oracle, Sybase, IBM etc invested heavily in tuning specifically for benchmarks like TPC-C just so they could post ads in the Wall St Journal saying theirs was faster.

I do sympathize with OP, though, their objection to measuring cold-start queries is incomplete without also describing how often cold start needs to happen. If you restart once every five years then it doesnt matter as much if it takes 20 minutes to be warm. Every hour, that would be a real problem.

Reminds me of the recent Terminal Bench controversy [1][2][3]

If theres a benchmark, people will cheat, lie and optimize for that benchmark. Honest depends on the compliance enforced on teams. But if, compliance itself is weak, it is going to be taken advantage of. Like growing up india, you would optimize for the exam and not what you learn from it.

[1] https://news.ycombinator.com/item?id=47920787

[2] https://www.tbench.ai/news/leaderboard-integrity-update

[3] https://debugml.github.io/cheating-agents/

Same with LLM benchmarks these days.

Really respectable writing and perspective. Questdb blog posts that get posted here never disappoint