LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques

This makes it impossible to really check their claims in practice.

The only way I've seen these debates settled is by using a fixed task and having someone act as a referee. That's what nanoGPT's speedrun did for optimizers.

The goal is to make a public record that shows which training tricks really save time and which ones don't work when tested again. Each record should explain how the method works, so over time, the leaderboard becomes a kind of lab notebook.

As more records are added, the leaderboard serves both as a ranking and as a detailed log of what was tried. I agree, and that's my next step: adding a second track with a different model family and task (SmolLM2 + SQuAD), so we can actually test if these methods transfer instead of just promising they do.

Honestly, your opus could probably build the framework too. But a leaderboard is more than just code. It also depends on someone being willing to review submissions for cheating, make decisions when things aren't clear, and keep adding new records even months later. That part can't be automated; it needs to be maintained by people.