Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

gandalfgeek•3 minutes ago

Big jump for sure, but definitely comes with a giant grain of salt lacking open-sourcing the harness itself and measuring performance on the held-out set.

stared•38 minutes ago

In the spirit of ARC-AGI-3-like challenges, we just tested if frontier AI models are able to solve a lovely puzzle game, Baba Is You: https://quesma.com/blog/baba-is-bench/

A year ago, Sonnet 4 barely solved the first level. Now, both Fable 5 and GPT-5.6 Sol beat the first two stages. GPT 5.2 is slow, but efficient, while Gemini 3.1 Pro and 3.5 Flash struggle.

daytonellwanger•18 minutes ago

Can someone tell me what the catch is? To outperform the state-of-the-art so drastically would be massive news, and surely the ARC Foundation would have tested this against the private data set, right?

levocardia•about 1 hour ago

(1) What does it score on the private test set? (2) Does this approach generalize to, e.g., Atari or NES games, or is it just hard-coding priors about the games into the model (as Chollet specifically warned was a chronic problem in benchmarks in the original Arc-AGI paper)

causal•27 minutes ago

We need to see private set results, but if this holds then it might represent a breakthrough in other domains as well.

Alifatisk•about 1 hour ago

What does it mean to reach 99% score on Arc-AGI-3? That the agent is able to tackle difficult problems?

modeless•about 1 hour ago

It doesn't necessarily mean anything to reach 99% on the public set. All of the public set is known in advance, so it's possible to hardcode rules that make this easy for the models. ARC-AGI-3 is supposed to measure generalization to unseen games, so the only score that matters is the score on the held out private test set that nobody outside the ARC prize foundation has access to. Also, I believe the private set is significantly harder than the public set.

westurner•about 1 hour ago

> Schema, the harness we introduce today, reaches 99% on the ARC‑AGI‑3 Public set using Claude Opus 4.8 and Fable 5, and 95.35% using GPT‑5.6 Sol.

Impressive results. Will this translate to coding agents (and training general purpose and for coding LLMs) too?

---

> When Michelson and Morley could not detect the medium light was supposed to wave in, Lorentz took the first route: keep the aether, patch the rules with contraction hypotheses that absorbed the null result. Einstein took the second: in special relativity, he discarded the aether as part of the state and made simultaneity frame-relative, yielding a simple electrodynamics of moving bodies.

BECs, SVT, Superfluid Quantum Gravity

Massful photons are modeled with Proca fields. Like Einstein, Proca was also a student of Minkowski. The Mass-Equivalence principle ~~does not~~ still holds if photons have mass.

(edit) Energy-momentum relation: https://en.wikipedia.org/wiki/Energy%E2%80%93momentum_relati...

> could not detect the medium light was supposed to wave in,

Superfluid Quantum Gravity (Fedi,) says that there is a medium that light waves through; there is not nothing in space, space is a quantum dilatant superfluid with near-zero viscosity.

Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

⚡ Community Insights

Discussion (8 Comments)Read Original on HackerNews