Nvidia AVO scores 100% on the ARC-AGI-3 interactive reasoning benchmark

ddsrtslnd23 about 5 hours ago 31 commentsRead Article on twitter.com

DE version is available. Content is displayed in original English for accuracy.

⚡ Community Insights

Discussion Sentiment

63% Positive

Analyzed from 943 words in the discussion.

Discussion (31 Comments)Read Original on HackerNews

mellosouls•about 2 hours ago

Underlying article should be the link:

https://developer.nvidia.com/blog/nvidia-avo-reaches-100-on-...

There you will find the extremely important qualifier it's the public set, not the private set (with the risk of overfitting, ie the results not repeating when submitted to be run in competition), and the detail that this is essentially a harness added to Opus 5, not Nvidia's own models.

Obviously still impressive, you would think.

woeirua•about 4 hours ago

I thought ARC-AGI-3 was explicitly a test of raw model performance excluding the harness? Adding the harness back in doesn't tell us anything new. We've known that agents are capable of long horizon reasoning with sufficient harnesses. GPT-4(?) was capable of beating Pokemon 18 months ago but models only became capable of beating it without a harness in the last six months...?

r0ze-at-hn•about 3 hours ago

These days calling to a server like openai or anthropic effectively results in a server side harness wrapping the weights.

dwohnitmok•about 2 hours ago

> GPT-4(?) was capable of beating Pokemon 18 months ago but models only became capable of beating it without a harness in the last six months...?

GPT-4 was decidedly not capable of beating Pokemon 18 months ago. I doubt it would be able to complete a single level. I don't think people realize how large the advances in model capabilities have been. GPT-4 in a modern harness is absolutely horrendous compared to modern models.

margalabargala•about 2 hours ago

> I doubt it would be able to complete a single level

Have you ever played pokemon?

altcognito•about 3 hours ago

I'm assuming you're referring to a harness that includes memory -- I generally think of the harness as anything beyond executing the generation loop, but I'm not an expert.

True as that may be, it may be better to optimize models for some amount of memory versus forcing some token count based on a reasoning level, right?

throwaway314155•about 3 hours ago

I don't think GPT-4 was ever used for beating pokemon with or without a harness. Successful attempts include Gemini 2.5 and Opus 4.7 both using relatively advanced custom harnesses that give access to game memory, notes systems, and one-off hacks to get around parts of the game the model gets stuck on. More recently, Fable 5 beat FireRed with a _very_ minimal harness (screenshots and button inputs). That's the only example I know of but that is a very sophisticated and very expensive model compared to GPT-4.

Most of this doesn't discredit your overall point, though.

dezgeg•about 1 hour ago

GPT-o3 was the first OpenAI one to beat Red. The harness used by GPT Plays Pokemon is the most featureful one of the main competitors (GPT, Claude, Gemini), IIRC.

Community maintained spreadsheet of the runs: https://docs.google.com/spreadsheets/d/e/2PACX-1vQDvsy5Dt_-P...

dist-epoch•about 3 hours ago

The intent in forbidding harnesses was to prevent an ARC-AGI specific harness, which for example presented the game interface in a more agent-friendly way.

What NVIDIA has here is a generic "evolution" harness, which can be used for any problem.

I think it would be fair game to allow OpenClaw, Hermes, Codex, Grok Bot, this NVIDIA thing, to compete, as long as they don't have ARC-AGI specific skills, toolset.

subzel0•about 3 hours ago

The 100% score was achieved on the 25 public set, not on the semi-private or private sets.

magicalhippo•about 4 hours ago

The blog post: https://developer.nvidia.com/blog/nvidia-avo-reaches-100-on-...

Using Claude Opus 5, but it can use others:

AVO is also designed to operate across frontier models. While our full public-set result used Claude Opus 5, we additionally paired AVO with GPT-5.6 Sol on a challenging subset of games. In these limited experiments, Sol reached matched levels faster in wall-clock time in several cases, while Opus used fewer environment actions in matched-level comparisons. These preliminary results suggest complementary operating profiles across models, and we leave a broader systematic comparison to future work

xnx•about 3 hours ago

Verified high score is just 40%: https://arcprize.org/leaderboard

moralestapia•about 1 hour ago

That's a different thing. No AVO in there.

embedding-shape•about 4 hours ago

None of the tweets, nor the press release, seems to mention how long time it actually took E2E to complete the evaluation, but they do mention it took "12% fewer actions" compared to just Opus 5 without AVO. Feels a bit suspicious they don't break down the timing involved, looking at the diagram from the press release, it gives the impression there is a lot of machinery here, and given they claim fewer actions, each action must be more carefully considered, doesn't it?

Curious to read more about it though, seems the paper for it is here: https://arxiv.org/pdf/2603.24517, I'm not sure I understand if it's better than just Codex with a /goal, as they talk about "can discover performance-critical micro-architectural optimizations" but leave Codex alone for a day or two and you'll get the same results without doing "additional autonomous adaptation" at all.

throwaway2027•about 3 hours ago

I wonder if these benchmarks swap words, meaning and more because you might as well be benchmaxxing for specific words. I notice a lot of recurring just structural sentences coming back in smaller LLM models where they're fit for a specific task which is fine because most of the work we do is repetitive and there are patterns to learn but they should be word agnostic which I wonder if LLM can really fix.

program_whiz•about 4 hours ago

Is this AGI? I don't think I can score 100% on ARC AGI.

daemonologist•about 3 hours ago

You could probably score 100% on ARC 3 if you were motivated enough. I find some of the current problems to be kind of like Chess - mechanically simple, and ~solvable, but it's difficult to force myself to think at length about a monotonous and meaningless problem. The machines do have an advantage on the "energy" front; they've become almost psychotically persistent (and don't get tired after too many prompts).

Anyway yes I think we've had AGI for a while now, even if the GI doesn't quite match up with what we expect from a human.

chris_st•about 4 hours ago

You'll find that those goalposts are very movable.

andriy_koval•about 2 hours ago

> Is this AGI? I don't think I can score 100% on ARC AGI.

100% is some "RHAE" metric: its performance of median human first time seeing those problem.

edgarvaldes•about 2 hours ago

>A 100% score means AI agents can beat every game as efficiently as humans. (0)

Yey, AGI is finally solved.

(0) https://arcprize.org/arc-agi/3

inerte•about 3 hours ago

It is until ARC-AGI-4. Maybe around 73 we will stop.

adastra22•about 3 hours ago

Yes, we’ve had AGI for years now.

program_whiz•about 1 hour ago

Its interesting because I didn't think it was, but then reading the NVIDIA approach, this kind of loop plus generating a program to explain things. Maybe that is AGI? I don't know, but it seems like an additional layer that maybe is a fundamental shift in capabilities (kind of like reinforcement learning and COT was).

Insanity•about 4 hours ago

Depends on which definition they’ll use today.

oblio•about 1 hour ago

What's AGI, at the end of the day? Equivalence to the average human?

m3kw9•about 4 hours ago

agi with a context of 250k-1mill tokens?

tiahura•about 4 hours ago

Could KITT? Cmdr. Data?

tiahura•about 4 hours ago

AVO: Agentic Variation Operators for Autonomous Evolutionary Search

https://arxiv.org/html/2603.24517v1

ru552•about 3 hours ago

Thoughts on Nvidia releasing AVO or even open sourcing it? They've been very open with their models.

angoragoats•about 3 hours ago

Can we please prioritize links to the papers, github repos, press releases, or blog articles for these types of posts? I don't use Twitter and I don't think anyone else should either.

AndrewKemendo•about 4 hours ago

Now we’re talking

The next year is going to be wild folks