Claude Opus 4.6 accuracy on BridgeBench hallucination test drops from 83% to 68%

Reubend•about 14 hours ago

Because the website doesn't seem to show any sample size of runs, I assume they ran it once across the suite.

The models are nondeterministic, and therefore it's pretty normal for different runs to give different results.

I don't see this as evidence that Opus 4.6 has gotten worse.

slurpyb•about 7 hours ago

I would love to know what you’re doing in the harness to not feel the total degradation in experience now in comparison to December & January.

bsder•about 10 hours ago

> The models are nondeterministic, and therefore it's pretty normal for different runs to give different results.

And how is that an excuse?

I don't care about how good a model could be. I care about how good a model was on my run.

Consequently, my opinion on a model is going to be based around its worst performance, not its best.

As such, this qualifies as strong evidence that Opus 4.6 has gotten worse.

senko•about 2 hours ago

>> The models are nondeterministic, and therefore it's pretty normal for different runs to give different results.

> And how is that an excuse? […] this qualifies as strong evidence…

This qualifies as nothing due to how random processes work, that’s what the gp is saying. The numbers are not reliable if it’s just one run.

If this is counter-intuitive, a refresher on basic statistics and probability theory may be in order.

dlahoda•about 10 hours ago

are models really non deterministic?

Rury•about 9 hours ago

People are describing the results when they say models are non-deterministic. Give it the same exact input twice, and you'll get two different outputs. Deterministic would mean the same input always gives the same output.

loneboat•about 10 hours ago

Yes. Look up LLM "temperature" - it's an internal parameter that tweaks how deterministic they behave.

csomar•about 10 hours ago

The models are deterministic, the inference is not.

ehtbanton•about 7 hours ago

Benchmarks like this one are designed to thoroughly test the model across several iterations. 15% is a MASSIVE discrepancy.

Come on Anthropic, admit what you're doing already and let us access your best models unhindered, even if it costs us more. At the moment we just all feel short-changed.

Claude Opus 4.6 accuracy on BridgeBench hallucination test drops from 83% to 68%

⚡ Community Insights

Discussion (12 Comments)Read Original on HackerNews