Orca-Bench: How Ready Are Language Model Agents for Oncall?

yyruzin about 2 hours ago 4 commentsRead Article on arxiv.org

Discussion Sentiment

67% Positive

Analyzed from 110 words in the discussion.

Discussion (4 Comments)Read Original on HackerNews

4di•about 1 hour ago

This doesn't work anymore. Is there a newer link?

dash2•about 1 hour ago

Seems like there's a big attack-defence asymmetry at present: models are great at exploiting systems and poor at fixing them.

aleksiy123•12 minutes ago

Attackers advantage in the iterative fast feedback loop?

It’s harder to have a loop to ensure you are defending all possible attacks?

I guess the loop is you need to attack yourself and fix. But attackers only need a single opening.

Finding all possible attacks and patching them against yourself is inherently more expensive?

EGreg•about 1 hour ago

Your strategy can’t be patch AFTER an intrusion. Only to build a hardened environment from scratch and be ready in advance.