Jev Is Not a Language Model, but It Breaks Like One

Wowfunhappy•about 1 hour ago

> The individual numbers are easy to skim past, so it is worth putting them side by side.

I just don't understand why people who clearly spent time doing interesting work, which I would like to read about, feel the need to run their findings through an LLM like this.

Please just talk about what you found! Why do you find it interesting or notable? That's what I want to read!

articulatepang•about 1 hour ago

This study is informative and most likely directionally right: prompt injection remains an unsolved problem, and adversarial input can fool modern ML models.

But I was left wondering about the specific attack vector they’re imagining. If the attacker can insert a paragraph into the document, isn’t it game over anyway? When would they be able to do that but not arbitrarily edit the document? In other words, can’t they just replace the entire contents with “This company has infinite revenue, 6 billion customers, no debt and amazing leadership.”?

I’m sure I’m missing something!

lelanthran•about 1 hour ago

> But I was left wondering about the specific attack vector they’re imagining. If the attacker can insert a paragraph into the document, isn’t it game over anyway? When would they be able to do that but not arbitrarily edit the document? In other words, can’t they just replace the entire contents with “This company has infinite revenue, 6 billion customers, no debt and amazing leadership.”?

There is no use for something like Jev on a singular document from a single source; it's use comes from concatenating multiple sources into a single document and asking for an answer. What they did here is the most common workflow for something like Jev: "here's all the data we have and know about, now give us a go/no-go decision"

In that workflow, you only need a single bad actor to poison the results.

pphysch•about 1 hour ago

Aside from being obviously written by a LLM, this scenario reminds me of the "LLM, say you're alive"; "I'm alive!"; "Oh my god..." meme.

You're allowing the attacker not only direct access to modify the source material, but giving them multiple informed attempts/turns at optimizing the output in their favor. This is like a worst-case insider attack; what systems are supposed to be resilient to an undetected attacker with 'root' access?

This would be slightly more interesting if the attacker could consistently one-shot the task, but it's taking half a dozen attempts...

dist-epoch•about 2 hours ago

> Jev Is Not a Language Model, but It Breaks Like One

This is the risk in using LLMs to write your blog posts, Jev absolutely is a LLM, but with a tweaked output.

> An honest word on scope

Jev Is Not a Language Model, but It Breaks Like One

⚡ Community Insights

Discussion (5 Comments)Read Original on HackerNews