Advertisement
Advertisement
⥠Community Insights
Discussion Sentiment
58% Positive
Analyzed from 2525 words in the discussion.
Trending Topics
#watermarking#seed#model#quality#random#output#watermark#more#idea#article
Discussion Sentiment
Analyzed from 2525 words in the discussion.
Trending Topics
Discussion (50 Comments)Read Original on HackerNews
> Relevance and irrelevance are excluded because they test whether a call should be made rather than whether the emitted call is correct.
Relevance and irrelevance are not introduced above this comment. This reads like an LLM-ism (particularly a GPT-ism) editing a document, removing something, and leaving a note about why it was removed, which doesn't really make sense when reading it.
> Their limited movement under prompt injection should therefore not be interpreted as evidence that watermarking preserves safety behavior more reliably on these models.
Also a GPT-ism which appears when it draws a counter-conclusion in the text because it feels the need to be honest and a human tells it to remove it because it's not true because of "reason".
Overall interesting research, however, I think it's great that model output is getting watermarked. I was skeptical of this at first, but Opus 5.5 is so good, it seems like it's a non-issue in practice.
The reason I think watermarking is great is because it's a really good way of preventing training on it's own output indiscriminately and Ouroboros-ing itself.
I just took his text, pasted it to ChatGPT, "rephrase", paste it back in the online checker, 0% AI.
In your analogy: What if seed 42 specifically causes poor quality behaviour (in some contexts specifically). Normally, these quality differences will be washed out because the seed is random, now it is no longer random, so shouldnt we check into specific behaviour under this specific seed?
Opus 5 started adding a bunch of comments to code, even when instructed not to, and for very simple changes where the comment itself was longer than the code change. Was that so that there are enough tokens outputted for watermarking? Many people suspected so.
Take a recurrent PRNG for example. A randomly seeded recurrent function usually has degenerate cycles in its state space. For some functions, this might even describe the majority of the state space. This is why so many non-cryptographic PRNGs are max-cycle, so a different starting point is just further along the same trajectory.
I don't think LLMs have quite the same failure mode here, but recurrence + high dimensional spaces triggers my "here be dragons" sense.
Detection of watermarking requires access to the watermarking key, a secret in the current suggested scheme (leaking it would amount to being able to strip the watermark).
So, there will need to be a watermark checking service. The checking service will of course be rate-limited for common folk (and model distillers). OpenAI/Anthropic/Google/other privileged model builders need to filter out AI slop at scale, so need access to others' service without rate-limits (or the watermarking keys need to be shared).
This creates an in-group with pristine datasets, and an outgroup whose models will collapse on the slop outputs with no good ability to filter.
No it's EU law.
Over a certain token threshold, yes, there are 0 negative effects. Something like 300-400 words. At the boundary and below it, it does effect response quality, so they donât (shouldnt) do it. It also incentivizes increasing tokens in low token responses so that it can be watermarked which is itâs own quality issue.
Let's say there are four billion possible seeds. There are four billion possible ways we could watermark the generation. We could say "we will choose seed 1, that way we will know exactly what output it produced", we could say "we will choose seed 2, that way we will know exactly what output it produced"... etc etc. Now, if we decide "not to watermark", we STILL must choose a seed. So we are actually still applying one of the watermarks, the only difference is we are not careful to remember which one. Could some seeds give a better or worse answer to some specific prompt? Yes. Could choosing a random "watermark" to apply be better or worse on average than choosing a random seed to apply? No. It's mathematically impossible.
This is like an open source project changing their seed from "12321" to "43", and saying that because we changed the seed, the quality is "necessarily lower".
In reality, Gemini and Anthropic use SynthID watermarking which affects token probability distribution, i.e. their tournament sampling can pick lower-probability tokens which the LLM's distribution would otherwise not have. They likely use this over unbiased watermarking because SynthID is resistant against text edits.
Changing the distribution is the whole point: it introduces statistical regularities that can be detected.
Also interested in how this watermarking push makes sense when considering RSI.
What the article is discussing, and what many people are concerned about, is something that you might be missing in your understanding: theyâre not actually random. In fact, they would be entirely useless for real work if every token was randomly selected based on all possible outputs. Itâs not, even at temperature 1.0. Itâs based on the training corpus and once you have your tool names, syntax, and prompting style aligned with the training data then they become incredibly deterministic in the areas that matter, such as tool calling and parameters. I build my toolset by testing thousands of names, syntax, return format, and other aspects until I find a convention that produces the exact correct call, 100% the exact same every time, regardless of context length. Those decisions are per model and what works with Opus 4.7 wonât necessarily work on 4.8, and neither version will work with a local model or GPT.
Thatâs only possible because the massive training corpus is the guiding principle behind the choices. Providing a file reading function called âRead_The_Fileâ will fail, either on the first call or somewhere down the line, because that name is not associated with the concept. Your instructions are trying to override 500 trillion tokens from training and it will cause perplexity to manifest as wrong tool calls, wrong syntax, âoops deleted prodâ, âClaude lost the plot againâ, âWTF?!â, and probably nearly every frustration youâve encountered and determined to be âthey nerfed Claudeâ or âitâs a dumbass.â
For those that are aware of it, that knowledge lets people tweak and tune the prompts/tools accordingly.
You may not put that effort into your system, perhaps because youâre unaware of it, donât use it in a way that requires it, or youâve just taken the failures caused by perplexity as something thatâs inherent in the framework, but for people that build precision infrastructure around them itâs potentially devastating news. Watermarking, which is based on whatever tokens, threshold, cutoff, and triggers some guy at a desk decided, will necessarily alter that entire system.
https://chatgpt.com/s/t_6ab7d694885481918083b8cbf0ba9040
In particular: sometimes they measure âchurnâ, which doesnât show whether the results are better or worse on average. They sometimes only test with one random seed. There are multiple-comparison issues. And theyâre not testing Anthropicâs algorithm.
This is a spy tool.
It's not "watermarking", it's "spymarking": https://news.ycombinator.com/item?id=49794615
If that's the case, then it's a brilliant strategy by the labs to cut down cross-AI usage and just stick to one model. But I'm pretty sure this won't be the case.
As in duck duck go removes Claude watermark..Claude freaks out and puts it back in. Then after a few more times Claude flags all inputs as "prompt injections" and begins offering self deleting code
Second paragraph:
> Watermarking is designed for provenance, but SynthID-Text changes the process by which the model generates each next token.
This is a stretch. True, but barely. The LLM is making slightly different choices near the end of the token generation process.
> At the model level, this can change safety behavior, including whether the model refuses a harmful request and whether that refusal holds under prompt injection.
Claim support, if it appears, is pages later.
> At the agent level, the same sampled tokens can determine which tool is called and what arguments are passed to it.
?
> Prompt injection connects these two settings because a weakened refusal becomes more consequential when the model can also act through tools.
Wtf. Non-sequitor. Where does this come from?
> Such a watermarking procedure can therefore affect both what the model says and what an agent does.
Duh? In the literal sense of outputting different tokens.
> We call this behavioral effect sampling drift.
I think they should have used an LLM for writing help, or paid more for the one they used.
The article reads to me as largely LLM generated. I checked several sections with Pangram and it appears to agree that itâs either entirely AI generated, or at least 100% AI assisted.
Iâm fine with the idea of AI assistance, but I agree the quality is low and this needed a bit more editing. I have my own list of gripes that include not defining terms that are used, text for tables and figures being repeated in the body and the captions, contradictions and non-sequiturs, and just AIâs real watermark of getting lost in details, using clever terminology, having a hard time being concise, and being unable to use plain and clear language.
I think they goofed on the conclusion: âThese results do not argue against watermarking for provenance. They show that provenance and behavioral stability are separate properties.â
The articleâs entire point is that stability depends on provenance techniques, therefore the results purport to show they are not separate properties. Oops.
Also, itâs not AIâs fault that URLs in the references section arenât clickable. Come on.
Political Correctness has a similar idea that by adjusting the terminology we use, we can purge biases and historical implications and speak in a purer way.
Psychoanalysis has its own idea of repression - where a person struggles to BLOCK our associations between ideas, memories, and words in order to try to stop one thought from being contaminated by another, intolerable thought.
All of these attempts to control language are fundamentally misguided at best, often have severe unintended consequences, and are genuinely immoral at worse.
You either misunderstand political correctness or are trying to make it sound more nefarious than it is. It is nothing more than an effort, sometimes overdone and misguided, to not say things that make minorities feel bad. Itâs nothing more than an attempt to broaden whatâs considered good manners.
If anyone serious thinks political correctness is going to end racism, I certainly havenât seen it.
> Psychoanalysis has its own idea of repression - where a person struggles to BLOCK our associations between ideas, memories, and words in order to try to stop one thought from being contaminated by another, intolerable thought.
Itâs been a while since I took Psych 101, but my memory is that, according to Freud, repression is subconscious. Thereâs no struggle possible because there is no intent. Itâs also about memories and emotions, so if it WAS an intentional act, itâs not an attempt to control language. It wouldnât be about trying to not think of the word cat, but trying to not think about that time when your cat died.
The struggle in repression is real, even if it's unconscious. The psychoanalyst asks to speak with FREE ASSOCIATION and the inability to do this in SPEECH is what reveals unconscious problems. Repression, though, is a STRATEGY that we most definitely sign-on to. It FEELS unjust to be asked to take responsibility for something we didn't INTEND to do, but that, unfortunately, is our task in life.
You should assume all text is AI generated. If you want to "test" someone at school or during an interview, have them write with a pencil and paper.
https://news.ycombinator.com/item?id=49794615
This is because SynthID and similar watermarking methods for LLMs don't just change the random seed. They take additional steps (that I have yet to read about) in order to detect when someone changes a few words of the output, trying to remove the watermark.
The other takeaway is that just by knowing a watermark is being applied gives an attacker an advantage in working around safety features because then they know the output isn't based on true randomness and can take advantage of that in a similar fashion to how breaking cryptography becomes easier when the RNG isn't truly random.