ZH version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
50% Positive
Analyzed from 3352 words in the discussion.
Trending Topics
#text#model#models#code#more#claude#watermark#watermarking#where#idea
Discussion (66 Comments)Read Original on HackerNews
The only practical use I can see for this in the world we actually live in is to prevent model collapse. Most people using AI don't care if people training future AI ignore them, so would have no incentive to switch to providers that do not watermark. Of course, this disencetivises all if the pro-social applications of this technology, and risks giving the big providers a monopoly on "known human" data, which has serious antitrust implications.
[0] Note that the watermark is not inherent to the model itself, but rather how the model is run. So this teqnique cannot be used by people providing open-weight models. It would need to be used by those actually running the models.
>However, in a world of commodity AI models, can simply use a model from an AI provider that does not watermark. Or download any open source model and run it themselves
That's seems above the skill of the majority of the population that might be tempted at using generative text. I'm thinking of students trying to "write" a paper, or any of the myriad of other things people are blissfully unaware of how genAI works that are using it every day. For those people, that just use the default prompt to blindly copy/paste away. For techy nerd types running from CLI, yeah, they could simply use a different model.
With a strong enough regulatory framework, we probably could suppress unlisenced providers and non-watermarking local harnesses to the point where bypassing them would take some technical know-how. However, I do not think there is anywhere near the political will for such a regime.
>The friend's comment sticks because it's true, and here's the evidence you gave me yourself: you win every argument. Of course you do — you're writing both parts. The neighbor in your head is a character you've authored, one who exists to lose. That's not deliberation; it's rehearsal. And people who are actually calm don't rehearse.
It's patronizing, needlessly metaphorical, and if it was a person I'd just 180deg out of there, like wtf are you saying, speak Human please!
No idea if it is accurate without the rest of the context, but that's my reading.
It drives me up the fucking wall.
The worst part is I can prompt Claude with “this is gobbledegook, simplify” and it will reword its previous answer perfectly but no amount of hooks or system prompt hacking will fix it otherwise.
It'll be interesting to see if all this causes a subtle shift in vernacular.
It’s also why I’m not a fan of using subagents for code review like Claude Code started doing by default recently. It’s getting output from another LLM, then summarizing it for me so by the time I see it it’s already been blended up once.
like if you have a long cli command or something will it still try to watermark it ?
Is there some way you can know which tokens are required to be syntactically correct vs not?
Here is a pure browser client-side demo: https://massa-ai.freeboxos.fr/textego/
No server, browser only
Of course, you can also run your own check service if you pay some API fees, and those checks can be a lot more convenient for users since the services can check multiple sources, to whom they are paying for the privilege.
Then someone washes the text through a local model that rewords it, the markers are lost, amd they're clear again.
Here's a visual representation of the watermark: https://i.imgur.com/JNUIykX.png
This is even worse in such forms of writing like coding, where there's even less choices the model can make on what the next token should be. Plain text in code will obviously be watermarked, that includes comments. But the code itself might get watermarked by choosing certain code over others more often.
I'm inclined to believe the models will be instructed to not watermark code, especially since it's harder to detect reliably because the shorter the body of text the harder it is to detect, but who knows what Anthropic and all the other AI labs will decide to do in the future.
EDIT: Also for those in the comments who are naive enough to think Anthropic is doing this just because the EU said so and not because it's beneficial to them (and all other AI labs), well, you are indeed naive. Identifying code will be paramount in training future models because the more synthetic data you feed it, the more cannibalization happens, the worse the models will perform over time due to lack of good data, among other such reasons as selling AI detection services to colleges, and a plethora of other reasons.
That being said, I do question how this will apply to code as opposed to prose. Even data dense text (ie, if you ask Claude to evaluate what running shoe to buy, and it spits back a list of options with reviews and prices) may struggle.
What it probably will work well at it flagging the current tsunami of entirely AI generated novels on Amazon/Kindle, which is...honestly not without value.
> Identifying code will be paramount in training future models
True, but note that this strictly allows providers to identify text generated by their own models. If Anthropic wants to filter out GPT generated text in their training data, they'll need to feed it through an OpenAI API, which is implausible. So it might help on the margins, but I don't think it solves the problem of model collapse.
The model essentially is just doing what it always does which is predict the next token, but the next token now is split into two and nudged towards one side more often than the other which is how over a body of text identifies if the text was in-fact AI or not. Which is also why the shorter the text the harder it is to identify.
As for the second half, I predict that almost all American, Japanese and European labs will use watermarking at some point (as well as SynthID for other generative AI). It doesn't matter if they all have unique internal seeds because they'll give people the tools to ID AI, be it by selling it (unlikely for most use cases , but likely gonna happen for academic where they'll provide some value app that bulk checks student works), or more likely make it free to check like SynthID where you simply ask Gemini if the image has SynthID. At which point they can just pay each or use each other's tools to check.
The wildcard are the Chinese models, but those will likely force some sort of watermarking as well, if not for the global market, for the CCP's benefit.
The models "natural" output is the result of a series of random numbers. The watermark works by biassing that series towards a different series of numbers. Assuming that second series is cryptographically secure psuedo-random, the even distinguishing the biased sequence from true random would be impossible with compromising the key or prng.
As an extreme, suppose your prompt was public, and the model seeded its PRNG with a secret key instead of a genuine random seed. Such an output is not meaningfully different from one based on a true RNG, but can be trivially fingerprinted by someone who knows the keys.
In practice, I am doubtful they have a scheme that is both practically useful and cryptographically secure. However, there is a lot of room below cryptographically secure that is still just as good for all other purposes.
And I pray it is because of the EU AI Act, because it is worth giving them my personal information to prove I am not in the EU so they can turn this off. Heck, I will pay more to avoid this crap.
If it were not so noticable, I would shrug it off. But it has made things clearly worse this year.
The only way this works is to use the same exact model and weights right? So that you can replay the text generation as it would have been originally done, and compare output?
And then what, if there is no match do you need to retry with all other known models that could have been used?
Or are models sufficiently similar that they are interchangeable for this type of watermark?
And what if a competing or open source model was used? I can't see how the watermark would work.
And if you have access to a non-watermarked output? How can you prove they are not simply using another key? How can you be sure the text is not watermarked? From the explanations, you can't.
Both will be proprietary for a closed model, which means the owner will have a monopoly on detecting their own model(s). (They may or may not offer API access, but if they do it will be a closed box.)
Because detection essentially means running the model again, the monopolist will probably charge their usual token rates for detection, which doubles their revenue. If they don't they'll be spending a lot more on compute with little/no extra revenue.
What's more likely to happen is that open models won't have the tech, they'll be used in paraphrase mode to strip watermarks.
But in fact most people will just skip the closed models and use open models by default.
The irony is that the EU legislation is primarily about video deepfakes and AI pseudo-journalism. Fiction, parody, satire, and other creative expressions are explicitly excluded from labelling requirements.
However you slice it, text watermarking is likely to end up being irrelevant.
(Music went through a similar process with MP3s and other audio formats. They were watermarked for a while, until everyone realised watermarked audio is almost entirely useless - although some companies did make a lot of money before the industry got there.)
I think this is trivial to implement for open weights models? The main issue is each open weight provider could choose their own randomisation key and so actually matching the synthID would be finding a needle in a haystack! Intractable at scale and a chore for even one chunk of text.
For example, someone inventing a new technology might use AI to assist with code prototypes/doc prep etc. Where would that leave the rights of the owner/creator/inventor. These text watermarks provide fuel for legal battles.
Am I wrong?
I'm having a hard time thinking of any other use case... What other reason is there to fingerprint your code?
I assume if they arrest you for vibe coding something that violated computing laws (hacking) they can then scan your code, send it to anthropic, anthropic confirms that Claude was used on your account, on x device to build malware.
Remember they busted a hacker because of his Windows unique install ID.
Imagine a patent drafted by AI under human direction. Regardless of the current laws, this would provide evidence for, and leave the door open for future laws/claims...
Claims would literally be on the balance of probabilities, and there would be argument about exactly where the line is, given that some passages could be heavily edited while some might not be edited at all, and does that mean only parts of the work can be copyrighted?
Other countries allow copyright for AI gen work, so there's no issue there.
I don't have an answer either way, but I'm taking this occasion to remind you (and HN generally) that many people use LLMs for things other than code
I'm a native English speaker and I have no idea what this means.
If you then know how the particular dice weights, you can later use that to confirm watermarked text.
What the article is trying to say:
"For AI-generated text, watermarking usually works by secretly biasing which words/tokens the model prefers while it writes."
Simple. Understandable. No "dice" or "leaning" or other convoluted explanations
“Provenance” seems to be the flavor of the day.
There was an odd key to unlocking it sounding like a human. I spent like a month on this topic last year, and I randomly stumbled upon something that ended up massively improving generated tests.
It was a shot in the dark after failing so often. And then it suddenly sounded good.
Rae bought all the candy at the store - Milky Way, Reese's, M&Ms, everything - and set out for Kay's party.
That can't be rewritten as,
Rae bought all the candy at the store; Milky Way, Reese's, M&Ms, everything; and set out for Kay's party.
Semicolons set off independent clauses - what would otherwise be complete sentences. Em dashes create breaks in the flow of the sentence; commas are more similar.
> The one idea in this step: a model writes by rolling weighted dice between several words that would each be fine.
> The one idea in this step: the key secretly colours the shortlist and gives one colour a gentle nudge. The text still reads normally.
> The one idea in this step: with the key, you can re-colour any text and simply count. Marked text lands green too often to be luck.
> The one idea in this step: the mark lives in runs of untouched wording. Editing erases it exactly where the runs break, and nowhere else.
> The one idea in this step: detection is private, probabilistic, and about processing, not authorship.
I suspect someone prompted the LLM with "have just one idea in each step", because apparently breaking down a process into steps is just toooo much wooooork these days.
Anthropic always likes blaming other people and countries for everything they do -- it's in their DNA -- but they're doing this worldwide for their own benefit, not some sort of extraterritorial influence of Brussels. No one in Europe forced them to do it outside the EU, but they wanted to for their own reasons.