Have a sneer percolating in your system but not enough time/energy to make a whole post about it? Go forth and be mid - welcome to the Stubsack, your first port of call for learning fresh Awful you’ll near-instantly regret.
Any awful.systems sub may be subsneered in this subthread, techtakes or no.
If your sneer seems higher quality than you thought, feel free to cut’n’paste it into its own post — there’s no quota for posting and the bar really isn’t that high.
The post Xitter web has spawned so many “esoteric” right wing freaks, but there’s no appropriate sneer-space for them. I’m talking redscare-ish, reality challenged “culture critics” who write about everything but understand nothing. I’m talking about reply-guys who make the same 6 tweets about the same 3 subjects. They’re inescapable at this point, yet I don’t see them mocked (as much as they should be)
Like, there was one dude a while back who insisted that women couldn’t be surgeons because they didn’t believe in the moon or in stars? I think each and every one of these guys is uniquely fucked up and if I can’t escape them, I would love to sneer at them.
(Credit and/or blame to David Gerard. Also just came back from Spider-Man: Brand New Day, movie was awesome)
Anthropic is now watermarking the outputs of its AI. For once this is some AI news that doesn’t completely piss me off, and it’s amusing to see all the uninformed boosters get in a tizzy about this.
I actually understand at a reasonable level how this watermarking works. A year ago, I watched Scott Aaronson give a talk about it, and from what I know he was somewhat involved in developing the theory behind it while working for OpenAI. But at the time my thought was, “He is naive if he thinks these companies would ever implement this out of the goodness of their hearts.” And I was right; Anthropic is only doing watermarking now thanks to the EU AI Act, even though the theory has long been developed.
Watermarking doesn’t mean adding an extra watermark that can be easily removed. It instead directly affects the output of the chatbot itself. Fundamentally, an LLM is still a most-likely-next-word-predictor. More precisely, an LLM produces a probability distribution of what the next word can be. For example, “my pet is a …” could give a distribution of 60% dog, 30% cat, and 10% axolotl. Normally, an LLM would randomly choose the next word based on this distribution, and this is one reason why LLMs are nondeterministic (there’s another parameter called “temperature” that affects this, but no need to get into that).
With watermarking, instead of a truly random choice, the randomness instead comes from a cryptographic pseudorandom generator seeded with a secret key from the AI company. If you don’t know the secret key, then you can’t really tell that watermarking was used. But if you do know the secret key, then the idea is you can tell when the text was generated by the LLM because you know exactly what word should be next. It would be a freak coincidence if some non-AI text just happened to choose the correct next word every time. Thus, you can provide a service to tell if some text was generated by the LLM. (This technically makes the LLM “deterministic”, in a completely useless sense.)
Now, I think this is a step in the right direction, but it has its limits. The biggest problem is that you don’t want people to just move to a different LLM without watermarking, and that’s exhibit #832593 why government regulation is important. Another issue is that sometimes there is very little randomness in what the next word should be (“The first president of the USA is George …”). Finally, watermarking can be defeated by editing the output, although you would have to break up most of the blocks of consecutive words. I have a feeling most AI users are not the type to put in extra effort after copy-pasting the output directly from the chat window.
I suppose it will discourage some of the “use cases” of LLMs, such as drowning the world with spam Slopstack essays. Ah, who am I kidding? Everyone could already tell it’s AI generated, they don’t care!
Does this in any way extend to generated code? The space of possible continuations for generating “useful” code seems laughably tiny for that
related
couldn’t you just change the first word to a synonym and completely thwart this? You’d never be able to compare subsequent next token predictions since you don’t have the internal state of the LLM at that token.
Some systems like SynthID (for Google’s AI) get around this problem. In fact you don’t need to know the LLM’s internal state, and defeating it would likely involve breaking up most blocks of 3 words. The oversimplified explanation is that it introduces a function g that gives a score to each word, with the score being (pseudo)randomly determined by your secret key. For each next word the LLM generates, the LLM produces a small list of candidate next words, and the one with the highest score according to g is selected. You should expect that the LLM will generally pick words with a high score, but the score itself is independent of the LLM. To detect a watermark, you need to know g and the secret key, and you check if the average score is much higher than expected from normal text.
Now, one question is, will this bias to the LLM to favor certain words? The solution is that for each next word, you append the last 3 words (nothing special about 3, just a small number) to the secret key for g, and this repeatedly scrambles which words have a high score. To defeat the watermark, you would need to break up most blocks of 3 words. I’m sure there are deeper issues with this, but I have not studied the topic that much.
Thank you for the technical explanation, that was absent from some mainstream coverage I’d seen.
Given that it works by controlling the PRNG, does that mean they could salt it by user id and track particular outputs back to specific accounts?
From a technical perspective, I don’t really know how to let it track specific users. The goal is that if some AI vendor provides their key, it is easy for them to prove that their AI produced the text. So I guess if a user wanted to show that they generated the text, they could prove it. But usually AI users don’t want people knowing that they used the slop cannon. In any case, Anthropic’s own announcement states that their watermark does not contain identifying information for users.
In any case, some proud AI users are now frightened that there might be a way for people to identify that they use such a wonderful tool. How will I ever deliver low quality slop for my contracts now?
Gee, if only there were some way to create code that doesn’t raise questions about authorship. Some sort of skill that people could learn that allowed them to deliver code to their clients that complied with their policies…