Will Claude's new AI "watermark" keep us all honest?

Is there a reliable way to tell something was written by AI?

If Claude wrote it, the answer is now "yes," says AI researcher Mustafa Ocal.

Anthropic announced this summer that it is watermarking all text produced by Claude to comply with the new EU AI Act. The mark is now integrated into Claude's newer models. Implementation for older models is on the way.

The watermark aids in the detection of Claude-generated text — but is "indistinguishable" to human readers, according to a company release.

"It's brilliant," says Ocal, a professor who researches AI detection at FIU's Knight Foundation School of Computing & Information Sciences. "People think, 'I can just get rid of the watermark by deleting em dashes.' But it's much more complex than that. It is never, ever going to be human detectable."

The text watermarking technique is not new. Google pioneered it in 2024 and is already using it. However, Ocal says that Anthropic's approach goes further because the company is implementing the watermark universally in its latest models while sharing a way to check for it: a proprietary scanner restricted to certain organizations. This has important practical implications, he says.

Ocal has spent weeks combing through Anthropic's recent publications and speaking with colleagues in the industry. He offers his analysis on the watermark — and why he thinks it will usher in a new era of AI transparency — in a Q&A for FIU News.

Who can see Claude’s watermark? 

Any text that was made or heavily edited by Claude will carry the watermark, as long as it's about five sentences or more in length. But your girlfriend won't see it, and your teacher won't either. We could gather all the AI professors I know and still not find it.

The watermark is only readable by the scanner. That scanner has to be given to you by Anthropic. Right now, it's in private preview. Only a few organizations have access to it, like law enforcement, regulators, media organizations and fact-checkers, along with a few researchers and businesses.

Does the watermark insert any hidden data that we can’t see, like invisible punctuation?

No. People expect some kind of hidden character or code, like the difference between a Turkish "İ" and an English "I" in binary. It's not that at all. The watermark is based on word choice.

How does the watermark work?

A very complex formula tells Claude to insert certain words in certain contexts. We can't notice them, but Anthropic's scanner can. If the scanner finds a lot of these words in the exact kind of places that Claude would have put them, it's a very strong indicator that Claude wrote it.

It's like this: Say you went to the casino and played 100 hands of blackjack. You might get blackjack about 5 percent of the time. But if you get blackjack every other hand, that's a very strong indication of cheating. The same logic applies to watermarking. If the scanner detects watermarked words at a high rate in a large sample, it's just not normal.

If Claude's watermark is based on word choice, how could it be indistinguishable to the human eye?

We won’t find the watermark by looking for “AI words” like “delves.” And not by looking for "AI constructions" like “It’s not just [X], it’s [Y],” either. Watermarking is not based on a visible style of writing.

The watermark is practically invisible because we don't know which words are watermarked. They might be adjectives, verbs, nouns, or even punctuation. And the watermarked words could change by topic. For example, for text about dogs, Claude might mark the nouns. For text about cats, it could watermark the verbs.

Can I remove the watermark from a Claude-generated document?

Since we don't know which words Claude is choosing, it will be very hard to remove the watermark from any long piece of text. You'd have to heavily edit or rewrite.

If I ask Claude to just lightly edit my document, will it insert the watermark?

It depends. If Claude makes changes to word choice, then it’s possible. But if only light proofreading is needed, it's unlikely the watermark will make its way into the text.

What makes watermarking unique from other AI detection methods?

Commercial AI detectors do something completely different. They are trained to detect differences in AI writing and human writing.

The watermark is not about a style of writing. Only word choice. 

I'm skeptical. Even if Claude’s watermark is “invisible” to us today, shouldn't we be able to detect it eventually?

I think nobody will. It’s a matter of math.

To show you where I'm coming from, I think it's important to pause here and look at how large language models like Claude work. Then, we can see why the watermark is so hard for people to find.

How large language models write

AI models don’t “think.” They predict.

Here’s an example. If you take out your iPhone, open your messaging app and type this:

        Tomorrow I will go to the _______

You’ll notice that before you’ve finished the sentence, your phone suggests three words, such as:

        gym      I    store     I     hospital

The app makes predictions based on its training data. Right now, it predicts that one of these three options is highly probable to go next.

AI works the same way. It's predicting what will likely go next. Yet unlike your messaging app, it has to decide on its own.

To do this, it lays out all words that could reasonably go next based on its training data. Each word has a probability attached to it, based on how well the AI predicts the options will fit. Then, the AI chooses a word at random based on these probabilities.

It’s a little bit like when, in school, you were asked, “If there are 3 red marbles, 4 blue marbles and 5 green marbles in a bag, what’s the probability of pulling a blue marble out without looking?”

Here's an example of how the word decision happens. Let's say you asked an AI model for a one-sentence story about fishing, and the model gave you this sentence:

        The old man's rod bent sharply as the bass fought against the line.

The model ultimately chose “sharply” here as the word. Not because it was the only word that worked here. Instead, it was randomly selected out of a “bag of marbles” — a pool of words that were all highly probable options.

The choices might have looked like this:

Our pretend AI model picked "sharply" here by randomly selecting from the choices available while considering how often "sharply" should be picked. The word had a 23% chance.

Another way to look at it: In a bag of 100 marbles, 23 marbles were the word "sharply." And so, when it came time for the model to randomly pluck a marble from the bag, "sharply" had a decent chance of being selected. This time, it was.

As you can see, the way that AI selects words is highly randomized. The model doesn't pick the No. 1 most-probable option every time. It can pick any of the words from the table. It doesn't really matter to us whether the rod bent "sharply" or "suddenly." Sure, the model chose "sharply," but any of these options could have worked. 

In this way, AI is constantly making word choices that readers never notice.

You can test this at home. Try asking the mainstream AI model of your choice, "What is a field goal kick?" If you do this 1,000 times, you'll get 1,000 answers. All different, yet all correct.

Here's the point: There is an astonishingly large number of word choices AI can use to convey an idea.

And so, if an AI company wants to weave in a watermark by being a little more selective, it can easily do so without readers noticing.

How the watermark works

Claude's watermark works by reducing randomness in word choice. The model takes advantage of its ability to say one idea in 1,000 ways to insert intricate word choice patterns.

To see this in action, let’s pretend we asked Claude to tell us what the greatest food in the world is. It responds:

Pizza is one of the most satisfying foods there is — crispy crust, gooey cheese, and endless topping combos.

Claude could have easily sewn its watermark into at least one word choice here. Let’s say it was the word “satisfying.”

There are multiple words that could replace "satisfying" here without changing the essence of the sentence. Before the watermark was deployed, Claude would have chosen from these options at random based on each word's likeliness to fit here, like in our previous example. Here’s what the choice might have looked like:

But now with the watermark, this word choice decision has changed.

Claude won't randomly select an option, while respecting each word's likeliness of going next.

There is no blindly reaching into the "bag of marbles."

Instead, Claude will be biased. And biased in a way that people will never notice. It's going to take advantage of its ability to communicate one idea in 1,000 different ways, all without telling us its method.

Essentially, Claude is implementing its watermark by "dyeing” some word choices blue and others red. The blue words will appear more, and the red words will appear less. A high ratio of blue words to red words indicates the watermark.

To see an example of how this might work in action, let's pretend we can pop open the hood on Anthropic's watermark and look at how it affects our pizza sentence.

Instead of the old choice, we might see Claude faced with a choice like this:

Now, Claude is biased heavily towards choosing blue words in this context.

A high proportion of blue words to red words in an extended sample makes it clear to Anthropic’s scanner: Claude wrote this text.

Like drawing blackjack 50% of the time at the casino, it’s extremely unlikely for a human to “draw” blue words at such a high rate. There are just too many other card combinations – or in this case, word choices – that a human writer would naturally choose from instead.

Meanwhile, what's "blue" and what's "red" stays hidden from us, and won't be detectable.

The watermark isn't as simple as Claude pushing a word like "satisfying" all the time. The words that are dyed blue and red are probably changing across contexts, in line with some extremely complicated formula that we need Anthropic's scanner to decode. For us, it's untraceable, because all the words look normal; but for the scanner, the watermark looks very visible, because it can see blue and red.

Here's an example.

Let’s say that you asked Claude to write a novel, and it began writing this:

The sky turned a brilliant shade of amber as the sun sank below the horizon, casting long shadows across the field while a cool breeze swept through the grass.

Here, there are multiple opportunities to weave in the watermark. For example:

The sky turned a brilliant shade of amber as the sun sank below the horizon, casting long shadows across the field while a cool breeze swept through the grass.

Here, 80% of dyed words are blue. Notice that each blue word doesn't seem out of place; however, they all  have at least one close, common replacement that a human might have used instead.

With so many synonyms available, the chance of a human choosing all these blue words at such a high rate is unlikely.

It's an early indicator that Claude may have written this. A larger sample would be needed to confirm.

This shows why Claude's watermark is so hard to remove.

We don't know which words to look for.

Final takeaway: Do you think watermarking is a good idea?

It's an excellent idea.

Now, companies will be a bit nervous knowing Claude's watermark is out there. They'll have to disclose when they use Claude to make hiring decisions. They can't just say, "Oh, our HR team made the decision," and hide the fact that they used the model. If there's Claude-generated text in the process, they'll have to disclose that AI was involved. And that's actually good for applicants, if you think about it.

This is also going to be great for rooting out forged documents and evidence. If the U.S. Department of Justice had this scanner, for example, they could inspect any piece of evidence and say when something is Claude-generated, which would be very helpful to us.

Right now, we don't know what's real and what's not. Now, at least for Claude, we can finally tell. 

Media contact: To schedule an interview with Professor Ocal, contact David Drucker at ddrucker@fiu.edu or (954) 401-3352.