Do AI detectors actually work?

An AI detector tells you a document is "87% likely AI-generated". It is worth understanding what that number is, because it is not what most people assume.

What they measure

Almost all commercial AI detectors are classifiers. They learn to separate two piles of text: one written by humans, one generated by models. The features they lean on are statistical properties of the prose:

Perplexity. How surprising each word is given the ones before it. Language models optimise for likely next words, so their output tends to have lower perplexity than human writing.

Burstiness. Variation in sentence length and complexity. Humans write erratically; models are more even.

Distributional quirks. Vocabulary and punctuation habits that differ between the two piles.

Note what is absent from that list. They are not reading a watermark. They are not detecting a hidden signal that was deliberately placed. They are guessing from style, and there is no ground truth in the document to check the guess against.

Where this breaks down

Non-native English speakers. This is the most robustly documented failure. A widely cited 2023 Stanford study found detectors flagged over half of TOEFL essays written by non-native speakers as AI-generated, while classifying essays by native-speaker students almost perfectly. The mechanism is straightforward: writing in a second language tends to use a more limited, more predictable vocabulary, which reads to the classifier exactly like a language model.

Anyone writing in a plain style. Technical documentation, legal writing, formulaic academic prose, the writing of neurodivergent students who favour consistent structure: all skew toward the same low-perplexity, low-burstiness profile.

Text that has been through any tool. Grammarly, a spellchecker, an editor's pass: anything that regularises prose pushes it toward the AI side of the boundary.

Heavily edited AI text. The reverse also holds. Take model output, rewrite the openings, vary the sentence lengths, add specific detail. Detection confidence collapses. The detectors are measuring style, and style is editable.

Why the percentage is misleading

The number looks like a probability. It is a classifier confidence score, which is a different thing.

"87%" does not mean there is an 87% chance a machine wrote this. It means the model's output layer produced 0.87 for a text with these statistical features. Whether that corresponds to any real-world likelihood depends on how similar your text is to the training distribution, and for the populations above it demonstrably does not.

The scores are also not reproducible across tools. Run the same document through four detectors and you can get 12%, 45%, 78% and 94%. At least three of those are wrong, and there is no way to tell which.

What institutions have done about it

Several universities have restricted or disabled AI detection since 2023. The reasons given are usually false positives and the difficulty of acting fairly on an unverifiable score. Vanderbilt disabled Turnitin's AI detector in 2023 and published its reasoning. Others have kept it available while barring it as sole evidence in misconduct proceedings.

That trend is the most useful signal available: the institutions with the most exposure to these tools' error rates have been the ones stepping back from them.

How this differs from watermark detection

This is the distinction worth internalising, because the two get conflated constantly.

An AI detector infers from style. It has no ground truth and its output is a guess with an unquantified error rate.

A watermark check observes what is in the file. A zero-width space at offset 412 either is present or is not. A C2PA manifest in an APP11 segment either exists or does not. These are facts about bytes, reproducible in a hex editor, and verifiable after removal by re-running the check.

The second kind of tool can be wrong about what a finding means, since an odd space is usually a formatting artefact rather than a watermark, but it cannot be wrong about whether the character is there. That is a meaningfully different epistemic position, and it is why our checker reports named characters and counts rather than a confidence percentage.

We do also produce a writing-style score, and it has exactly the limitations described above. Which is why it is labelled a heuristic, lists every signal that fed into it, and never claims to be a probability that a machine wrote your text. Nobody can produce that number honestly.

If you have been falsely accused

Practical, in rough order of usefulness:

1. Version history. Google Docs and Word both keep it. A document that accumulated over days with real revisions is strong evidence.
2. Drafts and notes. Messy intermediate work is hard to fake after the fact.
3. The research literature. The Stanford non-native-speaker study is directly citable, as are the institutional decisions to restrict these tools.
4. Ask which tool and which threshold. Detectors have configurable sensitivity. The specific number and its documented error rate are fair questions.
5. Offer to discuss the work. Someone who wrote an argument can defend it. This is usually the most persuasive thing available.

The short answer

Do they work? They separate the two piles better than chance, which is not nothing. They do not work well enough to justify the confidence with which their outputs are reported, and they fail hardest on people who are already disadvantaged.

Treat the percentage as a weak signal, not a finding. That is roughly how the more careful institutions have landed too.