Can AI review website copy, or judge your offer?
A GPT-4 judge agreed with expert human reviewers 85% of the time, slightly more often than the experts agreed with each other. The same judge reversed a third of its verdicts when the two options were simply swapped. What a model can and cannot tell you about your copy.
"AI cannot review my website copy. It does not know my business." That objection is aimed at the wrong target. Whether a model forms a view on your copy is not in question, because it always will. How often that view is right has been measured three times, and the answers do not agree.
Can AI review website copy accurately?
More accurately than the objection assumes, and less reliably than the number suggests. In the closest thing to a controlled test, a GPT-4 judge agreed with expert human reviewers on 85% of paired comparisons, against the 81% agreement the humans reached with each other. Accuracy is not the weak spot. Reproducibility is.
What can a model see in your copy without knowing your business?
Five things, all readable from the page. Whether your headline names an outcome or a feature. Whether a claim has its evidence within a screen of it. Whether your button uses a verb a buyer would say out loud. Whether the reading level spikes mid-page. Whether two pages contradict each other.
None of those need your market or your margin. They are properties of the page, like a broken link. A headline that says what the product is rather than what it does is a mechanical fault, and the most common one on a founder's homepage.
Want this run on your page? Paste your URL.
How often does an AI verdict match a human expert's?
On MT-Bench, 85% of the time excluding ties, measured across 80 questions, 859 first-turn votes and 58 expert-level human labellers. The same study found those labellers agreed with each other 81% of the time on 479 votes. A model reviewing copy is about as consistent with experts as experts are with each other.
- 85% GPT-4 judge agreement with 58 expert human labellers
- 81% agreement those human experts reached with each other
- 65.0% of verdicts that held when the two options were swapped
That third number is the one to carry. Swap the order of two versions, change nothing else, and the verdict reversed in roughly a third. The same study padded answers into repetitive lists, and GPT-4 took the bait 8.7% of the time while two older models took it 91.3%. Figures from Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, an academic paper that sells no tooling.
Why does that 85% flatter the model?
Because it was measured on a job humans are also bad at. Those experts agreed with each other only 81% of the time on open-ended preference. On a task with a defensible right answer, a 2025 study put agreement between human raters at 98.52%. On that study's coefficient, Scott's pi, the best model judge landed eight points below human judgement.
Read them together and the shape changes. A model looks human-level on the subjective call, which version reads better, and behind on the checkable one. Grading trivia answers is not reviewing a homepage, so take it as a caution about reading the 85%, not a measurement of your copy.
Human baseline from 1,200 questions and 13 judge models, in Thakur et al., Judging the Judges, GEM 2025.
A third paper adds the failure that matters most here. Shown the same substance with emoji and formatting added, human judges picked the dressed-up version 47% of the time against a 39% random baseline, and GPT-4o 7%. Under invented citations the models were the weaker party: GPT-4 accepted a fake reference 66% of the time, and every judge but one did no better than chance. From Chen et al., Humans or LLMs as the Judge?, EMNLP 2024.
So a model is harder to sway with pretty formatting than a person, and easier to fool with authority that does not exist. Worth knowing before reading which parts of an audit a model could invent.
Does Revslip know your market, or guess at it?
It researches it. Revslip has a separate market research step that reads competitors' claims, offers and friction, and which audience pains each site addresses or ignores. So it does form a view on how your offer stacks up. That view ships as an explicit hypothesis with its reasoning attached, never as a verdict.
The audit itself is narrower. It crawls your homepage, landing, product, pricing and contact pages, captures desktop and mobile renders, and runs 200 plus conversion signals across messaging, conversion, trust, technical, performance and content. What a signal is and how to count them is a separate post.
The money attached to each finding is a conservative monthly range, and the formula behind it changes with your business model. For a lead generation site it runs visitor-to-lead gap, qualified rate, close rate, deal value, and you can rebuild it by hand.
6,000 visitors × 0.6% lead gap × 40% qualified × 25% close × 2,000 = 7,200 a month
A range, not a forecast: it assumes your traffic and close rate hold, so halve it before it goes in a plan. Our sample is 134 businesses audited, every one arriving already suspecting a problem and mostly small, so copy faults turn up more there than across the web.
How do you use a copy verdict without over-trusting it?
Four moves, in order. Run the comparison both ways round and keep only what survives. Ask for the reasoning rather than the ranking. Ship the mechanical findings before the judgement calls. Take anything about your offer to real customers instead of to a model.
- Swap the order. Paste the two versions the other way round and ask again.
- Ask for the reason. "Which line names a feature" is checkable. "Which page is better" is not.
- Ship the mechanical faults. Headline naming an outcome, proof moved up beside its claim, a real verb on the button.
- Leave the offer question open. Take the positioning hypothesis to three customers and ask them.
Then watch the right thing. A headline change moves scroll depth and clicks on the first in-page link within about two weeks, long before the conversion rate says anything, and the sample that rate needs is bigger than most people assume.
Prefer the findings already ranked? Run a free audit.
When is a model the wrong reviewer for your copy?
When you need to know what a real person thought. Revslip runs no session recordings, no heatmaps and no moderated usability testing, so it reads what your page says rather than what a visitor did with it. Pair it with five minutes watching someone use the page.
The verdict we are not entitled to We will tell you your headline names a feature and show you the reasoning. We will not tell you your offer is wrong, because a third of paired verdicts reverse on a reorder and that is not a standard worth betting a price change on. Treat our copy notes as a shortlist to test and keep the pricing call yourself.
Questions people ask about AI copy reviews
Three keep coming up, and all ask where the line sits.
Can ChatGPT review my website copy?
It can review the text you paste. It cannot see what you did not paste: the layout, the image carrying the claim, everything below the fold. Why pasting a URL into ChatGPT is not an audit covers the gap.
Does an AI copy review replace a copywriter?
No. It replaces the first pass where someone names what is obviously broken. A writer decides what to say instead, and that choice is the part the measurement above never covers.
Will it understand my industry?
Well enough to spot a feature-led headline in any industry. Whether the outcome you name is the one your buyers want is a question for your buyers, and any tool's honest answer is a hypothesis.