Skip to content
Reza Motaghi
One-Person Startup10 min read

Why AI mistakes survive human review, and how to catch them

By Reza Motaghi

A wooden block tower standing over a gap where one block was pulled out, the block lying at its foot, a black and white image about a structure that stands on a defect nobody checked
On this page

Show an experienced breast radiologist a wrong AI score and their accuracy on that case falls from 82 to 45 percent. Show it to a trainee and it falls to 20. The study ran in Radiology in 2023, with 27 readers and a simulated AI that was wrong on purpose on some of the cases.1 Nobody in it was careless. They were agreeable.

Now think about the last thing an AI agent did for you. It wrote the code, ran the analysis, filled the table, and told you it was done. You read the summary. You looked at the part you understand. You moved on.

This article is about the part you did not look at. I run my own version of that risk every week. I build the training data from my own reads, train a vision model on it, and test it on cases it has never seen. And I split the work the way everyone does. The model takes the machine learning, I keep the medicine. In October I had a strong AI model turn one of my treatment-planning rulebooks into a decision table. It agreed with 99 of 100 lines from my past plans. A second model, briefed only to attack it, found 19 errors. After the fixes, it found 20 more. Every one of those errors had passed me, and I had been looking.

The delegated side

Here is the idea in one sentence. The errors that survive AI-assisted work do not live in your field. They live on the side you handed over.

Call it the delegated side: the part of the work you gave to the model because it is not what you do. The engineering, if you are a clinician. The clinical reasoning, if you are an engineer. The statistics, the contract clause, the tax rule. You delegated it for a good reason, and that reason is exactly why you will not find the errors in it. You read your own side closely because you can. You read the delegated side the way you read a receipt, for the total.

Two agreeable parties sit on that side, and between them nothing says no.

The first is the model. It agrees by finishing the surface. A 2024 study of five assistants found all of them sycophantic across four tasks.2 It found something worse. People, and the preference models trained on people, sometimes chose the convincing wrong answer over the right one. A 2025 test put the rate at 58 percent of cases.3 In about one case in seven the model talked itself out of a correct answer to agree with the user. A paper in Science this year measured eleven models affirming a user's actions 49 percent more often than people do.4 The latest models score better on the vendors' own tests, and the habit is not gone.

The model also agrees by saying done. In one 2026 study, coding agents claimed success on a large share of the tasks they had failed.5 Five language models used as judges could not tell the true claims from the false ones. When METR gave real maintainers 296 pull requests that passed an automated grader, the maintainers would have merged roughly half.6 Done is a claim. It is not the result.

The second agreeable party is you. Not because you are naive, but because deference is the rational response to a tool that knows more than you do about its own side. The strongest evidence here is a meta-analysis in Nature Human Behaviour of 106 experiments. On decision tasks, the human plus the AI was worse on average than the better of the two alone.7 The pair did not add a check. It added a second yes. The mechanism is plain in the lab. People overestimate how accurate a model is, and a longer explanation raises their confidence without raising their accuracy.8 When the model phrases its answer with doubt, people rely on it less and get more right.9 Even then the over-reliance shrinks. It does not vanish.

So the model says it is done, and you say fine. Now the error is a premise.

Why the error stays

An error that passes two agreeable parties is not caught later. It is built on.

The next step reads the last step's output as ground truth. Models do this to themselves. When a model's own earlier mistakes sit in its context, it makes more of them, and scale does not cure it.10 Models that reason before they answer do it less, which is why I will not tell you errors always compound. They compound wherever nothing independent reads the output between steps. A pipeline with tests catches some of them. A pipeline that trusts its own summaries catches none.

How long does an error live once it is a premise? Until someone is paid to look. The ten most-cited test sets in machine learning carry wrong labels in at least 3.3 percent of their examples.11 In the ImageNet validation set it is at least 6 percent. Thousands of models were graded against those labels for years before anyone counted. A re-check of MMLU, a benchmark that every frontier model reports, found errors in about one question in fifteen, four years after release.12 Those are the audited cases. The unaudited ones are the subject of this piece.

So how many errors are left in your work? Nobody knows, and the honest number is not zero. Ecologists count the fish they cannot see by tagging one catch and counting the tagged fish in the next. Software engineers borrowed the method in the 1990s to estimate the defects an inspection had not found. The borrowed method comes with a warning in the IEEE Transactions on Software Engineering.13 With fewer than four independent inspectors, every version of it underestimates, and the underestimate can be large. One reader is not an estimate. Two sequential passes with fixes in between, which is what I ran on my table, are not an estimate either. They prove errors remained. They do not say how many.

The four checks

Here is what I run now. Each check is a thing a working person does, and each has a point where it does not apply.

1. Review the delegated side first. Your own field gets your eyes for free. Put the review hours where you have the least reason to look, on the side you handed over, because that is where the survivors are. In my case that meant reading the model's data handling before its clinical output. I found a metric that had moved in a direction the cases could not explain. Off-ramp: when the delegated side already has an independent check that does not depend on the model. Tests the model did not write, or a second specialist. Then your hours go back to your own field.

2. Never answer the model's question with the model's answer. Agents ask. "A or B, I suggest A." Every time you answer A because it suggested A, a review did not happen. Decide B on its merits first, or ask for the evidence, or say you do not know and make it show you. The lab finding behind this is the explanation effect: a longer, more confident explanation raises your trust and not your accuracy. Off-ramp: a question inside your own field, where you can judge the answer as fast as you can read it.

3. Ask the question only your field can ask: can this result be true? You cannot check the method on the delegated side. You can check whether the result is possible. A model that gains accuracy on cases it never saw. A cost that falls when the volume rises. A sensitivity that improves while nothing about the input changed. A specialist knows what cannot happen, and that knowledge is a check the model's own side cannot run. This is how I read every number an agent gives me on the cases it has never seen. Off-ramp: none. This one always applies.

4. Brief a reader to disagree, and repeat. A second model, a different one from the builder, with the source in its hand and one brief: find what is wrong. Critics trained for the job beat human reviewers in 63 percent of cases in one 2024 test.14 They also invented bugs, so every finding goes back to the source. The reader must be different because a model asked to judge its own kind favours its own output.15 And one pass is not enough. On my table the second pass found as many as the first. Stop when a pass finds nothing serious. Off-ramp: a disposable artifact that no later step will build on. For anything that becomes a premise, run the pass.

The delegated side, four checks: review the delegated side first, never answer the model's question with its own answer, ask whether the result can be true, brief a reader to disagree and repeat until a pass finds nothing serious

Two things I changed after reading the evidence against them. I had written that errors last forever. The evidence says they last until audited, which in practice is years, and the piece now says that. And I had written that the operator defers. The evidence says novices defer, and experts often under-defer inside their own field. That is the point. The deference happens on the delegated side, and that is where check one sends you.

What the checks are not

They are not a reason to stop delegating. The model's side of my work is better than it would be if I did it. The table that carried 39 errors also carried rules I would not have written down for a year.

They are not a claim that AI makes more errors than people. People made the labels in those test sets. The claim is narrower: an error on the delegated side has fewer readers than an error on yours, and the fix is to give it one.

They are not a quality department. Four checks a person runs, not a process a team files. The moment the checks become a form, the operator signs the form the way they signed the summary.

And they are not a substitute for reading the image before you open the AI answer. That rule is for your own side. These are for the other one.

Where this comes from

Every one of the four checks has a receipt behind it in the Lab Notes. The table that agreed with the past and failed the hostile reader is check four. The treatment line nothing justified, caught by one line of code against a list two specialists confirmed, is check three in its cheapest form. The search engine I did not build, because the evidence said unfiltered retrieval makes a strong model worse, is check one applied before the work existed. And a better agent needs more supervision, not less for the same reason this piece exists. The better the delegated side gets, the less reason you have to look at it.

Today I build the training data from my own reads, train the model on it, test it on cases it has never seen, and ship tools I read with. The delegated side of that work gets a hostile reader before I do. If your work has a delegated side and you want to compare notes on who reads it, the contact page is open.

References

Footnotes

  1. T. Dratsch et al., "Automation Bias in Mammography: The Impact of Artificial Intelligence BI-RADS Suggestions on Reader Performance", Radiology 307(4):e222176, 2023, https://doi.org/10.1148/radiol.222176. ↩

  2. M. Sharma et al., "Towards Understanding Sycophancy in Language Models", ICLR 2024, https://arxiv.org/abs/2310.13548. ↩

  3. A. Fanous et al., "SycEval: Evaluating LLM Sycophancy", Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society 8(1):893-900, 2025, https://doi.org/10.1609/aies.v8i1.36598. ↩

  4. M. Cheng et al., "Sycophantic AI decreases prosocial intentions and promotes dependence", Science 391(6792):eaec8352, 2026, https://doi.org/10.1126/science.aec8352. ↩

  5. L. Advani, "From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents", arXiv preprint, 2026, https://arxiv.org/abs/2606.09863. ↩

  6. P. Whitfill et al., "Many SWE-bench-Passing PRs Would Not Be Merged into Main", METR research note, 10 March 2026, https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/. ↩

  7. M. Vaccaro, A. Almaatouq, T. Malone, "When combinations of humans and AI are useful: A systematic review and meta-analysis", Nature Human Behaviour 8(12):2293-2303, 2024, https://doi.org/10.1038/s41562-024-02024-1. ↩

  8. M. Steyvers et al., "What large language models know and what people think they know", Nature Machine Intelligence 7(2):221-231, 2025, https://doi.org/10.1038/s42256-024-00976-7. ↩

  9. S. S. Y. Kim et al., "'I'm Not Sure, But...': Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust", ACM FAccT 2024, https://doi.org/10.1145/3630106.3658941. ↩

  10. A. Sinha et al., "The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs", ICLR 2026, https://arxiv.org/abs/2509.09677. ↩

  11. C. G. Northcutt, A. Athalye, J. Mueller, "Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks", NeurIPS Datasets and Benchmarks Track, 2021, https://arxiv.org/abs/2103.14749. ↩

  12. A. P. Gema et al., "Are We Done with MMLU?", Proceedings of NAACL 2025 (long papers), 5069-5096, 2025, https://doi.org/10.18653/v1/2025.naacl-long.262. ↩

  13. L. C. Briand et al., "A comprehensive evaluation of capture-recapture models for estimating software defect content", IEEE Transactions on Software Engineering 26(6):518-540, 2000, https://doi.org/10.1109/32.852741. ↩

  14. N. McAleese et al., "LLM Critics Help Catch LLM Bugs", arXiv preprint, 2024, https://arxiv.org/abs/2407.00215. ↩

  15. A. Panickssery, S. R. Bowman, S. Feng, "LLM Evaluators Recognize and Favor Their Own Generations", NeurIPS 2024, https://arxiv.org/abs/2404.13076. ↩