Skip to content
Reza Motaghi
One-Person Startup10 min read

How to check an AI answer that sounds sure

By Reza Motaghi

A large arrow sign pointing one way and a small house standing the other way in the dark, a black and white image about looking where the confident pointer does not
On this page

In 2025 a research group put eye trackers on twelve radiologists and gave them AI support while they read screening mammograms. The readers looked at the real cancers longer, not shorter. Their fixation on the lesions rose from 4.4 to 5.4 seconds, and they found more of them.1 A year later a British group ran the mirror experiment in Radiology. Ten radiologists read mammograms with AI prompts that were sometimes wrong. With a wrong prompt on screen, their sensitivity for the cancers the AI had missed fell from 71 to 39 percent.2 Their fixations on the case dropped too.

Same instrument, same signal, opposite result. The only difference was whether the tool was right that day, and nothing on the screen said which day it was.

I evaluate frontier vision models blind, against a two-radiologist reference. I also train a vision model on my own reads. The first one I trained learned how I write a report and nothing about the image, and the output read as fluently as my own. The test caught it. The tone never did. That is the failure this article is about, and it is not confined to radiology.

What the separate channel is

Confidence and correctness travel on separate channels. Every answer a model gives carries two things at once: what it claims, and how sure it sounds. The second is written into the prose, the formatting, the absence of hedges. The first is only in the world, and the world is not on the screen.

A person reads both channels together, because in a human colleague they are usually coupled. A senior reader who sounds sure has earned it, most days. A model's tone is not coupled to anything. It is trained to sound the same on a good day and a bad one, which is why the same tone that sharpens your attention when the model is right dulls it when the model is wrong. The separate channel is the name for that gap, and the four checks below are built to ignore the second channel entirely.

The report that read like a good one

The receipt is a model, not a patient. The first vision model I trained on my own reads came back writing reports in my exact format. It used my headings, my phrasing, my order of findings. Its loss curve fell the way a good run's loss curve falls. Reading its output, nothing wavered.

It had not looked at the image. Cases it had trained on scored the same as cases it had never seen, which is the proof that the picture never entered the learning. Every surface signal of success was intact the whole time. The only thing that told me was a held-out test I had written before training started, with the gates decided in advance. I wrote about the three runs it took to learn that in the lab notes. The point here is narrower. If I had graded that model by how its reports sounded, it would have passed.

Where the pull comes from

The pull is measurable, and it lands on attention before it lands on judgment. In the eye-tracking study above, the wrong prompt did not only change the readers' conclusions. It changed where they looked. Fixation rate and fixation duration both fell when the wrong prompt was present.2 The eyes left the cancer before the mind did.

A second team corrupted the AI's advice on purpose. In a European Radiology study, six radiologists read mammograms with AI support, and in 30 percent of the cases the advice had been shifted by one category.3 Readers accepted the wrong call about a third of the time. Junior readers were caught more often than senior ones, and the senior ones were not immune.

Both studies were stress tests. The AI was wrong far more often than a deployed tool is, on case sets built to be hard. They show how bad it gets when the tool is wrong, not how often it happens on an ordinary morning. Read the numbers as a ceiling, not an average.

The cleanest test of the channel itself comes from outside medicine. A user study at AAAI in 2026 had 184 people solve problems with an AI helper that showed a confidence score.4 When the confidence was calibrated, meaning high when the helper was right and low when it was wrong, people's accuracy rose by about 20 points. When the confidence was unearned, it rose by 2. And the unearned kind was the one people followed. The same study found the mirror error: false modesty made people reject advice that was correct.

So the persuasive ingredient is not confidence. It is unearned confidence, and you cannot tell the two apart by listening. A larger study of reliance found the pull is strongest in people who already trust the tool.5 That is a fair description of anyone who has used a good tool for a year.

Every field has a tone

The tone that gets past a radiologist gets past everyone else too. Close to 30 percent of AI-generated software dependency suggestions name package versions that do not exist, according to a 2026 supply-chain report.6 One invented package was downloaded more than 30,000 times before anyone checked that it was real. The suggestion was formatted like every real one.

A consulting firm delivered a footnoted report to a national government in 2025. It carried a fabricated quote from a court judgment and citations to papers that do not exist.7 The report was accepted and paid for until an outside academic checked the references. The firm refunded part of the fee. The tone was professional throughout.

Lawyers, whose job is checking citations, have filed briefs built on case law that does not exist, and courts have fined them for it.8 If trained citation checkers can be talked past their own discipline by tone, the door is wide.

The mirror case exists too, and it matters for the checks. A cockpit runway alert that fires falsely loses the crew's trust, and a maker's own engineer said so in 2025.9 A single visible error from a tool can make people abandon it even when it remains better than they are. Blanket suspicion is not a fix. It misfires exactly as often as blind trust does.

Four checks that ignore tone

The checks run the same on an answer that sounds sure and on one that hedges. That is the whole design. Two of them I changed after reading the evidence against my first version.

1. Write what would change your mind before you look. One line, before the model's answer is on screen: what would make me accept this, what would make me reject it. Then the tone of the answer cannot move the bar, because the bar was set first. This is the older habit of reading before you open the AI answer, applied to the decision rather than the read. A study of AI-assisted workflows found that documenting the initial call before seeing the AI's output was one of the few things that protected the reader afterward.10 Off-ramp: on a routine case with a tool whose misses list you know, you do not need the line. Write it where the answer would change what you do.

2. Ask what it gets wrong, not what it got right. Earned confidence comes with a misses list: the kinds of cases the tool is known to miss, measured on cases it had never seen. If the vendor, the colleague or the model itself cannot show you one, treat every confident call as unearned and run the other three checks on it. My first version of this check said to treat confidence as a warning sign. The AAAI study changed it. Calibrated confidence is the best single help on record, so the question is not whether the answer sounds sure but whether the sureness was earned. Off-ramp: a misses list is a document, not a feeling. A colleague's assurance that the tool is good is not one.

3. Look where it does not point. A confident prompt pulls your eyes to it. In the eye-tracking study, the cancers that were missed were the ones the AI had left quiet. So after you have looked where the tool points, sweep the region it left alone, the way an incidental finding is found in the part of the image nobody ordered. Off-ramp: when the tool is right most of the time, its prompts are a real help, and the 2025 study shows readers used them well.1 The sweep is for the categories on the misses list, not for every case.

4. Audit the confident calls hardest. In your own review, whether it is a sample of reports, a batch of merged pull requests or a month of approved drafts, pull the answers that sounded surest and check those first. Those are the ones that got past you. The corrupted-advice study found the readers most often caught were the ones with the least experience of the tool, but nobody was immune.3 Off-ramp: this is a sample, not a re-read of everything. Ten confident answers a month, checked against the source, is enough to learn the tool's tone.

The separate channel, four checks: write what would change your mind before you look, ask what it gets wrong, look where it does not point, audit the confident calls hardest

What the checks are not

They are not a mood. A reader who distrusts every confident answer makes the mirror error, rejecting good advice and abandoning good tools, and the evidence on that is as solid as the evidence on blind trust. The checks are specific acts, and they end.

They are not an argument against AI support. The 2025 study is the proof: when the tool is usually right, readers look longer at the right places and find more. A tool with a known misses list and calibrated confidence is worth having. The checks exist because you cannot hear which kind you have.

They are not everyday rates. The two mammography studies made the AI wrong on purpose, far more often than any deployed system is. Quote them as what they are, a demonstration of how far attention moves when the tone is wrong, not a measurement of a normal week.

What I sign for

I read medical images every day, and I evaluate frontier vision models blind, against a two-radiologist reference, before I trust anything they say. A frontier general model I tested writes in a tone that never wavers, and it stated invented findings in the same calm tone as true ones. Nothing in the tone told me which sentences. My own read, written before I opened the model's, did.

A radiologist signs for what was seen. The tone of a report was never the thing being signed. That is the same line I built into my own CBCT viewer, where the agent moves the camera and the human reads. Run the second check on the tool you rely on most. If it cannot show you a misses list, you now know what its confidence is worth. If you build or evaluate models that read images and want a radiologist who builds on your side of the table, the contact page reaches me directly.

References

Footnotes

  1. Gommers JJJ et al. Influence of AI decision support on radiologists' performance and visual search in screening mammography. Radiology, 2025. Twelve radiologists, 150 cases; area under the curve 0.93 unaided to 0.97 with AI support; fixation time on lesion regions 4.4 to 5.4 seconds. https://pmc.ncbi.nlm.nih.gov/articles/PMC12314766/ ↩ ↩2

  2. Taib AG et al. Automation bias in action: eye tracking of humans reading screening mammograms with and without AI prompts. Radiology, 2026. Ten breast radiologists, sixty cases seeded with false-negative and false-positive prompts; median sensitivity for AI-missed cancers 71 to 39 percent with the wrong prompt present; fixation rate 0.47 to 0.44 per second and mean fixation duration 0.56 to 0.54 seconds. The authors describe the work as a pilot. Read via AuntMinnie's report, 14 July 2026, the journal page being access-walled. https://www.auntminnie.com/clinical-news/womens-imaging/article/15830012/incorrect-ai-suggestions-influence-reader-performance-on-mammography ↩ ↩2

  3. Pesapane F et al. Evaluating cognitive biases in AI-assisted mammography interpretation: a simulation reader study of explainable AI across radiologist experience levels. European Radiology, 2026. Six radiologists at one centre, AI advice shifted by one BI-RADS category in 30 percent of cases; automation bias 36.1 percent, anchoring 33.9; juniors 41.7 versus seniors 30.0; saliency heatmaps roughly halved both, to 17.8 and 17.2. Read via AuntMinnieEurope, 2 June 2026. https://www.auntminnieeurope.com/imaging-informatics/artificial-intelligence/article/15826533/ai-can-mislead-radiologists-and-make-the-error-more-persuasive ↩ ↩2

  4. Fregosi A, Vicente A, Campagner A, Cabitza F. Too sure for our own good: a user study on AI confidence and human reliance. Proceedings of the AAAI Conference on Artificial Intelligence, 2026. 184 participants; well-calibrated confidence raised decision accuracy by about 20 percentage points, miscalibrated confidence by about 2; miscalibrated low confidence pushed participants to reject correct advice. https://ojs.aaai.org/index.php/AAAI/article/view/38798 ↩

  5. Pearson J et al. Examining human reliance on artificial intelligence in decision making. Scientific Reports, 2026. 295 participants; most used AI guidance strategically; over-reliance concentrated in those with favourable prior attitudes toward AI. https://pmc.ncbi.nlm.nih.gov/articles/PMC12881489/ ↩

  6. Sonatype, State of the Software Supply Chain, 2026, and Lasso Security research, as reported by Steve Poole, foojay.io, 9 April 2026. Close to 30 percent of AI-generated dependency recommendations reference package versions that do not exist; one hallucinated package downloaded more than 30,000 times in three months. https://foojay.io/today/why-java-developers-over-trust-ai-dependency-suggestions/ ↩

  7. CFO Dive. Deloitte refunds over $60K for report with AI errors, Australian government says. 21 October 2025. About AU$97,000 of a AU$440,000 contract; a fabricated court quote and citations to papers that do not exist. https://www.cfodive.com/news/deloitte-refunds-60k-report-ai-errors-australian-government-accounting/803321/ ↩

  8. PlatinumIDS. 1,227 fabricated citations and counting: inside the AI hallucination crisis hitting courts worldwide. 2 April 2026. Sixth Circuit sanctions in Whiting v. City of Athens, Tennessee, March 2026, $15,000 per attorney plus fees for more than two dozen fake or misrepresented citations. A legal-industry compilation of named court rulings. https://blog.platinumids.com/blog/ai-hallucination-crisis-courts-2026 ↩

  9. Insurance Journal. Will airport runway close calls spur wider adoption of cockpit alert technology? 7 October 2025. A Honeywell engineer on false alarms: "confidence in our system is eroding". https://www.insurancejournal.com/news/national/2025/10/07/842942.htm ↩

  10. Song, Bernstein, Sheppard, Bruno, Baird. Brown Radiology, Psychology and Law Lab. medRxiv preprint, posted 22 May 2026, not yet peer reviewed. 2,347 mock jurors, 22 workflow conditions; documenting the radiologist's initial interpretation before AI review lowered perceived liability, p = 0.0125. https://www.medrxiv.org/content/10.64898/2026.05.20.26353717v1.full.pdf ↩