Skip to content
Reza Motaghi
One-Person Startup11 min read

How to tell whether a medical AI ever looked at the image

By Reza Motaghi

An empty picture frame on a bare wall, a black and white image about an answer given to a picture that was never looked at
On this page

In 2021 three researchers audited AI detectors for COVID-19 on chest radiographs. The detectors had passed their tests. Then the team moved one small thing between a positive film and a negative one. It was the laterality marker, the L or R a technologist places in the corner. The verdict moved with the marker, and it moved more than it did for a random patch of the same size.1

The letter was not the point. The style of the marker was, because the positive and negative films had come from different sources with different markers. The detector had looked at the image and used the wrong part of it. Testing on films from a new hospital did not reliably show that either, the authors wrote in Nature Machine Intelligence.

That is the expensive failure, and the field has known about it for five years. This article is about a cheaper one that hides behind the same kind of score. A medical AI can pass an imaging exam without looking at the image at all. The score cannot tell you which happened. A changed image can, and it takes an afternoon.

I met the cheaper failure on my own vision model first. The first one I trained on my own reads wrote reports in my exact shape. Then I tested it on cases it had never seen. It had learned how I write and nothing about what I look at. The test caught it, and nothing before the test had.

What the changed image is

The changed image is a test in four moves, run on an imaging AI before its score is believed. Take the image away and ask the question again. Swap in another patient's real scan that shows the opposite answer, and keep the question. Cover the region the answer depends on, then a region it does not. Then report what the answers did, beside the score.

It is not an accuracy test. Accuracy records that the answer was right. The changed image shows what the answer was right about: the picture, the words around it, or a corner that should have meant nothing. Any buyer can ask for it.

The exam that never needed the image

The outside evidence arrived at scale this year, and it says what my test said. An audit of nine systems on chest radiograph yes-or-no questions, posted as a preprint in June, ran the removal test on all of them.2 A text-only model that never saw the image came within 5.7 points of the best image-reading one. Of the six vision-language models, four used the image in part, one was unstable, and one ignored it.

Even the four that used the image leaned on it for a minority of their right answers. Cover the region a radiologist had marked, and between a sixth and a third of those answers flipped. The authors call the advantage of the image modest and uneven. They do not call it zero.

The headline version came earlier. In 2023 the journal Radiology reported that a chatbot scored 69 percent on a radiology board-style exam and nearly passed.3 The paper states its method plainly: the questions were written without images. Every one of its wrong answers used confident wording.

A year later a study put the Japan Diagnostic Radiology Board Examination to a vision model and to its own text-only version.4 The image model scored 45 percent and the text-only one 41. The difference was not significant, the authors reported in the Japanese Journal of Radiology. The image model was right alone on some questions and the text model on others.

A right answer does not settle it either. On the New England Journal's image challenge a vision model matched physicians, a study in npj Digital Medicine found.5 In about a third of the cases it got right, its reasoning was flawed. In about a quarter, its reading of the image was.

Why a text path always exists

None of these systems cheated. Paired images and text always offer a text path to the answer, because medicine writes the text that way. In 1975 a study in the British Medical Journal followed 80 new outpatients and found that the history alone gave the final diagnosis in 66 of them.6 The examination and the laboratory added little.

A question stem written by a physician carries the history, and an AI model that reads the stem well is doing what a clinician does. The problem is not the text path. The problem is that the score cannot say which path was taken, and a buyer of an imaging AI is paying for the image path.

The horse, the blinders and the second reader

The oldest version of this test is from 1911. A horse named Hans appeared to answer questions, and an investigator named Oskar Pfungst tested him by changing what the horse could see.7 When the questioner knew the answer, Hans was right 98 percent of the time. When nobody present knew it, 8 percent. Blinders that stopped the horse seeing the questioner had the same effect as an ignorant questioner.

In 2019 a group writing in Nature Communications found the same horse inside an image classifier.8 The classifier called a picture a horse because of a source tag printed on about one fifth of the horse photographs. Remove the tag and the horse call went away. Paste it onto a car and the car became a horse. Standard accuracy metrics could not tell that classifier from one that looked at the animal.

Radiology has its own blinders. The second reader in a screening programme is blinded to the first read, so the second opinion is about the image and not about the first opinion. The changed image is that instinct turned on an AI model. Change what it is shown, and watch.

The changed image, four checks

Each check is something a working person does with a question set, a vision model and an afternoon. One of them I changed after reading the evidence against it, and the check says which.

1. Remove the image and ask again. Same question, same options, no picture. If the score barely moves, one of two things is true. The AI model never looked, or the question never needed it. The exam cannot tell you which, so the first job is to sort the questions. On Polish specialist-exam questions the same model lost about one point without the image where the text could answer. Where only the image could, it lost about 29.9 Write beside the drop how you asked, because the runtime is part of the instrument: told to guess or not, reasoning on or off, which build. Score refusals apart from wrong answers. Off-ramp: a set built on the history, like most physician vignettes, gives a small drop that describes the exam and not the AI model. Take the image away and an AI model does what it does with any missing fact. It writes the most common case, and the score barely notices.

2. Swap in the opposite scan. This is the check that changed. As I first drafted it, the swap put in another patient's scan with the same finding, and a reader was expected to hold its answer. An AI model that follows the words also holds its answer, so a same-label swap cannot separate a reader from a word-follower. The chest audit's own authors add that a genuine reader can be scored as ignoring the image on a diffuse or bilateral finding.2 So the check is now the opposite scan. Same question, unchanged words, another patient's real scan that shows the opposite answer. For a finding question that is a normal scan, and for a normal question a scan with the finding. A reader has to change its answer. A word-follower gives the same answer and is now wrong. Score how often the answer followed the words. The design has a precedent. In 2017 a computer vision group paired every question with two similar images that had different answers, and every model it tested scored lower on the pairs.10 Off-ramp: a flip can mean an unfamiliar image rather than lost evidence, which is why the scan is real and never edited. And someone who reads scans has to choose the opposite scan for every question. That person is the scarce part.

3. Cover the region that matters, then one that does not. This is the check that catches the marker. Have a radiologist mark the region each answer depends on. Cover it and ask again, then cover a region of the same size that should not matter and ask again. A reader's answer moves under the first cover and holds under the second. The detector in the opening would have held its verdict with the lungs covered and lost it with the corner covered. Off-ramp, and it is the one that matters most: a zero is not proof. In the chest audit the reference radiologist's own answers moved under the cover on about a quarter of cases, and a second radiologist's on none.2 The authors call their rate a conservative lower bound. So compare the vision model against a radiologist's own run of the same test, never against zero. And never write that an AI model which held its answer did not look.

4. Report the drop beside the score. The score goes in the report. Beside it goes the drop from each of the three checks. Beside that goes the same model's drop on the image-dominant questions alone, so an easy exam is not mistaken for a blind model. On Japanese physician licensing questions, proprietary models scored 87.5 percent with the image and 81.8 without it.11 The authors traced the small drop to question stems that already carried enough of the answer. On Italian questions with a blank in place of the image, one model lost 27.9 points and another 2.4.12 The largest drop was largely a model declining to answer, and every model still described findings it could not have seen. Both numbers are honest, and neither means anything without the other. Off-ramp: when a set has no image-dominant items, the report says so. The drop is then a fact about the exam and not the AI model.

The changed image, four checks: remove the image and ask the question again, swap in another patient's real scan with the opposite answer and keep the question, cover the region that matters and then one that does not, report the drop beside the score and beside the drop on image-only questions

What the checks are not

They are not proof that AI models do not look. In the chest audit four of the six vision-language models used the image. On the Polish image-dominant questions a frontier model gained nearly 30 points from seeing it. The claim is narrower. A score cannot tell you, and the checks can.

They are not a replacement for accuracy. The audit's authors call their test a complement, and they do not conclude that the systems are unfit for use. An AI model that looks and is wrong has passed the changed image and failed the patient.

They are not a memorisation check. Public exam questions may sit in the training data. A general benchmark paper found a model scoring far above its own language backbone with no image at all, and read that as memorised items.13 A small drop can be a memorised item as well as a text path. The changed image does not separate those two. Only a set the AI model has never seen does.

They are not a case against the text path. Clinicians read the history first, and an AI model that does the same is not cheating. The checks ask only that the buyer know which path the score came from.

What I sign for

Today I build the training data from my own reads, train the model on it, test it on cases it has never seen, and ship tools I read with. The first one learned how I write a report and nothing about the image. The test caught it. It is the test I now run before I believe a score, my own included.

Ask for the drop before the demo. The exam score is what the vendor brings. The drop is what you ask for, and whether it exists tells you whether anyone ever asked the question. If you are weighing an imaging AI on its score and want a reader who caught this on his own model, the contact page reaches me directly.

References

Footnotes

  1. DeGrave AJ, Janizek JD, Lee SI. AI for radiographic COVID-19 detection selects shortcuts over signal. Nature Machine Intelligence 3(7):610-619, 2021. Detectors relied on confounders such as laterality markers, looked accurate, and external testing alone did not reliably reveal it. Moving the marker between a positive and a negative film moved the verdict more than a random patch swap did. https://doi.org/10.1038/s42256-021-00338-7 ↩

  2. Lotfinia M, Ziegelmayer S, Adams L, Truhn D, Maier A, Tayebi Arasteh S. Vision-language models for chest radiography do not always need the image. arXiv preprint, 2026 (v1 16 June, v2 19 June). A text-only model came within 5.7 accuracy points of the best multimodal one. The audit sorts nine systems, including text-only and image-only controls, by how their answers respond to image interventions. Of the six vision-language models, four used the image in part, one was unstable and one ignored it. Causal grounding rates for the four ran from 17.5 to 33.5 percent, and the reference radiologist's own rate was 23.1 percent. https://arxiv.org/abs/2606.17710 ↩ ↩2 ↩3

  3. Bhayana R, Krishna S, Bleakney RR. Performance of ChatGPT on a Radiology Board-style Examination: Insights into Current Strengths and Limitations. Radiology 307(5):e230582, 2023. A chatbot scored 69 percent on 150 board-style questions written without images and used confident wording in all 46 of its wrong answers. https://doi.org/10.1148/radiol.230582 ↩

  4. Hirano Y, Hanaoka S, Nakao T, Miki S, Kikuchi T, Nakamura Y, Nomura Y, Yoshikawa T, Abe O. GPT-4 Turbo with Vision fails to outperform text-only GPT-4 Turbo in the Japan Diagnostic Radiology Board Examination. Japanese Journal of Radiology 42(8):918-926, 2024. On 139 board questions the image model scored 45 percent and the text-only model 41 percent, P = 0.44. Sixteen questions were answered correctly only by the image model and 11 only by the text model. https://doi.org/10.1007/s11604-024-01561-z ↩

  5. Jin Q, Chen F, Zhou Y, et al. Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine. npj Digital Medicine 7(1):190, 2024. A vision model scored 81.6 percent against 77.8 for physicians on the NEJM Image Challenge, with flawed rationales in 35.5 percent of its correct answers and flawed image comprehension in 27.2 percent. https://doi.org/10.1038/s41746-024-01185-7 ↩

  6. Hampton JR, Harrison MJ, Mitchell JR, Prichard JS, Seymour C. Relative contributions of history-taking, physical examination, and laboratory investigation to diagnosis and management of medical outpatients. British Medical Journal 2(5969):486-489, 1975. History alone gave the eventual diagnosis in 66 of 80 new outpatients, the examination in 7 and laboratory tests in a further 7. https://pubmed.ncbi.nlm.nih.gov/1148666/ ↩

  7. Pfungst O. Clever Hans (The Horse of Mr. Von Osten). Translated by Rahn CL. Henry Holt and Company, 1911. The horse answered 98 percent correctly when the questioner knew the answer and 8 percent when nobody present did, and blinders that hid the questioner had the same effect. Project Gutenberg text. https://www.gutenberg.org/ebooks/33936 ↩

  8. Lapuschkin S, Wäldchen S, Binder A, Montavon G, Samek W, Müller KR. Unmasking Clever Hans predictors and assessing what machines really learn. Nature Communications 10:1096, 2019. A horse classifier keyed on a source tag present in about one fifth of the horse pictures. Removing the tag removed the call and adding it to a car produced a horse call. https://doi.org/10.1038/s41467-019-08987-4 ↩

  9. Pokrywka J, Grzybowski Ł, Lasik A, Kubis M, Kaczmarek JI, Kusa W. Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence. arXiv preprint, 2026 (13 August). The best model scored 42.9 percent from the answer options alone, against a chance level of 20. Table 4, no extended reasoning: 82.19 to 83.56 percent on text-sufficient and complementary items, 42.86 to 72.14 on image-dominant items, a difference of 1.4 against 29.3 points by the author's subtraction. The models were not told the image was missing. https://arxiv.org/abs/2608.12928 ↩

  10. Goyal Y, Khot T, Summers-Stay D, Batra D, Parikh D. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. CVPR 2017, pp. 6325-6334. Pairing each question with two similar images that have different answers exposed models that had learned language priors. https://doi.org/10.1109/CVPR.2017.670 ↩

  11. Xun Y, Liu J, Niu Q, et al. JMed48k: A Multi-Profession Japanese Medical Licensing Benchmark for Vision-Language Model Evaluation. arXiv preprint, 2026 (21 May, revised 28 May). On 568 image-bearing physician items, proprietary models scored 87.5 percent with the image and 81.8 without. Across professions the net image-removal effect ran from 5.7 to 39.8 points, tracking how much the question stem already said. https://arxiv.org/abs/2605.22080 ↩

  12. Felizzi F, Riccomi O, Ferramola M, et al. Are Large Vision Language Models Truly Grounded in Medical Images? Evidence from Italian Clinical Visual Question Answering. Workshop on Multimodal Representation Learning for Healthcare (MMRL4H), EurIPS, 2025. Blank images cost four models between 2.4 and 27.9 points on 60 questions while all still explained visual findings confidently. The largest drop was largely a model refusing to answer. https://arxiv.org/abs/2511.19220 ↩

  13. Chen L, Li J, Dong X, et al. Are We on the Right Way for Evaluating Large Vision-Language Models? arXiv preprint, 2024 (29 March). Many multimodal benchmark items are answerable without the image. One model scored 43.6 percent on MMMU without images against 17.9 for its own language backbone, which the authors read as memorised training samples. https://arxiv.org/abs/2403.20330 ↩