What not to teach an AI model from your own expert reads

In the study that validated the imaging criteria for jaw joint disorders, three examiners rated the same joints for degenerative change on three kinds of image. On the panoramic radiograph they agreed on a positive call 19 percent of the time.¹ On CT, 84 percent. Same joints, same experts, same disease. What changed was what the image could show. That number is the whole idea of this article. What a good reader can call is set by the input, not by the reader.

It matters now because experts are being asked to write their judgment down as training data. The instinct is to write everything you see. You are the expert, that is why they asked. But a model trained on your labels learns whatever you wrote, and it cannot tell which of your calls came from the image and which came from you. Twenty years of pattern memory lets an expert see things the image does not support. Write those down as findings and you have taught the machine to assert what it cannot see. Hallucination installed on purpose, by the most qualified person in the room.

What the callable floor is

The callable floor is the set of calls you would make every time from this input alone, and could defend to a peer with the input and nothing else. That trains. The calls that need your history with a thousand similar cases are the ceiling. The ceiling is real, it is yours, and it is not label material.

A model trained on your work learns your floor, not your ceiling. So the first decision in teaching it is not what to teach. It is what not to. Made before the first example is written, or made for you by the first wrong answer.

The joint I left blank

A joint on a radiograph. I was certain it showed degenerative change. I left it blank. The image did not support the call. My history with a thousand similar joints did. The study above says what that history is worth on that image. A positive call that other experts would share one time in five. Label it and the model learns to name degenerative change from an image that does not show it. Then it does so every time, confidently, on someone else's patient.

Writing training examples from my own reads taught me that the hardest part of teaching a model is deciding what not to teach it. After that joint the same rule ran across every structure on the image, the sinuses, the jaws, the whole field. Where the input carried the call, I wrote it. Where I carried it, I wrote what I would do next.

Where the floor comes from

The floor is not a radiology idea. It is what supervised learning does. A model trained by imitation learns the demonstrator's mistakes along with the skill, and the mistakes compound.² Fine-tune a language model on facts it cannot ground and it learns them slowly, and as it learns them its hallucination rate rises in step.³ Its hallucinations then mirror those examples. What it was taught to assert without grounds, it asserts elsewhere.⁴

Medical labels are that kind of teaching whenever they go past the image. When the chest X-rays of 500 patients were relabelled by several radiologists, at least two disagreed on some label in over 90 percent of patients.⁵ Named diagnoses agreed far less than plain descriptions.⁶ An audit of one large public chest X-ray dataset found its labels frequently did not match what was in the images.⁷ Even the test sets that rank general machine learning models carry wrong labels, enough of them to reorder which model looks best.⁸

None of that says experts are bad. It says the confidence of an expert is not evidence. When physicians worked through easier and harder cases, accuracy fell from 55 percent to under 6, while their confidence barely moved.⁹ The model gets the confidence either way. Only the person who knows the case can tell what it was worth, which is the same tell I described in the default patient.

Radiology already wrote the rule

Every radiology report has two parts. Findings, what the image shows. Impression, what the radiologist thinks it means. The split is written into the profession's own standard for communicating imaging results, and it exists because the two are different kinds of statement.¹⁰ The findings section is the callable floor. The impression is where the ceiling is allowed to speak, in the radiologist's own name.

The field has been less tidy about certainty. Eleven clinicians once rated the 18 phrases radiologists use for doubt. The probability they read into "probable" or "cannot exclude" varied widely between them, and within the same person a week later.¹¹ Twenty-five years on, the ACR's quality commission still records that no endorsed framework exists for stating diagnostic certainty in a report.¹² That is a second reason not to write the ceiling into a label. There is no agreed way to write it.

The ceiling is real

Radiologists shown a mammogram for half a second can tell normal from abnormal better than chance. That is gist perception, and it is measured.¹³ It is also unreliable from one person to the next. When 39 radiologists repeated the half-second task, only 13 were consistent enough with themselves to meet the usual bar for reliability, and none reached good reliability.¹⁴ So the eye that sees past the image exists, and it is not the same eye in every expert. That is exactly why the framework keeps it and does not label it. A gut call is worth keeping in your own file. It is worth testing. It is not worth teaching as a fact.

Two honest counters. Models can learn signal in an image that no human sees. From retinal photographs, one predicted age, sex and blood pressure, none of which an ophthalmologist can read from the picture.¹⁵ But those labels came from the patient record, not from an expert's eye. That is a different kind of label, and the floor rule does not apply to it. And a strong model trained on a weaker supervisor's labels can outperform that supervisor.¹⁶ It recovers part of the gap. It does not learn to see what the input does not show.

Beyond the reading room

Every system trained on human decisions has a floor, and usually nobody chose it. Amazon's recruiting model, trained on ten years of the company's own hiring, learned to penalise résumés that contained the word "women's".¹⁷ A widely used health-risk algorithm learned medical need from the cost of care. At the same risk score, its Black patients were considerably sicker than its White patients.¹⁸ Toxicity ratings track the annotator's own politics.¹⁹ In each case the model learned exactly what it was shown, and what it was shown was the labeller's floor. The only choice is whether you set it on purpose.

Five decisions before the first example

The decisions are made before the first example because the pull runs the other way once you start. Each carries the evidence that shaped it, and two carry a limit.

1. From this input alone, would I make this call every time. If no, it is not a label. This is where named diagnoses fall and plain descriptions survive, which is what the agreement studies show. It has one off-ramp. When the label is an outcome, a biopsy, a follow-up scan, an entry in the record, the input may carry more than you can see. The model may find it. That is not your ceiling. It is a different label, set by the record and not by your eye.

2. Sure, but the input does not show it. Record what you would do next. Ask for more views. Defer to the next study. Describe what you see rather than name what you suspect. That is honest output, and it trains too. Learning when to defer is a research field of its own. Models that hand the uncertain case to a human have beaten the same models forced to answer everything.²⁰ Uncertainty is not deleted by the floor. It is written as the next action.

3. Keep a second file for the calls you were sure of and did not record. That is your ceiling and your value. It is the file a competitor cannot get by hiring your labels. It is also the file the gist studies say to be careful with: your gut is real, and only sometimes calibrated. Keep it, and check it against outcomes over time. Do not teach it.

4. Review a sample as a hostile peer would. Each label against the input alone, the way you would read a case cold before looking at anyone's report. If it needs your history to stand, back to step two. Real labelling programmes already gate this way. In the RSNA challenges, annotators had to agree with the reference cases at a set rate before they were allowed to label the full set.²¹ Confidence is not the test. Defensibility is.

5. Raise the floor with better input, never with looser calls. More views, more history, a second modality. The jaw joint study is the whole argument in one row: the same experts went from 19 percent positive agreement on a panoramic radiograph to 84 on CT. On high-resolution cone beam CT two radiologists agreed on every osseous sign of joint degeneration with kappa above 0.8.²² The floor rose because the input did. Nothing was relaxed.

Two things the decisions are not. They are not an argument for thinner data. Disagreement between labellers is signal, not noise. Models trained on the full spread of human labels are better calibrated than models trained on the majority vote.²³ Models that learn each annotator separately beat the averaged label.²⁴ The floor removes the calls the input cannot support. It keeps the honest uncertainty. And they are not a claim that a model can never exceed you. It can, on the floor. It cannot be taught the ceiling by being handed it as fact.

What I sign for

I read images every day, and I write training examples from my own reads. The line between the two files is the most useful thing I own. It is the same line I built into my own CBCT viewer, where the agent moves the camera and never signs the read. What the input supports, the machine may learn. What I know from twenty years of looking stays with my name on it, and I test it, on my own cases, blind, as my main research work.

Decide your floor before the first example. If you are the expert being asked to label, or the builder asking, the contact page reaches me directly. I can show you how a daily reader draws that line.

References

Footnotes

1. Ahmad M, Hollender L, Anderson Q, Kartha K, Ohrbach R, Truelove EL, John MT, Schiffman EL. Research diagnostic criteria for temporomandibular disorders (RDC/TMD): development of image analysis criteria and examiner reliability for image analysis. Oral Surgery, Oral Medicine, Oral Pathology, Oral Radiology and Endodontology, 2009. Three examiners, kappa for osteoarthritis 0.16 on panoramic radiography, 0.46 on MRI, 0.71 on CT. Positive percent agreement 19, 59 and 84 percent. https://pubmed.ncbi.nlm.nih.gov/19464658/ ↩

2. Ross S, Gordon G, Bagnell JA. A reduction of imitation learning and structured prediction to no-regret online learning. AISTATS, 2011. https://arxiv.org/abs/1011.0686 ↩

3. Gekhman Z et al. Does fine-tuning LLMs on new knowledge encourage hallucinations? EMNLP, 2024. https://aclanthology.org/2024.emnlp-main.444/ ↩

4. Kang K, Wallace E, Tomlin C, Kumar A, Levine S. Unfamiliar finetuning examples control how language models hallucinate. 2024. https://arxiv.org/abs/2403.05612 ↩

5. Hurdles to artificial intelligence deployment: noise in schemas and "gold" labels. Radiology: Artificial Intelligence, 2023. 500 patients, at least two radiologists disagreed on one or more labels in 460 of them. https://pubs.rsna.org/doi/10.1148/ryai.220056 ↩

6. Inter- and intra-observer agreement in a diagnostic labelling scheme for chest radiographs. Diagnostics, 2022. Kappa from 0.40 for interpretive labels such as atelectasis to 0.99 for descriptive ones. https://pmc.ncbi.nlm.nih.gov/articles/PMC9776917/ ↩

7. Oakden-Rayner L. Exploring large-scale public medical image datasets. Academic Radiology, 2020. https://arxiv.org/abs/1907.12720 ↩

8. Northcutt C, Athalye A, Mueller J. Pervasive label errors in test sets destabilize machine learning benchmarks. NeurIPS Datasets and Benchmarks, 2021. An average of 3.3 percent label errors across ten test sets. https://arxiv.org/abs/2103.14749 ↩

9. Meyer AND, Payne VL, Meeks DW, Rao R, Singh H. Physicians' diagnostic accuracy, confidence, and resource requests. JAMA Internal Medicine, 2013. 118 physicians, 55.3 percent correct on easier cases against 5.8 on more difficult ones, confidence 7.2 against 6.4 out of 10. https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/1731967 ↩

10. ACR Practice Parameter for Communication of Diagnostic Imaging Findings, American College of Radiology. On the report's structure and history: Brady AP. Radiology reporting, from Hemingway to HAL? Insights into Imaging, 2018. https://link.springer.com/article/10.1007/s13244-018-0596-3 ↩

11. Hobby JL, Tom BD, Todd C, Bearcroft PW, Dixon AK. Communication of doubt and certainty in radiological reports. British Journal of Radiology, 2000. https://pubmed.ncbi.nlm.nih.gov/11064655/ ↩

12. Potential frameworks for communicating diagnostic certainty in radiology reports: from the ACR Commission on Quality and Safety. Journal of the American College of Radiology, 2025. https://doi.org/10.1016/j.jacr.2025.07.027 ↩

13. Evans KK, Haygood TM, Cooper J, Culpan AM, Wolfe JM. A half-second glimpse often lets radiologists identify breast cancer cases even when viewing the mammogram of the opposite breast. PNAS, 2016. https://doi.org/10.1073/pnas.1606187113 ↩

14. Gandomkar Z et al. Reliability of radiologists' first impression when interpreting a screening mammogram. PLOS ONE, 2023. 39 radiologists, two rounds. Only 13 reached an intra-class correlation of 0.6, none reached 0.75. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0284605 ↩

15. Poplin R et al. Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning. Nature Biomedical Engineering, 2018. https://arxiv.org/abs/1708.09843 ↩

16. Burns C et al. Weak-to-strong generalization: eliciting strong capabilities with weak supervision. OpenAI, 2023. https://arxiv.org/abs/2312.09390 ↩

17. Dastin J. Amazon scraps secret AI recruiting tool that showed bias against women. Reuters, 10 October 2018. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G ↩

18. Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 2019. https://www.science.org/doi/10.1126/science.aax2342 ↩

19. Sap M, Swayamdipta S, Vianna L, Zhou X, Choi Y, Smith NA. Annotators with attitudes: how annotator beliefs and identities bias toxic language detection. NAACL, 2022. https://aclanthology.org/2022.naacl-main.431/ ↩

20. Madras D, Pitassi T, Zemel R. Predict responsibly: improving fairness and accuracy by learning to defer. NeurIPS, 2018. https://arxiv.org/abs/1711.06664. Mozannar H, Sontag D. Consistent estimators for learning to defer to an expert. ICML, 2020. https://arxiv.org/abs/2006.01862. Dvijotham K et al. Enhancing the reliability and accuracy of AI-enabled diagnosis via complementarity-driven deferral to clinicians. Nature Medicine, 2023. https://www.nature.com/articles/s41591-023-02437-x ↩

21. Lessons learned in building expertly annotated multi-institution datasets and hosting the RSNA AI challenges. Radiology: Artificial Intelligence, 2024. Annotators had to reach 60 percent agreement with ground-truth cases before labelling the full set. https://pubs.rsna.org/doi/full/10.1148/ryai.230227 ↩

22. Radiographic interpretation using high-resolution CBCT to diagnose degenerative temporomandibular joint disease. PLOS ONE, 2021. Two radiologists, Cohen's kappa 0.80 to 0.87 for osteophyte, erosion, cysts, flattening and sclerosis. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0255937 ↩

23. Peterson JC, Battleday RM, Griffiths TL, Russakovsky O. Human uncertainty makes classification more robust. ICCV, 2019. https://arxiv.org/abs/1908.07086 ↩

24. Davani AM, Díaz M, Prabhakaran V. Dealing with disagreements: looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 2022. https://aclanthology.org/2022.tacl-1.6/ ↩