What the best general model read on a specialist's ruler

A single accuracy score hides the one thing a clinician needs to know: how much the model made up.

I gave the strongest general AI model I could reach my own panoramic radiographs, blind. One board-certified oral and maxillofacial radiologist, n = 128 cases, my signed reads as the answer key. Every statement the model made was compared to my read, finding by finding, by a judge that could not see which model had written it. By design there was no overall score. Hits and inventions were counted apart.

It read roughly three findings in ten of what I read. Of the findings it asserted, roughly three in four were wrong. The prose was fluent throughout. A reader who trusted the fluency would have signed a report with an inventory of the wrong teeth in it.

Put those two facts into one accuracy number and the invention disappears. A model that reads three in ten and invents three in four can average out to a figure that reads as a pass. Gate on the inventions first and the same model stops at the door.

So the ruler I keep is this. Grade per finding, never per case. Count what the model invented before you count what it found. Never average safety into a score.

General models are not built to read radiographs and nobody should expect them to. That is exactly why the test matters: they are the models people paste images into. The specialist's answer key is the only instrument that shows what a fluent answer left out and what it made up.

One caveat I owe the reader. My own answer key has since been revised, because my labels drifted while I read the set, which is the next note. The revision may move the figures. It does not change what a single number can hide.

What I would do differently: put a version number on the answer key before the first test, so the caveat above is a footnote, not a paragraph.

The check

Three questions to ask of any accuracy number before you believe it:

1. Per finding or per case? A case with one hit and four inventions can count as correct.
2. Where did the inventions go? If they were averaged into the score, the score cannot show them. Ask for hits and inventions as two numbers.
3. Who wrote the answer key, and had they seen the model's output first? An answer key written after seeing the model grades the model's fluency, not its sight.