Skip to content
Reza Motaghi
Lab Notes3 min read

What the best general model read on a specialist's ruler

By Reza Motaghi

A single accuracy score hides the one thing a clinician needs to know: how much the model made up.

I gave the strongest general AI model I could reach my own panoramic radiographs, blind. One board-certified oral and maxillofacial radiologist, n = 128 cases, my signed reads as the answer key. Every statement the model made was compared to my read, finding by finding, by a judge that could not see which model had written it. By design there was no overall score. Hits and inventions were counted apart.

It read roughly three findings in ten of what I read. Of the findings it asserted, roughly three in four were wrong. The prose was fluent throughout. A reader who trusted the fluency would have signed a report with an inventory of the wrong teeth in it.

Put those two facts into one accuracy number and the invention disappears. A model that reads three in ten and invents three in four can average out to a figure that reads as a pass. Gate on the inventions first and the same model stops at the door.

So the ruler I keep is this. Grade per finding, never per case. Count what the model invented before you count what it found. Never average safety into a score.

General models are not built to read radiographs and nobody should expect them to. That is exactly why the test matters: they are the models people paste images into. The specialist's answer key is the only instrument that shows what a fluent answer left out and what it made up.

One caveat I owe the reader. My own answer key has since been revised, because my labels drifted while I read the set, which is the next note. The revision may move the figures. It does not change what a single number can hide.

What I would do differently: put a version number on the answer key before the first test, so the caveat above is a footnote, not a paragraph.

The check

Three questions to ask of any accuracy number before you believe it:

  1. Per finding or per case? A case with one hit and four inventions can count as correct.
  2. Where did the inventions go? If they were averaged into the score, the score cannot show them. Ask for hits and inventions as two numbers.
  3. Who wrote the answer key, and had they seen the model's output first? An answer key written after seeing the model grades the model's fluency, not its sight.

About the author

DDS, MSc · Board-certified specialist in oral and maxillofacial radiology · Independent researcher

0009-0005-3194-1879

Reza Motaghi is a board-certified specialist in oral and maxillofacial radiology who reads medical images every day, 100,000+ reports signed, and runs enterprise AI in production as Chief Innovation Officer. He builds the training data from his own reads, made together with fellow radiologists under one written standard, trains models on it, and tests them on cases they have never seen.

He published Radigrade, the registered benchmark for AI-written radiograph reports: invented findings counted first, graded by an AI judge calibrated against his own verdicts, general models in public and vendors’ systems privately on request. He also built CBCTScope, the first CBCT reading viewer with native AI-agent control.

Research recordAbout RezaContactLinkedIn (opens in a new tab)

The newsletter

Get the essays as they’re written.

Imaging AI, evaluation, and building alone, from a radiologist who reads every day and tests before he trusts. A few essays a month, nothing else.

Join for FreeFree · Unsubscribe anytime

Related

  1. Inventory before interpretation

    Blind on my own radiographs, the strongest general model reported a root canal in a tooth that was not there, then kept writing. Count before you interpret.

  2. Saying less is not seeing more

    The newest general model invented half as much on 128 of my own radiographs and missed more bone loss and caries. A single safety number cannot tell a safer reader from a quieter one. The two-direction test, in three steps.

  3. It counts the teeth and misses the disease

    Blind on 128 of my own radiographs, the strongest general model I could reach counted every implant exactly and never once called the jaw joint. The parts list is solved, the call is not. The check, in three steps.