What AI companies pay radiologists for, and what to check
By Reza Motaghi

On this page
In 2011 a group of radiologists finished the reference database that a generation of lung-nodule software would be built and tested on. Four thoracic radiologists read every scan, first blind, then with each other's marks in view, and nobody was made to agree. Of 2,669 lesions that at least one of them marked as a nodule, 928 were marked by all four.1 The authors wrote in Medical Physics that fewer would have been marked with three readers, and more might have been with five.
The answer key changed with the number of people who wrote it, and the people who wrote it said so. That is the fact to hold when the next forecast says the machine no longer needs the specialist. Ten years ago serious people said stop training radiologists. In September 2026 the job postings asked radiologists for something else: write the reference report, write the grading criteria, write the hard cases.
This piece is for any specialist whose expertise is now being asked for as grading material, and for the founder writing that budget. It says what the postings ask for and why an AI model cannot check its own answer key. Then four moves to make before your expertise goes into someone else's exam.
I read medical images every day, and I wrote the answer key for my own AI model before any machine was graded against it. I wrote the label contract for my dataset, read every case, and took each read to consensus with a second reader. The reference the two of us sign is frozen before any AI report is graded against it. When the model and that reference disagree, one of them is wrong, and the two of us say which.
What the answer key is
The answer key is everything that has to exist before an AI model's read can be graded. A reference read for each case, written by someone qualified to write it. A written standard for what a correct read must state and must not invent. A set of cases chosen as the exam and kept out of training. And a person who decides when the reference and the machine disagree. The grading itself is the last and smallest step, and on the postings it is one line of six.
An AI model cannot produce its own key. A key the model wrote is the model marking its own homework. A key nobody checked is a guess with a column heading. A key that is only a report can be matched by learning the report and not the image. Someone who reads the image has to write it. However capable an AI model becomes, someone qualified has to write the answer key and decide who is right when model and key disagree. That sentence is the whole argument.
Note what the sentence does not say. It does not say the person who writes the key out-answers the machine, and on one lab's benchmark the machine won. The claim is smaller and harder to get around: someone decides what right looks like, and someone checks whoever does the marking.
What the market advertised in September 2026
In September 2026 one employer's posting asked for radiologists at 200 to 400 dollars an hour.2 Board certified or board eligible, a residency completed, two years of reading independently. The posting reached radiologists through an expert-staffing marketplace, and it named no client. By the end of the month it was closed. Advertised, not hiring, and a dated range, never a paid rate.
Six tasks were printed on it. Three of them write the key or the exam: reference report authoring, grading criteria and guideline authoring, difficult-case writing. One of them is marking: output review, which the posting described as judging AI reads for invented findings, missed findings and unsupported certainty. The other two are reading and labelling. The posting described reference report authoring as writing the report a study should produce, to serve as ground truth. That is a count of lines, not of hours, and it still says what the buyer thinks is scarce.
The posting also said why: imaging is where a confidently wrong answer costs most, and the standard the AI models are graded against is written by radiologists.2 Recruiting copy persuades as well as describes, so take it as the buyer's own view of the job.
Two other postings say the same thing from other angles. In April 2026 a posting for an unnamed AI lab asked radiologists for prompt design, reference answers at attending level, rubric grading and calibration sessions.3 In September a posting open to any specialty offered 150 dollars an hour.4 It described the core task as breaking the ideal answer into checkable criteria, so that grading is consistent instead of impressionistic. On all three lists, writing sits at or above marking. Paid to write the key is truer than paid to grade.
Graded truth is the scarce input, and the postings are that scarcity with a price attached. The price is for judgment written down, not for reading speed.
Why an AI model cannot check its own key
The clearest evidence comes from a lab that tried hard to build a good key. In April 2026 one lab published a clinical benchmark of 525 tasks, each authored by a physician, reviewed, then adjudicated, three or more physicians in all.5 For the difficult examples two physicians had to confirm independently that the AI model had really erred. Authoring physicians sometimes flagged flaws the others did not agree with. The last phase cut the criteria that were subjective or a matter of preference.
On that benchmark the best system scored 59.0 against 43.7 for physicians writing answers to the same tasks, specialty matched, unhurried and with the web at hand.5 And the marking was done by an AI model. So the doctors did not out-answer the machine, and a human did not do the marking. What the humans did was write the key, three deep, and confirm each hard-case error twice. Out-answering the doctor is not the end of the job. It is where the job starts.
Read that result at its size. The benchmark is chat text, not images, radiology is a handful of its tasks, and the lab that built it also built the top-scoring system. None of that changes what the humans had to do to make the score mean anything.
One year earlier the same lab's first benchmark had its criteria written mostly by single physicians and not checked by others.6 It reported that an AI grader agreed with physicians about as well as physicians agreed with each other on the shared criteria. In one year the builders moved from one physician per criterion to three phases of physicians. Grading did not become less expert labour as the AI models improved. It became more.
Whether an AI model can do the marking at all is contested, and I will not state it as settled. A blinded evaluation of 620 real clinical queries used practising physicians, each matched to the question's specialty.7 The language-model judges differed from the experts systematically, though they agreed on which system was best. A multicentre study of more than 400 physicians found that rankings shifted with the graders' seniority and practice environment.8 Its authors concluded that AI graders could pre-screen but not substitute. The reconciling reading is narrow. An AI model marks against a written criterion about as well as one physician does. It does not reproduce an expert's open side-by-side judgment. So mark its work becomes check the marker.
The same job in law, software and the sciences
The door is wider than radiology. In law, practising attorneys scored rubrics and compared pairs of answers on real legal tasks. The method of grading changed the result.9 In the sciences nearly 1,000 experts wrote the questions for an exam meant to be the hardest there is.10 Its own audits found the key needed several expert readers. In software 93 experienced developers screened a coding benchmark in 2024.11 By July 2026 the lab behind it had stopped reporting on it, after an audit found flaws in most of its hardest tasks.12
Same mechanism in all three. The people who do the job are paid to write the exam and audit the key. The key is weakest exactly where the AI model and the key disagree, and that is where the four moves come from.
The answer key, four moves
These are the moves a specialist makes when their expertise is asked for as grading material. Each carries its evidence and its off-ramp, the case where not to run it. One of them I changed after reading the evidence against my first version, and I say which.
1. Write the reference read before you see the AI model's. A reference written after the model's answer is a correction, not a reference. Very experienced radiologists reading mammograms with a purported AI suggestion in view fell from 82.3 to 45.5 percent correct when the suggestion was wrong.13 That was a reading task, not a reference-writing task, and it is the same mechanism. Read the image before you open the AI answer, and write the reference down before you open it. Off-ramp: if the contract only asks for output review, you are marking, not writing the key. Then ask who wrote the reference, when, and whether that person ever saw the model's answer first.
2. Grade against a written contract, not an impression. What must be stated, what must not be invented, what counts as callable. Written once, applied every time, and checked whoever marks against it, a person or an AI model. The case for the contract is consistency and an audit trail, never accuracy. A contract presupposes what good is, which is why the three-phase lab cut its subjective criteria in its last phase.5 Your own key drifts too, and a sparse key scores true findings as inventions, so measure your drift before you grade anyone. Off-ramp: when the question is which of two answers is better, compare them side by side. In one law preprint practising attorneys recovered the intended quality order far better by comparing pairs than by scoring a rubric.9 Its authors keep the rubric as an analytical tool, and so do I: the contract for what must be stated, the comparison for which answer is better.
3. Keep the hard cases as the exam, out of training, and put the most readers on them. This is the move I changed. My first version said keep the hard cases as the exam and never let them be trained on. The evidence added a third clause, because a case an AI model gets wrong is the case most likely to carry a wrong key. The hardest exam above took only questions that had first defeated the frontier models. Its authors' own audit then found about 18 percent of a biology, chemistry and health subset disputed by their reviewers.10 An independent audit put the share of chemistry and biology keys in conflict with published evidence near 29 percent, and named the selection as the cause.14 The coding benchmark above was retired after its hardest tasks failed an audit.12 The three-phase lab picked its hard cases by where its own models failed, then had two physicians confirm each one.5 No study I read compares the key-error rate of hard and easy cases inside one benchmark. The inference rests on the selection logic and the size of those rates.
The second clause has its own history. The lung-nodule database chose not to seal a test set, and its authors said why: they could not anticipate how it would be used.1 That is the hold-out as a decision someone has to make and enforce. The three-phase lab made it, with a canary string in the data and a private set it keeps back.5 A test set answers honestly once. Off-ramp: if hard means your own read of difficulty rather than the AI model's failure, say so in the contract. The selection effect comes from choosing cases by what the model got wrong.
4. Match the grader to the question's specialty. A generalist grading a specialist's call is a second opinion from the wrong room. The blinded evaluation above matched each of its 149 physicians to the question's specialty, and the radiology posting matched its workstreams by subspecialty.72 I state this as a design principle with an untested effect. No study I read compares matched graders with unmatched ones. The one direct test of grading inside your own specialty found no effect and calls itself underpowered.15 Seniority also matters: the multicentre study found rankings shifted with the graders' seniority and practice environment.8 Off-ramp: when the question is general, a documentation task or a care consult, the matched specialist may cost you on clinical utility. The authors of the matched study say so themselves.7 Match the grader to the question, then record who graded, because the record is what lets the next reader weigh the result.

What the moves are not
They are not a job ad, and neither is this piece. The ranges above are advertised, dated, closed, and posted for clients the marketplace did not name. They say what a buyer thought the work was worth in one month of 2026, and nothing about what anyone was paid.
They are not a claim that the specialist out-answers the machine. On one lab's own benchmark the machine did, and the key still took three physicians per criterion. The moves are about the key and the marker.
They are not a verdict on whether an AI model can mark, which is contested, so the honest version is check the marker. And they are not a ruling that specialty matching works. It is the right design, with an untested effect, and the record of who graded is what will settle it.
What I sign for
I read medical images every day. The reference my own AI model is graded against was written by two readers and frozen before any AI report was graded against it. Today I build the training data from my own reads, train the model on it, test it on cases it has never seen, and ship tools I read with. The key is the part of that sentence no AI model could have written for me.
If your expertise is being asked for, the four moves are the audit for the contract you are offered. Write the reference first, grade against a written standard, keep the hard cases as the exam with the most readers on them, and match the grader to the question. What I am testing now, and how, is on the research page.
References
Footnotes
-
Armato SG III, McLennan G, Bidaut L, et al. The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): A Completed Reference Database of Lung Nodules on CT Scans. Medical Physics 38(2):915-931, 2011. Four thoracic radiologists per scan, a blinded read then an unblinded read, no forced consensus, the word nodule deliberately left undefined, no sealed test set by design. Of 2,669 lesions marked as a nodule of at least 3 mm by at least one radiologist, 928 were marked by all four. https://doi.org/10.1118/1.3528204 (full text https://pmc.ncbi.nlm.nih.gov/articles/PMC3041807/) ↩ ↩2
-
Mercor. Radiologist (MD), Medical Imaging AI Annotation and Evaluation, job posting, 2026. Advertised at 200 to 400 dollars an hour, hourly contract, remote, client unnamed. Six printed tasks: case interpretation with written reasoning, structured finding labelling, reference report authoring, output review, grading criteria and guideline authoring, difficult-case writing. Board certified or board eligible with residency complete, two or more years of independent reading, fifteen hours a week minimum, workstreams matched by subspecialty. Dated 10 September 2026 in the page's own data and closed by 30 September 2026. https://work.mercor.com/jobs/list_AAABoIoJXpi-r75u4e5G-4jT ↩ ↩2 ↩3
-
Mercor. Radiology Expert, job posting, 2026. Prompt design, golden reference responses at attending level, rubric grading, onboarding office hours and specialty calibration sessions, for an unnamed AI lab. Its rate contradicts itself on the page, 130 to 180 dollars in the header and 180 to 400 in the body, so it is not quoted. Dated 29 April 2026, closed. https://work.mercor.com/jobs/list_AAABndgXCEQamyh0JGJI4Jsj/radiology-expert ↩
-
Weekday AI. General Clinician (MD/DO), HLS, job posting on Workable, 2026. 150 dollars an hour. Grading criteria development, clinical dialogue evaluation, clinical reasoning annotation, output review, guideline authoring and difficult-case writing, for an unnamed client. Posted 17 September 2026. https://apply.workable.com/weekday-1/jobs/view/8282B5187B.md ↩
-
Hicks RS, Trofimov M, Lim D, Arora RK, et al. HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats. arXiv 2604.27470, 30 April 2026. Vendor-authored. 190 physicians, 525 tasks chosen from 15,079 candidates, three physician phases (author, review, adjudicate), two independent physicians to confirm each model error on difficult examples, subjective and preference-based criteria cut in the last phase. Best system 59.0 against physician-written responses 43.7, a model-based grader, radiology 8 of 525, chat text not images, a canary string and a private held-out set. https://arxiv.org/abs/2604.27470 ↩ ↩2 ↩3 ↩4 ↩5
-
Arora RK, Wei J, Hicks RS, et al. HealthBench: Evaluating Large Language Models Towards Improved Human Health. arXiv 2505.08775, 13 May 2025. Vendor-authored. 262 physicians, 48,562 criteria, example-specific criteria written by individual physicians and largely unvalidated by others, model-physician agreement in the same range as physician-physician agreement on consensus criteria. https://arxiv.org/abs/2505.08775 ↩
-
Feng J, Patel V, Heagerty P, et al. Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries. arXiv 2606.28960, 27 June 2026. 149 practising physicians in 36 states, 620 real queries across 30 specialties, graders matched to each question's specialty, pairwise comparison. Language-model judges differed systematically from the experts (Kendall tau from minus 0.200 to 0.467) while agreeing on the top system. No arm compares matched with unmatched graders, and the authors warn that matching may lower quality on other dimensions such as clinical utility. https://arxiv.org/abs/2606.28960 ↩ ↩2 ↩3
-
Shi P, Li J, Yang Z, et al. Physicians and artificial intelligence diverge in evaluating large language models on real clinical cases. npj Digital Medicine, 2 July 2026. More than 400 physicians across seven specialties, rankings shifted with grader seniority and practice environment, AI agents could pre-screen but not substitute for physician evaluation. Abstract read. https://doi.org/10.1038/s41746-026-02942-6 ↩ ↩2
-
Yang R, Chen R, Kelaita P, et al. JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment. arXiv 2605.25240, May 2026. Practising attorneys, including at major US law firms, on 30 real legal tasks, about 242 hours of attorney time. Comparative judgment recovered the constructed quality order far better than rubric scoring (Spearman 0.908 against 0.150). Quality levels were induced by prompting, one benchmark, US only, and the authors keep rubrics as an analytical instrument. https://arxiv.org/abs/2605.25240 ↩ ↩2
-
Phan L, Gatti A, Han Z, et al. Humanity's Last Exam. arXiv 2501.14249, v11 read, published in Nature 649:1139-1146, 2026. Questions from nearly 1,000 experts, each first tested against frontier models and forwarded to expert review only when the models failed. The authors' audit estimates expert disagreement at 15.4 percent on the public set and about 18 percent on a biology, chemistry and health subset, 25 percent if one dissenter counts, and states that multiple reviewers are needed. https://arxiv.org/abs/2501.14249 ↩ ↩2
-
OpenAI. Introducing SWE-bench Verified. 13 August 2024. 93 experienced software developers screened 1,699 samples into a 500-sample verified set. Opened through the Wayback capture, the live page returns 403. https://openai.com/index/introducing-swe-bench-verified/ ↩
-
Dufresne A. OpenAI drops SWE-Bench Verified, backs Pro after flaw audit. AI Weekly, 8 July 2026. Secondary report: 59.4 percent of the 138 hardest tasks had material flaws in test design or problem description. https://aiweekly.co/alerts/openai-drops-swe-bench-verified-backs-pro-after-flaw-audit ↩ ↩2
-
Dratsch T, Chen X, Rezazade Mehrizi M, et al. Automation Bias in Mammography: The Impact of Artificial Intelligence BI-RADS Suggestions on Reader Performance. Radiology 307(4):e222176, May 2023. 27 radiologists, 50 mammograms, a purported AI suggestion. Very experienced readers 82.3 percent correct with a correct suggestion against 45.5 percent with a wrong one. https://doi.org/10.1148/radiol.222176 ↩
-
Skarlinski M, Laurent J, Bou A, White A. About 30% of Humanity's Last Exam Chemistry/Biology Answers are Likely Wrong. FutureHouse, 23 July 2025, updated 16 September 2025. 29 percent, plus or minus 3.7, of text-only chemistry and biology questions had answers in direct conflict with peer-reviewed evidence, attributed to the requirement that questions defeat the models. https://www.futurehouse.org/research/hle-exam ↩
-
Borgohain S, Mariathas R. Decomposing Physician Disagreement in HealthBench. arXiv 2602.22758, February 2026. 22.5 percent of cases show physician disagreement. Grading within one's own specialty had no detectable effect in a test the authors call underpowered, with specialties assigned by a model. https://arxiv.org/abs/2602.22758 ↩
