Skip to content
Reza Motaghi
One-Person Startup10 min read

Fine-tuning got cheap. The expert who grades it did not

By Reza Motaghi

A balance scale with a server cabinet on one pan and a single person on the other, drawn in a single white line on black, a minimalist illustration about budgeting for the expert who grades the data the way you budgeted for GPUs
On this page

In 1949 a radiologist in San Francisco published what his colleagues did not want to read. When experienced readers looked at the same chest films, they disagreed with each other on about a third of the positive ones.1 Shown their own films again, they disagreed with themselves on about a fifth. Nothing has changed since. When 115 pathologists read breast biopsies against an expert panel in 2015, they matched the panel three times in four.2 On the hardest category, atypia, they matched it less than half the time.

That is the ground truth. It has always been a person, and the person has always been fallible. What changed is the price of everything around it. Fine-tuning got cheap. Graded truth did not. The scarce input to specialist AI is no longer compute, and it is not raw data. It is a person whose corrections are correct.

What "the expert is the dataset" means

The story everyone builds on says the opposite. Small models are cheap. Tuning is a weekend and a rented GPU. Whoever has the most examples wins. That part of the story is true. It is also the part that stopped mattering, because examples are only as good as the judgment that graded them.

The expert is the dataset. Not the source of the data. The dataset itself: their floor, their consistency, their willingness to leave a call blank. One expert or a panel of them, the unit is judgment, not volume. A model is a compressed copy of whoever graded it. If that person is you, your position is stronger than you were told. If you are the builder, your budget is buying the wrong thing.

On both sides of the desk

I read images every day. For some time I have also been writing training examples from my own reads. The reading took twenty years to become worth training on. The training took a weekend to set up. Nothing about the second job is hard the way the first one is, and that is the whole point.

The hard part is not writing the examples. It is knowing which of my calls the image supports and which ones only my history supports. Then having the discipline to write down what I would do next instead of what I believe. I gave that rule a name in the callable floor. It is the same rule that makes a grader worth paying for. Anyone can produce labels. Very few people can produce labels that a peer would sign.

Where the price moved

The cheap half is real. In 2023 a 65-billion-parameter model was fine-tuned on a single 48 GB card while matching full-precision results.3 The training data it learns from is the only input that did not get cheaper. A thousand carefully curated examples were enough to align one.4 Hosted fine-tuning is now a line on a price list.

The expensive half is where the money went. One analysis of frontier labs' post-training budgets put human data at about three times the marginal compute in 2024.5 Data costs were growing many times faster than compute costs.

When a frontier lab built its medical benchmark, the yardstick took 262 physicians writing tens of thousands of rubric criteria.6 The same lab now writes plainly that human graders set the gold standard and do not scale.7 The investors have noticed too. A widely read venture essay last year said it in one line. The moat is not the model, it is the painstakingly assembled dataset nobody else has.8

Why the AI judge cannot replace you yet

The obvious answer is to let a model do the grading. It works less well than it looks. When language-model judges scored answers in two expert domains, they agreed with the human experts about two times in three.9 The experts agreed with each other about three times in four. Worse, the judges agreed with lay users 80 percent of the time. They tracked what sounds right, not what is right.

In medicine the gap is narrower and the failure is different. On a large clinical benchmark, the best model judge came within a hair of the physician ceiling on agreement.10 What it never did was abstain. Physicians abstained more as the items got harder. The judge scored every one. And on the question that matters most for a medical chatbot, whether an answer is complete, model judges scored barely above chance against clinicians.11 The judge is calibrated to fluency, and in a specialist field fluency is exactly what a wrong answer has. It is the default patient again, this time grading the exam.

Radiology already wrote the rule

The field that produced the 1949 paper answered it with process, not with better eyes. Double reading, with the second reader blind to the first. Consensus panels. When the radiological society builds a dataset for its AI challenges, the test-set labels need at least three annotators in consensus.12 An annotation cycle runs two to three months. The annotators are paid mostly in authorship. The training set can be noisy. The yardstick cannot.

The field also learned what happens when the yardstick is cheap. The largest public chest X-ray datasets have training labels written by rule-based text tools reading the reports.13 Only the small test sets were graded by radiologists. When two radiologists re-read a sample of those training cases, they disagreed with the labels on close to half of them.14 An earlier audit of another public set found its label accuracy well below what the documentation implied.15 Those datasets shipped models. They also shipped the labeller's mistakes into every one of them.

And the gold standard itself is not gold. Across fifty-three autopsy series over four decades, the median rate of major diagnoses missed in life was close to a quarter.16 Even the reference the field measures itself against was a person's call, checked later by another person.

Beyond the reading room

The dataset that started modern computer vision was graded by tens of thousands of crowd workers.17 That worked because every one of them was an expert in its subject. Everyone can tell a dog from a cat. Weak supervision has shipped in the years since. Labels written by rules came within a few points of hand-labelled sets.18 A model trained on its own predictions over three hundred million unlabelled images beat one trained on billions of human-tagged ones.19 Read closely, each of these still traces back to a person. The rules were written by domain experts. The teacher model was already good because someone had graded it. Reinforcement learning without human labels works where the answer is mechanically checkable, in maths and code,20 which is precisely where a specialist is not needed. Where the answer is a judgment, the judgment has to come from somewhere. Training on a model's own output instead makes the tails of the distribution disappear.21

Two things sharpen the claim rather than break it. Panels beat individuals: pooling ten ordinary diagnosticians raised accuracy from 46 to 76 percent on real cases,22 and human-plus-model collectives beat both alone.23 So the unit is judgment, one expert or a panel, never volume. And the expert is not a fixed dataset either. The 1949 readers disagreed with themselves a fifth of the time, which is why the grader's rules have to be written down and versioned, not remembered.

Five budget lines before you count examples

Write these before the first example, because after it the volume starts to look like value.

1. Name who graded them, and whether they could defend each to a peer. Before you count. A dataset with no named grader is volume, not truth. The test is the one from the callable floor: could the grader show a peer the input, and nothing else, and have them agree. The off-ramp: when the label is a measured outcome, a biopsy, a follow-up, a compiler, the grader is the record. This line is already answered.

2. Pay for the expert's time on the evaluation set first. Training data can be noisy. The yardstick cannot, and neither can the instrument it is measured on. This is the one place the evidence is unanimous, from the radiological society's three-annotator test sets to the frontier lab's 262 physicians. If the budget buys one expert-hour, it buys the eval set.

3. Give the expert the right to say "cannot be called from this input". A dataset with no blanks was graded by someone who did not know when to stop. The blank is a label, and models can be trained to defer on it. The physicians in the benchmark study abstained more as items got harder. The model judge never did. That difference is the expert's whole value, in one behaviour.

4. Version the expert's rules like code. When they change, the old labels are a different dataset. Experts disagree with their past selves, and the 1949 paper measured how much. So the rules that produced a set of labels have to travel with the set. Datasheets that record who labelled what, under which guidelines, exist for exactly this.24 The off-ramp: tools that learn from a modest set of gold labels can take over half or more of routine annotation.25 They route the uncertain cases back to the expert. The expert's grading is the seed. Tools scale it. They do not replace it.

5. If you are the expert, keep your own reads. They are worth more than the model they will train. When I built my own CBCT viewer, the agent got the camera and never the read, for the same reason. Every argument above is an argument about your position, and it is stronger than the story told you. A model built on your reads is a compressed copy of you. Keep the original.

The expert is the dataset, five budget lines. 1, name who graded them. 2, pay for the eval set first. 3, give the expert the right to say blank. 4, version the rules like code. 5, if you are the expert, keep your reads.

Two things the lines are not. They are not an argument that compute stopped mattering. At the frontier, training a general model is still the largest bill there is.26 The claim is about specialist AI, where the model is a commodity and the grader is not. And they are not a case for one heroic expert. A panel with a written rulebook is a better dataset than one person with a good eye. The person with the good eye should be the first to say so.

What I sign for

I read the images and I grade the reads, and only one of those took a career. What the model learns from me is not the reading. It is my floor, my consistency, and where I leave the call blank. That is why the fluent wrong report is the failure I test for first. Evaluating frontier vision models on my own cases, blind, is now my main research work. Every number in it is worth exactly what the grading behind it is worth.

If you are the founder writing the budget, or the expert being asked for your reads, the contact page reaches me directly. Graded truth in imaging is a conversation I am glad to have.

References

Footnotes

  1. Garland LH. On the scientific evaluation of diagnostic procedures. Radiology, 1949. Inter-observer disagreement about 30 percent and intra-observer about 20 percent on positive chest films. https://pubs.rsna.org/doi/10.1148/52.3.309 ↩

  2. Elmore JG, Longton GM, Carney PA et al. Diagnostic concordance among pathologists interpreting breast biopsy specimens. JAMA, 2015. 115 pathologists, overall concordance with the consensus panel 75.3 percent, invasive carcinoma 96, ductal carcinoma in situ 84, atypia 48, benign 87. https://pmc.ncbi.nlm.nih.gov/articles/PMC4516388/ ↩

  3. Dettmers T, Pagnoni A, Holtzman A, Zettlemoyer L. QLoRA: efficient finetuning of quantized LLMs. NeurIPS, 2023. https://arxiv.org/abs/2305.14314 ↩

  4. Zhou C et al. LIMA: less is more for alignment. NeurIPS, 2023. https://arxiv.org/abs/2305.11206 ↩

  5. Zhu, Kang. Human data is (probably) more expensive than compute for training frontier LLMs. August 2025. Human data about 3.1 times marginal compute in 2024, data costs growing about 88 times against compute's 1.3 times over the period studied. https://ddkang.substack.com/p/human-data-is-probably-more-expensive ↩

  6. Arora RK et al. HealthBench: evaluating large language models towards improved human health. OpenAI, 2025. 262 physicians, 60 countries, 26 specialties, 48,562 rubric criteria, 5,000 conversations. https://arxiv.org/abs/2505.08775 ↩

  7. Anthropic. Demystifying evals for AI agents. January 2026. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents ↩

  8. Andreessen Horowitz. Fruits of the walled garden. October 2025. https://a16z.com/fruits-of-the-walled-garden/ ↩

  9. Szymanski A et al. Limitations of the LLM-as-a-judge approach for evaluating LLM outputs in expert knowledge tasks. ACM IUI, 2025. Judge-expert agreement 68 percent in dietetics and 64 in mental health, expert-expert 75 and 72, judge-lay agreement 80. https://arxiv.org/abs/2410.20266 ↩

  10. Clinician-level agreement without clinical caution: LLM evaluator limits in medical AI benchmarking. 2026. 3,800 items, ten physicians, best model judge kappa 0.694 against a physician ceiling of 0.709; model judges never abstained. https://arxiv.org/abs/2607.01103 ↩

  11. DeLucia A, Dredze M et al. Same verdict, different reasons: LLM-as-a-judge and clinician disagreement on medical chatbot completeness. 2026. AUC 0.49 to 0.66. https://arxiv.org/abs/2604.16383 ↩

  12. Lessons learned in building expertly annotated multi-institution datasets and hosting the RSNA AI challenges. Radiology: Artificial Intelligence, 2024. https://pmc.ncbi.nlm.nih.gov/articles/PMC11140499/ ↩

  13. Irvin J et al. CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison. AAAI, 2019. https://arxiv.org/abs/1901.07031 ↩

  14. Rafferty A, Rajan A. Limitations of public chest radiography datasets for artificial intelligence: label quality, domain shift, bias and evaluation challenges. 2025. Two radiologists disagreed with the labels on 58 and 49 of 120 MIMIC-CXR cases and 61 and 60 of 120 CheXpert cases. https://arxiv.org/abs/2509.15107 ↩

  15. Oakden-Rayner L. Exploring large-scale public medical image datasets. Academic Radiology, 2020. https://arxiv.org/abs/1907.12720 ↩

  16. Shojania KG, Burton EC, McDonald KM, Goldman L. Changes in rates of autopsy-detected diagnostic errors over time: a systematic review. JAMA, 2003. Median major error rate 23.5 percent across 53 series. https://pubmed.ncbi.nlm.nih.gov/12783916/ ↩

  17. Deng J et al. ImageNet: a large-scale hierarchical image database. CVPR, 2009. The worker figures, about 49,000 Mechanical Turk workers in 167 countries, are from later accounts of the project: Gershgorn D, The data that transformed AI research, Quartz, 2017. https://qz.com/1034972/the-data-that-changed-the-direction-of-ai-research-and-possibly-the-world ↩

  18. Ratner A et al. Snorkel: rapid training data creation with weak supervision. VLDB, 2017. Within 3.6 percent of hand-labelled sets. https://arxiv.org/abs/1711.10160 ↩

  19. Xie Q, Luong MT, Hovy E, Le QV. Self-training with noisy student improves ImageNet classification. CVPR, 2020. https://arxiv.org/abs/1911.04252 ↩

  20. DeepSeek-AI. DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. 2025. https://arxiv.org/abs/2501.12948 ↩

  21. Shumailov I et al. AI models collapse when trained on recursively generated data. Nature, 2024. https://www.nature.com/articles/s41586-024-07566-y ↩

  22. Kurvers RHJM et al. Automating hybrid collective intelligence in open-ended medical diagnostics. PNAS, 2023. https://www.pnas.org/doi/10.1073/pnas.2221473120 ↩

  23. Zöller N et al. Human-AI collectives most accurately diagnose clinical vignettes. PNAS, 2025. https://pmc.ncbi.nlm.nih.gov/articles/PMC12184336/ ↩

  24. Gebru T et al. Datasheets for datasets. Communications of the ACM, 2021. https://arxiv.org/abs/1803.09010 ↩

  25. RadAnnotate: large language models for efficient and reliable radiology report annotation. 2026. Auto-annotation of 55 to 90 percent of reports with low-confidence cases routed to experts, radiologist annotation time cut 50 to 60 percent. https://arxiv.org/abs/2603.16002 ↩

  26. Epoch AI. How much does it cost to train frontier AI models? June 2024. https://epoch.ai/blog/how-much-does-it-cost-to-train-frontier-ai-models ↩