Skip to content
One-Person Startup10 min read

How to check the ground truth a model was graded on

By Reza Motaghi

A brass balance with two identical weights and the beam still tipped, a black and white image about an argument settled by a procedure rather than found
On this page

The pass mark on a licensing exam is not a fact about the exam. A panel of experts sits down and estimates, item by item, how a barely competent candidate would do. They look at each other's numbers, then at how real candidates scored, and revise over several rounds until the estimates converge.1 Two panels scoring the same exam land on different pass marks. The standard is the place where the argument ended.

Every gold standard is made the same way. The labels a model learns from, the rubric a judge scores against, the benchmark number quoted in a meeting. Each is the endpoint of an argument between people. The argument is the part nobody prints. This article is about how to ask for it.

I read medical images every day, and I built the dataset my own model learns from. It was labelled by two readers under a contract written before the first label. On a share of cases we disagreed, and we settled each one and kept the record. That record is the reason I no longer read a gold standard the way I read a lab value.

What the settled argument is

A gold standard is a settled argument: a set of labels that a group of people stopped disagreeing about, by some procedure, on some date. The procedure and the date are the standard's real specification. The number on top is a summary of both.

That is not a complaint. A standard has to be settled, or nothing can be measured against it. The trouble starts when the settlement is mistaken for a discovery. Then nobody asks who was in the room, what the tie-break was, or whether anyone has looked since.

The contract before the first label

The receipt is the rule, not a score. Before the first case was labelled, the labelling contract was written. It said what counts as a finding, what does not, and what happens when the two readers differ. Every case was read by me and by a second reader. Where we disagreed, we argued to a consensus and wrote down what settled it.

The argued cases turned out to be the most useful thing in the dataset. They are where the labels are thinnest. There, the expert's judgment is the dataset rather than a fact behind it, and a model trained on those labels will be least reliable. A standard that keeps its arguments tells you where to look. A standard that shows only a number does not.

Where the argument shows

The clearest public example is a chest radiograph dataset the field has trained on for years. Eight radiologists labelled the same 500 patients for fourteen findings.2 In 460 of those patients, at least two readers disagreed on at least one finding. Complete agreement, everyone on everything, occurred in 40 patients.

In two patients of five the gold label for at least one finding was, in the authors' own words, effectively a coin flip. The authors are careful about the cause, and so should you be. A large share of the disagreement came from the label schema itself, which forced overlapping and hierarchical findings into a flat list. The readers were not unreliable. They were answering a question the schema had settled badly.

The second example is from language models, and it cuts the other way. Two annotator pools, expert and crowd, graded the same model answers.3 On one benchmark their majority labels differed on nearly one item in four, and they named the opposite winner on nearly one in ten. The six-model leaderboard built from either pool came out identical.

The paper draws its own sharp line. Reporting a leaderboard is safe, because a rank averages the argument away. Training a reward model, routing cases for annotation, or validating a judge all consume the individual label, and there the disagreement is fully live. The authors also project that the stability does not scale, and that a twenty-model board would almost certainly reshuffle under the same swap. That projection is a model, not an event, so I quote only what happened.

Live leaderboards show how thin the settlement can be. On one public arena, dropping two votes out of 57,477 changed the top-ranked model.4 A benchmark scored by paid expert annotators needed 92 dropped votes to move. Fragility tracks how the votes were collected and how close the competitors are, both of which can be inspected.

Medicine's own bedrock behaves the same way. Autopsy is the closest thing the field has to objective diagnostic truth. It still disagrees with the living diagnosis in one case in five.5 Compiling that figure required a pathologist to re-read the original slides. Seven heart failure trials, nearly 47,000 patients between them, sent every reported event to a blinded committee working under a charter written before the trials began.6 The committee confirmed between six and nine of every ten hospitalisations the investigators had called heart failure. The trials' answers did not change. The aggregate survived the argument. The individual event did not.

Every field settles its standards in a room

The engineers who run the internet wrote their settlement procedure down. A protocol becomes a standard by rough consensus. The governing document defines that as a chair judging that every technical objection has been heard and answered. It sums the idea up in one line: lack of disagreement is more important than agreement.7 A majority may not outvote an unaddressed objection. The standard is the argument's end, not its vote count.

Sometimes the settlement costs the arbiter. Python's assignment expression carried months of public argument before its creator accepted it in 2018. He stepped down as the language's final authority the same week. He wrote that he never again wanted to fight so hard for a change and find so many people despised his decision.8 The feature is in every Python since. The price of settling it is in the mailing list.

Even the cleanest reference standards keep a thin human layer. The genome benchmark most sequencing tools are checked against is built mainly by cross-checking independent technologies against each other.9 That is a more mechanical kind of agreement than three physicians arguing over a rubric. Its structural variant calls are still described by the programme itself as crowd-sourced and expert-curated. Nobody found a domain where the label is free of judgment. Only domains where the judgment is thinner, cross-checked by an instrument, and written down.

Four questions for any gold standard

The questions are cheap because each one is answered by a document the standard's owner either has or does not. One of them I changed after reading the screening audits.

1. Who disagreed? Ask for the disagreement rate between annotators, and for the cases behind it. If the answer is that nobody disagreed, either the task was trivial or the disagreement was never recorded, and both answers are useful. A rubric adjudicated by three physicians across three rounds is a different object from a rubric one person wrote on a Friday.10 The first kind will say so. Off-ramp: some standards are built by instrument rather than by people. A culture that grew or did not, a base pair that is present or not. Those answer this question with the instrument's error rate instead. Ask for that.

2. How was it settled? Majority vote, senior reader, a discussion round, a coin. Each leaves a different kind of error in the labels. A structured panel that argued its disagreements over two blinded rounds resolved most of them and measurably improved agreement over a simple majority vote.11 A coin flip settled 40 percent of one dataset's patients. The procedure is the standard's real specification, so ask for it in writing. Off-ramp: a standard whose settlement procedure is published, as trial charters and consensus guidelines are, has already answered. Read the charter.

3. What is the version number? A standard nobody has amended is a standard nobody has checked. Benchmarks are now being rebuilt as maintained artefacts.12 Each refresh brings a fresh set of items and a rule that keeps new scores comparable with old ones. Your own labels are no different, and even a single careful reader's labels drift between the first case and the last. Ask what changed since the version you were shown, and who changed it. Off-ramp: a standard that is frozen on purpose, like a sealed test set, has a version number of one. It only answers honestly while nobody has tuned to it. Ask how many times it has been looked at instead.

4. Does it show its arguments? My first version of this question assumed that forcing a consensus throws information away. The screening audits changed it. In double-read mammography, the arbitrating reader alone caught most of the cancers the first reader would have missed.13 A settled disagreement adds yield rather than erasing it. What gets erased is the record of which cases needed the arbitration. So the question is not against consensus. It asks for the argued cases, kept and marked, because they are where a model, a judge and an audit will disagree too. In one clinical annotation study, the small slice of cases where four models disagreed held most of the human labelling errors.14 Off-ramp: a standard that cannot produce its argued cases can still be used. Use it as a rank, never as a per-label truth, which is exactly the line the leaderboard paper draws.

The settled argument, four questions: who disagreed, how it was settled, what the version number is, whether the standard shows its arguments

What the questions are not

They are not a claim that there is no truth. Whether a culture grew, whether a base pair is present, whether a patient died. These are not manufactured by adjudication in the way a rubric score is. A careful reader will supply the counterexample at once. The questions are for what a benchmark calls ground truth, which is where the argument lives.

They are not an argument against adjudication. The best-run standards in this article were all adjudicated: three physicians over three rounds, a blinded panel over two, a committee under a charter. Adjudication is how a standard gets settled well. The questions ask to see it.

They are not a reason to trust medicine more than benchmarks, or the reverse. Autopsy and histopathology carry disagreement rates in the same range as the annotation numbers above. That is the reason to run the same four questions on a lab value that you would run on a leaderboard.

What I keep

I read images every day, I built a gold standard, and I keep its arguments. The consensus method's own practitioners put the limit better than I can: a well-run consensus can be stable, well documented, and nonetheless wrong.15 Settled is not the same as verified. The version number is what lets someone check.

Ask the next gold standard you are shown how it was settled. If the answer is a number, ask again. If you build or evaluate models on expert judgment, the contact page reaches me directly. I have sat on both sides of that argument. The disagreement itself is what gets tried, and the next piece is about how to leave a record of it.

References

Footnotes

  1. Standard-setting study, the Angoff method as described in 1971: a panel of subject-matter experts estimates item by item what a minimally competent candidate would score, then revises over rounds; different panels reach different passing standards. https://en.wikipedia.org/wiki/Standard-setting_study ↩

  2. Abdalla M, Fine B. Hurdles to artificial intelligence deployment: noise in schemas and "gold" labels. Radiology: Artificial Intelligence 5(2), 2023. 500 patients, eight radiologists, fourteen labels; disagreement in 460 of 500 patients; complete agreement in 40; a coin-flip gold label for at least one finding in 201 of 500 (40 percent); much of it attributed to the flat label schema. https://pubs.rsna.org/doi/10.1148/ryai.220056 ↩

  3. Jha A. Whose gold? Annotator-pool disagreement is large at the item level, and hidden by small leaderboards. arXiv 2608.15980, 17 August 2026. Majority labels differed on 23.6 percent of MultiPref items and 30.5 percent of MT-Bench cells, reversing the winner on 9.2 and 8.5 percent; the six-model leaderboard identical (Kendall tau 1.00); projected displacement 86 percent at ten models and 99.97 percent at twenty. https://arxiv.org/abs/2608.15980 ↩

  4. Dropping just a handful of preferences can change top large language model rankings. arXiv 2508.11847, August 2025. Two of 57,477 votes (0.003 percent) on Chatbot Arena; 92 votes (2.74 percent) on MT-Bench. https://arxiv.org/html/2508.11847v4 ↩

  5. Khaba, Maaga, Kgokong. Frontiers in Medicine, July 2026. Major discrepancy between clinical diagnosis and autopsy in 51 of 267 cases (19.1 percent). https://pmc.ncbi.nlm.nih.gov/articles/PMC13444749/ ↩

  6. Wandel S, et al. Centrally adjudicated versus investigator-reported outcomes in randomised heart failure trials. European Heart Journal 45(47):5087, December 2024. Seven trials, 46,691 patients; cardiovascular death confirmed in 87.9 to 99.2 percent, heart failure hospitalisation in 61.6 to 88.0 percent; "central adjudication of events did not alter the results". Committee definition: an independent group of clinical experts, blinded to treatment assignment, reviewing events against criteria pre-specified in a charter (CASRAI, endpoint adjudication committee, updated 6 September 2026). https://academic.oup.com/eurheartj/article/45/47/5087/7887814 ↩

  7. Resnick P. On consensus and humming in the IETF. RFC 7282, June 2014. https://www.rfc-editor.org/rfc/rfc7282 ↩

  8. PEP 572, assignment expressions, 2018, and van Rossum G, "Transfer of power", python-committers, 12 July 2018. https://peps.python.org/pep-0572/ and https://mail.python.org/pipermail/python-committers/2018-July/005664.html ↩

  9. National Institute of Standards and Technology. Genome in a Bottle. Benchmark updated November 2025. https://www.nist.gov/programs-projects/genome-bottle ↩

  10. Hicks et al. HealthBench Professional: evaluating large language models on real clinician chats. arXiv 2604.27470, 30 April 2026. Rubrics "written and iteratively adjudicated by three or more physicians across three phases". https://arxiv.org/abs/2604.27470 ↩

  11. Duggan GE, Reicher JJ, Liu Y, Tse D, Shetty S. Improving reference standards for validation of AI-based radiography. British Journal of Radiology 94(1123), June 2021. 1,100 images, six findings, panels of three; 88 percent of disagreements resolved within two rounds; fracture agreement from kappa 0.64 to 0.74. https://pmc.ncbi.nlm.nih.gov/articles/PMC8248225/ ↩

  12. Liu, Zhao, Zhou, Liang, Ma. MMBench-Live: a continuously evolving benchmark for multimodal models. arXiv 2607.01813, 2 July 2026. About 5,900 instances per refresh; a distribution-consistent update rule to keep versions comparable. https://arxiv.org/abs/2607.01813 ↩

  13. Audit of screening-detected breast cancers with discordant interpretations on double-read screening mammography, cases April 2010 to March 2012. 57 of 670 screen-detected cancers discordant (8.5 percent); the arbitrating reader alone caught 69 percent of the cancers the first reader would have missed. https://pmc.ncbi.nlm.nih.gov/articles/PMC3980438/ ↩

  14. Wittlinger S, et al. Multi-LLM disagreement as a scalable detector of human annotation errors in structured data from clinical free-text. medRxiv, 6 May 2026. 910 German colonoscopy reports, 4,550 annotations; the lowest-agreement 6.5 percent of annotations held an estimated 80 percent of the errors. Single centre, one annotator, one language; the models agreed with each other more than with the human, so their flags may share blind spots. https://www.medrxiv.org/content/10.64898/2026.05.04.26352392v1.full ↩

  15. CASRAI. The Delphi method: building expert consensus in rounds. Practitioner guide, accessed 16 September 2026: the method "measures convergence of expert opinion, not ground truth". https://casrai.org/guides/delphi-method ↩