Agreement with past cases is not a check

Ninety-nine lines in a hundred agreed with my past cases. Thirty-nine of them were wrong.

In October I had a strong AI model turn one of my treatment-planning rulebooks into a decision table. My rules, applied by plain code, with no model deciding anything at the moment of use. One board-certified oral and maxillofacial radiologist, one rulebook, one hundred lines of my past plans as the comparison.

The first check was the cheap one. Compare what the table says with what was planned for the same cases before. Ninety-nine of a hundred lines agreed. By that number the table was finished.

Then I gave the table to a second model with one brief: find what is wrong. It found nineteen errors. The fixes went in, five of them on rulings I had to make myself. Then the same brief again, on the fixed table. Twenty more, six of them serious. The six were fixed, the other fourteen are listed as known, and the table still reads ninety-seven of a hundred against the past. It is not finished. It is unchecked in a way I can now name.

Here is the shape of it. Agreement with the past measures the common path. My past cases walk the same few branches a hundred times. A table that gets those branches right scores ninety-nine and never meets the branch that matters once a year. Most of the errors sat in how the table read words in free text. A phrase that means one thing in my rulebook and another in a note. The cheap comparison cannot see those, because the past cases never asked.

The statisticians have a name for the first half of this. Two readers can agree on almost every case and still agree no better than chance, when nearly every case falls in the same category. The paradox was set out in a 1990 paper in the Journal of Clinical Epidemiology.¹ It is the reason an agreement rate without a denominator of hard cases means little. Medical imaging has its own version. A model can post a high overall score while failing on the rare subset that matters. A 2020 study of this hidden stratification found gaps of more than twenty percent behind a clean headline number.² Software engineers found the mirror image in production systems. In a study of 198 real failures, ninety-two percent of the catastrophic ones came from code that handled a rare error badly.³ The tests had never walked that code. Common paths pass. Rare paths fail. Agreement counts the common ones.

A reviewer briefed to disagree sees the other half, and one pass is not enough. The second pass found as many as the first. There is evidence behind both halves of that sentence. A 2024 study trained a model to criticise code. Its critiques were preferred over a human's in sixty-three percent of cases on natural errors.⁴ One warning came with it: critics also invent bugs, so every finding is checked against the source. The reviewer must also be a different reader. A model asked to grade its own kind favours its own output, and the bias tracks how well it recognises itself⁵. And two passes do not give you a count of what remains. The method that estimates hidden defects from the overlap between reviewers needs independent readers, at least four of them, and underestimates badly with fewer⁶. My passes were sequential, with fixes in between, so there is no overlap to read. They prove that errors remained. They do not say how many. Even the old inspection data says the second session matters. In one 1994 study, repair between two review sessions found about ten percent more defects than one long session.⁷

I stopped at two because the clock said so, not because the table was clean. The honest count of what is left is not zero. It is a number I do not have.

This is not a radiology rule. Any rulebook a model turns into code will agree with yesterday's decisions and fail on the case the record never held. A pricing policy, a triage protocol, a contract playbook.

One caveat I owe the reader. The comparison ran against plan lines a model had drafted earlier for those cases and I had checked, not against a signed ground truth. That makes the ninety-nine weaker, not stronger. The point stands either way.

What I would do differently: run the hostile pass before the agreement check, and budget for a third.

References

Footnotes

1. A. R. Feinstein, D. V. Cicchetti, "High agreement but low kappa: I. The problems of two paradoxes", Journal of Clinical Epidemiology 43(6):543-549, 1990, https://doi.org/10.1016/0895-4356(90)90158-L. ↩

2. L. Oakden-Rayner et al., "Hidden stratification causes clinically meaningful failures in machine learning for medical imaging", ACM Conference on Health, Inference, and Learning (CHIL '20), 151-159, 2020, https://doi.org/10.1145/3368555.3384468. ↩

3. D. Yuan et al., "Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems", OSDI 14, 2014, https://www.usenix.org/conference/osdi14/technical-sessions/presentation/yuan. ↩

4. N. McAleese et al., "LLM Critics Help Catch LLM Bugs", arXiv preprint, 2024, https://arxiv.org/abs/2407.00215. ↩

5. A. Panickssery, S. R. Bowman, S. Feng, "LLM Evaluators Recognize and Favor Their Own Generations", NeurIPS 2024, https://arxiv.org/abs/2404.13076. ↩

6. L. C. Briand et al., "A comprehensive evaluation of capture-recapture models for estimating software defect content", IEEE Transactions on Software Engineering 26(6):518-540, 2000, https://doi.org/10.1109/32.852741. ↩

7. A. Porter, H. Siy, C. Toman, L. Votta, "An experiment to assess the cost-benefits of code inspections in large scale software development", interim report, 1994, https://ntrs.nasa.gov/citations/19950024821. ↩