Skip to content
Reza Motaghi
One-Person Startup9 min read

Why a reused test set stops telling the truth

By Reza Motaghi

A sealed envelope with a black wax seal on a dark desk under one lamp, a black and white image about a test drawn once and not opened
On this page

In 2017 a contestant in a cervical cancer screening competition reached fourth place of 848 entrants without building a model that could tell the images apart. He did it by querying the scoreboard.1 Each submission returned a score. Each score leaked a little about the hidden answers. After enough submissions the leaderboard had told him what it was supposed to be testing. Nobody stole anything. The test had simply been looked at too many times, by someone who kept count.

Most teams never keep count. They keep the test set separate, which is the right first move, and then consult it after every change, which quietly undoes the first move. This article is about what each look costs and how to keep the bill.

I train a vision model on my own reads. Before the first run I drew the exam once, sealed it, and wrote the gates it had to pass in a fixed order. Not ceremony. It is the only way I know to keep a test honest against the person most tempted to bend it. That person is me, after the third failed run.

What the look count is

The look count is the number of decisions made after seeing a test's score. A test set answers honestly while nobody has tuned to it. The moment a score changes what you do next, the next score is partly a measure of your tuning, not of the model. That is the whole mechanism, and it does not need bad faith. It needs a curious person and a scoreboard.

The mathematics is a proof, not a study. Reuse a held-out set adaptively, choosing each next question from the last answer, and the statistical guarantee it gives decays with each look.2 The same six authors also showed the decay can be held off.3 The price is calibrated noise on every answer and a fixed budget of queries. That is a real instrument. Almost nobody outside a lab runs it. So the free version of the guarantee, draw once and seal, is the one most people actually have.

The exam that cannot be looked at

The receipt is a discipline, not a case. Before training the first model on my own reads, I drew a held-out exam once and sealed it. The gates were fixed in advance, in a written order, with no single number to average them into. Every tuning decision since has run on the training side. The exam has been looked at zero times by a model, and the day it is scored will be the day the count starts.

It was not a comfortable rule to keep. The first fine-tune learned my report format and nothing about the image. After a failure like that, the temptation is to look at the exam, just once, to see how far off the model really is. That look would have spent the exam before it had been used. What stopped me was not virtue. It was having written the rule down before I had a reason to break it.

Where the spending shows up

The clean result is a proof. The messy results are the field's, and they cut both ways.

The reassuring half first. A study looked at more than 100 real competitions on the same platform, where thousands of adaptive submissions were scored against a public leaderboard.4 It found little evidence of substantial overfitting. The gap between the much-consulted public score and the once-only private score stayed small. The most reused test set in computer vision was checked with a fresh set built the same way a decade later.5 Models' gains carried over rather than evaporating. Under protected conditions, a private split held back and a bounded number of looks, reuse did less damage than the theorem allows.

The other half is the one to plan for. A benchmark of real clinical cases is rebuilt weekly, so that nothing in it could have entered a model's training data.6 It found the best of 38 models scoring 39 percent. Most of the models scored worse on cases dated after their own training cutoff. That is the cousin failure: the test leaking into the training instead of into your decisions. Its grader agrees with human graders less often than humans agree with each other, so read the number as a signal, not a measurement. The cure is the same either way.

The spending also hides well. Even a short round of extra training on a reasoning model can conceal the statistical traces that contamination detectors look for.7 A model that has seen the test can pass an audit for it. A planted copy of a test set inflates a score in proportion to how big the model is and how hard it was trained.8 The models most worth testing are the ones most able to remember.

Every field has a scoreboard

The pattern is older than machine learning. Teams that watch a live experiment and stop it the moment it looks significant are making the same adaptive error on a product decision. The fix is statistics that stay valid under continuous peeking, and it is now built into commercial testing platforms.9 The California bar exam of February 2025 had its questions online within days.10 The makeup sitting was pushed back to rewrite them. An exam whose questions circulate can no longer certify anyone who saw them. Aviation regulators rewrite the pilot knowledge test bank on purpose, to defeat candidates who memorise harvested questions instead of learning the material.11

And the newest scoreboards spend fastest. A frontier maths benchmark disclosed in 2025 that its funder, also a model maker scored on it, had seen most of the problems.12 The organisation now holds fifty problems back that the funder will only ever receive unsolved. A model maker submitted a specially tuned chat variant to a public leaderboard.13 The plain version everyone could download ranked 32nd when it was tested days later. A 2026 proposal for evaluating models asks for what medicine has asked of trials for years.14 Seal the problems until grading, freeze the model before, publish the set only after.

Four rules that keep the count

The rules are cheap because they are about bookkeeping, not tooling. One of them I changed after reading the evidence against my first version.

1. Draw once, seal it, date it. The test is drawn one time, by a rule written before the draw, and the date goes into every result it produces. My own exam was drawn by a rule that puts the rarest findings first. That way the sealed set could not be quietly easier than the training set. Off-ramp: a test drawn from a stream that refreshes itself, the weekly-rebuilt benchmark above, has no seal because it has no reuse. Date each refresh instead.

2. Decide what passes before the first look. The gates, in a fixed order, written down, with no single average to hide a failure inside. This is pre-registration, and it exists because a gate moved after a look is a look. Off-ramp: on an engineering run, where the point is to learn rather than to certify, the gates can move. Say so in the log, and never report that run's score as if it were the exam.

3. Count the looks. Every decision made after seeing a score is a look, and the count belongs next to the score. My first version of this rule said a test answers honestly only once. The reusable-holdout result changed it. You can look more than once if each look is paid for with noise and a budget. That is a real instrument, and almost nobody on a small team runs it. So the honest rule is not never look twice. It is know how many times you have looked, and price each one. Off-ramp: a leaderboard with a private split and a bounded submission count is already paying. That is why the competition study found so little damage.

4. When it is spent, draw fresh. Never top up the old set. A topped-up test remembers what you tuned to. The answer key you wrote last year has drifted from the one you would write today. Draw a new set by the same rule, date it, and let the old one become history. Off-ramp: if a fresh draw is impossible, because the cases are finite, then the count is the only honesty left. Report it.

The look count, four rules: draw once and seal it, decide the gates before the first look, count the looks, draw fresh when it is spent

What the rules are not

They are not a claim that every reused test is worthless. The field's own evidence says reuse under protected conditions did little harm, and the field's own instruments can make repeated looks safe. The rules are for the ordinary case, a team with no private split, no budget of looks and a scoreboard everyone can see.

They are not about contamination alone. A test leaking into a model's training data is the loud version of the problem, and the weekly-rebuilt benchmark is one answer to it. The quiet version is the test leaking into your own decisions, and no refresh fixes that. Only the count does.

They are not a rule against learning from failures. Engineering runs are for looking. The exam is not.

What I sign for

I read medical images every day, and I evaluate frontier vision models blind, against a two-radiologist reference. The exam those models will sit was drawn once, sealed, and dated, with its gates written in a fixed order before the first training run. I know its look count because it is zero, and I will publish the count with the score.

A radiologist's signature is worth something because the read was made before the result was known. A score is worth something for the same reason. If you build or evaluate models that read images, the contact page reaches me directly. A radiologist who keeps the count can sit on your side of the table.

References

Footnotes

  1. Whitehill J. Climbing the Kaggle leaderboard by exploiting the log-loss oracle. arXiv, 6 July 2017. Rank 4 of 848 entrants on the Intel and MobileODT Cervical Cancer Screening competition, by repeated queries to the public leaderboard, without a classifier that solved the task. https://arxiv.org/abs/1707.01825 ↩

  2. Dwork C, Feldman V, Hardt M, Pitassi T, Reingold O, Roth A. Generalization in adaptive data analysis and holdout reuse. NeurIPS, 2015. https://arxiv.org/abs/1506.02629 ↩

  3. Dwork C et al. The reusable holdout: preserving validity in adaptive data analysis. Science, 2015, volume 349, page 636. The Thresholdout mechanism, differential privacy applied to holdout access, with a query budget set by the holdout's size. https://www.cis.upenn.edu/~aaroth/reusable.html ↩

  4. Roelofs R, Shankar V, Recht B, Fridovich-Keil S, Hardt M, Miller J, Schmidt L. A meta-analysis of overfitting in machine learning. NeurIPS, 2019. More than 100 Kaggle competitions; little evidence of substantial overfitting from adaptive leaderboard reuse. https://proceedings.neurips.cc/paper_files/paper/2019/file/ee39e503b6bedf0c98c388b7e8589aca-Paper.pdf ↩

  5. Recht B, Roelofs R, Schmidt L, Shankar V. Do ImageNet classifiers generalize to ImageNet? ICML, 2019. New test sets built by the original collection process; accuracy drops of 3 to 15 points tracked harder images, not reuse. https://arxiv.org/abs/1902.10811 ↩

  6. Yan et al. LiveMedBench: a contamination-free medical benchmark for LLMs with automated rubric evaluation. arXiv, 10 February 2026. 2,756 cases across 38 specialties, refreshed weekly from clinician forums; best of 38 models 39.2 percent; 84 percent degraded on post-cutoff cases; automated grader Macro F1 0.76 against a human-to-human baseline of 0.89. https://arxiv.org/abs/2602.10367 ↩

  7. Wang, Li, Ko, Zhang. On the fragility of benchmark contamination detection in reasoning models. arXiv, September 2025, revised March 2026. https://arxiv.org/abs/2510.02386 ↩

  8. Quantifying the effect of test set contamination on generative evaluations. arXiv, January 2026, revised February 2026. Score inflation scales with model capacity and training intensity, shrinks with further training on fresh data. https://arxiv.org/abs/2601.04301 ↩

  9. Johari R, Pekelis L, Walsh DM. Always valid inference: bringing sequential analysis to A/B testing. arXiv, December 2015; applied as "Peeking at A/B tests", KDD 2017. https://arxiv.org/abs/1512.04922 ↩

  10. ABA Journal. California bar hunts for who leaked bar questions; applicants sue test administrator. 2025. The February 2025 exam's questions posted online; the March makeup sitting moved from 3 to 4 March to 18 to 19 March. https://www.abajournal.com/web/article/cali-bar-hunts-for-who-leaked-bar-questions-applicants-sue-test-administrator ↩

  11. Flying Magazine. FAA updates sample knowledge test questions. The question bank rewritten in 2021 to defeat question harvesting. https://www.flyingmag.com/faa-updates-sample-knowledge-test-questions/ ↩

  12. Epoch AI. Clarifying the creation and use of the FrontierMath benchmark. 23 January 2025. Fifty problems held back from the funder. https://epoch.ai/latest/openai-and-frontiermath ↩

  13. TechCrunch. Meta's "vanilla" Maverick AI model ranks below rivals on a popular chat benchmark. 11 April 2025. https://techcrunch.com/2025/04/11/metas-vanilla-maverick-ai-model-ranks-below-rivals-on-a-popular-chat-benchmark ↩

  14. Cruz, Aji. LLM Olympiad: why model evaluation needs a sealed exam. arXiv, 24 March 2026. https://arxiv.org/abs/2603.23292 ↩