Skip to content
Reza Motaghi
One-Person Startup9 min read

Same AI model, different answer: record the runtime too

By Reza Motaghi

A CT scanner ring with a plain test block resting on its table where a patient would lie, drawn in a single white line on black, a minimalist illustration about the phantom every scanner images before it images a patient, the canary for the whole instrument
On this page

In 2012 a group in Maastricht ran the same brain MRIs through the same analysis software and got different brains back. A version change in the software moved measured volumes by 8.8 percent on average, and one structure by 64.1 Running the identical version on a different workstation, or on the next release of the Mac operating system, moved them again. Nothing about the patients had changed. The instrument had.

That paper is fourteen years old and it describes exactly what happens to anyone evaluating an AI model today. Everyone records the model. Almost nobody records the software the model ran on. Which means the number you are comparing this quarter's model against may not exist any more.

What the reproducibility record is

The reproducibility record is five lines kept next to every evaluation number. The whole stack that produced it. That stack pinned for the length of the comparison. A canary input run before each session. One change at a time when a result flips. The log stored with the result. The idea behind it fits in one sentence. The runtime is part of the instrument. Weights are the lens. The inference engine, its version, the drivers, the quantisation, the hardware, the prompt and the seed are the rest of the microscope. A measurement taken through a different microscope is a different measurement.

Same image, opposite answer

Same image, same weights, same prompt, same settings, same seed, checked twice. Weeks apart, a local vision model gave me a clinically opposite answer on one of my own cases. Same-day runs came back identical every time. The only thing that had moved between the two dates was the inference software underneath. Without the first run's stack written down, I had one experiment with two results and no way to know which to believe.

I read images every day, and I evaluate frontier vision models on my own cases, blind. The flip did not cost me a diagnosis. It cost me every earlier number, because I could no longer say what they had been measured with. Every check I run on a model, reading cold or handing it an off-average case, assumes the instrument held still between runs.

Where the drift comes from

The drift is mechanical, and it has been measured. Floating-point addition is not associative, so the order in which a GPU sums a row of numbers changes the result in the last digits.2 Most inference kernels choose that order by the size of the batch, and the batch depends on who else is using the server. A lab that ran a thousand identical greedy requests through one open model got 80 different completions, diverging from the hundred-and-third token.3 Batch-invariant kernels made all thousand identical. The fix was real engineering, which is the proof that the problem was real.

The engines say so themselves. vLLM's own documentation promises reproducibility only on the same hardware and the same vLLM version.4 When one open-weight model was served by a dozen providers, its score on the same maths benchmark ranged from 37 to 93 percent.5 One outlier traced to a serving-engine commit that had changed a default. A study that held weights, decoding and hardware constant and changed only the inference backend found scores shifting by up to 16.6 percentage points.6 The same study read 35,000 machine learning papers and found the inference stack is rarely reported.

Two details matter for anyone who evaluates rather than benchmarks. The divergence concentrates on tokens the model is unsure about, and on long reasoning chains.7 That is to say, on the cases you actually care about. And quantisation is the most common thing a provider changes without saying. It shifts a biomedical model's accuracy by a few points in either direction depending on the prompt template.8

Radiology already wrote the rule

Radiology treated the machine as part of the read long before anyone trained a model. Under US mammography law, the image quality test is a phantom, a plastic block imaged at least weekly on every unit.9 CT scanners are checked daily against water and air, the two ends of the Hounsfield scale. Accreditation requires phantom images that prove the numbers still mean what they meant.10 When quantitative imaging wanted to compare a measurement across hospitals, the field wrote metrology standards that specify scanner, protocol and software before the number counts.11 Nobody compares two scans without knowing the reconstruction kernel. With the same kernel, 84 percent of quantitative features reproduce. Across kernels, 15.12

The field also knows what a setting moving underneath looks like. In 2009 a Los Angeles hospital found that its brain perfusion CT protocol had been delivering about eight times the intended dose. It had done so for eighteen months, to more than two hundred patients, after a configuration change nobody had connected to the output.13 The scans looked fine. A patient's hair fell out. That is what an unlogged change in the stack looks like from the outside. The result looks normal until something you were not measuring tells you it was not.

Beyond the reading room

Software has been quietly part of the instrument in every computational field. Chemists computing NMR shifts with a widely used script got different answers on Linux, Windows and macOS.14 The script paired files by directory listing order, and the three systems sort directories differently. Deep reinforcement learning results changed with the random seed alone.15 Across seventeen fields, a review found hundreds of published machine learning papers whose results rested on leaked data.16 The reproducibility programme at a major machine learning conference exists because the field could not rerun itself.

The counter-evidence is worth stating plainly, because it sharpens the record rather than weakening it. The largest invisible variable is not the runtime. It is the prompt: formatting alone, with the words unchanged, moved one model's accuracy by up to 76 points.17 It is why a prompt copied between models fails in the first place. So the prompt and its settings are a line in the record, not an afterthought. And a flip is not by itself a verdict. In 2023 a study reported one frontier model getting worse at arithmetic between March and June. The sharper reading was that its behaviour had drifted while its capability had not.18 A flip tells you something moved. Only the record tells you what.

The reproducibility record, in five lines

The lines are short because you have to keep them on every run, not just the ones that go wrong. Each carries its evidence, and two carry a limit.

1. Record the whole stack, not the model. Runtime and version, drivers, quantisation, hardware, exact prompt and settings, seed. This is what MLPerf already requires of a submitted result, the software versions of every component. Its rules put it plainly: a result that cannot be replicated is not a valid result.19 The model card, the record most people keep, has no field for any of it.20 The limit: a record is not a promise. The engines only guarantee reproducibility on the same hardware and version. The record's job is to let you notice a change, not to prevent one.

2. Pin it for the length of the comparison. Nothing in the stack changes between the two numbers you compare. If it must, the comparison starts again from the first number. This is not a call to freeze forever. Production practice upgrades a stack through shadow runs and canary releases, new stack beside old on the same inputs, and only then switches.21 The record makes that possible. Without it you cannot tell an upgrade from a drift.

3. Keep a canary. One input from your own set whose answer you know cold, run first, every session, same settings, its answer logged next to the runtime version. It is the same test that catches a model update changing behaviour your workflow depends on, pointed at the stack instead of the model. This is the phantom, the glass of ice water for the thermometer. The day it moves, nothing else you run that day counts until you know why.

4. When a result flips, change one thing at a time until it flips back. That line of the record is your cause. Not the model. Not "it got worse". The 2023 arithmetic story is the cautionary tale: the flip was real and the wrong cause was attached to it. Deterministic modes now exist in the major serving engines. Turn them on, and write down that you did.22

5. Store the log with the result. A number without its stack is an anecdote. The regulators of medical AI have arrived at the same place from the other side. Version control of the deployed software is a guiding principle, and a device that changes must say how.23 Of the radiology AI devices the FDA has cleared, close to half have had software version updates.24 The adverse-event database has no field to record what a version change did.

The reproducibility record, five lines. 1, record the whole stack. 2, pin it for the comparison. 3, keep a canary. 4, change one thing at a time. 5, store the log with the result.

Two things the record is not. It is not a claim that every number moves. On a single machine, same day, my runs were identical every time, and rank order between models often survives even when absolute scores shift. It is not paperwork for its own sake either. It is the difference between "the model got better" and "something got better", and only one of those is a finding.

What I sign for

A radiologist does not compare two scans without knowing the scanner. I stopped comparing two model outputs without knowing the stack. The record above is what I keep now. The flip that taught it to me is why my results carry their runtime the way a report carries its protocol. When I built my own CBCT viewer, the agent got the camera and the human kept the read. The same discipline holds here. The machine's part of the work is recorded so that the human's part can be trusted. It is one half of how I evaluate models on my own cases. The other half is the reading, and neither is worth much if the instrument moved between runs.

Save the record. Run the canary before you trust tomorrow's first number. If you evaluate models and want to compare notes on what the stack does to a result, the contact page reaches me directly.

References

Footnotes

  1. Gronenschild EHBM, Habets P, Jacobs HIL, Mengelers R, Rozendaal N, van Os J, Marcelis M. The effects of FreeSurfer version, workstation type, and Macintosh operating system version on anatomical volume and cortical thickness measurements. PLOS ONE, 2012. Version change: volume differences 8.8 percent on average, range 1.3 to 64.0, cortical thickness 2.8 percent. Workstation and OS version effects roughly half that. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0038234 ↩

  2. PyTorch documentation, Reproducibility. Floating-point addition is not associative, and several operations are non-deterministic by default unless deterministic algorithms are forced. https://docs.pytorch.org/docs/stable/notes/randomness.html ↩

  3. Thinking Machines Lab. Defeating nondeterminism in LLM inference. September 2025. One thousand greedy runs of one open model, 80 distinct completions, first divergence at token 103, root cause batch-size dependence in reduction kernels. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ ↩

  4. vLLM documentation, Reproducibility. "vLLM only provides reproducibility when it runs on the same hardware and the same vLLM version." https://docs.vllm.ai/en/latest/usage/reproducibility/ ↩

  5. Willison S. Open weight LLMs exhibit inconsistent performance across providers. August 2025, on data from Artificial Analysis. gpt-oss-120b scored between 36.7 and 93.3 percent on AIME 2025 across providers. One provider's shortfall traced to old vLLM commits that did not respect the reasoning-effort setting. https://simonwillison.net/2025/Aug/15/inconsistent-performance/ ↩

  6. Pape, Evertz, Schönherr. The silent hyperparameter: quantifying the impact of inference backends on LLM reproducibility. 2026. Five engines including vLLM, SGLang and llama.cpp, weights, decoding and hardware held constant, shifts of up to 16.6 percentage points. Survey of 35,000 publications. https://arxiv.org/abs/2605.19537 ↩

  7. Beyond reproducibility: token probabilities expose large language model nondeterminism. 2026. https://arxiv.org/abs/2601.06118. Understanding and mitigating numerical sources of nondeterminism in LLM inference. 2025. Up to 9 percent accuracy variation for a distilled reasoning model under bf16 greedy decoding from GPU count, type and batch size. https://arxiv.org/abs/2506.09501 ↩

  8. Rasmussen, Qin. Quantization effects on biomedical LLM reliability. 2026. INT8 within one to two points of FP16 for specialised models, up to plus 4.2 points on one template. https://arxiv.org/html/2608.03854v1 ↩

  9. 21 CFR 900.12(e)(2), Mammography Quality Standards Act regulations: "an image quality evaluation test, using an FDA-approved phantom, at least weekly." Digital units follow the manufacturer's quality control programme, which keeps the weekly phantom. https://www.law.cornell.edu/cfr/text/21/900.12 ↩

  10. American College of Radiology, CT Accreditation Program requirements (phantom images for CT number accuracy, uniformity and low-contrast resolution, with an annual physicist survey). https://www.acr.org/Accreditation/Modalities/CT ↩

  11. Sullivan DC, Obuchowski NA, Kessler LG et al. Metrology standards for quantitative imaging biomarkers. Radiology, 2015. https://pubs.rsna.org/doi/10.1148/radiol.2015142202 ↩

  12. Choe J et al. Deep learning-based image conversion of CT reconstruction kernels improves radiomics reproducibility for pulmonary nodules or masses. Radiology, 2019. 592 of 702 features reproducible with the same kernel (84.3 percent), 107 of 702 across kernels (15.2 percent). https://pubs.rsna.org/doi/10.1148/radiol.2019181960 ↩

  13. Wintermark M, Lev MH. FDA investigates the safety of brain perfusion CT. American Journal of Neuroradiology, 2010. About 206 patients at one facility, roughly eight times the expected dose over eighteen months, discovered after a patient reported hair loss. https://pubmed.ncbi.nlm.nih.gov/19892810/ ↩

  14. Bhandari Neupane J et al. Characterization of leptazolines A to D, polar oxazolines from the cyanobacterium Leptolyngbya sp., reveals a glitch with the Willoughby-Hoye scripts for calculating NMR chemical shifts. Organic Letters, 2019. https://pubs.acs.org/doi/10.1021/acs.orglett.9b03216 ↩

  15. Henderson P et al. Deep reinforcement learning that matters. AAAI, 2018. https://ojs.aaai.org/index.php/AAAI/article/view/11694 ↩

  16. Kapoor S, Narayanan A. Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 2023. 329 papers across 17 fields. https://arxiv.org/abs/2207.07048. Pineau J et al. Improving reproducibility in machine learning research: a report from the NeurIPS 2019 reproducibility program. JMLR, 2021. https://www.jmlr.org/papers/volume22/20-303/20-303.pdf ↩

  17. Sclar M, Choi Y, Tsvetkov Y, Suhr A. Quantifying language models' sensitivity to spurious features in prompt design. ICLR, 2024. Up to 76 accuracy points from formatting alone in a few-shot setting. https://arxiv.org/abs/2310.11324 ↩

  18. Chen L, Zaharia M, Zou J. How is ChatGPT's behavior changing over time? 2023. https://arxiv.org/abs/2307.09009. Narayanan A, Kapoor S. Is GPT-4 getting worse over time? July 2023, the capability-versus-behaviour reading. https://www.normaltech.ai/p/is-gpt-4-getting-worse-over-time ↩

  19. MLCommons, MLPerf Inference rules. Submissions must specify the software version of all components, hardware configuration and settings. "Results that cannot be replicated are not valid results." https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc ↩

  20. Mitchell M et al. Model cards for model reporting. FAT*, 2019. https://arxiv.org/abs/1810.03993 ↩

  21. Breck E, Cai S, Nielsen E, Salib M, Sculley D. The ML test score: a rubric for ML production readiness and technical debt reduction. IEEE Big Data, 2017. https://research.google/pubs/the-ml-test-score-a-rubric-for-ml-production-readiness-and-technical-debt-reduction/ ↩

  22. LMSYS. Towards deterministic inference in SGLang and reproducible RL training. September 2025. https://www.lmsys.org/blog/2025-09-22-sglang-deterministic/. vLLM documentation, batch invariance. https://docs.vllm.ai/en/latest/features/batch_invariance/ ↩

  23. FDA, Health Canada, MHRA. Good machine learning practice for medical device development: guiding principles. October 2021. https://www.gov.uk/government/publications/good-machine-learning-practice-for-medical-device-development-guiding-principles/good-machine-learning-practice-for-medical-device-development-guiding-principles. FDA, Marketing submission recommendations for a predetermined change control plan for AI-enabled device software functions, final guidance, December 2024. ↩

  24. The absence of full lifecycle risk management for AI-based medical devices in radiology. npj Digital Medicine, 2026. Of 956 FDA-cleared radiology AI devices, 429 had software version updates. https://www.nature.com/articles/s41746-026-02712-4 ↩