Skip to content
Reza Motaghi
One-Person Startup10 min read

Why your AI validation has an expiry date

By Reza Motaghi

A tamper seal torn across the seam of a metal panel under one raking light, a black and white image about a change nobody announced
On this page

In August 2026 the American regulator recalled a software version used by radiologists to view their images. The reason on the enforcement report was one sentence: a software issue may cause the ruler measurement to be inaccurate.1 The ruler is the tool a radiologist uses to size a lesion. Six units were affected, across five states and one other country. The people using them had a viewer that looked exactly like the one they had validated. Only the ruler had changed.

That recall is the good case. It was found, reported and published. This article is about the ordinary case, where the change ships, nothing fires, and the validation you rely on has quietly expired.

I run enterprise AI in production, and I train a vision model on my own reads. The exam that model sits was written with a date on it and its gates in a fixed order, because I know the ground under any model moves. When it moves, I can re-run the same test and see the move. That is the whole difference between a validation and a hope.

What the expiry date is

The expiry date is the day a validation stops describing the tool in front of you. Every validation has one. It is set by whoever can change the tool, and in a hosted model or a cleared device that is never you. The date is real whether or not anyone prints it, and almost nobody prints it.

A validation is a statement about a tool at a moment. The tool is a moving thing with a stable name. The expiry date is where the two come apart, and the four rules below exist to put the date back on the label.

The exam with a date on it

The receipt is a rule I keep, not a case. Before the first training run on my own reads, I drew a held-out exam once, sealed it, and wrote the gates in a fixed order. The date went into the record. When I later found that the same image on the same weights could give an opposite answer weeks apart, because only the inference software had moved, the dated exam was what made the move visible. Without it I would have had two results and no way to know which day to believe.

The systems I sign for in production update on their vendors' schedules, not mine. So the same discipline applies there in reverse: the validation I hold is dated, and when the date passes, it is re-run or it is retired.

Where the changes come from

The public record on cleared devices is the place to start, because it is measurable. A 2026 study tracked 956 radiology AI devices cleared in the United States since 1995.2 Of those, 429 had a software version update after clearance.

The recalls tell the second half. Of 182 recall reports, 124 were tied to software defects. Only 38 of those recalls were followed by a corrective software update that could be traced, across eight products.

Two honest caveats travel with that number. The study was built from the regulator's public summaries, so it sees only the updates significant enough to be filed. It cannot see the changes a manufacturer routed through its own private record. And a thirty-year analysis of the adverse-event database shows the biggest jumps in reports track changes in reporting rules, not in devices.3 So the finding is not that devices are getting worse. It is that the industry does not close its own loop, and a user cannot see when it does.

Hosted models are the same gap without the paperwork. A 2026 study of sixteen model providers asked one question: could an outsider verify that the model being served was the model being documented.4 None disclosed enough. Five things change a served model with no version identifier changing: fine-tuning, filter updates, system-prompt revisions, retrieval changes, and routing. One provider kept a single identifier stable across ten documented releases over nineteen months.5

The clearest evidence comes from a vendor's own hand. One frontier lab published two postmortems a year apart. The first traced weeks of degraded output to infrastructure bugs under an unchanged model name, and said plainly that its evaluations did not capture the degradation users were reporting.6 The second traced a fresh regression to three product-layer changes, none of them a model change, that neither internal use nor evals reproduced at first.7 Both are honest documents. Both arrived after the fact, which is when every disclosure arrives.

Where disclosure does exist

The counter-case is real and the rules have to fit it. The same labs publish deprecation calendars. One commits to at least sixty days' notice before retiring a model and keeps a dated lifecycle table down to the snapshot.8 Another commits to six months.9 A third publishes shutdown dates by model version.10 For the change a vendor names and schedules, disclosure is good and getting better.

The drift nobody flags as a release is a different animal, and the rulebook treats it differently too. A device maker must report a malfunction likely to cause serious injury.11 It need not report a bug fix. European guidance names bug fixes and security updates as changes that are not significant, so they ship without review.12 A change below the threshold goes into the manufacturer's own file, which nobody outside sees unless the regulator later asks.13 A device with an approved plan for future changes carries a line on its label saying it may receive periodic updates.14 That is consent to change, given once. It is notice of no particular one.

Europe's rule for high-risk AI is still being built. As of mid-2026 no notified bodies had been designated under the AI Act, and devices already regulated as medical devices have until August 2028 to meet its obligations.15 The radiologists' own society says some device providers rely on voluntary, periodic reporting by the hospitals that use them.16

So the sharp claim is not that nobody discloses. It is that every mechanism fires after harm, after the maker's own private judgment that a change was minor, after complaints pile up, or after a scandal. None fires on the change itself. The gap is a timing gap and a threshold gap, and it is yours to close.

Every field has a stable name over a moving thing

An eleven-line package with fifteen million downloads was pulled from a registry in 2016, and build systems at some of the largest companies in the world broke within hours.17 Nobody who depended on it had touched their code. In 2019 a signing certificate inside a browser expired at midnight and every extension signed under it stopped loading for millions of people who had changed nothing.18 In 2023 a platform shut off the free tier of its programming interface with days of notice and thousands of tools built on it went dark.19

Each time the thing kept its name. Each time the change was found by the people it broke. A stable name is not a stable thing, and an update can change behavior your workflow depends on without a single line of your own changing.

Four rules before the ground moves

The rules are cheap because they are bookkeeping, not tooling. One of them I changed after reading the deprecation calendars.

1. Put an expiry date on every validation. Ninety days, six months, or the vendor's release cadence, whichever is shortest. Write it on the validation itself. When the date passes, the validation is re-run or it is retired, and until then it is not cited. My first version of this rule said nobody discloses changes. The calendars changed it: the vendors with the best discipline disclose the change they name and schedule, so the date can follow their cadence. It cannot follow the drift they never name. Off-ramp: a tool that is frozen, pinned and run on your own hardware has no vendor calendar. Date it to your own next change.

2. Ask the vendor three questions. What changed since I validated it, when, and how would I have known if you had not told me. The third question is the only one that matters. A vendor who can answer it has a change log you can read. A vendor who cannot has told you where the change log lives, which is with you. Off-ramp: a vendor with a published lifecycle table and a scorecard of disclosures, the kind the 2026 study proposes, has already answered. Read the table instead.

3. Keep a fixed suite and re-run it on their calendar and yours. The same cases, the same order, the same gates, dated. The 2025 postmortem is the proof that a vendor's evaluations are not yours: their suite did not catch the degradation their users reported. Your suite is the one built for your cases, and it only answers honestly while nobody has tuned to it, so keep it sealed between runs. The day a number moves, stop until you know why. Off-ramp: a suite too expensive to re-run monthly can be a canary, one fixed input with a known output, run before every session. The runtime piece carries that one.

4. Treat a silent change as a recall you must find yourself. When the suite moves and no notice came, log it, date it, and count it. The count is the vendor's real change log, and after a year it is also your renewal argument. Off-ramp: none. A silent change you did not log is one you will find again, later, in a complaint.

The expiry date, four rules: date every validation, ask the vendor three questions, re-run a fixed suite on their calendar and yours, count the silent changes

What the rules are not

They are not a claim that vendors hide changes. Two of the clearest documents in this article are a vendor's own postmortems, published because the scope of the problem justified it. The rules are for the changes no postmortem will ever be written about.

They are not an argument against updates. A fixed tool decays with the world around it. The rules ask for a dated validation, not a frozen product.

They are not a substitute for the regulator. The device rules exist and the recall in the first paragraph shows them working. They fire on harm. The rules here fire on the date.

What I sign for

I read medical images every day, I run enterprise AI in production, and I evaluate frontier vision models blind and dated, against a two-radiologist reference. The machine is part of the read. The machine changes on a calendar I do not own. The dated exam is how I find out, and what does not depreciate when the model changes is the record I keep around it.

Date the last validation you still trust. If the date is missing, it has already expired, and the only question is what changed in the meantime. If you build or evaluate models that read images, the contact page reaches me directly. A radiologist who keeps the date can sit on your side of the table.

References

Footnotes

  1. FDA enforcement report Z-3030-2026, Class II, FUJIFILM Synapse PACS software version 7.4.310, 26 August 2026: "a software issue may cause the ruler measurement to be inaccurate", six units across five US states and Australia. Reproduced by SoftwareCPR. https://www.softwarecpr.com/2026/08/fda-recall-software-issue-may-cause-the-ruler-measurement-to-be-inaccurate/ ↩

  2. Li J, Guo Z, Guo Y, Xiao R, Ding Y, Shi W, Liu W. The absence of full lifecycle risk management for AI-based medical devices in radiology. npj Digital Medicine, 4 May 2026. 956 devices, 1995 to 2025; 429 (44.87 percent) updated; 182 recall reports, 124 (68.13 percent) software defects, 38 (30.65 percent, eight products) followed by a corrective update. Built from public decision summaries and enforcement databases. https://pmc.ncbi.nlm.nih.gov/articles/PMC13346426/ ↩

  3. Mishali, Sheffer, Mishali, Negev. Evaluation of reporting trends in the MAUDE database, 1991 to 2022. Digital Health, January 2025. Report surges track mandatory electronic reporting from 2015 and a 2018 summary-reporting programme. https://pmc.ncbi.nlm.nih.gov/articles/PMC11755539/ ↩

  4. Abraham S, Bucknall B. Silent updates: measuring and closing the post-deployment disclosure gap. AIES-26, arXiv, 12 August 2026. Sixteen providers; five mechanisms; a proposed disclosure scorecard. https://arxiv.org/abs/2608.11803 ↩

  5. Pebblous. Silent model updates: the disclosure gap. 14 August 2026, summarising Abraham and Bucknall; one identifier across ten releases over nineteen months. https://blog.pebblous.ai/report/silent-model-updates-disclosure-gap/en/ ↩

  6. Anthropic. A postmortem of three recent issues. 17 September 2025. https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues ↩

  7. Anthropic. An update on recent Claude Code quality reports. 23 April 2026. https://www.anthropic.com/engineering/april-23-postmortem ↩

  8. Anthropic. Model deprecations. Accessed 16 September 2026; at least sixty days' notice; dated lifecycle table. https://platform.claude.com/docs/en/about-claude/model-deprecations ↩

  9. OpenAI. API deprecations. Accessed 16 September 2026; at least six months' notice for generally available models. https://developers.openai.com/api/docs/deprecations ↩

  10. Google. Gemini API deprecations. Accessed 16 September 2026. https://ai.google.dev/gemini-api/docs/deprecations ↩

  11. US Food and Drug Administration. Medical device reporting: how to report medical device problems. 21 CFR Part 803. https://www.fda.gov/medical-devices/medical-device-safety/medical-device-reporting-mdr-how-report-medical-device-problems ↩

  12. Sidley Austin LLP. Important new EU MDR guidance on "significant change" and clinical evidence for software, summarising MDCG 2020-3. 20 March 2020. https://www.sidley.com/en/insights/newsupdates/2020/03/important-new-eu-mdr-guidance-on-significant-change-and-clinical-evidence-for-software ↩

  13. US Food and Drug Administration. Deciding when to submit a 510(k) for a software change to an existing device. Guidance, October 2017. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/deciding-when-submit-510k-software-change-existing-device ↩

  14. Foley and Lardner LLP. AI-enabled device software functions: FDA's final guidance for predetermined change control plans. January 2025. https://www.foley.com/insights/publications/2025/01/fda-final-guidance-ai-device-software-predetermined-change-control-plan/ ↩

  15. Morgan Lewis. Recent EU and UK medical devices developments. 27 May 2026. https://www.morganlewis.com/blogs/asprescribed/2026/05/recent-eu-and-uk-medical-devices-developments-eu-mdr-ivdr-eu-ai-act-uk-medical-devices-regime ↩

  16. Cuocolo et al., European Society of Radiology. Consensus recommendations on AI medical device post-market surveillance. Insights into Imaging, December 2025. https://pmc.ncbi.nlm.nih.gov/articles/PMC12701188/ ↩

  17. The npm left-pad incident, 22 March 2016. https://en.wikipedia.org/wiki/Npm_left-pad_incident ↩

  18. VentureBeat. Mozilla issues Firefox fix after expired certificate disabled all add-ons. 4 May 2019. https://venturebeat.com/mobile/mozilla-issues-firefox-fix-after-expired-certificate-disabled-all-add-ons/ ↩

  19. Engadget. Twitter shut off its free API and it's breaking a lot of apps. 6 April 2023. https://www.engadget.com/twitter-shut-off-its-free-api-and-its-breaking-a-lot-of-apps-222011637.html ↩