A receipt for every model
Summary
When a model finishes training, can you tell whether the machines actually did the work — every step, on every processor, exactly as claimed? Today, no one can. The result is simply believed, and when one of a thousand processors is broken or dishonest, the error hides in the model until someone notices weeks later, if they ever do. We have built the check. A training run becomes a file a few hundred kilobytes long. From that file, an ordinary computer with no special hardware states in advance what the machines will produce, confirms afterwards that they did, and — if a machine lied or failed — names which one and at which step. We have done this on four kinds of processor, writing each prediction into a public record before the hardware was rented. It agreed every time.
The problem, and who it is for
An AI model is made through a long conversation between machines. One processor returns a number; another uses it to take the next step. Across a large training run, these small exchanges accumulate into something people will later rely on. Much of that work happens on hardware its eventual users will never see.
Failures are part of this landscape. Meta’s account of training Llama 3 describes faulty processors, memory problems, and silent data corruption. Operators already monitor their machines and recover from many interruptions. The harder question begins with an apparently ordinary result: was the computation behind it actually carried out as intended? A dishonest operator makes the same question more urgent: the result may describe work that never happened.
Even a healthy machine can make the answer difficult to compare. Computer arithmetic has finite precision, and the order of operations matters. PyTorch’s numerical accuracy guide explains that the same inputs need not produce identical results on a CPU and a GPU. A difference can be expected behaviour, an error, or something that needs closer investigation.
This matters to the labs that train models, the networks that rent out computing power, and the regulators asking how a model was made. Each needs evidence that can travel beyond the room where the work happened. A useful receipt would let someone else check the calculation without having to trust the original operator.
Key words
- Training run
- The sequence of computations through which a model is fitted to its training data.
- Receipt
- Our shorthand for a compact account that lets another computer check the work described.
- Checksum
- A compact value calculated from data, used here to compare a recomputed result with the one recorded.
- Checkpoint
- A saved state of a model and its training process, used to resume or examine a run.
- Numerical precision
- The limited detail with which a computer represents and calculates numbers.
- Hardware architecture
- The design of a processor, including how it carries out its operations.
The papers
- GPU MechanismDraftSame Architecture, Same SM Count Is a Sufficient Condition for One Kernel Family and Neither Necessary Nor Sufficient for Another: Measuring, Naming and Removing the Device Dependence of GPU Training Bytes
- The step oraclePaper 2 · Revision 3A GPU training step, stated before the card runs and verified on a CPU
Further reading
The problem Machines that are wrong and do not say so.
- Meta — The Llama 3 Herd of Models, §3.3.4: Reliability and Operational Challenges
A 54-day training snapshot: 419 unexpected interruptions, six attributed to silent data corruption.
- Dixit et al. — Silent Data Corruptions at Scale (2021)
CPU errors that leave no hardware alarm, found by comparing computations with known answers.
- Hochschild et al. — Cores that don’t count (2021)
Cores that pass manufacturing tests and still compute wrongly: a few per several thousand machines.
- Ma et al. — Understanding Silent Data Corruption in LLM Training (2025)
The fault followed through training, where weights can drift even while the loss looks much the same.
The arithmetic Why two correct machines disagree.
- PyTorch — Numerical accuracy
The same inputs need not give the same answer on a CPU and a GPU, even with randomness controlled.
- NVIDIA — cuBLAS, §2.1.4: Results Reproducibility
The vendor’s reproducibility promise has boundaries: architecture, SM count, and software version.
- PyTorch issue #105726 — A100 and 3090 arithmetic
An A100/3090 disagreement, followed through to the library’s choice of arithmetic.
The prior art Checking a machine you did not watch.
- Jia et al. — Proof-of-Learning: Definitions and Practice (2021)
A protocol for checking a training history, with room for the numerical differences between machines.
- Fang et al. — Proof-of-Learning is Currently More Broken Than You Think (2023)
What can slip through when a verifier must allow for those differences.
- Srivastava, Arora & Boneh — Optimistic Verifiable Training by Controlling Hardware Nondeterminism (2024)
A route to exact training replication, using higher precision and recorded rounding decisions.
- Gensyn — Verde: Verification via Refereed Delegation for Machine Learning Programs (2025)
Exact agreement through reproducible operators, with the cost of replacing them measured.
- Prime Intellect — TOPLOC: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference (2025)
Compact checks for inference, where the question is whether the promised model actually answered.
- Horace He / Thinking Machines — Defeating Nondeterminism in LLM Inference (2025)
The inference side of the question: why a different batch can change an answer, and how to prevent it.
The origin A machine whose organisation is not yet finished.
- Alan Turing — Intelligent Machinery (1948)
The unorganised machine, and the possibility of learning through what happens to it.