Research note
A receipt for every model
The name, and the goal
We took the name because we are early, and because the questions remain open. Unorganized Machines is a research company that builds from the foundations. Sometimes that means asking how a computation can be checked. Sometimes it means returning to the browser, or reconsidering how people keep and organise their work. Each project gives one part of the machine a shape.
The goal of the present work is simple to say: a world where every model comes with a receipt, and nothing a machine computes has to be taken on faith. Two papers are written. The first partners are being chosen. The rest of the machine is still deciding what it is.
The papers
- GPU MechanismDraftSame Architecture, Same SM Count Is a Sufficient Condition for One Kernel Family and Neither Necessary Nor Sufficient for Another: Measuring, Naming and Removing the Device Dependence of GPU Training Bytes
- The step oraclePaper 2 · Revision 3A GPU training step, stated before the card runs and verified on a CPU
Further reading
The problem Machines that are wrong and do not say so.
- Meta — The Llama 3 Herd of Models, §3.3.4: Reliability and Operational Challenges
A 54-day training snapshot: 419 unexpected interruptions, six attributed to silent data corruption.
- Dixit et al. — Silent Data Corruptions at Scale (2021)
CPU errors that leave no hardware alarm, found by comparing computations with known answers.
- Hochschild et al. — Cores that don’t count (2021)
Cores that pass manufacturing tests and still compute wrongly: a few per several thousand machines.
- Ma et al. — Understanding Silent Data Corruption in LLM Training (2025)
The fault followed through training, where weights can drift even while the loss looks much the same.
The arithmetic Why two correct machines disagree.
- PyTorch — Numerical accuracy
The same inputs need not give the same answer on a CPU and a GPU, even with randomness controlled.
- NVIDIA — cuBLAS, §2.1.4: Results Reproducibility
The vendor’s reproducibility promise has boundaries: architecture, SM count, and software version.
- PyTorch issue #105726 — A100 and 3090 arithmetic
An A100/3090 disagreement, followed through to the library’s choice of arithmetic.
The prior art Checking a machine you did not watch.
- Jia et al. — Proof-of-Learning: Definitions and Practice (2021)
A protocol for checking a training history, with room for the numerical differences between machines.
- Fang et al. — Proof-of-Learning is Currently More Broken Than You Think (2023)
What can slip through when a verifier must allow for those differences.
- Srivastava, Arora & Boneh — Optimistic Verifiable Training by Controlling Hardware Nondeterminism (2024)
A route to exact training replication, using higher precision and recorded rounding decisions.
- Gensyn — Verde: Verification via Refereed Delegation for Machine Learning Programs (2025)
Exact agreement through reproducible operators, with the cost of replacing them measured.
- Prime Intellect — TOPLOC: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference (2025)
Compact checks for inference, where the question is whether the promised model actually answered.
- Horace He / Thinking Machines — Defeating Nondeterminism in LLM Inference (2025)
The inference side of the question: why a different batch can change an answer, and how to prevent it.
The origin A machine whose organisation is not yet finished.
- Alan Turing — Intelligent Machinery (1948)
The unorganised machine, and the possibility of learning through what happens to it.