Here’s a test that takes two minutes and that almost nobody runs before putting a model into production.
Ask it the same question. Then ask it again. Then a third time. Compare the answers.
Across twenty-one models, ten gave the same answer every time. Seven were all over the place. Four landed somewhere in between — mostly stable, occasionally not.
So a bit over half the models I tested will, given identical input, sometimes give you different output.
Why this is worse than it sounds
If you’re using AI to draft an email, variation is fine. Arguably it’s a feature — you’d rather not send the same six sentences to everyone.
But most business automation isn’t drafting. It’s deciding. Categorising a support ticket. Extracting a total from an invoice. Deciding whether a lead is worth a call. Marking a document as needing review. And in every one of those, the same input producing different output isn’t flavour, it’s a defect.
Consider what it does to your ability to operate:
You can’t reproduce a result. A customer disputes how their case was handled. You re-run it to see what happened. You get a different outcome. You now have no idea what the system did on the day, and no way to find out.
You can’t test a change. You tweak a prompt and re-run your examples. Half the differences you see are your change. Half are noise. You cannot tell which is which, so you’re tuning blind — and any confidence you gain from “it looks better now” is unearned.
You can’t audit. “Why was this application declined?” is a question with no stable answer. That’s uncomfortable in most contexts and legally serious in a few.
Your edge cases move. The thing you carefully verified works is a thing that worked that time. It might not be the behaviour in production, and you’d never know, because nothing failed.
What actually causes it
Some of it is settings. Models have a randomness dial — temperature — and it usually ships turned up, because a bit of variation makes conversation feel more natural. For anything that makes a decision, turn it down. That’s a config change and it’s free.
But not all of it is settings, and this is the part that catches people. Some models are far more stable than others at the same settings. Batching, hardware differences, and how the model was built all leak variation in through the sides. Turning the dial to zero reduces the spread; it doesn’t reliably eliminate it. Seven of my twenty-one were scattered enough that I wouldn’t put them behind a decision at all.
What to do
Test it. Same prompt, three runs, compare. Two minutes. Do this before you build anything on top of a model, not after you’ve discovered your ticket routing is 15% random.
Match the model to the job. Use the stable ones where output feeds a process. Save the variable ones for drafting and brainstorming, where variation is genuinely useful. This is a real distinction and it should drive which model goes where — not “which one is best”, but which one is right for this specific job.
Constrain the output shape. If you need one of four categories, make the model choose from four categories rather than write a sentence you then have to interpret. Structure removes a lot of room for drift.
Log what actually happened. If you can’t reproduce it, at minimum record it — the input and the output, every time. That turns “we can’t explain this” into “here’s exactly what it did”, which is usually what you actually needed.
The general lesson
Consistency is the least glamorous of the five things I check and the one that causes the most quiet damage, because inconsistent output never announces itself. There’s no error. Nothing goes red. The system just behaves slightly differently than it did last time, and everyone assumes the last time was the fluke.
Under half the models I tested are actually dependable in this respect. Whether yours is one of them is knowable in about two minutes — which makes not knowing a strange thing to accept.
Measured across twenty-one open-weight models on my own hardware in July 2026, three identical runs per model. Part of the Five Pillars methodology.