AI Non-Determinism and Why the Same Prompt Gives a Different Answer
Key takeaways
|
It usually starts with a screenshot. A CEO asks ChatGPT for the best vendors in the company's category, doesn't see the company named and forwards the image to marketing with a one-word message. Someone in marketing runs the same question, gets an answer that does name the company and screenshots that one. Now there are two screenshots, two people who feel vindicated and one very unproductive afternoon.
Both screenshots are real. Neither one tells you anything useful on its own.
AI answers change every time you ask. That isn't a glitch in your tracking or a sign the model is broken. It's how these systems are built. Once a leadership team understands that, the conversation shifts from arguing about screenshots to measuring something that actually means something, and that shift is worth more than any single optimization you'll make this year.
What is AI non-determinism?
AI non-determinism is the tendency of AI models like ChatGPT, Claude and Gemini to produce different answers to the same prompt across separate runs. Differences can include which brands are named, their order, the sources cited and the wording. It happens because models sample from probabilities and because retrieved sources, personalization and model versions change. |
Why the answers keep changing
Four forces are at work, and they stack on top of each other. The first is sampling. Language models don't always pick the single most likely next word. They choose from a range of likely options, which is what makes their writing sound natural rather than robotic. It also means two runs of the same prompt can diverge from the very first sentence and end up in different places.
The second is retrieval. When an assistant searches the web before answering, the results it gets back and the passages it decides to use can differ from one run to the next, the process described in search grounding.
The third is personalization. Memory, custom instructions, location and conversation history all shape the answer, so your answer and your buyer's answer may differ for reasons neither of you can see, a topic covered in AI memory and personalization.
And the fourth is change over time. Providers update their models and systems regularly, sometimes without much announcement, so an answer from last month may have come from a meaningfully different system. Any one of these would create variation. Together, they make a single answer close to meaningless as a measurement.
What the research actually shows
The best public evidence comes from SparkToro and Gumshoe, who had roughly 600 volunteers run the same brand-recommendation prompts through AI tools, collecting 2,961 responses. The results are striking. The same list of brands came back less than once in a hundred runs. The same list in the same order came back less than once in a thousand.
But the most important finding is the one that's easy to miss. Underneath all that chaos, the brands that genuinely owned a category kept showing up. The lists were different and the order was different, but the core was remarkably consistent. Variation sits on top of a stable layer, and that stable layer is what you're trying to measure and, over time, to move.
Other research points the same way. Ahrefs found that Google's AI Mode and AI Overviews cited different sources 87% of the time for the same query. If two products from one company can disagree that often, it's a reminder that single answers were never going to be reliable evidence of anything.
The screenshot fallacy
I've started calling it the screenshot fallacy, treating one AI answer as evidence of where you stand. It drives a surprising number of bad decisions, from panicked strategy changes after one disappointing answer to false confidence after one flattering one.
The way I explain it to leadership teams is as a ladder of confidence. One answer is an anecdote; it might be typical or it might be a fluke. The same prompt run three to five times gives you a hint about whether you appear at all. Ten or more prompts, each run several times on one assistant, gives you a real pattern for that assistant. And thirty or more prompts, run several times across several assistants and tracked over time, gives you a measurement you can make decisions on.
So when someone forwards a screenshot, the most useful response isn't to argue about it. It's to ask, kindly, where it sits on that ladder.
Measuring something that won't sit still
The method isn't complicated, but it does require discipline. Start with a fixed set of buyer prompts, somewhere between thirty and a hundred, written the way buyers actually ask questions, so that changes in results reflect the market rather than your wording. AI search prompts covers how to build that set.
Run each prompt several times, three at a minimum and more for the prompts that matter most, across several assistants. ChatGPT, Perplexity, Google's AI features and at least one of Claude or Gemini.
Test in clean sessions wherever possible, logged out or with memory turned off, so your own history doesn't skew the results. Then stop thinking in answers and start thinking in rates. The percentage of responses that name you is your share of model. Track how often you're named first, how you're described and whether your pages are cited. And report the trend quarterly. Month-to-month movement is mostly noise. Quarter-over-quarter movement is where real signal shows up.
Tools like Semrush's AI Visibility Toolkit, Profound and Peec AI can automate a lot of this. A disciplined spreadsheet works too. What matters is consistency, not sophistication. AI search visibility tracking compares the options.
Reading what stays stable
Once you start measuring properly, a useful pattern emerges. Some things tend to stay stable across runs, such as whether category leaders appear at all, the broad framing of well-known brands, the main facts about well-documented companies and the answers to simple factual questions. Other things tend to be volatile, such as the order brands appear in, which specific sources get cited, the exact wording and the answers to open-ended recommendation questions.
That distinction is strategically useful. If you're missing from the stable layer entirely, you have an authority problem, and the fix is building presence across the web, the work described in brand mentions in AI answers. If you're present but your position and citations keep bouncing around, you're on the edge of the core, and the fix is usually content that's easier to retrieve and quote, as covered in content chunking for AI.
Explaining it to leadership
Executives don't need the mechanics of sampling and retrieval. They need a frame they can repeat. The one that works best in my experience is this. AI visibility is like polling, not like rankings. Nobody judges an election from a single phone call. You look at many polls over time and watch the trend.
Then give them that trend, one number, share of model this quarter versus last, alongside your top competitors. And agree in advance that single screenshots, good or bad, don't change the plan on their own. That one agreement will save you more meetings than anything else in this post.
AI non-determinism in the Engine Optimization Matrix™
Non-determinism is the reason the Citation column of the Engine Optimization Matrix™ is measured as rates rather than positions. In the SEO row, you can check a ranking. In the GEO and LLMO rows, whether your brand is named is a probability, and the work is to make that probability as high and as stable as it can be.
For related reading, see anatomy of an AI response, Perplexity vs ChatGPT citations and AI visibility.
Stop guessing which engine to fix first. Mastering AI Visibility puts AI SEO, AEO, GEO and LLMO in one Demand Doctrine™ edition, built on the Engine Optimization Matrix™. Out November 18.
Frequently asked questions
What is AI non-determinism?
It's the tendency of AI models to give different answers to the same prompt across runs, including different brands, orders, sources and wording.
Why does ChatGPT give different answers to the same question?
Because models sample from probabilities, retrieved sources vary, personalization differs and the underlying systems are updated over time.
How often do AI answers change?
Very often. In SparkToro and Gumshoe's research, the same list of brands came back less than once in 100 runs.
Is anything stable in AI answers?
Yes. The core brands in a category tend to appear consistently, even when order and sources change.
How do you measure AI visibility if answers change?
Run a fixed set of prompts several times across several assistants and measure the percentage of responses that name your brand.
How many times should I run a prompt?
At least three times per assistant, and more for your most important prompts. Base decisions on many prompts, not one.
Does personalization affect AI answers?
Yes. Memory, custom instructions, location and conversation history can all change what an assistant says, so test in clean sessions where possible.
Should I worry if AI didn't mention us once?
Not on its own. A single answer is an anecdote. Look at your appearance rate across many prompts and runs before drawing conclusions.
About the author. Harold Bell is the founder of MQL Magnet and has spent about 16 years in enterprise technology marketing, working with brands like Cisco, Rubrik and Nutanix in different capacities. He's a member of the Forbes Communications Council, the creator of the Engine Optimization Matrix™ and the author of Partner Over Product. |
