Ask ChatGPT the Same Question Twice and It Names a Different Brand First Three Times in Four, Study Finds
Published on
A study by BusySeed, a marketing agency in New York, found that ChatGPT often changes which brand it recommends first when people ask the same shopping question again.
The study compared 172,624 pairs of answers to the same question. The answers were collected between April 27 and October 16, 2026, across 172 types of products and services. In only 25.4% of the pairs did ChatGPT name the same brand first both times. In the other 74.6%, a different brand came first in the second answer.
The time between the two answers made almost no difference. When the answers were less than a week apart, the same brand came first 23.9% of the time. When more than four weeks passed between answers, the same brand came first 25.9% of the time.
This pattern appeared across many types of products and services. In 143 categories with enough data, the percentage of repeated questions that produced a different top brand ranged from 34.3% to 92.8%. The middle value was 77.3%. No single category made up more than 1.7% of all the data.
For every comparison, researchers used the same version of the model, the same web search setting, and exactly the same question wording. This means the changes were not caused by OpenAI updating the model.
A separate BusySeed study lasting 15 weeks looked at ChatGPT, Claude, and Gemini and found a similar pattern in all three AI assistants. The company named first changed in 56.4% of comparisons from one week to the next. Only 85 of the 480 sets of questions tracked kept the same top company for the entire study.
“If you check where you rank in ChatGPT on a Monday, you’ve learned something about Monday,” said Omar Jenblat, BusySeed’s founder and chief executive. “What holds up is how often you show up across many asks, not where you placed once.”
This finding matters because more companies are using tools that give them a single ranking in AI answers. The study suggests that such a number may show where a company appeared in one answer, rather than its position over time.
Founded in 2013 and based in New York City, BusySeed provides services in search, advertising, and finding potential customers. The company publishes its studies and the data behind them on its website.
The study used OpenAI’s developer interface, not the ChatGPT app that people log into. The app can personalize its answers. The study also could not tell whether the changes came from randomness inside the model or from real changes in the web pages the model reads. Methodology and data: https://busyseed.com/research/ask-chatgpt-the-same-question-twice.