Can AI disrupt market research? Colgate publishes paper: letting AI "role-play" can perfectly predict market demand.
Some market perspectives accordingly suggest that the business model of traditional research is facing disruptioncompanies no longer need to spend weeks recruiting respondents, and can complete thousands of targeted interview simulations and pricing A/B tests in a very short time. Beyond the hype, however, the paper's authors and frontline observers offer a more cautious judgment. The research team explicitly states that synthetic consumers are not a comprehensive replacement for human research; multiple commenters on social platform X also caution that the human-like data distributions generated by this method may contain statistical noise, and that it remains to be tested in high-stakes decisions beyond mass consumer goods.
Title context: Can AI disrupt market research? Colgate publishes paper: letting AI "role-play" can perfectly predict market demand.
Text:
A new study jointly released by Colgate-Palmolive and PyMC Labs shows that simply changing the way questions are asked so that large language models "role-play" consumers can make AI predictions of real purchase intent reliable enough to reach 90% of human test-retest reliability.
The paper, posted on the preprint platform arXiv, has recently caused a stir in overseas technology and business circles. Companies worldwide spend billions of dollars annually on consumer research, and some market observers believe the business model of traditional research is coming under pressurecompanies no longer need to spend weeks recruiting respondents and can complete thousands of targeted interview simulations and pricing A/B tests in a very short time.
Beyond the hype, however, the paper's authors and frontline observers offer a more cautious judgment. The research team explicitly states that synthetic consumers are not a comprehensive replacement for human research; multiple commenters on social platform X also warn that the human-like data distributions generated by this method may contain statistical noise and still need to be tested in high-stakes decisions beyond mass daily consumer goods.
Solving AI's "fence-sitting" problem: moving away from direct scoring
Before putting fast-moving consumer goods into production, companies usually test product concepts with representative consumer samples, and the core metric is purchase intent (PI) measured on a 5-point Likert scale (1 = "definitely would not buy," 5 = "definitely would buy"). This decades-old process is costly, and respondents often answer perfunctorily or show acquiescence bias and positivity bias, leaving demand signals full of noise.
Academia and industry have long attempted to use AI as "synthetic consumers," but directly asking models to score from 1 to 5 (direct Likert rating, DLR) often yields little. After testing OpenAI's GPT-4o and Google's Gemini-2.0-flash, the paper found that when scoring directly, the models tend to retreat to the middle "safe score" of 3 and rarely give 1 or 5, while real human answers are mostly concentrated at 4 and 5.
As a result, the response distribution similarity (KS similarity) of the two models was as low as 0.26 and 0.39, respectively. If prompts are modified to force the models toward the high-score range, the models become "over-corrected," distorting the relative ranking signal of product concepts.
The research team believes the problem is not the large language models themselves, but the way questions directly demand numbers. To address this, the paper proposes semantic similarity rating (SSR): first input specific demographic characteristics and product concept images into the model and let it freely write down purchase thoughts; then use OpenAI's text embedding model (text-embedding-3-small) to extract text vectors, calculate their cosine similarity with 5 preset anchor statements, average across 6 reference statement sets, and finally convert the result into a probability distribution on the 5-point scale.
This zero-shot framework requires no historical research data for training or fine-tuning and offers the low-cost advantage of being plug-and-play. Its Python code has been open-sourced on GitHub.
Tested on 9,300 real samples: beats traditional algorithms, but cannot do without "personas"
The research team used 57 real concept studies completed by a leading personal care company in the U.S. market as a benchmark. Each study included 150 to 400 independent respondents, for a total sample of 9,300 people.
Under full profiles and image-text concept stimuli, SSR raised the correlation achievement rate of GPT-4o and Gemini-2.0-flash on relative product rankings to 90.2% and 90.6%, respectively (using the correlation upper bound from simulating 2,000 random human group retests as the reference). The distribution KS similarity of the two models also jumped to 0.88 and 0.80, far ahead of the direct scoring method and also higher than the follow-up rating method (FLR), in which text is first written and then translated by an AI "scoring expert" (0.72 and 0.59 for the two models, respectively).
SSR also performed steadily in generalization and dimensionality-reduction tests. Faced with the new question "How relevant is the product concept to you," Gemini-2.0-flash achieved a correlation achievement rate of 82% and a distribution similarity of 0.81; when the image-text concept was simplified to plain text, model performance declined only slightly.
A more critical comparison came from traditional machine learning. The researchers randomly selected 28 studies as a training set and 29 as a test set, repeatedly training 300 gradient-boosted decision tree models (LightGBM) using 5 demographic features and 3 product attributes. Even with in-sample data, LightGBM's correlation achievement rate was only 64.6%, with a distribution similarity of 0.80; under the same split, zero-shot GPT-4o with SSR achieved 88.3% and 0.88, respectively.
But the experiment also drew a red line: "personas" are indispensable. Once all demographic characteristics were removed from the Gemini-2.0-flash prompt, although the synthetic samples could produce an average score consistent with the real data (4.00.1) and a distribution similarity as high as 0.91, under the best reference set the correlation achievement rate for product rankings plunged from 92% to 50%. AI lacking profile constraints only gives positive reviews and loses the core function of screening products.
On segmented dimensions, synthetic consumers reproduced real consumption patterns fairly well. In real research, purchase intent showed an inverted U-shaped distribution with age (higher in middle age than in youth and old age), and GPT-4o accurately reproduced this trend; when faced with income settings under budget constraints and different categories and price tiers, AI's preference rankings also closely matched humans.
The accompanying free text also brought an unexpected benefit. Real respondents' open-ended comments often stop at "pretty good," but AI can provide detailed criticism with little positivity bias, and the span of its predicted means is wider, which instead provides more discriminative signal for early screening. Faced with an unpopular concept, Gemini-2.0-flash said bluntly: "Sounds expensive, I'm not really buying those 'microbiome' claims, I'd rather stick with products I'm familiar with."
Disrupting the industry or a "stochastic parrot"?
In their conclusion, the paper's authors outline a new R&D path: companies need not pay for every early idea, but can first use synthetic consumers for low-cost broad screening and reserve expensive human samples for the most promising candidates. This can accelerate iteration and also give small and medium-sized brands access to consumer insights they could not previously afford.
However, the research team did not avoid the method's hard flaws. The 6 anchor statement sets used in the experiment were manually optimized for these 57 studies, and their generalizability across studies remains to be seen; although AI can simulate age and income differences fairly well, it performs unstably on gender, region of residence, and ethnicity dimensions.
A more fundamental constraint comes from the training corpus. SSR works in oral care and personal care largely because online forums and consumer reviews contain massive amounts of relevant discussion; once it enters unfamiliar categories with scarce corpus, large models will still hallucinate. Real-world hard budget constraints, cultural context, and marketing exposure are also difficult for synthetic samples to fully capture.
The debate on social platform X reflects industry divisions over this trend. User James He called semantic similarity elicitation "a fun little trick," noting that the way it creates human-like distributions "is essentially just noise, and loses all covariance between items," and that "it may be enough for deciding toothpaste options, but not necessarily for the most critical decisions."
User James Sutton saw this as a typical example of a "stochastic parrot": what the large model outputs is merely "the mean of the internet's means," and role-playing is essentially simplified world-building based on conditional logicnot real thinking, yet still useful.
Other observers saw the potential of this success. User MartianCats commented that commercial advertising ubiquitously uses "choice architecture" to guide consumption, and now large models are beginning to directly model human behavior: "It is not yet clear how behavioral economists view large models, but they may disrupt economics just as they recently changed mathematics."
This article is republished from the "Wallstreetcn" APP, author: Gao Zhimou; GMTEight editor: Song Zhiying.
Related Articles

New stock news | Hearing passed! Kunlun New Materials' half-year results exceed the whole of last year, and the world's third-largest independent electrolyte manufacturer is about to list on the Hong Kong stock market.

Title context: Goldman Sachs' "fancy move" to raise financing for Musk to buy GPUs
Text:
Goldman Sachs' "fancy move" to raise financing for Musk to buy GPUs

BAMA TEA (06980) responds to suspension of military procurement qualification: it is a qualification adjustment in a specific field and has no impact on the group's public-facing sales, operations, and core business.
New stock news | Hearing passed! Kunlun New Materials' half-year results exceed the whole of last year, and the world's third-largest independent electrolyte manufacturer is about to list on the Hong Kong stock market.

Title context: Goldman Sachs' "fancy move" to raise financing for Musk to buy GPUs
Text:
Goldman Sachs' "fancy move" to raise financing for Musk to buy GPUs

BAMA TEA (06980) responds to suspension of military procurement qualification: it is a qualification adjustment in a specific field and has no impact on the group's public-facing sales, operations, and core business.






