LLM Synthetic Panels (Purchase Intent)
LLMs can simulate survey purchase intent at ~90% of human test-retest reliability — but only when you elicit free text and map to Likert via Semantic Similarity Rating (SSR), not when you ask for 1–5 directly.
Colgate-Palmolive + PyMC Labs (arXiv v3, 2025-10-27): 57 personal-care surveys, 9,300 human responses. Direct numeric Likert from LLMs yields collapsed, unrealistic distributions; SSR (text -> embeddings -> cosine similarity to anchor statements -> PMF over 5-point scale) matches human patterns (KS similarity >0.85). Source: arXiv 2510.08338 v3, https://arxiv.org/abs/2510.08338, 2025-10-27; Source: PyMC Labs blog, https://www.pymc-labs.com/blog-posts/AI-based-Customer-Research, 2026-06-13
Method (SSR)
- Give LLM a demographic persona + product concept.
- Ask purchase intent; require natural language answer (not a number).
- Embed response; compare to anchor statements per Likert point.
- Output probability mass over 1–5 (preserves uncertainty).
Implementation: pymc-labs/semantic-similarity-rating (ResponseRater, get_response_pmfs). Source: GitHub pymc-labs/semantic-similarity-rating, 2026-06-13
Viral framing correction
@HowToAI_ thread (11,375 bookmarks) oversold "90% accuracy." Precise claim: 90% correlation attainment vs human test-retest on survey purchase intent, not revenue prediction or novel categories. Source: X/@HowToAI_, 2026-06-11
The artifact image is the paper's first page and confirms the authorship and method framing. The self-thread is the more important review object: it corrects the viral number and names the scope limit. Treat synthetic panels as a cheap screening/evaluator layer, not as a replacement for demand evidence or actual buyer behavior. Source: local X image artifact and self-thread, 2026-06-30
| Works well | Does not substitute |
|---|---|
| CPG / familiar brands | Novel products with weak model priors |
| Concept screening before $50k panels | In-market behavioral validation |
| Qual + quant in one pass | Ethnographic / usability research |
Services-as-software angle
Autopilot research vendor: run SSR panels on every copy/positioning variant; human panels only for finalists. Margin widens as models improve if you sell outcomes (rankings, launch decisions), not the SSR library alone. Source: compiled from Services-as-Software, 2026-06-13
Stack fit (Kevin)
- Dedalus / Agent Machines GTM — synthetic positioning tests on landing copy before spend; pair with The Eval Loop (Slop Is an Output Problem) rubric (humanizer + kevin-voice as judges for your copy, SSR for market copy).
- Content pipeline — Security and Review Skills variants screened by SSR-style panel before posting (experimental).
- Not a wiki default — cite SSR when discussing research automation; don't treat PMF as proven from synthetic panels alone.
Evaluator Pattern
SSR is useful for agent loops because it converts subjective language into a distribution without asking the model to emit a fake number directly. The transferable pattern is: elicit qualitative text first, then score it with a separate similarity or rubric layer. That is safer than making the same model both roleplay the customer and declare the metric, which invites numeric anchoring and evaluator hacking.
Timeline
- 2026-06-30 | Deep-reviewed the paper screenshot, arXiv page, self-thread correction, and artifact reply. Confirmed arXiv v3 last revised 2025-10-27 and strengthened the page's evaluator-risk framing: SSR is a screening layer, not proof of market demand. Source: X/@HowToAI_, 2026-06-11; Source: arXiv 2510.08338 v3, 2025-10-27
- 2026-06-11 | @HowToAI_ viral thread; author corrected accuracy framing in self-thread. Source: X/@HowToAI_, 2026-06-11
- 2026-06-13 | Wiki concept page from bookmark absorb. Source: compiled, 2026-06-13
- 2026-06-13 | Quality remediation: arXiv + SSR method, GitHub package, corrected metrics. Source: arXiv 2510.08338; pymc-labs, 2026-06-13