Skip to content
Kun-Yu Lee
CV (PDF)

Auditing Brand Recommendations in LLMs

A framework that audits open-ended LLM brand recommendations as a stochastic retrieval-and-ranking process.

Northwestern University · Jan 2026 – Present · arXiv preprint, 2026; related manuscripts under review

arXivPDF

Research question

When an LLM recommends brands without an explicit candidate set, which brands does it retrieve, how prominently are they ranked, and what does its implicit “brand image” look like?

Figures

Five-step framework: define competitive set; measure prevalence and prominence with category-only prompts; explore marketplace correlates; match brands with consumer needs using needs-based prompts; diagnose LLM brand understanding with positioning probes
Figure 1. A five-step framework for auditing LLM brand recommendations.
Scatter plots of MRR@5 against log advertising spend, faceted by product category
Figure 2. LLM recommendation prominence (MRR@5) vs. advertising expenditure, by product category.

Figures from Evaluating Brand Retrieval and Ranking in Large Language Model Recommendations.

Methods

  • Six commercial LLMs across five product and service categories
  • Repeated stateless sampling of category-only and needs-based queries
  • BRP@k and MRR@k for recommendation prevalence and prominence; NDCG for need-matching
  • Coding of 17,821 LLM justifications into eight brand-association types

My contribution

  • Second author. Designed and ran the evaluation experiments and analyzed the results.
  • Contributed to the research framing and co-wrote the manuscripts.

Outcomes

  • Category-only queries omit established brands (e.g., Craftsman, BRP@5 = 0%).
  • Search interest is the most consistent predictor of recommendation prominence.
  • Needs-based NDCG varies widely across brand-positioning dimensions.
  • In the “machine brand image,” price tier is the strongest marketplace correlate, while advertising spending shows no association.

Collaborators

Edward C. Malthouse (Northwestern University), Jing Yang (Boston University), Sanchary Pal, Xueyan Feng

Publications