LLM Recommendations and the Decision Value of Missing Preferences

When a consumer leaves a preference unstated, do LLM recommenders raise it because the recommendation depends on it, or only because the consumer mentioned it?

• Benchmark of 72 hotel-choice scenarios across 61 cities, built from HotelRec ratings
• Exact oracle for whether a consumer's undecided preferences can change the recommendation
• Paired requests (“book one” vs. “the best three”) that make the same missing preference decision-relevant in one condition but not the other
• 2,160 queries to three frontier LLMs (GPT-5.6 Luna, Gemini 3.6 Flash, Claude Sonnet 5)
• Response types coded by three LLM annotators from providers outside the study, validated against blind human coding (κ = 0.98, n = 100)
• 95% scenario-cluster bootstrap intervals

• First author. Designed the benchmark and oracle, ran the evaluation, and analyzed the results.

• LLMs raised missing preferences mainly when consumers mentioned them, not when the recommendation depended on them.
• 70–93% of responses addressed open preferences stated as undecided, but only 10–27% did when the same preferences went unmentioned.
• With complete preferences, the same models chose correctly 97–100% of the time.

Edward C. Malthouse (Northwestern University)

• Evaluating Whether LLM Recommendations Respond to the Decision Value of Missing Preferences

  Kun-Yu Lee, Edward C. Malthouse. Manuscript, 2026.