Test-Time Reasoners Are Strategic Multiple-Choice Test-Takers

arXiv:2510.07761v2 Announce Type: replace Abstract: Large language models (LLMs) now give reasoning before answering, excelling in tasks like multiple-choice question answering (MCQA). Yet, a concern is that LLMs do not solve MCQs as intended, as work finds LLMs sans reasoning succeed in MCQA without using the question, i.e., choices-only. Such partial-input success is often linked to trivial shortcuts, but reasoning traces could reveal if choices-only strategies are truly shallow. To examine these strategies, we have reasoning LLMs solve MCQs in full and choices-only inputs; test-time reasoning often boosts accuracy in full and in choices-only, half the time. While possibly due to shallow shortcuts, choices-only success is barely affected by the length of reasoning traces, and after finding traces pass faithfulness tests, we show they use less problematic strategies like inferring missing questions. In all, we challenge claims that partial-input success is always a flaw, so we propose how reasoning traces could separate problematic data from less problematic reasoning.

Related

• Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation
• Do LLMs Encode Functional Importance of Reasoning Tokens?
• Calibrated Confidence Estimation for Tabular Question Answering
• Disentangling Mathematical Reasoning in LLMs: A Methodological Investigation of Internal Mechanisms
• Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models
• Self-Calibrating Language Models via Test-Time Discriminative Distillation

Source: arXiv cs.CL | 2026-04-21