Recognition-Refusal Misalignment in LLMs: Why Models Answer Structurally Unanswerable Questions

arXiv:2608.29109v1 Announce Type: new Abstract: Large language models often answer structurally unanswerable questions, such as computing cot(-540{eg}) or evaluating (1).startswith("1"), instead of abstaining. We ask whether this failure reflects missing recognition or failed routing from recognition to abstention. Across instruction-tuned models from 1.7B to 70B parameters, a single linear direction in the hidden state separates answerable from structurally impossible math and code prompts, showing that models represent impossibility before generation. Yet this recognition direction is nearly orthogonal to the canonical safety-refusal direction that mediates trained harmful-content refusal. An in-domain behavior-defined invalidity-aware direction is closer to recognition, but only partially aligned with it, and remains near-orthogonal to safety refusal. Generation-time steering along the recognition direction changes invalidity-aware behavior bidirectionally and dose-responsively on structural math and code cells, while random directions do not. Base/instruct comparisons further show that the low-cosine geometry is already present at the pretraining endpoint. The confident-on-impossible failure is therefore better explained as a routing failure than as an encoding failure: the model has a usable "no admissible answer" signal, but the safety-refusal pathway is not aligned to use it.

Related

• Beyond 'I cannot fulfill this request': Alleviating Rigid Rejection in LLMs via Label Enhancement
• From Prompt Risk to Response Risk: Paired Analysis of Safety Behavior of Large Language Model
• Popular but Wrong: Understanding and Mitigating LLM Overconfidence through Knowledge Popularity
• When Linguistic and Internal Confidence Diverge in Large Language Models
• Question Difficulty Estimation for Large Language Models via Answer Plausibility Scoring

Source: arXiv cs.CL | 2026-09-01