Beyond Bias Scores: Answerability Confounds in Activation Steering with Gradient-Constrained Direction Design (GCD)

arXiv:2610.17641v1 Announce Type: new Abstract: Bias scores in language-model evaluations often conflate what a model prefers with whether it answers at all. We identify this confound in extsc{BBQ}, whose ambiguous score factorizes into answerability and signed preference. Across 11 categories, 28,}824 items, and five models, contrastive steering lowers the signed ambiguous score in 11/11 categories but lowers the signed disambiguated preference score in only 8/11. Across 99 configurations, the answerability channel exceeds half of the observed score change in 17% and accounts for the entire change in 11%. In a six-category disambiguated evaluation, 20.8% of 1,500 items flip to the unknown option, where abstention is incorrect; extsc{StereoSet} shows no analogous answer-rate shift. The effect reflects selection and near-orthogonality between the contrastive and preference-gradient directions. We introduce Gradient-Constrained Direction Design (extsc{GCD}), an option-gradient construction with a first-order unknown-logit constraint. Under a matched Qwen-7B protocol, GCD reduces answer-rate distortion 14-fold and yields a 4-fold larger preference change, while ablations show similar gains for alternative and unconstrained option-gradient directions; nulling the unknown gradient alone is not responsible for the gain. We recommend reporting answerability alongside published scores.

Source: arXiv cs.AI | 2026-10-15

Related