Does Faithfulness-Guided Alignment Hurt Accuracy? Unlocking Accurate and Faithful Post-Retrieval Reasoning

arXiv:2602.01348v3 Announce Type: replace Abstract: Retrieval-augmented generation (RAG) can achieve strong answer accuracy on multi-hop questions, but outcome-level rewards often leave reasoning traces weakly grounded and difficult to audit. Under noisy retrieval, models may exhibit right-answer-wrong-reason failures, where the final answer is correct but the supporting rationale exploits shortcuts or unsupported evidence. We therefore ask whether faithfulness-guided alignment hurts answer accuracy in post-retrieval reasoning. To study this question, we propose CRAFT (Calibrated Reasoning with Answer-Faithful Traces), a reinforcement learning framework for the response-generation stage of retrieval-augmented multi-hop question answering. CRAFT trains models to produce structured reasoning traces with configurable auditability, while combining deterministic rewards for format compliance, answer correctness, and citation validity with a judge-based reward for semantic faithfulness. Experiments across model scales and benchmarks show that CRAFT unlocks task-specific reasoning capacity from 1.5B upward, improving both answer accuracy and Faithfulness; at 0.5B, performance remains sharply template-dependent. At 7B, CRAFT improves Faithfulness over the Base model in all evaluated settings and remains competitive with strong closed-source models. Code is available at https://github.com/Ameame1/CRAFT.

Related

• Quantifying Retriever-Generator Alignment in RAG with Local Explanations
• Evidence-Augmented Policy Optimization with Reward Co-Evolution for Long-Context Reasoning
• Does Reasoning Preserve Alignment? On the Trustworthiness of Large Reasoning Models
• Adaptive Information Control for Search-Augmented LLM Reasoning

Source: arXiv cs.CL | 2026-07-28