Safety
Guided Speculative Inference for Efficient Test-Time Alignment of LLMs
arXiv:2506.04118v3 Announce Type: replace Abstract: We propose Guided Speculative Inference (GSI), a novel algorithm for efficient reward-guided decoding in large language models. GSI combines soft be
arXiv:2506.04118v3 Announce Type: replace Abstract: We propose Guided Speculative Inference (GSI), a novel algorithm for efficient reward-guided decoding in large language models. GSI combines soft best-of-n test-time scaling with a reward model r(x,y) and speculative samples from a small auxiliary model pi_S(ymid x). We provably approximate both the optimal tilted policy pi_{eta,B}(ymid x) propto pi_B(ymid x)exp(eta,r(x,y)) of soft best-of-n under the base model pi_B, as well as the expected reward under the optimal policy. In experiments on reasoning benchmarks (MATH500, OlympiadBench, Minerva Math, MMLU-STEM, GSM8K) and across different model families, our method achieves higher accuracy than standard soft best-of-n with pi_S and reward-guided speculative decoding (Liao et al., 2025), and in certain settings even outperforms soft best-of-n with pi_B, while reducing end-to-end latency by up to 28%. The code is available at https://github.com/j-geuter/GSI .
Related
- Test-Time Alignment via Hypothesis Reweighting
- MGDA-Decoupled: Geometry-Aware Multi-Objective Optimisation for DPO-based LLM Alignment
- GIFT: Group-Relative Implicit Fine-Tuning Integrates GRPO with DPO and UNA
- Chain-in-Tree: Back to Sequential Reasoning in LLM Tree Search
- TEMPO: Scaling Test-time Training for Large Reasoning Models
- Deliberative Alignment is Deep, but Uncertainty Remains: Inference time safety improvement in reasoning via attribution of unsafe behavior to base model
Source: arXiv cs.LG | 2026-04-28