Agents
SubSearch: Intermediate Rewards for Unsupervised Guided Reasoning in Complex Retrieval
arXiv:2604.07415v1 Announce Type: cross Abstract: Large language models (LLMs) are probabilistic in nature and perform more reliably when augmented with external information. As complex queries often
arXiv:2604.07415v1 Announce Type: cross Abstract: Large language models (LLMs) are probabilistic in nature and perform more reliably when augmented with external information. As complex queries often require multi-step reasoning over the retrieved information, with no clear or predetermined reasoning path, they remain challenging. Recent approaches train models using reinforcement learning on the model's outcome, showing promise in improving how models handle complex information. We introduce SubSearch, a specialized framework that shifts from outcome-only supervision to intermediate reward signals that incentivize planning high-quality reasoning. Unlike previous work on process reward modeling, which focuses on training a separate reward model with annotated trajectories by either human annotators or large LLM judges, SubSearch directly optimizes the generator using intrinsic process rewards, which we define as internally-derived rewards, eliminating the need for external supervision, and moving towards autonomous information-intensive reasoning. Experiments on seven benchmarks show that rewarding intermediate reasoning steps with intrinsic rewards leads to more robust reasoning traces in both QA and multi-hop QA datasets over using only outcome rewards. SubSearch can help in building reasoning traces that allow agents to better integrate search engines for complex query answering, while offering a data-efficient alternative to supervised process modeling.
Related
- Reasoning Graphs: Deterministic Agent Accuracy through Evidence-Centric Chain-of-Thought Feedback
- Guaranteeing Knowledge Integration with Joint Decoding for Retrieval-Augmented Generation
- Search-R3: Unifying Reasoning and Embedding in Large Language Models
- ReCellTy: Domain-Specific Knowledge Graph Retrieval-Augmented LLMs Reasoning Workflow for Single-Cell Annotation
- A Decomposition Perspective to Long-context Reasoning for LLMs
Source: arXiv cs.CL | 2026-04-10