HiMPO: Hindsight-Informed Memory Policy Optimization for Less-Entangled Credit in Long-Horizon Agents

arXiv:2606.16285v2 Announce Type: replace Abstract: Long-horizon agents rely on memory mechanisms to compress interaction history, but optimizing memory writing faces a distinct credit assignment challenge: a memory update may be rewarded or penalized due to downstream tool failures, noisy observations, or reasoning errors rather than its own contribution. We propose HiMPO, a Hindsight-Informed Memory Policy Optimization framework for assigning less-entangled credit to memory-writing actions in long-horizon agents. HiMPO first estimates the local utility of a memory update by comparing the task-relevant information recoverable from the previous and updated memories under the same pre-write state. It then uses hindsight relevance as a bounded retrospective filter that attenuates memory credit when local utility is not supported by the target outcome. The resulting memory-specific advantage is applied only to memory tokens, while trajectory-level rewards optimize the rest of the agent's behavior. Across judge-based open-domain tasks and objective compressive-memory QA, HiMPO improves over strong memory-based and RL-based baselines while preserving compressed-context efficiency. Controlled interventions and live replay studies further show that HiMPO reduces blame leakage from tool-induced errors, assigns memory credit that aligns with the functional impact of memory writes, and remains robust to noisy training targets.

Related

• Hindsight Memory-PRM: Supervising Memory Management with Auditable Hindsight Credit
• Rewarding Beliefs, Not Actions: Consistency-Guided Credit Assignment for Long-Horizon Agents
• Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents
• Learning How and What to Memorize: Cognition-Inspired Two-Stage Optimization for Evolving Memory

Source: arXiv cs.CL | 2026-09-02