Test-Time Memory Construction

arXiv:2610.13348v1 Announce Type: new Abstract: Large Language Models (LLMs) struggle to answer questions over long interaction histories, where relevant evidence may be scattered across distant sessions. Existing memory systems typically construct and maintain sophisticated memory representations offline, before user queries are observed, incurring substantial overhead and limiting their ability to adapt memory construction to query-specific information needs. In this paper, we propose T^2Mem, a Test-Time Memory construction framework that defers memory construction to inference and progressively builds memory along the reasoning process. Specifically, T^2Mem extracts lightweight episode-level metadata offline to capture key events and temporal associations as retrieval cues, while retaining links to the original interactions to facilitate the reconstruction of dependencies across episodes. At test time, the LLM adaptively retrieves and organizes metadata and raw interactions, progressively constructing query-specific memory as reasoning unfolds. Experiments on LoCoMo and LongMemEval show the effectiveness of T^2Mem by consistently outperforming strong baselines across different backbone LLMs while substantially reducing the overhead of memory construction. Further analysis demonstrates that deferring memory construction to inference enables the memory construction process to dynamically adapt to the information needs of individual queries. All code is available at https://github.com/OpenBMB/T2Mem.

Related

• DeferMem: Query-Time Evidence Distillation via Reinforcement Learning for Long-Term Memory QA
• ConvMem: Convolutional Memory for Long-Context Reasoning
• LaTER: Efficient Test-Time Reasoning via Latent Exploration and Explicit Verification
• SGD-KV: Summarization Guided KV Cache Compression

Source: arXiv cs.CL | 2026-10-12