Semantic-TVM: Structure-Preserving Trustworthy Virtual Memory for Memory-Augmented and Tool-Using Agents

arXiv:2609.15011v1 Announce Type: new Abstract: Memory-augmented and tool-using agents expose exact private values when remote LLMs process retrieved memory, tool actions, and intermediate observations. One-way masking limits direct exposure but removes values needed for trusted execution and can leak them through later observations. We propose Trustworthy Virtual Memory (TVM), a closed-loop runtime that keeps exact-value state local while presenting a protected view to the remote model. Within this single runtime, Rule-TVM replaces whole protected fields with locally recoverable handles, and Semantic-TVM instead replaces only sensitive spans predicted by a trusted local model, preserving surrounding task-relevant context. On Memory-EHR and Memory-RAP across two providers, span-level projection recovers most of the EHR utility lost under whole-field replacement (Task Success 84.17% vs. 52.33% on DeepSeek) while measured exposure stays low and workflows remain executable.

Related

• ClawVM: Harness-Managed Virtual Memory for Stateful Tool-Using LLM Agents
• SiLR: Structure-Preserving Admission and Process Reward for LLM Tool Agents
• MemToC: Benchmarking Memory-Tool Conflict Resolution in Large Language Models
• Learning to Remember: End-to-End Training of Memory Agents for Long-Context Reasoning

Source: arXiv cs.AI | 2026-09-15