TokenMem: Faithful Knowledge Injection for Frozen LLMs

arXiv:2607.22625v1 Announce Type: new Abstract: Retrieval-augmented generation (RAG) enhances large language models (LLMs) with external knowledge, but suffers from knowledge conflicts: when retrieved information contradicts parametric memory, the shared self-attention pathway produces unpredictable outputs. We present TokenMem, a lightweight memory system that injects knowledge into frozen LLMs through a dedicated cross-attention channel, bypassing competition with parametric memory in the residual stream. TokenMem trains only a thin gating adapter (sim3-7M parameters) via a two-phase curriculum: first learning general knowledge utilization, then strengthening faithful compliance under counterfactual knowledge. In controlled experiments on five models spanning three families (Qwen3-4B/8B/14B, LLaMA-3.1-8B, OLMo-3-7B), TokenMem achieves 69-70% Knowledge Compliance (KC) on counterfactual benchmarks, compared to 20-52% for vanilla RAG, a gap of up to 49 percentage points. Ablation studies show that the two-phase curriculum is critical: removing Phase 2 collapses KC to near-zero. Mechanistic analysis reveals that the gate adapter learns a conflict-aware, layer-specific injection strategy without explicit supervision.

Related

• Exploring Knowledge Conflicts for Faithful LLM Reasoning: Benchmark and Method
• ParamMute: Suppressing Knowledge-Critical FFNs for Faithful Retrieval-Augmented Generation
• TriShieldRAG: A Three-Ring Defense-in-Depth Framework Against Knowledge Corruption in Retrieval-Augmented Generation
• Does RAG Know When Retrieval Is Wrong? Diagnosing Context Compliance under Knowledge Conflict
• Illocutionary Explanation Planning for Source-Faithful Explanations in Retrieval-Augmented Language Models

Source: arXiv cs.AI | 2026-07-28