Tokenizer-Agnostic Engram Module

arXiv:2607.29065v1 Announce Type: new Abstract: Deepseek's Engram, a conditional memory module, was introduced to trade-off storage versus reasoning in large language models. However, the module relies on token-level N-gram hashing for Engram embedding lookup, introducing a tight coupling to the tokenizer used: a model with a different tokenizer would have to train its own Engram embeddings from scratch. To improve the reusability of Engram embeddings, we propose a change to the hashing routine, enabling compatibility between Engram models using different tokenizers. Instead of modelling disjoint N-gram spaces, we treat N-gram as a method to sample potentially useful byte sequences, from all possible byte sequences across tokens. We replace the XOR-based hashing with the general polynomial hashing with a joint embedding space across N. This work investigates the possible trade-offs and shows that this simple substitution produces comparable performance and achieves tokenizer-agnosticism: hash equivalence for byte-equivalent token sequences.

Related

• Learning When to Attend: Conditional Memory Access for Long-Context LLMs
• TF-Engram: A Train-Free Engram with SSD-Backed Memory for Large Language Models
• REAL: A Reasoning-Enhanced Graph Framework for Long-Term Memory Management of LLMs

Source: arXiv cs.CL | 2026-08-03