Beyond N-gram: Data-Aware X-GRAM Extraction for Efficient Embedding Parameter Scaling

arXiv:2604.21724v1 Announce Type: new Abstract: Large token-indexed lookup tables provide a compute-decoupled scaling path, but their practical gains are often limited by poor parameter efficiency and rapid memory growth. We attribute these limitations to Zipfian under-training of the long tail, heterogeneous demand across layers, and "slot collapse" that produces redundant embeddings. To address this, we propose X-GRAM, a frequency-aware dynamic token-injection framework. X-GRAM employs hybrid hashing and alias mixing to compress the tail while preserving head capacity, and refines retrieved vectors via normalized SwiGLU ShortConv to extract diverse local n-gram features. These signals are integrated into attention value streams and inter-layer residuals using depth-aware gating, effectively aligning static memory with dynamic context. This design introduces a memory-centric scaling axis that decouples model capacity from FLOPs. Extensive evaluations at the 0.73B and 1.15B scales show that X-GRAM improves average accuracy by as much as 4.4 points over the vanilla backbone and 3.2 points over strong retrieval baselines, while using substantially smaller tables in the 50% configuration. Overall, by decoupling capacity from compute through efficient memory management, X-GRAM offers a scalable and practical paradigm for future memory-augmented architectures. Code aviliable in https://github.com/Longyichen/X-gram.

Related

• TLoRA: Task-aware Low Rank Adaptation of Large Language Models
• Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference
• LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning
• Aletheia: Gradient-Guided Layer Selection for Efficient LoRA Fine-Tuning Across Architectures
• TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Models

Source: arXiv cs.CL | 2026-04-24