Linear-Time and Constant-Memory Text Embeddings Based on Recurrent Language Models

arXiv:2604.18199v1 Announce Type: new Abstract: Transformer-based embedding models suffer from quadratic computational and linear memory complexity, limiting their utility for long sequences. We propose recurrent architectures as an efficient alternative, introducing a vertically chunked inference strategy that enables fast embedding generation with memory usage that becomes constant in the input length once it exceeds the vertical chunk size. By fine-tuning Mamba2 models, we demonstrate their viability as general-purpose text embedders, achieving competitive performance across a range of benchmarks while maintaining a substantially smaller memory footprint compared to transformer-based counterparts. We empirically validate the applicability of our inference strategy to Mamba2, RWKV, and xLSTM models, confirming consistent runtime-memory trade-offs across architectures and establishing recurrent models as a compelling alternative to transformers for efficient embedding generation.

Related

• Optimal Decay Spectra for Linear Recurrences
• Latent-Condensed Transformer for Efficient Long Context Modeling
• AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention
• DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language Models

Source: arXiv cs.CL | 2026-04-21