E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning

arXiv:2409.06679v3 Announce Type: replace Abstract: Processing long contexts is increasingly important for Large Language Models (LLMs) in tasks like multi-turn dialogues, code generation, and document summarization. This paper addresses the challenges of achieving high long-context performance, low computational complexity, and compatibility with pretrained models -- collectively termed the ``impossible triangle''. We introduce E2LLM (Encoder Elongated Large Language Models), a novel approach that effectively navigates this paradox. E2LLM divides long contexts into chunks, compresses each into soft prompts using a pretrained text encoder, and aligns these representations with a decoder-only LLM via an adapter. To enhance the LLM's reasoning with these soft prompts, we employ two training objectives: encoder output reconstruction and long-context instruction fine-tuning. Extensive experiments reveal that E2LLM not only outperforms 8 state-of-the-art (SOTA) methods in effectiveness and efficiency for document summarization and question answering, but also achieves the best performance on LongBench v2 among models of comparable size.

Related

• DYCP: Dynamic Context Pruning for Long-Form Dialogue with LLMs
• PaceLLM: Brain-Inspired Large Language Models for Long-Context Understanding
• Latent-Condensed Transformer for Efficient Long Context Modeling
• Sparse or Dense? A Mechanistic Estimation of Computation Density in Transformer-based LLMs

Source: arXiv cs.CL | 2026-04-15