FlexEE: Self-Speculative and KV-Compatible Early Exiting for Offloading-Aware LLM Inference

arXiv:2609.17008v1 Announce Type: new Abstract: Large language model (LLM) inference is often constrained by both computation and memory, especially in offloading-based deployments where model weights are transferred across memory hierarchies during autoregressive decoding. In this setting, reducing the number of executed layers can lower per-token latency while also avoiding costly weight movement. Motivated by this observation, we present FlexEE, an early exiting framework for resource-constrained and offloading-based LLM inference. FlexEE makes early exiting practical for LLM decoding through layer-wise exit supervision for reliable intermediate-layer prediction, self-speculative decoding over a Top-K local vocabulary for low-cost exit decisions, and dynamic hidden state management for KV-cache-correct and memory-aware execution. Across generative and downstream tasks, FlexEE enables efficient early exit with minimal accuracy degradation, delivering up to 1.27imes/3.16imes and 1.25imes/2.83imes end-to-end speedups on Llama2-7B and Llama3-8B under 0%/50% weight offloading, respectively.

Related

• MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices
• Output-Aware Rotation for INT2 KV-Cache Quantization
• CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
• Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective
• SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration

Source: arXiv cs.AI | 2026-09-16