Research
Sub-Token Routing in LoRA for Adaptation and Query-Aware KV Compression
arXiv:2604.21335v1 Announce Type: cross Abstract: Sub-token routing offers a finer control axis for transformer efficiency than the coarse units used in most prior work, such as tokens, pages, heads,
arXiv:2604.21335v1 Announce Type: cross Abstract: Sub-token routing offers a finer control axis for transformer efficiency than the coarse units used in most prior work, such as tokens, pages, heads, or layers. In this paper, we study routing within a token representation itself in LoRA-adapted transformers. The motivation is that a relevant token need not be internally uniform: under a retention budget, preserved value groups are distributed unevenly both across tokens and within tokens, which suggests that KV compression need not be an all-or-nothing decision at token level. We study this fine-grained routing mechanism in two settings. For compression-aware language modeling, we introduce a query-independent design that combines routed subspace LoRA with value-group routing on the KV path. For downstream-task-preserving KV compression, we introduce a query-aware design in which a predictor-based selector allocates a global retention budget over context-token/value-group pairs using query-conditioned relevance. Experiments show that the query-independent design improves the quality-compression tradeoff for language modeling, while the query-aware design preserves downstream behavior under reduced KV budgets. We further examine the relation between token-level and sub-token-level query-aware routing, and show that they form complementary compression axes: token-level methods determine which tokens survive globally, while sub-token routing determines how the surviving tokens are compressed internally.
Related
- MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression
- SpectralLoRA: Is Low-Frequency Structure Sufficient for LoRA Adaptation? A Spectral Analysis of Weight Updates
- Transactional Attention: Semantic Sponsorship for KV-Cache Retention
- Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey
- Cosine-Similarity Routing with Semantic Anchors for Interpretable Mixture-of-Experts Language Models
Source: arXiv cs.CL | 2026-04-24