Model Releases
Learning a Single Token to Replace Long System Prompts in LLMs
arXiv:2511.23271v2 Announce Type: replace Abstract: Long system prompts are widely used to steer Large Language Models (LLMs), but repeatedly processing them at inference time is inefficient and consu
arXiv:2511.23271v2 Announce Type: replace Abstract: Long system prompts are widely used to steer Large Language Models (LLMs), but repeatedly processing them at inference time is inefficient and consumes valuable context budget. This motivates a central question: can the behavioral effect of a long system prompt be retained using only a minimal learned representation? To enable this, we propose a lightweight training framework that learns a single Behavior-Equivalent Token ([BE]). The framework first trains [BE] to encode the semantic content of the original system prompt via reconstruction, and then distills the prompt's downstream behavior into this single token. Importantly, our method requires no update to the pretrained LLM weights, no auxiliary compression models, and no labeled responses. Empirical evaluations on three datasets show that replacing long prompts with a single [BE] token yields up to a 3000imes prompt compression ratio, while retaining about 98% of the downstream performance of the original system prompts. This substantially reduces inference cost and frees nearly the entire context window for user inputs and model outputs.
Related
- GroupGPT: A Token-efficient and Privacy-preserving Agentic Framework for Multi-User Chat Assistant
- Are Single-Token Sparse Autoencoder Features Causally Necessary? Layer-Depth and SAE-Family Effects
- Aligning Tree-Search Policies with Fixed Token Budgets in Test-Time Scaling of LLMs
Source: arXiv cs.CL | 2026-08-31