DeCoVec: Building Decoding Space based Task Vector for Large Language Models via In-Context Learning

arXiv:2604.11129v1 Announce Type: new Abstract: Task vectors, representing directions in model or activation spaces that encode task-specific behaviors, have emerged as a promising tool for steering large language models (LLMs). However, existing approaches typically require fine-tuning or invasive manipulation of internal states, limiting their flexibility and scalability. We propose extsc{DeCoVec} (Decoding Space based Task Vector), a training-free and non-invasive framework that constructs task vectors directly in the extit{decoding space} by leveraging in-context learning (ICL). Specifically, extsc{DeCoVec} captures the task essence as the difference between the output logit distributions of few-shot and zero-shot prompts, then steers generation by injecting this vector into the decoding process. Experiments across seven LLMs (0.5B--9B) on TruthfulQA, Math-500, and AQUA-RAT show that extsc{DeCoVec} consistently outperforms standard few-shot baselines, with gains up to +5.50 average accuracy. Further analysis demonstrates that extsc{DeCoVec} effectively suppresses generation degeneration and logical flaws while exhibiting strong robustness to demonstration ordering, all without incurring additional input token costs. Our method offers a training-free and non-invasive solution for LLM steering without requiring weight updates or auxiliary models.

Related

• Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insight
• Localizing Task Recognition and Task Learning in In-Context Learning via Attention Head Analysis
• Sparse or Dense? A Mechanistic Estimation of Computation Density in Transformer-based LLMs
• Guaranteeing Knowledge Integration with Joint Decoding for Retrieval-Augmented Generation

Source: arXiv cs.CL | 2026-04-14