Why Training-Free Token Reduction Collapses: The Inherent Instability of Pairwise Scoring Signals

arXiv:2604.16745v1 Announce Type: cross Abstract: Training-free token reduction methods for Vision Transformers (ToMe, ToFu, PiToMe, and MCTF) employ different scoring mechanisms, yet they share a closely matched cliff-like collapse at high compression. This paper explains why. We develop a diagnostic framework with two tools, ranking consistency rho_s and off-diagonal correlation rho_ext{off}, that decomposes the collapse into (1)a signal-agnostic error amplifier inherent to layer-wise reduction, predicting convex Pareto curves and r_{ext{crit}} propto 1/L; and (2)shared reliance on pairwise similarity signals whose ranking consistency degrades from rho_s{=}0.88 to 0.27 in deep layers. Pairwise rankings are inherently unstable (O(N_p^2) joint perturbations) while unary signals enjoy greater stability (O(N_p) perturbations, CLT). From three design principles derived from this diagnosis, we construct CATIS as a constructive validation: unary signals raise the trigger threshold, triage suppresses the gain. On ViT-Large at 63% FLOPs reduction, CATIS retains 96.9% of vanilla accuracy (81.0%) on ImageNet-1K where all baselines collapse to 43--65%.

Related

• Tarot-SAM3: Training-free SAM3 for Any Referring Expression Segmentation
• VisPCO: Visual Token Pruning Configuration Optimization via Budget-Aware Pareto-Frontier Learning for Vision-Language Models
• 2D or 3D: Who Governs Salience in VLA Models? -- Tri-Stage Token Pruning Framework with Modality Salience Awareness
• Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Models

Source: arXiv cs.CV | 2026-04-21