Learning Decision-Stump Thresholds in Context: Dynamics of Softmax Attention

arXiv:2610.07074v1 Announce Type: cross Abstract: Estimating a decision threshold requires locating observations near an unknown boundary. We study how gradient-based pretraining learns this statistical rule in a two-parameter softmax-attention model with a fixed feature and inequality direction. Pretraining uses labeled contexts and their true thresholds; a fresh threshold must be inferred from context alone. Under a large-resolution initialization, constant-step gradient descent on m tasks with n examples each produces a frozen estimator with error widetilde O((mwedge n)^{-1}+N^{-1}) for each fixed interior threshold and every fresh-context size N. The two terms separate finite-pretraining accuracy from fresh-context localization. The mechanism is coordinated parameter divergence: population training calibrates the relative label and feature scores, then increases the attention scale as t^{1/4}, giving population threshold error O(t^{-1/4}). To transfer this mechanism to a fixed finite corpus, we control gradient errors relative to the shrinking directions of progress at successive parameter scales. This certifies a growing training interval without requiring long-time tracking of the population trajectory. We also identify the boundary limitation of the one-head model and explain statistically what a reflected symmetrization could achieve.

Related

• Beyond Linear Attention: Softmax Transformers Implement In-Context Reinforcement Learning
• In-Context Learning for Data-Driven Censored Inventory Control
• Attention-Only White-Box Transformer via LeJEPA-Based Self-Supervised Pretraining

Source: arXiv cs.LG | 2026-10-07