Safety
PolicyAttention: Softmax Attention Implements Policy Mirror Descent for Closed-Loop Control
arXiv:2609.30500v1 Announce Type: cross Abstract: Can causal softmax attention implement policy mirror descent as a repeated controller rather than a one-step algebraic identity? Negative-entropy poli
arXiv:2609.30500v1 Announce Type: cross Abstract: Can causal softmax attention implement policy mirror descent as a repeated controller rather than a one-step algebraic identity? Negative-entropy policy mirror descent (PMD) has the statewise update operatorname{PMD}_eta(pi,Q)=operatorname{softmax}(logpi+eta Q). Building on the known Q-TD-PMD recursion, we construct one fixed causal-softmax actor--environment--one-step-critic protocol with explicit actor, routing, sampling, and normalization residuals, and propagate them to the policy actually returned. The construction states the finite-logit/full-support domain, the external tokenization and sampling boundary, and the mean-zero LayerNorm carrier conditions required by the normalized compilation. Separately trained pre-LN Transformers recover the target computation empirically. A frozen one-step audit model is closest to PMD among the tested fixed rules; in a preregistered five-run S=4 repeated-control test, the learned actor with an exact one-step critic reaches median returned-policy loss 1.052imes the Exact PMD oracle and retains the criterion across four no-retraining shifts. The same checkpoints with their learned critic give descriptive median 1.050imes the oracle (no registered margin). At S=8, replacing the exact critic by the learned critic raises median T=20 loss to 0.0225 yet leaves the Liang--Lai and Algorithm Distillation adaptations 20.2--24.2imes higher-loss; this is a one-sided sampled-critic bound because PolicyAttention consumes 144 generative transitions per round versus 20 on-policy transitions for the adaptations. The strict 20-transition comparison remains open. At S=8,16, the exact-critic common-harness comparison remains 17.7--28.2imes lower-loss than those adaptations, with the information asymmetry stated locally.
Related
- StaQ: a Finite Memory Approach to Discrete Action Policy Mirror Descent
- Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policies
- Contraction-Aware Reinforcement Learning for Nonlinear Control with Statistical Robustness
Source: arXiv cs.AI | 2026-09-28