Reward-Conditioned Attention: How Reward Design Shapes What Autonomous Driving Agents See

arXiv:2606.25127v1 Announce Type: new Abstract: We investigate how reward design shapes the internal attention patterns of reinforcement learning agents trained for autonomous driving. Using three Perceiver-based agents that share identical architectures and training data but differ only in their reward configurationsnicode{x2014}ranging from basic violation penalties to continuous proximity penaltiesnicode{x2014}we analyze cross-attention allocation across 50 real-world scenarios from the Waymo Open Motion Dataset. A central methodological finding is that naive pooling of timesteps across episodes substantially underestimates the attentionnicode{x2013}risk relationship; within-episode correlation with Fisher z-transform aggregation is the appropriate statistic and reveals a robustly positive link between collision risk and agent-directed attention. Building on this validated methodology, we demonstrate two reward-conditioned effects: agents trained with navigation rewards allocate up to 2.0imes more attention to GPS-path tokens than those trained with additional proximity penaltiesnicode{x2014}and 4.7imes more than agents with no navigation incentivenicode{x2014}revealing that reward content directly determines which scene elements the encoder prioritizes, and continuous time-to-collision penalties create a extit{learned vigilance prior}nicode{x2014}elevated resting agent surveillance maintained throughout collision-free phases. In several scenarios, the complete-reward and minimal-reward models exhibit opposite attentionnicode{x2013}risk correlation directions, demonstrating that reward design can qualitatively reverse attentional strategy rather than merely modulating its magnitude. These results suggest that attention analysis is a practical diagnostic for verifying that a reward function produces the intended representational behaviour in safety-critical RL systems.

Source: arXiv cs.LG | 2026-06-25