Spiking Local Interaction and Adaptive Complementary Fusion for Spiking Transformer

arXiv:2608.19238v1 Announce Type: cross Abstract: Spiking Transformers model token interactions primarily through spiking self-attention (SSA). However, binary query and key representations map continuous similarities to sparse and discrete relation responses, which may suppress weak relations and limit the propagation of local spatial context. To address this limitation, we introduce Spiking Local Interaction (SLI) and Adaptive Complementary Fusion (ACF). SLI establishes an attention-independent pathway for direct information exchange among neighboring spiking tokens using lightweight depthwise--pointwise transformations. ACF integrates SSA and SLI through layer-specific, channel-wise coefficients that adaptively balance their contributions at different network depths. The proposed design preserves the original attention formulation and can be incorporated into different Spiking Transformer architectures with modest parameter overhead. Experiments on ImageNet-1K, CIFAR-10, CIFAR-100, CIFAR10-DVS, and ADE20K show consistent improvements across image classification, event-based recognition, and semantic segmentation. In particular, QKFormer with SLI and ACF achieves 84.37% Top-1 accuracy on ImageNet-1K and 37.5% mIoU on ADE20K, where the segmentation model is trained without ImageNet pretraining. Ablation studies and qualitative analyses further indicate that SSA and SLI capture complementary interaction patterns and that learnable fusion consistently outperforms fixed weighting.

Related

• Rethinking Attention Locality in Spiking Transformers
• Fusion Complexity Inversion: Why Simpler Cross View Modules Outperform SSMs and Cross View Attention Transformers for Pasture Biomass Regression
• Adaptive Cross-Modal Fusion with Sparse Attention for Pedestrian Crossing Intention Prediction

Source: arXiv cs.CV | 2026-08-21