When Attention Beats Fourier: Multi-Scale Transformers for PDE Solving on Irregular Domains

arXiv:2605.08318v1 Announce Type: cross Abstract: We study the problem of architecture selection for deep learning models trained to solve partial differential equations (PDEs), asking when transformer-based architectures with learned attention outperform Fourier-domain neural operators. We introduce the extbf{Multi-Scale Attention Transformer} (msat{}), a deep learning architecture that encodes spatiotemporal solution histories as token sequences and trains end-to-end via a composite supervised objective with optional physics-informed regularization terms. We conduct a comprehensive empirical evaluation against nine baselines -- including physics-informed neural networks (PINNs), neural operators (FNO, DeepONet, GNOT), and state-space models (Mamba-NO) -- across five benchmark problems from the PINNacle suite, using identical train/test splits and reference data for all methods. msat{} achieves state-of-the-art generalization on complex geometry problems (L^2_rel = 0.0101 on Heat2D-CG, a 3.7imes improvement over FNO) at 34,s total inference vs. 120{,}812,s for Mamba-NO. Ablation studies over the physics regularization component reveal a precise inductive bias tradeoff: physics priors reduce test error on diffusion-dominated problems but degrade generalization on chaotic and recirculating-flow regimes, directly characterizing the prior misspecification boundary. Approximation error bounds as a function of domain boundary complexity kappa provide a theoretical basis for these empirical findings and a principled rule for architecture selection.

Source: arXiv cs.AI | 2026-05-12