Neural expectation-maximization and meta-models

arXiv:2610.12741v1 Announce Type: new Abstract: We interpret Transformer networks as learned approximations to a hybrid probabilistic-inference procedure that combines max-product Loopy Belief Propagation (LBP) with Expectation--Maximization (EM). In this view, a Transformer performs inference in partially observed Markov Random Fields (MRFs) whose structure and parameters are not given a priori but are inferred from data. We show that self-attention implements a differentiable approximation to max-product LBP on an input-conditioned, dynamically structured MRF, with attention weights encoding learned structural affinities. We further establish an EM correspondence at the level of depth: each layer updates latent representations in an E-step-like forward pass, while the transition heta^{(l)} o heta^{(l+1)} across layers acts as an implicit M-step. Building on this interpretation, we introduce meta-models, a design paradigm in which inputs generate instance-specific parameters, unifying mechanisms such as self-attention, mixture-of-experts, and gated units within a single framework. Empirical results support the proposed perspective.

Related

• Efficient Autoregressive Inference for Transformer Probabilistic Models
• Gradient Descent as Implicit EM in Distance-Based Neural Models
• Generating Hadamard matrices with transformers

Source: arXiv cs.LG | 2026-10-12