Mode Connectivity Beyond Classifiers: Evidence from Generative and Contrastive Models

arXiv:2608.30366v1 Announce Type: new Abstract: The loss landscape of Deep Neural Networks (DNNs) exhibits highly complex and non-convex properties. Recent studies have revealed the phenomenon of mode connectivity, demonstrating that independently trained network modes can be connected via a continuous low-loss path. However, existing mode connectivity research is predominantly confined to classifier-based models, leaving it an open question whether similar geometric properties exist in modern complex models. In this paper, we extend the boundaries of mode connectivity to generative and contrastive domains (specifically DDPM and NanoCLIP). Addressing the unique architecture of DDPM and CLIP, we propose an architecture-aware connection building algorithm. Extensive empirical results demonstrate for the first time that we successfully discover mode connectivity between independently trained DDPM and NanoCLIP modes. Our work provides a novel perspective for understanding the geometric properties of the loss landscapes in modern generative and contrastive models.

Related

• From Approximation Rates to Loss-Landscape Barrier Decay in Shallow ReLU Networks
• How the Hessian-Spectrum of Neural Networks Depends on Data
• On the global convergence of gradient descent for wide shallow models with bounded nonlinearities

Source: arXiv cs.LG | 2026-09-01