Decoupled Vision-Language System for Multimodal Understanding and Generation

arXiv:2608.20382v1 Announce Type: new Abstract: We introduce a new architecture design for multimodal large language models (MLLMs), Libra, capable of both multimodal understanding and generation. Libra architecture contains one vision system and one language system, connected by cross-modal bridges. This design decouples self-modal modeling and cross-modal interaction, enabling each modality to learn its unique representations while maintaining effective cross-modal comprehension. The decoupling is mainly achieved in a switch attention module and a switch FFN module, which dynamically routes the computation flow for self-modal modeling and cross-modal interaction scenarios. We evaluate the effectiveness in two important settings: extbf{Libra-1} for the understanding-only image-to-text setting, and extbf{Libra-2} for unified image-to-text understanding and text-to-image generation. In addition to the architecture design, we discuss various improvements on tokenization, positional encoding, and supervision. Experiments demonstrate that the dedicated Libra design enables mutual improvements on multimodal understanding and generation, achieving strong performance on both understanding and generation benchmarks.

Related

• Visually Grounded Self-Reflection for Vision-Language Models via Reinforcement Learning
• See, Think, Learn: A Self-Taught Multimodal Reasoner
• PixVL: Self-Supervised Training of Pixel-Level MLLMs via a Unified Mask--Text Consistency Cycle
• Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformers
• Addressing Overthinking in Large Vision-Language Models via Gated Perception-Reasoning Optimization

Source: arXiv cs.CL | 2026-08-24