Search: “multimodal”

Rosetta: Composable Native Multimodal Pretraining

arXiv:2607.00293v1 Announce Type: cross Abstract: Achieving true artificial general intelligence requires foundation models capable of integrating new modalities without forgetting prior knowledge. Ho

researcharxiv-cs-cl

2 Jul 2026

XSkill: Continual Learning from Experience and Skills in Multimodal Agents

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

DGX agent

arXiv:2603.12056v3 Announce Type: replace Abstract: Multimodal agents can now tackle complex reasoning tasks with diverse tools, yet they still suffer from inefficient tool use and inflexible orchestr

model-releasesarxiv-cs-ai

2 Jul 2026

AA: A Multi-view Multimodal Dataset for Screen-based Gaze Estimation

arXiv:2606.31211v1 Announce Type: new Abstract: We present AA, a multi-view multimodal dataset for screen-based gaze estimation. The dataset captures synchronized facial observations from eight fixed

Breaking Failure Cascades: Step-Aware Reinforcement Learning for Medical Multimodal Reasoning

arXiv:2606.31825v1 Announce Type: cross Abstract: Recent multimodal large language models have shown great promise in clinical image reasoning, but existing post-training pipelines remain predominantl

Tutorials

Building a Multimodal Dataset of Academic Paper for Keyword Extraction

arXiv:2606.31069v1 Announce Type: new Abstract: Up to this point, keyword extraction task typically relies solely on textual data. Neglecting visual details and audio features from image and audio mod

tutorialsarxiv-cs-cl

Learning to Deny: Action Denial in Multimodal Large Language Models

model-releasesarxiv-cs-cv

arXiv:2606.31187v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have rapidly advanced video understanding, achieving strong zero-shot and few-shot recognition across standard

Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models

arXiv:2606.29689v1 Announce Type: new Abstract: Open-ended aesthetic critique is a challenge for multimodal large language models (MLLMs): unlike multiple-choice aesthetic benchmarks, it has no single

safetyarxiv-cs-cl

COMPASS: Grounding Composition-Intent Guidance in Unified Multimodal Models

arXiv:2606.28696v1 Announce Type: new Abstract: Composition is a high-level visual intent that governs where subjects are placed and how a scene is organized, yet current unified multimodal models rem

Personalizing MLLMs via Reinforced Multimodal Reference Game

arXiv:2606.28845v1 Announce Type: new Abstract: Personalizing Multimodal Large Language Models (MLLMs) aims to recognize users' unique concepts from visual data and provide personalized responses. Alt

Tutorials

Residual-Guided Expert Specialization for Incomplete Multimodal Learning

arXiv:2606.30355v1 Announce Type: cross Abstract: As real-world prediction systems often face missing modalities at inference, incomplete multimodal learning (IML) remains a practical challenge. While

tutorialsarxiv-cs-ai

Agents

RESOURCE2SKILL: Distilling Executable Agent Skills from Human-Created Multimodal Resources

arXiv:2606.29538v1 Announce Type: cross Abstract: Skills are a useful abstraction for software agents, turning human and agent experience into reusable procedural knowledge. Yet existing skill librari

agentsarxiv-cs-ai

Context-specific Credibility-aware Multimodal Fusion with Conditional Probabilistic Circuits

model-releasesarxiv-cs-lg

arXiv:2603.26629v2 Announce Type: replace Abstract: Multimodal fusion requires integrating information from multiple sources that may conflict depending on context. Existing fusion approaches typicall

29 Jun 2026

From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models

arXiv:2606.26196v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have recently made remarkable progress in unifying vision-language understanding and reasoning, especially fo

hisao{}: A GPU-Native Parallel Optimizer for Multimodal Black-Box Functions via Convergence-Anticonvergence Oscillation

model-releasesarxiv-cs-lg

arXiv:2606.26164v1 Announce Type: new Abstract: Finding all modes of a multimodal black-box function is a fundamental challenge in optimization, Bayesian inference, and scientific computing. Existing

Tutorials

Modeling Local, Global, and Cross-Modal Context in Multimodal 3D MRI

arXiv:2606.26894v1 Announce Type: new Abstract: Brain MRI poses a fundamental challenge for machine learning: models must learn from high-dimensional 3D data spanning multiple co-registered modalities

tutorialsarxiv-cs-cv

Position Rebinding Cache Reuse: Replay-Free Visual Revisiting for Interleaved Multimodal Reasoning

arXiv:2606.26631v1 Announce Type: new Abstract: Interleaved multimodal reasoning improves visual grounding by revisiting visual evidence during multi-step generation, yet existing methods typically re

SyncLoop: A Multimodal Dual-Loop Framework for Self-Improving Mathematical Reasoning

model-releasesarxiv-cs-cl

arXiv:2507.16518v3 Announce Type: replace-cross Abstract: Recent advances in multimodal large language models (MLLMs) have shown impressive reasoning capabilities. However, further enhancing existing

25 Jun 2026

Accelerating Multimodal Large Language Models with Prior-Corrected Token Reduction

arXiv:2606.24156v1 Announce Type: new Abstract: Visual token reduction has emerged as an effective strategy for accelerating Multimodal Large Language Models (MLLMs). Many existing methods prune token

FlowR2A: Learning Reward-to-Action Distribution for Multimodal Driving Planning

arXiv:2606.24231v1 Announce Type: new Abstract: Multimodal driving planning faces a long-standing tension between two paradigms: scoring-based methods benefit from dense reward supervision but are con

Mind the Heads: Topological Representation Alignment for Multimodal LLMs

arXiv:2606.23885v1 Announce Type: cross Abstract: Representation alignment has emerged as an effective approach to improve Multimodal Large Language Models (MLLMs) by regularizing their internal repre

Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models

arXiv:2606.24165v1 Announce Type: new Abstract: Reducing visual token redundancy is critical for accelerating Multimodal Large Language Models (MLLMs) without degrading cross-modal reasoning performan

safetyarxiv-cs-cv

A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Models

arXiv:2511.15098v2 Announce Type: replace Abstract: Discrete diffusion-based multimodal large language models (dMLLMs) have emerged as a promising alternative to autoregressive MLLMs thanks to their a

Beyond Time Series: Spatial Reasoning for Epidemic Forecasting via Multimodal Learning

arXiv:2606.22171v1 Announce Type: new Abstract: Epidemic forecasting models typically rely on surveillance data reported over administrative regions, treating them as atomic units, thereby obscuring s

applicationsarxiv-cs-lg

Agents

Dynamic multi-agent deep reinforcement learning-based pricing and incentivization approach in multimodal transportation networks

arXiv:2606.23257v1 Announce Type: new Abstract: In multimodal transportation systems, shared mobility services (SMSs) are promoted for their potential to enhance flexibility and reduce congestion. How

agentsarxiv-cs-lg

Local Ai

EnTrust: Modeling Inter-Modal Conflict for Trustworthy Multimodal Medical Image Analysis

arXiv:2606.21384v1 Announce Type: new Abstract: Multimodal medical imaging fuses complementary anatomical and functional information, yet modalities frequently disagree in pathologically heterogeneous

local-aiarxiv-cs-cv

Extraction and Analysis of Multimodal Concepts in Vision Language Models through Sparse Autoencoders

arXiv:2606.21197v1 Announce Type: new Abstract: Vision Language Models (VLMs) have demonstrated impressive performance in tasks requiring joint understanding of images and text, such as image captioni

safetyarxiv-cs-cv

Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reuse

model-releasesarxiv-cs-cv

arXiv:2606.23581v1 Announce Type: cross Abstract: Multimodal agents repeatedly re-examine the same video frames, UI screenshots, and rendered artifacts as their context window slides and reasoning ite

EvoLMM: Self-Evolving Large Multimodal Models with Continuous Rewards

arXiv:2511.16672v4 Announce Type: replace Abstract: Recent advances in large multimodal models (LMMs) have enabled impressive reasoning and perception abilities, yet most existing training pipelines s

Human-Guided Agentic AI for Multimodal Clinical Prediction: Lessons from the AgentDS Healthcare Benchmark

arXiv:2602.19502v2 Announce Type: replace Abstract: Agentic AI systems are increasingly capable of autonomous data science workflows, yet clinical prediction tasks demand domain expertise that purely

InternVideo3: Agentify Foundation Models with Multimodal Contextual Reasoning

arXiv:2606.12195v1 Announce Type: new Abstract: Recent progress in foundation models has shifted toward agentic behavior involving multi-step reasoning and tool use. However, open-source efforts large

safetyarxiv-cs-cv

Latent World Recovery for Multimodal Learning with Missing Modalities

arXiv:2606.12362v1 Announce Type: cross Abstract: We study multimodal learning under missing modalities, with particular motivation from bioscience applications in which heterogeneous modalities are o

M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset

arXiv:2510.23508v3 Announce Type: replace Abstract: Existing real-world datasets for multimodal fact-checking have multiple limitations: they contain few instances, cover on only one or two languages,

applicationsarxiv-cs-cl

Local Ai

MultiToP: Learning to Patch Visual Tokens to Mitigate Hallucinations in Video Large Multimodal Models

arXiv:2606.11792v1 Announce Type: cross Abstract: Video Large Multimodal Models have achieved remarkable progress in video understanding, yet they remain prone to hallucinations, where generated respo

local-aiarxiv-cs-ai

Parameter-Efficient Adapter Tuning for Tabular-Image Multimodal Learning

model-releasesarxiv-cs-cv

arXiv:2606.11682v1 Announce Type: new Abstract: Tabular-image multimodal learning aims to improve predictive modeling by jointly using structured tabular attributes and visual data. Although pretraine

Agents

ChartAgent: A Multimodal Agent for Visually Grounded Reasoning in Complex Chart Question Answering

arXiv:2510.04514v3 Announce Type: replace Abstract: Recent multimodal LLMs have shown promise in chart-based visual question answering, but their performance declines sharply on unannotated charts-tho

agentsarxiv-cs-ai

LongMoE: Longitudinal Multimodal Learning via Trajectory-Aware Mixture-of-Experts

arXiv:2606.09907v1 Announce Type: cross Abstract: Multimodal clinical learning is increasingly important for integrating diverse patient data, including imaging, text, and personalised health records.

One Token per Multimodal Evidence: Latent Memory for Resource-Constrained QA

arXiv:2606.10572v1 Announce Type: new Abstract: External memory effectively grounds large language models (LLMs) and vision-language models (VLMs)-based question answering (QA) in relevant multimodal

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning

arXiv:2606.09853v1 Announce Type: new Abstract: A central objective in multimodal learning is to capture synergy: task-relevant information that arises only from the joint use of multiple modalities,

applicationsarxiv-cs-lg

COMPASS: Complete Multimodal Fusion via Proxy Tokens and Shared Spaces for Ubiquitous Sensing

arXiv:2604.02056v2 Announce Type: replace Abstract: Missing modalities in multimodal sensing cause not only information loss but also a fusion-interface mismatch: a fusion head trained on a canonical

Introducing Gemma 4 12B: a unified, encoder-free multimodal model

model-releasesgoogle-deepmind

Gemma 4 12B is a unified, encoder-free multimodal model designed to bring high-performance intelligence to laptops and released under an Apache 2.0 license. It eliminates separate encoders by projecti

Late-Layer Fusion is Enough: Dual-Path Vision Token Routing for Multimodal Large Language Models under Visual Saturation

arXiv:2606.09131v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) commonly inherit the deep, symmetric Transformer backbone designed for unimodal text modeling, and apply the sa

SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks

arXiv:2606.09669v1 Announce Type: new Abstract: Spatial reasoning is a foundational capability for multimodal large language models (MLLMs) to perceive and operate within the physical world. However,

Test-Time Scaling in Multimodal Foundation Models: A Comprehensive Survey of Generation and Reasoning

arXiv:2606.08231v1 Announce Type: new Abstract: Test-time Scaling (TTS) has emerged as a pivotal research direction for enhancing model performance by dynamically allocating computational resources du

Multi-FRuGaL: Multimodal Flexible Redundancy-aware Decomposed Gated Learning for Cancer Diagnosis and Prognosis

arXiv:2606.06867v1 Announce Type: new Abstract: Modern medicine relies on heterogeneous data sources spanning radiology, pathology, text reports, and structured clinical information. However, real-wor

applicationsarxiv-cs-cv

8 Jun 2026

MVCL-DAF++: Enhancing Multimodal Intent Recognition via Prototype-Aware Contrastive Alignment and Coarse-to-Fine Dynamic Attention Fusion

arXiv:2509.17446v3 Announce Type: replace-cross Abstract: Multimodal intent recognition (MMIR) suffers from weak semantic grounding and poor robustness under noisy or rare-class conditions. We propose

8 Jun 2026

TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents

arXiv:2606.05784v1 Announce Type: new Abstract: We identify and formally characterize credit misassignment as a systematic failure mode of GRPO in tool-augmented multimodal search agents: its uniform

6 Jun 2026

Where Should Knowledge Enter? A Layered Framework for Knowledge Infusion in Multimodal Iterative Generative Mo

arXiv:2606.06356v1 Announce Type: new Abstract: Multimodal generative models produce fluent outputs but remain unreliable when generation must respect structured, domain-specific, or safety-critical k

6 Jun 2026

Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models