Search: “multimodal”

DisasterBench: A Multimodal Benchmark for UAV-Based Disaster Response in Complex Environments

model-releasesarxiv-cs-cv

arXiv:2606.06217v1 Announce Type: new Abstract: When a disaster unfolds, responders must answer not only what is happening, but also why it is happening, what will happen next, and what to do now, oft

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

LoomVideo: Unifying Multimodal Inputs into Video Generation and Editing

DGX agent

arXiv:2606.06042v1 Announce Type: new Abstract: Developing unified video generation and editing models capable of interpreting interleaved multimodal inputs is a promising yet challenging frontier fie

model-releasesarxiv-cs-cv

5 Jun 2026

Agents

MARDoc: A Memory-Aware Refinement Agent Framework for Multimodal Long Document QA

arXiv:2606.05749v1 Announce Type: new Abstract: Iterative retrieval-reasoning agents have recently shown promise for multimodal long-document question answering. However, most existing systems maintai

agentsarxiv-cs-cl

Local Ai

Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads

arXiv:2606.05843v1 Announce Type: new Abstract: While Multimodal Large Language Models (MLLMs) demonstrate remarkable proficiency on complex vision-language tasks, the mechanisms by which they extract

local-aiarxiv-cs-cl

Seeing is Believing? Evaluating Vision-Language Model Susceptibility in Agent-to-Agent Multimodal Persuasion

arXiv:2510.22768v2 Announce Type: replace Abstract: As autonomous agents increasingly interact, they inevitably attempt to influence one another. While prior work in text-only settings has explored th

BreastGPT: A Multimodal Large Language Model for the Full Spectrum of Breast Cancer Clinical Routine

arXiv:2606.04911v1 Announce Type: cross Abstract: Breast cancer remains a leading cause of cancer-related mortality among women. Its clinical management requires multimodal reasoning across a clinical

4 Jun 2026

FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMs

model-releasesarxiv-cs-cv

arXiv:2606.04282v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are predominantly evaluated on free-form vision-language tasks such as visual question answering, captioning, a

4 Jun 2026

MM-BizRAG: Rethinking Multimodal Retrieval-Augmented Generation for General Purpose Enterprise Q&A

arXiv:2606.04231v1 Announce Type: cross Abstract: Recent advances in multimodal retrieval-augmented generation (MM-RAG) have shifted toward minimal parsing, relying on page-level images for producing

4 Jun 2026

Best of Both Worlds: Multimodal Reasoning and Generation via Unified Discrete Flow Matching

arXiv:2602.12221v2 Announce Type: replace Abstract: We propose UniDFlow, a unified discrete flow-matching framework for multimodal understanding, generation, and editing. It decouples understanding an

safetyarxiv-cs-cv

CORE: Conflict-Oriented Reasoning for General Multimodal Manipulation Detection

arXiv:2606.03066v1 Announce Type: new Abstract: The rapid rise of generative AI has made multimodal fake news increasingly realistic and pervasive, posing severe threats to public trust and social sta

MemVerse: Multimodal Memory for Lifelong Learning Agents

arXiv:2512.03627v2 Announce Type: replace Abstract: Despite rapid progress in large-scale language and vision models, AI agents still suffer from a fundamental limitation: they cannot remember. Withou

Social Caption: Evaluating Social Understanding in Multimodal Models

arXiv:2601.14569v2 Announce Type: replace Abstract: Social understanding abilities are crucial for multimodal large language models (MLLMs) to interpret human social interactions. We introduce SOCIAL

researcharxiv-cs-cl

Spectral-Progressive Thought Flow for Lightweight Multimodal Reasoning

arXiv:2606.02842v1 Announce Type: new Abstract: Multimodal spatial reasoning often relies on long chains of intermediate textual and visual thoughts, where accumulating visual tokens and dense cross-m

researcharxiv-cs-lg

Amazing deep dive from the @togethercompute team on serving MiniMax M3 in production. M3 with its 1M context, native multimodality and MiniM…

applicationstogether-ai--x

Amazing deep dive from the @togethercompute team on serving MiniMax M3 in production. M3 with its 1M context, native multimodality and MiniMax Sparse Attention requires real work across paged decode,

CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning

arXiv:2606.02502v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) unify heterogeneous vision-language tasks under a shared generative framework via instruction tuning, yet real-

DraDDP: A Multimodal Multi-Party Dialogue Discourse Parsing Dataset

arXiv:2606.00012v1 Announce Type: cross Abstract: Multi-party dialogue discourse parsing aims to identify dependency structures and relation types between utterances in conversations. Previous studies

Hardware

MiniMax-M3 combines 1M context, native multimodality, and MiniMax Sparse Attention. The next layer is serving it efficiently: KV-block-major…

MiniMax-M3 combines 1M context, native multimodality, and MiniMax Sparse Attention. The next layer is serving it efficiently: KV-block-major sparse attention, paged MSA decode, optimized index scoring

hardwaretogether-ai--x

MLLM-Microscope: Unlocking Hidden Structure Within Multimodal Large Language Models

arXiv:2606.00909v1 Announce Type: cross Abstract: This work presents MLLM-Microscope, a novel system designed for analyzing the hidden representations within Multimodal Large Language Models (MLLMs).

Multimodal Action Diffusion for Robust End-to-End Autonomous Driving

model-releasesarxiv-cs-cv

arXiv:2606.02105v1 Announce Type: new Abstract: End-to-End Autonomous Driving (E2E-AD) systems have largely converged on predicting intermediate trajectory waypoints, delegating final control to hand-

ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning

arXiv:2606.02576v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually a

applicationsarxiv-cs-cv

Tools

Serving MiniMax-M3 for efficient inference: Unlocking 1M-Token Context and Multimodality Without Regrets

MiniMax-M3 is a large language model capable of handling 1 million token contexts and multimodal inputs while maintaining efficient inference performance. Together AI's blog post discusses techniques

toolstogether-ai-blog

Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition

arXiv:2606.00959v1 Announce Type: new Abstract: Understanding modality interaction in multimodal large language models (MLLMs) is central to reliable deployment. We introduce Partial Information Decom

UrbanFusion: Stochastic Multimodal Fusion for Contrastive Learning of Robust Spatial Representations

arXiv:2510.13774v2 Announce Type: replace-cross Abstract: Forecasting urban phenomena such as housing prices and public health indicators requires the effective integration of various geospatial data.

researcharxiv-cs-cv

Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearning

arXiv:2606.00105v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have achieved remarkable progress on vision-language tasks, but they may also memorize and expose sensitive o

BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMs

arXiv:2605.30900v1 Announce Type: new Abstract: Current multimodal models handle static image recognition well, but intuitive physical reasoning remains a weakness. Predicting how objects will move an

ConSensus: Multi-Agent Collaboration for Multimodal Sensing

arXiv:2601.06453v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly grounded in sensor data to perceive and reason about human physiology and the physical world. However,

PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection

arXiv:2502.12119v4 Announce Type: replace-cross Abstract: Visual instruction tuning adapts pre-trained Multimodal Large Language Models (MLLMs) to follow human instructions for real-world applications

applicationsarxiv-cs-ai

TRINE: A Token-Aware, Runtime-Adaptive FPGA Inference Engine for Multimodal AI

arXiv:2603.22867v1 Announce Type: cross Abstract: Multimodal stacks that mix ViTs, CNNs, GNNs, and transformer NLP strain embedded platforms because their compute/memory patterns diverge and hard real

Empathic Prompting: Non-Verbal Context Integration for Multimodal LLM Conversations

arXiv:2510.20743v2 Announce Type: replace-cross Abstract: We present Empathic Prompting, a novel framework for multimodal human-AI interaction that enriches Large Language Model (LLM) conversations wi

MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encoding

arXiv:2605.29850v1 Announce Type: new Abstract: Recent progress in task-optimized neural networks has established encoding models as a powerful tool for predicting brain responses to naturalistic stim

researcharxiv-cs-lg

TANDEM: Temporal-Aware Neural Detection for Multimodal Hate Speech

arXiv:2601.11178v2 Announce Type: replace Abstract: Social media platforms are increasingly dominated by long-form multimodal content, where harmful narratives are constructed through a complex interp

TRACER: Persistent Regularization for Robust Multimodal Finetuning

arXiv:2605.29380v1 Announce Type: cross Abstract: Mainstream strategies for finetuning pretrained multimodal models often degrade out-of-distribution (OOD) robustness, a phenomenon known as catastroph

EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents

arXiv:2605.27820v1 Announce Type: new Abstract: As AI agents increasingly operate in open, real-world environments, they require a deep synergy of multimodal perception, tool invocation with multi-hop

Information-theoretic Multimodal Representation Learning for Electrocardiogram Signals

arXiv:2605.27583v1 Announce Type: new Abstract: Electrocardiograms (ECGs) are widely used non-invasive measurements of cardiac activity and play a central role in clinical diagnosis. Recent multimodal

safetyarxiv-cs-lg

Local Ai

MMRad-22K: A Structured Multimodal Evidence Dataset for Chest X-ray Report Generation

arXiv:2602.12843v2 Announce Type: replace Abstract: Chest X-ray (CXR) reporting follows a region-based clinical workflow in which radiologists inspect anatomical regions and integrate localized findin

local-aiarxiv-cs-cv

Reasoning Matters: Mitigate Hallucination in Multimodal Large Reasoning Models via Reasoning-Conditioned Preference Optimization

arXiv:2605.27906v1 Announce Type: new Abstract: Multimodal Large Reasoning Models introduce the reasoning paradigm, demonstrating strong capabilities on complex vision-language tasks. However, they st

SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning

arXiv:2602.01990v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to con

SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding

arXiv:2601.21666v2 Announce Type: replace Abstract: Multimodal Large Language Models (MLLMs) are a major focus of recent AI research. However, most prior work focuses on static image understanding, wh

Utility-Aware Multimodal Contrastive Learning for Product Image Generation

arXiv:2605.28733v1 Announce Type: new Abstract: Product images strongly influence consumer decision-making in online marketplaces. Empowered by multimodal contrastive learning, generative AI can outpu

ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attention

arXiv:2602.07574v2 Announce Type: replace-cross Abstract: Modern multimodal large language models (MLLMs) adopt a unified self-attention design that processes visual and textual tokens at every Transf

researcharxiv-cs-cl

Agents

Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactions

arXiv:2605.26256v1 Announce Type: new Abstract: Multimodal large language model (MLLM)-based embodied agents have shown strong potential for solving complex tasks in physical environments. However, pe

agentsarxiv-cs-ai

27 May 2026

Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks

arXiv:2505.24876v2 Announce Type: replace-cross Abstract: Deep reasoning is fundamental for solving complex tasks, especially in vision-centric scenarios that demand sequential, multimodal understandi

AnatomiX, an Anatomy-Aware Grounded Multimodal Large Language Model for Chest X-Ray Interpretation

arXiv:2601.03191v3 Announce Type: replace-cross Abstract: Multimodal medical large language models have shown substantial progress in chest X-ray interpretation but continue to face challenges in spat

Hardware

Beyond the Aggregation Dilemma: Prior-Retaining Decoupled Learning for Multimodal Graphs

arXiv:2605.24684v1 Announce Type: cross Abstract: Multimodal Attributed Graph Learning (MAGL) integrates intrinsic node attributes with structural topology via graph aggregation. However, as pretraine

hardwarearxiv-cs-ai

BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training

arXiv:2605.25451v1 Announce Type: new Abstract: Training multimodal large language models (MLLMs) is challenged by both model and data heterogeneity. Existing systems redesign the training pipeline to

researcharxiv-cs-lg

ConceptM^3oE: Concept-Guided Multimodal Mixture of Experts for Interpretable Computational Pathology

arXiv:2605.24399v1 Announce Type: new Abstract: Healthcare models are transitioning from unimodal prediction toward multimodal reasoning over heterogeneous diagnostic inputs. In computational patholog

applicationsarxiv-cs-ai

Multimodal Functional Maximum Correlation for Emotion Recognition

arXiv:2512.23076v2 Announce Type: replace-cross Abstract: Emotional states manifest as coordinated yet heterogeneous physiological responses across central and autonomic systems, posing a fundamental

Tutorials

MultiPUFFIN: A Multimodal Domain-Constrained Foundation Model for Molecular Property Prediction of Small Molecules

arXiv:2603.00857v2 Announce Type: replace-cross Abstract: MultiPUFFIN is a domain-informed multimodal foundation model for predicting thermophysical properties of small molecules, addressing a critica

tutorialsarxiv-cs-ai