Search: “multimodal”

ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding

arXiv:2603.27064v2 Announce Type: replace-cross Abstract: Understanding charts requires models to jointly reason over geometric visual patterns, structured numerical data, and natural language -- a ca

safetyarxiv-cs-cl

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

MAny: Merge Anything for Multimodal Continual Instruction Tuning

DGX agent

arXiv:2604.14016v1 Announce Type: new Abstract: Multimodal Continual Instruction Tuning (MCIT) is essential for sequential task adaptation of Multimodal Large Language Models (MLLMs) but is severely r

model-releasesarxiv-cs-lg

16 Apr 2026

MERRIN: A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments

arXiv:2604.13418v1 Announce Type: new Abstract: Motivated by the underspecified, multi-hop nature of search queries and the multimodal, heterogeneous, and often conflicting nature of real-world web re

Tools

Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers

This guide covers how to train and fine-tune multimodal embedding and reranker models using the Sentence Transformers library, enabling systems to work with both text and image data simultaneously. It

toolshugging-face

Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks

arXiv:2604.13403v1 Announce Type: new Abstract: In-context learning (ICL) enables models to adapt to new tasks via inference-time demonstrations. Despite its success in large language models, the exte

safetyarxiv-cs-cv

A Survey of Multimodal Mathematical Reasoning: From Perception, Alignment to Reasoning

arXiv:2603.08291v3 Announce Type: replace Abstract: Multimodal Mathematical Reasoning (MMR) has recently attracted increasing attention for its capability to solve mathematical problems involving both

15 Apr 2026

BoxTuning: Directly Injecting the Object Box for Multimodal Model Fine-Tuning

arXiv:2604.11136v1 Announce Type: cross Abstract: Object-level spatial-temporal understanding is essential for video question answering, yet existing multimodal large language models (MLLMs) encode fr

researcharxiv-cs-ai

C2F-Thinker: Coarse-to-Fine Reasoning with Hint-Guided Reinforcement Learning for Multimodal Sentiment Analysis

arXiv:2604.00013v2 Announce Type: replace-cross Abstract: Multimodal sentiment analysis aims to integrate textual, acoustic, and visual information for deep emotional understanding. Despite the progre

Tutorials

Controlling Multimodal Conversational Agents with Coverage-Enhanced Latent Actions

arXiv:2601.07516v2 Announce Type: replace-cross Abstract: Vision-language models are increasingly employed as multimodal conversational agents (MCAs) for diverse conversational tasks. Recently, reinfo

tutorialsarxiv-cs-ai

Improving Pediatric Emergency Department Triage with Modality Dropout in Late Fusion Multimodal EHR Models

arXiv:2604.09905v1 Announce Type: new Abstract: Emergency department triage relies heavily on both quantitative vital signs and qualitative clinical notes, yet multimodal machine learning models predi

researcharxiv-cs-lg

TorchUMM: A Unified Multimodal Model Codebase for Evaluation, Analysis, and Post-training

arXiv:2604.10784v1 Announce Type: new Abstract: Recent advances in unified multimodal models (UMMs) have led to a proliferation of architectures capable of understanding, generating, and editing acros

Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMs

arXiv:2604.08846v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have been shown to be vulnerable to malicious queries that can elicit unsafe responses. Recent work uses prom

13 Apr 2026

Mosaic: Multimodal Jailbreak against Closed-Source VLMs via Multi-View Ensemble Optimization

arXiv:2604.09253v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) are powerful but remain vulnerable to multimodal jailbreak attacks. Existing attacks mainly rely on either explicit visu

13 Apr 2026

Unified Multimodal Uncertain Inference

arXiv:2604.08701v1 Announce Type: new Abstract: We introduce Unified Multimodal Uncertain Inference (UMUI), a multimodal inference task spanning text, audio, and video, where models must produce calib

13 Apr 2026

Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding

arXiv:2603.18472v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) perform strongly on natural images, yet their ability to understand discrete visual symbols remains u

Commander-GPT: Dividing and Routing for Multimodal Sarcasm Detection

arXiv:2506.19420v2 Announce Type: replace Abstract: Multimodal sarcasm understanding is a high-order cognitive task. Although large language models (LLMs) have shown impressive performance on many dow

Applications

CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs

arXiv:2510.12184v2 Announce Type: replace Abstract: Recently, efficient Multimodal Large Language Models (MLLMs) have gained significant attention as a solution to their high computational complexity,

applicationsarxiv-cs-cv

Mitigating Visual Context Degradation in Large Multimodal Models: A Training-Free Decoupled Agentic Framework

arXiv:2509.23322v2 Announce Type: replace Abstract: With the continuous expansion of Large Language Models (LLMs) and advances in reinforcement learning, LLMs have demonstrated exceptional reasoning c

Tutorials

Steering the Verifiability of Multimodal AI Hallucinations

arXiv:2604.06714v1 Announce Type: new Abstract: AI applications driven by multimodal large language models (MLLMs) are prone to hallucinations and pose considerable risks to human users. Crucially, su

tutorialsarxiv-cs-ai

Following Delegation Traces to Study Human Oversight in Interaction with Multimodal AI

arXiv:2610.21948v1 Announce Type: cross Abstract: As multimodal AI systems are becoming our everyday thought partners, the shift towards studying how users transfer control to these systems becomes es

MMSearch-OPD: Easy Made Hard, Hard Made Learnable for Long-Horizon Multimodal Search Agents

arXiv:2610.22240v1 Announce Type: new Abstract: Multimodal deep search is moving from short queries to long-horizon investigations that continually acquire evidence. However, training such agents rema

safetyarxiv-cs-cv

Agents

TRACER: A Multimodal Harness for Patient Care in Hospitals

arXiv:2610.21266v1 Announce Type: cross Abstract: Continuous video and clinical records provide complementary information for understanding how patient behavior evolves over a hospital stay. A central

agentsarxiv-cs-ai

UniFGVC: Universal Training-Free Few-Shot Fine-Grained Vision Classification via Attribute-Aware Multimodal Retrieval

arXiv:2508.04136v4 Announce Type: replace-cross Abstract: Few-shot fine-grained visual classification (FGVC) aims to discriminate between subtly different categories with very limited data. While Mult

Measuring Epistemic Humility in Multimodal Large Language Models

arXiv:2509.09658v3 Announce Type: replace Abstract: Hallucinations in multimodal large language models (MLLMs), where a model generates content inconsistent with the input image, pose significant risk

PrismaCoder: Pushing the Boundaries of Open Multimodal Code Intelligence through Decomposed Rubric Synthesis

arXiv:2610.20538v1 Announce Type: new Abstract: Multimodal code intelligence connects the symbolic world of programs with the visual world their execution creates. Interest in this capability is burge

researcharxiv-cs-cl

Local Ai

Toward Fine-Grained Forgetting:Attribute Unlearning for Multimodal Large Language Models

arXiv:2608.01008v2 Announce Type: replace Abstract: Multimodal large language models (MLLMs) can memorize and disclose sensitive information, motivating machine unlearning to remove designated knowled

local-aiarxiv-cs-ai

Towards a Data-Centric Understanding of Spatial Reasoning in Multimodal Large Language Models

arXiv:2610.19108v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have been widely applied in embodied agents and robotics for vision tasks. However, MLLMs suffer from limitat

Understanding Unified VLMs: From Attention Probing to Guidance Tuning for Multimodal Generation

arXiv:2610.15227v1 Announce Type: new Abstract: Unified vision-language models (VLMs) perform both text-to-image (T2I) and image-to-text (I2T) generation with one shared Transformer backbone, yet how

researcharxiv-cs-ai

Heading Hierarchy Reconstruction for Complex and Long Documents with Level-Wise Multimodal Memory

arXiv:2610.18238v1 Announce Type: new Abstract: Vision-language models (VLMs) have been widely adopted for document parsing and optical character recognition (OCR), as they can accurately extract page

14 Oct 2026

Tutorials

MDN-VJEPA: Multimodal Probabilistic Joint-Embedding World Models

arXiv:2610.13956v1 Announce Type: new Abstract: Joint-Embedding Predictive Architectures (JEPAs) learn world models by predicting future representations rather than reconstructing observations. Variat

tutorialsarxiv-cs-lg

14 Oct 2026

Applications

1M-token, multimodal decision models are here! But there is more to this one. You can classify, extract, and summarize with 1M tokens of tex…

1M-token, multimodal decision models are here! But there is more to this one. You can classify, extract, and summarize with 1M tokens of text and image input. ScaleDown's Constrained Decoding Models r

applicationsdair-ai--x

13 Oct 2026

JoyInside-VLM-Agent: Product-Conditioned Multimodal Policy Learning for Embodied Agents

arXiv:2610.12957v1 Announce Type: cross Abstract: Embodied agents must translate multimodal observations and user requests into executable actions while adapting to product constraints, changing goals

12 Oct 2026

S-Space: Exploring Spatial Workspace in Multimodal Language Models

arXiv:2610.13527v1 Announce Type: cross Abstract: Multimodal language models (MLLMs) exhibit promising spatial reasoning capabilities, yet the internal mechanisms supporting them remain poorly underst

researcharxiv-cs-cl

12 Oct 2026

The Evidence Anchoring Boundary: Where Visual Interventions Matter and When They Help in Multimodal LLMs

arXiv:2610.12981v1 Announce Type: cross Abstract: Visual interventions provide a training-free way to reduce hallucinations in multimodal large language models. However, an intervention may have littl

researcharxiv-cs-ai

12 Oct 2026

From Surface to Depth: Towards Cognitive Appraisal Reasoning in Multimodal Emotion Understanding

arXiv:2610.11918v1 Announce Type: cross Abstract: Recent multimodal large language models (MLLMs) increasingly incorporate explainable reasoning for emotion understanding. However, reasoning based mai

9 Oct 2026

HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling

arXiv:2610.12393v1 Announce Type: new Abstract: Self-supervised multimodal representation learning has achieved remarkable success across diverse domains, yet capturing synergistic information remains

9 Oct 2026

Self-correction Optimization for Interleaved Multimodal Generation

arXiv:2610.10400v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have made significant progress in visual understanding and generation. However, generating interleaved image--t

researcharxiv-cs-cv

8 Oct 2026

AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking

arXiv:2601.17645v2 Announce Type: replace-cross Abstract: Internet audio-visual clips convey meaning through time-varying sound and motion, which extend beyond what text alone can represent. To examin

Dual-Modality Multi-Stage Adversarial Safety Training: Robustifying Multimodal Web Agents Against Cross-Modal Attacks

arXiv:2603.04364v2 Announce Type: replace-cross Abstract: Multimodal web agents that process both screenshots and accessibility trees are increasingly deployed to interact with web interfaces, yet the

Hardware

Efficient Multimodal Inference through Adaptive Acquisition and Sequential Fusion

arXiv:2610.07466v1 Announce Type: new Abstract: Multimodal systems often encode every available input, even when a subset suffices for prediction. Adaptive acquisition can reduce this cost by using pr

hardwarearxiv-cs-lg

Quantum Entangled Multimodal Fusion Networks (QEMFN): Resource-Aware Hybrid Vision-Language Fusion via Trainable Entanglement

arXiv:2610.08216v1 Announce Type: new Abstract: Multimodal vision-language systems typically fuse image and text embeddings through classical operators such as concatenation, attention, bilinear pooli

Reading, Not Manipulating: Leveraging Router Logits for Multimodal Safety in MoE Vision-Language Models

arXiv:2610.07774v1 Announce Type: new Abstract: Vision-language models (VLMs) face compositional safety risks where harmful intent emerges from the interaction between visual and textual inputs. As mi

A Multimodal Sequence-to-Sequence Model for Cross-Subject Prediction of Brain Responses to Naturalistic Stimuli

arXiv:2507.18104v3 Announce Type: replace Abstract: Brain encoding models predict time-resolved neural activity from computational representations of ongoing experience, providing a principled framewo

researcharxiv-cs-cv

6 Oct 2026

Introducing EmbeddingGemma 2: A best-in-class open model for natively multimodal embeddings | Google

model-releasesr-localllama

https://huggingface.co/google/embeddinggemma-2 EmbeddingGemma 2 is an open multimodal embedding model built by Google DeepMind which maps text (incl. code), images, video, and audio inputs—and combina

6 Oct 2026

HyperBrowseComp: A Multilingual and Multimodal Stress Test for Web-Browsing Agents

arXiv:2610.03574v1 Announce Type: new Abstract: We introduce HyperBrowseComp, a multilingual and multimodal browsing benchmark comprising 423 manually authored and human-validated questions across 13

Kinematics-Induced Multimodal 3D Human Pose Estimation with Subject-Level Privacy

arXiv:2610.02943v1 Announce Type: new Abstract: Multimodal 3D Human Pose Estimation (3D HPE) combines complementary information from RGB, LiDAR, and mmWave radar, but models trained on correlated obse

Seeing, Saying, but Not Using: From Reportable Spatial Facts to Usable States in Multimodal Large Language Models

arXiv:2610.02876v1 Announce Type: new Abstract: A multimodal large language model that correctly reports a spatial fact does not necessarily use that fact in subsequent reasoning. To study this distin

The Percept-V Challenge: Can Multimodal LLMs Crack Simple Perception Problems?

arXiv:2508.21143v4 Announce Type: replace Abstract: Cognitive science research treats visual perception, the ability to understand and make sense of a visual input, as one of the early developmental s

researcharxiv-cs-cl