Search: “multimodal”

SeePhys Pro: Diagnosing Modality Transfer and Blind-Training Effects in Multimodal RLVR for Physics Reasoning

arXiv:2605.09266v1 Announce Type: new Abstract: We introduce SeePhys Pro, a fine-grained modality transfer benchmark that studies whether models preserve the same reasoning capability when critical in

12 May 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

Self-Captioning Multimodal Interaction Tuning: Amplifying Exploitable Redundancies for Robust Vision Language Models

DGX agent

arXiv:2605.08145v1 Announce Type: cross Abstract: Current vision language models face hallucination and robustness issues against ambiguous or corrupted modalities. We hypothesize that these issues ca

researcharxiv-cs-ai

12 May 2026

UniShield: Unified Face Attack Detection via KG-Informed Multimodal Reasoning

arXiv:2605.08709v1 Announce Type: new Abstract: Unified face attack detection (UAD) requires recognizing physical spoofing and digital forgery within a shared decision space, yet existing discriminati

12 May 2026

AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models

arXiv:2605.07478v1 Announce Type: new Abstract: Speech-driven facial animation requires accurate correspondence between acoustic signals and facial motion, especially for articulation-related mouth mo

applicationsarxiv-cs-cv

Fidel-TS: A High-Fidelity Multimodal Benchmark for Time Series Forecasting

model-releasesarxiv-cs-lg

arXiv:2509.24789v4 Announce Type: replace Abstract: The evaluation of time series forecasting models is hindered by a lack of high-quality benchmarks, leading to overestimated assessments of progress.

MIST: Multimodal Interactive Speech-based Tool-calling Conversational Assistants for Smart Homes

arXiv:2605.06897v1 Announce Type: cross Abstract: The rise of Internet of Things (IoT) devices in the physical world necessitates voice-based interfaces capable of handling complex user experiences. W

Multimodal Diffusion Transformer with Memory Bank for Scalable Long-Duration Talking Video Generation

arXiv:2411.16748v5 Announce Type: replace Abstract: Long-duration talking video synthesis faces enduring challenges in achieving high video quality, portrait consistency, temporal coherence, and compu

Physics-Based Benchmarking Metrics for Multimodal Synthetic Images

arXiv:2511.15204v3 Announce Type: replace-cross Abstract: Current state of the art measures like BLEU, CIDEr, VQA score, SigLIP-2 and CLIPScore are often unable to capture semantic or structural accur

safetyarxiv-cs-ai

UniD-Shift: Towards Unified Semantic Segmentation via Interpretable Share-Private Multimodal Decomposition

arXiv:2605.07356v1 Announce Type: new Abstract: Semantic segmentation of large-scale 3D point clouds is crucial for applications such as autonomous driving and urban digital twins. However, the sparse

safetyarxiv-cs-cv

Gated Multimodal Learning for Interpretable Property Energy Performance Prediction and Retrofit Scenario Analysis

arXiv:2605.05088v1 Announce Type: new Abstract: Achieving resilient and sustainable cities requires scalable approaches to decarbonising residential buildings, which account for about 20% of UK greenh

applicationsarxiv-cs-lg

7 May 2026

Agents

Material Database Agent: A Multimodal Agentic Framework for Scientific Literature Mining

arXiv:2605.04278v1 Announce Type: new Abstract: Materials science workflows rely on structured and unstructured data from the vast body of available scientific literature. However, most of the experim

agentsarxiv-cs-cl

7 May 2026

EO-Gym: A Multimodal, Interactive Environment for Earth Observation Agents

arXiv:2605.01250v1 Announce Type: new Abstract: Earth Observation (EO) analysis is inherently interactive: resolving uncertainty often requires expanding the region of interest, retrieving historical

Erase Persona, Forget Lore: Benchmarking Multimodal Copyright Unlearning in Large Vision Language Models

arXiv:2605.03547v1 Announce Type: new Abstract: Large Vision-Language Models (LVLMs), trained on web-scale data, risk memorizing and regenerating copyrighted visual content such as characters and logo

LangPrecip: Language-Aware Multimodal Precipitation Nowcasting

arXiv:2512.22317v2 Announce Type: replace-cross Abstract: Short-term precipitation nowcasting is an inherently uncertain and under-constrained spatiotemporal forecasting problem, especially for rapidl

UnAC: Adaptive Visual Prompting with Abstraction and Stepwise Checking for Complex Multimodal Reasoning

arXiv:2605.03950v1 Announce Type: new Abstract: Although recent LMMs have become much stronger at visual perception, they remain unreliable on problems that require multi-step reasoning over visual ev

Boosting Multimodal Remote Sensing Image Classification with Transformer-based Heterogeneously Salient Graph Representation

arXiv:2311.10320v3 Announce Type: replace Abstract: Data collected by different modalities can provide a wealth of complementary information, such as hyperspectral image (HSI) to offer rich spectral-s

Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning

arXiv:2605.02378v1 Announce Type: new Abstract: In-context learning (ICL) allows large models to adapt to tasks using a few examples, yet its extension to vision-language models (VLMs) remains fragile

From Where Things Are to What They Are For: Benchmarking Spatial-Functional Intelligence in Multimodal LLMs

arXiv:2605.02130v1 Announce Type: new Abstract: Human-level agentic intelligence extends beyond low-level geometric perception, evolving from recognizing where things are to understanding what they ar

LinMU: Multimodal Understanding Made Linear

arXiv:2601.01322v2 Announce Type: replace Abstract: Modern Vision-Language Models (VLMs) achieve impressive performance but are limited by the quadratic complexity of self-attention, which prevents th

Metric Unreliability in Multimodal Machine Unlearning: A Systematic Analysis and Principled Unified Score

arXiv:2605.02206v1 Announce Type: new Abstract: Machine unlearning in Vision-Language Models (VLMs) is required for compliance with the General Data Protection Regulation (GDPR), yet current evaluatio

Local Ai

Semantic Context-aware mOdality fUsion Transformer (SCOUT): A Context-Aware Multimodal Transformer for Concept-Grounded Pathology Report Generation

arXiv:2605.01144v1 Announce Type: new Abstract: Whole-slide images (WSIs) present a fundamental challenge for computational pathology due to their extreme resolution, multi-scale heterogeneity, and th

local-aiarxiv-cs-cv

Probing Multimodal Large Language Models on Cognitive Biases in Chinese Short-Video Misinformation

model-releasesarxiv-cs-cl

arXiv:2601.06600v2 Announce Type: replace Abstract: Short-video platforms have become major channels for misinformation, where deceptive claims frequently leverage visual experiments and social cues.

4 May 2026

Agents

OptimusKG: Unifying biomedical knowledge in a modern multimodal graph

arXiv:2604.27269v1 Announce Type: new Abstract: Biomedical knowledge graphs (KGs) are widely used in the life sciences, yet many are derived from unstructured documents and therefore lack schema-level

agentsarxiv-cs-ai

1 May 2026

Attribution-Guided Multimodal Deepfake Detection via Cross-Modal Forensic Fingerprints

arXiv:2604.26453v1 Announce Type: new Abstract: Audio-visual deepfakes have reached a level of realism that makes perceptual detection unreliable, threatening media integrity and biometric security. W

safetyarxiv-cs-cv

CheXthought: A global multimodal dataset of clinical chain-of-thought reasoning and visual attention for chest X-ray interpretation

arXiv:2604.26288v1 Announce Type: cross Abstract: Chest X-ray interpretation is one of the most frequently performed diagnostic tasks in medicine and a primary target for AI development, yet current v

GoViG: Goal-Conditioned Visual Navigation Instruction Generation via Multimodal Reasoning

arXiv:2508.09547v2 Announce Type: replace-cross Abstract: We introduce Goal-Conditioned Visual Navigation Instruction Generation (GoViG), a new task that aims to generate contextually coherent navigat

M2R2: MultiModal Robotic Representation for Temporal Action Segmentation

arXiv:2504.18662v3 Announce Type: replace-cross Abstract: Temporal action segmentation (TAS) has long been a key area of research in both robotics and computer vision. In robotics, algorithms have pri

researcharxiv-cs-ai

QEVA: A Reference-Free Evaluation Metric for Narrative Video Summarization with Multimodal Question Answering

arXiv:2604.24052v1 Announce Type: cross Abstract: Video-to-text summarization remains underexplored in terms of comprehensive evaluation methods. Traditional n-gram overlap-based metrics and recent la

28 Apr 2026

Tutorials

Multimodal Diffusion to Mutually Enhance Polarized Light and Low Resolution EBSD Data

arXiv:2604.22212v1 Announce Type: cross Abstract: In spite of the utility of 3-D electron back-scattered diffraction (EBSD) microscopy, the data collection process can be time-consuming with serial-se

tutorialsarxiv-cs-cv

27 Apr 2026

Thinking Like a Botanist: Challenging Multimodal Language Models with Intent-Driven Chain-of-Inquiry

arXiv:2604.20983v1 Announce Type: cross Abstract: Vision evaluations are typically done through multi-step processes. In most contemporary fields, experts analyze images using structured, evidence-bas

24 Apr 2026

CCTVBench: Contrastive Consistency Traffic VideoQA Benchmark for Multimodal LLMs

arXiv:2604.20460v1 Announce Type: new Abstract: Safety-critical traffic reasoning requires contrastive consistency: models must detect true hazards when an accident occurs, and reliably reject plausib

23 Apr 2026

VTouch++: A Multimodal Dataset with Vision-Based Tactile Enhancement for Bimanual Manipulation

arXiv:2604.20444v1 Announce Type: cross Abstract: Embodied intelligence has advanced rapidly in recent years; however, bimanual manipulation-especially in contact-rich tasks remains challenging. This

23 Apr 2026

Multimodal embodiment-aware navigation transformer

arXiv:2604.19267v1 Announce Type: new Abstract: Goal-conditioned navigation models for ground robots trained using supervised learning show promising zero-shot transfer, but their collision-avoidance

safetyarxiv-cs-ro

22 Apr 2026

PuzzleWorld: A Benchmark for Multimodal, Open-Ended Reasoning in Puzzlehunts

arXiv:2506.06211v2 Announce Type: replace-cross Abstract: Puzzlehunts are a genre of complex, multi-step puzzles lacking well-defined problem definitions. In contrast to conventional reasoning benchma

22 Apr 2026

REVEAL: Multimodal Vision-Language Alignment of Retinal Morphometry and Clinical Risks for Incident AD and Dementia Prediction

arXiv:2604.18757v1 Announce Type: cross Abstract: The retina provides a unique, noninvasive window into Alzheimer's disease (AD) and dementia, capturing early structural changes through morphometric f

safetyarxiv-cs-ai

22 Apr 2026

Agents

A Discordance-Aware Multimodal Framework with Multi-Agent Clinical Reasoning

arXiv:2604.16333v1 Announce Type: new Abstract: Knee osteoarthritis frequently exhibits discordance between structural damage observed in imaging and patient-reported symptoms such as pain. This misma

agentsarxiv-cs-lg

BasketHAR: A Multimodal Dataset for Human Activity Recognition and Sport Analysis in Basketball Training Scenarios

arXiv:2604.17065v1 Announce Type: new Abstract: Human Activity Recognition (HAR) involves the automatic identification of user activities and has gained significant research interest due to its broad

DGSSM: Diffusion guided state-space models for multimodal salient object detection

arXiv:2604.17585v1 Announce Type: new Abstract: Salient object detection (SOD) requires modeling both long-range contextual dependencies and fine-grained structural details, which remains challenging

EgoWalk: A Multimodal Dataset for Robot Navigation in the Wild

arXiv:2505.21282v2 Announce Type: replace Abstract: Data-driven navigation algorithms are critically dependent on large-scale, high-quality real-world data collection for successful training and robus

applicationsarxiv-cs-ro

GoCoMA: Hyperbolic Multimodal Representation Fusion for Large Language Model-Generated Code Attribution

arXiv:2604.16377v1 Announce Type: new Abstract: Large Language Models (LLMs) trained on massive code corpora are now increasingly capable of generating code that is hard to distinguish from human-writ

researcharxiv-cs-cl

MegaRAG: Multimodal Knowledge Graph-Based Retrieval Augmented Generation

arXiv:2512.20626v2 Announce Type: replace-cross Abstract: Retrieval-augmented generation (RAG) enables large language models (LLMs) to dynamically access external information, which is powerful for an

researcharxiv-cs-cl

More Than Sum of Its Parts: Deciphering Intent Shifts in Multimodal Hate Speech Detection

model-releasesarxiv-cs-cl

arXiv:2603.21298v2 Announce Type: replace Abstract: Combating hate speech on social media is critical for securing cyberspace, yet relies heavily on the efficacy of automated detection systems. As con

Multimodal Fusion of Histopathology Images and Electronic Health Records for Early Breast Cancer Diagnosis

arXiv:2604.17122v1 Announce Type: new Abstract: Breast cancer is a leading cause of cancer-related mortality worldwide, and timely accurate diagnosis is critical to improving survival outcomes. While

ZSG-IAD: A Multimodal Framework for Zero-Shot Grounded Industrial Anomaly Detection

arXiv:2604.17949v1 Announce Type: new Abstract: Deep learning-based industrial anomaly detectors often behave as black boxes, making it hard to justify decisions with physically meaningful defect evid

Local Ai

ExoNet: Multimodal Deep Learning for TESS Exoplanet Candidate Identification via Phase-Folded Light Curves, Stellar Parameters, and Multi-Head Attention Fusion

arXiv:2604.15560v1 Announce Type: cross Abstract: NASA's Transiting Exoplanet Survey Satellite (TESS) has identified thousands of exoplanet candidates, yet many remain unconfirmed due to the limitatio

local-aiarxiv-cs-lg

20 Apr 2026

MM-Telco: Benchmarks and Multimodal Large Language Models for Telecom Applications

arXiv:2511.13131v2 Announce Type: replace Abstract: Large Language Models (LLMs) have emerged as powerful tools for automating complex reasoning and decision-making tasks. In telecommunications, they

20 Apr 2026

Hoi! -- A Multimodal Dataset for Force-Grounded, Cross-View Articulated Manipulation

arXiv:2512.04884v3 Announce Type: replace Abstract: We present a dataset for force-grounded, cross-view articulated manipulation that couples what is seen with what is done and what is felt during rea

researcharxiv-cs-ro

17 Apr 2026

MISID: A Multimodal Multi-turn Dataset for Complex Intent Recognition in Strategic Deception Games

arXiv:2604.12700v1 Announce Type: new Abstract: Understanding human intent in complex multi-turn interactions remains a fundamental challenge in human-computer interaction and behavioral analysis. Whi