Search: “multimodal”

LithoBench: Benchmarking Large Multimodal Models for Remote-Sensing Lithology Interpretation

arXiv:2605.07640v1 Announce Type: cross Abstract: Remote sensing lithology interpretation is fundamental to geological surveys, mineral exploration, and regional geological mapping. Unlike general lan

11 May 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

OmicsLM: A Multimodal Large Language Model for Multi-Sample Omics Reasoning

DGX agent

arXiv:2605.06728v1 Announce Type: cross Abstract: Interpreting transcriptomic data is one of the most common analytical tasks in modern biology. Yet most current models either consume expression profi

model-releasesarxiv-cs-ai

11 May 2026

Local Ai

Think Before You Drive: World Model-Inspired Multimodal Grounding for Autonomous Vehicles

arXiv:2512.03454v4 Announce Type: replace-cross Abstract: Interpreting natural-language commands to localize target objects is critical for autonomous driving (AD). Existing visual grounding (VG) meth

local-aiarxiv-cs-ai

11 May 2026

Densification and forecasting of Sentinel-2 time series from multimodal SAR and Optical satellite data using deep generative models

arXiv:2605.04239v1 Announce Type: new Abstract: Optical satellite image time series are extensively used in many Earth observation applications, including agriculture, climate monitoring, and land sur

7 May 2026

Applications

MIRAGE: Retrieval and Generation of Multimodal Images and Texts for Medical Education

arXiv:2605.04772v1 Announce Type: new Abstract: Access to diverse, well-annotated medical images with interactive learning tools is fundamental for training practitioners in medicine and related field

applicationsarxiv-cs-cv

7 May 2026

A-CODE: Fully Atomic Protein Co-Design with Unified Multimodal Diffusion

arXiv:2605.03360v1 Announce Type: cross Abstract: We present A-CODE, a fully atomic unified one-stage protein co-design model that simultaneously refines discrete atom types and continuous atom coordi

researcharxiv-cs-lg

6 May 2026

RoboAlign-R1: Distilled Multimodal Reward Alignment for Robot Video World Models

model-releasesarxiv-cs-ro

arXiv:2605.03821v1 Announce Type: new Abstract: Existing robot video world models are typically trained with low-level objectives such as reconstruction and perceptual similarity, which are poorly ali

6 May 2026

Bolek: A Multimodal Language Model for Molecular Reasoning

arXiv:2605.02745v1 Announce Type: new Abstract: Molecular property models increasingly support high-stakes drug-discovery decisions, but their outputs are often difficult to audit: classical predictor

safetyarxiv-cs-lg

5 May 2026

Applications

Multimodal Confidence Modeling in Audio-Visual Quality Assessment

arXiv:2605.01219v1 Announce Type: cross Abstract: Audio-visual quality assessment (AVQA) is essential for streaming, teleconferencing, and immersive media. In realistic streaming scenarios, distortion

applicationsarxiv-cs-cv

5 May 2026

Self-Supervised Learning for Multimodal Non-Rigid 3D Shape Matching

arXiv:2303.10971v2 Announce Type: replace Abstract: The matching of 3D shapes has been extensively studied for shapes represented as surface meshes, as well as for shapes represented as point clouds.

5 May 2026

Hypergraph and Latent ODE Learning for Multimodal Root Cause Localization in Microservices

model-releasesarxiv-cs-lg

arXiv:2605.00351v1 Announce Type: new Abstract: Root cause localization in cloud native microservice systems requires modeling complex service dependencies, irregular temporal dynamics, and heterogene

4 May 2026

Introducing WARM-VR: Benchmark Dataset for Multimodal Wearable Affect Recognition in Virtual Reality

model-releasesarxiv-cs-lg

arXiv:2605.00184v1 Announce Type: new Abstract: With the growing integration of human-computer interaction into everyday life, advances in machine learning have enabled systems to better perceive and

4 May 2026

Structural Prognostic Event Modeling for Multimodal Cancer Survival Analysis

arXiv:2512.01116v3 Announce Type: replace Abstract: The integration of histology images and gene profiles has shown great promise for improving survival prediction in cancer. However, current approach

4 May 2026

GuideDog: A Real-World Egocentric Multimodal Dataset for Blind and Low-Vision Accessibility-Aware Guidance

arXiv:2503.12844v2 Announce Type: replace Abstract: For people affected by blindness and low vision (BLV), safe and independent navigation remains a major challenge, impacting over 2.2 billion individ

Iterative Multimodal Retrieval-Augmented Generation for Medical Question Answering

arXiv:2604.27724v1 Announce Type: new Abstract: Medical retrieval-augmented generation (RAG) systems typically operate on text chunks extracted from biomedical literature, discarding the rich visual c

Learning Generalizable Multimodal Representations for Software Vulnerability Detection

arXiv:2604.25711v2 Announce Type: replace-cross Abstract: Source code and its accompanying comments are complementary yet naturally aligned modalities-code encodes structural logic while comments capt

Linguistically Informed Multimodal Fusion for Vietnamese Scene-Text Image Captioning: Dataset, Graph Framework, and Phonological Attention

arXiv:2604.27712v1 Announce Type: cross Abstract: Scene-text image captioning requires fusing three information streams -- visual features, OCR-detected text, and linguistic knowledge -- to generate d

safetyarxiv-cs-cl

VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking

arXiv:2601.08611v2 Announce Type: replace-cross Abstract: The growing scale of online misinformation urgently demands Automated Fact-Checking (AFC). Existing benchmarks for evaluating AFC systems, how

A Multimodal Pre-trained Network for Integrated EEG-Video Seizure Detection

arXiv:2604.26379v1 Announce Type: new Abstract: Reliable seizure detection in mouse models is essential for preclinical epilepsy research, yet manual review of synchronized video-EEG recordings is lab

safetyarxiv-cs-cv

30 Apr 2026

FASH-iCNN: Making Editorial Fashion Identity Inspectable Through Multimodal CNN Probing

arXiv:2604.26186v1 Announce Type: new Abstract: Fashion AI systems routinely encode the aesthetic logic of specific houses, editors, and historical moments without disclosing it. We present FASH-iCNN,

30 Apr 2026

Curriculum-guided multimodal representation learning enables generalizable prediction of nanomaterial-protein interactions

arXiv:2507.14245v2 Announce Type: replace Abstract: Nanomaterial-protein interactions (NPI) are pivotal to realizing the therapeutic and diagnostic potential of nanomaterials. Although AI promises to

researcharxiv-cs-lg

29 Apr 2026

Multimodal Contextualized Support for Enhancing Video Retrieval System

arXiv:2412.07584v2 Announce Type: replace Abstract: Current video retrieval systems, especially those used in competitions, primarily focus on querying individual keyframes or images rather than encod

29 Apr 2026

The Thinking Pixel: Recursive Sparse Reasoning in Multimodal Diffusion Latents

arXiv:2604.25299v1 Announce Type: new Abstract: Diffusion models have achieved success in high-fidelity data synthesis, yet their capacity for more complex, structured reasoning like text following ta

29 Apr 2026

DeepTaxon: An Interpretable Retrieval-Augmented Multimodal Framework for Unified Species Identification and Discovery

arXiv:2604.24029v1 Announce Type: cross Abstract: Identifying species in biology among tens of thousands of visually similar taxa while discovering unknown species in open-world environments remains a

28 Apr 2026

Hierarchical Prototype-based Domain Priors for Multiple Instance Learning in Multimodal Histopathology Analysis

arXiv:2604.23982v1 Announce Type: new Abstract: Digital pathology has fundamentally altered diagnostic workflows by enabling the computational analysis of gigapixel Whole Slide Images (WSIs), yet effe

safetyarxiv-cs-cv

28 Apr 2026

One Identity, Many Roles: Multimodal Entity Coreference for Enhanced Video Situation Recognition

arXiv:2604.23173v1 Announce Type: new Abstract: Video Situation Recognition (VidSitu) addresses the challenging problem of 'who did what to whom, with what, how, and where' in a video. It tests thorou

28 Apr 2026

AromaGen: Interactive Generation of Rich Olfactory Experiences with Multimodal Language Models

arXiv:2604.01650v2 Announce Type: replace-cross Abstract: Smell's deep connection with food, memory, and social experience has long motivated researchers to bring olfaction into interactive systems. Y

MTT-Bench: Predicting Social Dominance in Mice via Multimodal Large Language Models

arXiv:2604.22492v1 Announce Type: cross Abstract: Understanding social dominance in animal behavior is critical for neuroscience and behavioral studies. In this work, we explore the capability of Mult

Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Models

arXiv:2510.07632v2 Announce Type: replace Abstract: Frontier AI models have achieved remarkable progress, yet recent studies suggest they struggle with compositional reasoning, often performing at or

UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents

arXiv:2602.07038v2 Announce Type: replace-cross Abstract: Key Information Extraction (KIE) from real-world documents remains challenging due to substantial variations in layout structures, visual qual

Focus on What Matters: Fisher-Guided Adaptive Multimodal Fusion for Vulnerability Detection

arXiv:2601.02438v3 Announce Type: replace-cross Abstract: Software vulnerability detection can be formulated as a binary classification problem that determines whether a given code snippet contains se

researcharxiv-cs-ai

24 Apr 2026

Learning Physics from Pretrained Video Models: A Multimodal Continuous and Sequential World Interaction Models for Robotic Manipulation

arXiv:2603.00110v2 Announce Type: replace Abstract: The scarcity of large-scale robotic data has motivated the repurposing of foundation models from other modalities for policy learning. In this work,

safetyarxiv-cs-ro

24 Apr 2026

OmniGen2: Towards Instruction-Aligned Multimodal Generation

arXiv:2506.18871v4 Announce Type: replace-cross Abstract: In this work, we introduce OmniGen2, a versatile and open-source generative model designed to provide a unified solution for diverse generatio

22 Apr 2026

FLiP: Towards understanding and interpreting multimodal multilingual sentence embeddings

arXiv:2604.18109v1 Announce Type: new Abstract: This paper presents factorized linear projection (FLiP) models for understanding pretrained sentence embedding spaces. We train FLiP models to recover t

Multimodal In-context Learning for ASR of Low-resource Languages

arXiv:2601.05707v2 Announce Type: replace Abstract: Automatic speech recognition (ASR) still covers only a small fraction of the world's languages, mainly due to supervised data scarcity. In-context l

SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models

model-releasesarxiv-cs-lg

arXiv:2604.16883v1 Announce Type: new Abstract: In long-context decoding for LLMs and LMMs, attention becomes increasingly memory-bound because each decoding step must load a large amount of KV-cache

STEP-PD: Stage-Aware and Explainable Parkinson's Disease Severity Classification Using Multimodal Clinical Assessments

arXiv:2604.17611v1 Announce Type: new Abstract: Parkinson's disease (PD) is a progressive disorder in which symptom burden and functional impairment evolve over time, making severity staging essential

researcharxiv-cs-lg

CAVERS: Multimodal SLAM Data from a Natural Karstic Cave with Ground Truth Motion Capture

model-releasesarxiv-cs-ro

arXiv:2604.15052v1 Announce Type: new Abstract: Autonomous robots operating in natural karstic caves face perception and navigation challenges that are qualitatively distinct from those encountered in

17 Apr 2026

A Multimodal Clinically Informed Coarse-to-Fine Framework for Longitudinal CT Registration in Proton Therapy

arXiv:2604.13397v1 Announce Type: new Abstract: Proton therapy offers superior organ-at-risk sparing but is highly sensitive to anatomical changes, making accurate deformable image registration (DIR)

16 Apr 2026

AIM-CoT: Active Information-driven Multimodal Chain-of-Thought for Vision-Language Reasoning

arXiv:2509.25699v2 Announce Type: replace Abstract: Interleaved-Modal Chain-of-Thought (I-MCoT) advances vision-language reasoning, such as Visual Question Answering (VQA). This paradigm integrates sp

16 Apr 2026

Agents

M2HRI: An LLM-Driven Multimodal Multi-Agent Framework for Personalized Human-Robot Interaction

arXiv:2604.11975v1 Announce Type: new Abstract: Multi-robot systems hold significant promise for social environments such as homes and hospitals, yet existing multi-robot works treat robots as functio

agentsarxiv-cs-ro

15 Apr 2026

ReefMapGS: Enabling Large-Scale Underwater Reconstruction by Closing the Loop Between Multimodal SLAM and Gaussian Splatting

arXiv:2604.11992v1 Announce Type: cross Abstract: 3D Gaussian Splatting is a powerful visual representation, providing high-quality and efficient 3D scene reconstruction, but it is crucially dependent

15 Apr 2026

A Mamba-Based Multimodal Network for Multiscale Blast-Induced Rapid Structural Damage Assessment

arXiv:2604.11709v1 Announce Type: new Abstract: Accurate and rapid structural damage assessment (SDA) is crucial for post-disaster management, helping responders prioritise resources, plan rescues, an

safetyarxiv-cs-ai

Bidirectional Cross-Attention Fusion of High-Res RGB and Low-Res HSI for Multimodal Automated Waste Sorting

arXiv:2603.13941v2 Announce Type: replace Abstract: Growing waste streams and the transition to a circular economy require efficient automated waste sorting. In industrial settings, materials move on

Hardware

Can LLMs Reason About Attention? Towards Zero-Shot Analysis of Multimodal Classroom Behavior

arXiv:2604.03401v2 Announce Type: replace-cross Abstract: Understanding student engagement usually requires time-consuming manual observation or invasive recording that raises privacy concerns. We pre

hardwarearxiv-cs-ai

VisText-Mosquito: A Unified Multimodal Dataset for Visual Detection, Segmentation, and Textual Explanation on Mosquito Breeding Sites

arXiv:2506.14629v3 Announce Type: replace-cross Abstract: Mosquito-borne diseases pose a major global health risk, requiring early detection and proactive control of breeding sites to prevent outbreak

researcharxiv-cs-cl

Multimodal Anomaly Detection for Human-Robot Interaction

arXiv:2604.09326v1 Announce Type: cross Abstract: Ensuring safety and reliability in human-robot interaction (HRI) requires the timely detection of unexpected events that could lead to system failures

safetyarxiv-cs-cv

13 Apr 2026

On the Spectral Geometry of Cross-Modal Representations: A Functional Map Diagnostic for Multimodal Alignment