Search: “multimodal”

EggHand: A Multimodal Foundation Model for Egocentric Hand Pose Forecasting

arXiv:2605.07642v1 Announce Type: new Abstract: Forecasting future 3D hand pose sequences from egocentric video is essential for understanding human intention and enabling embodied applications such a

11 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

LithoBench: Benchmarking Large Multimodal Models for Remote-Sensing Lithology Interpretation

DGX agent

arXiv:2605.07640v1 Announce Type: cross Abstract: Remote sensing lithology interpretation is fundamental to geological surveys, mineral exploration, and regional geological mapping. Unlike general lan

model-releasesarxiv-cs-ai

11 May 2026

OmicsLM: A Multimodal Large Language Model for Multi-Sample Omics Reasoning

arXiv:2605.06728v1 Announce Type: cross Abstract: Interpreting transcriptomic data is one of the most common analytical tasks in modern biology. Yet most current models either consume expression profi

11 May 2026

Local Ai

Think Before You Drive: World Model-Inspired Multimodal Grounding for Autonomous Vehicles

arXiv:2512.03454v4 Announce Type: replace-cross Abstract: Interpreting natural-language commands to localize target objects is critical for autonomous driving (AD). Existing visual grounding (VG) meth

local-aiarxiv-cs-ai

11 May 2026

Densification and forecasting of Sentinel-2 time series from multimodal SAR and Optical satellite data using deep generative models

arXiv:2605.04239v1 Announce Type: new Abstract: Optical satellite image time series are extensively used in many Earth observation applications, including agriculture, climate monitoring, and land sur

7 May 2026

Applications

MIRAGE: Retrieval and Generation of Multimodal Images and Texts for Medical Education

arXiv:2605.04772v1 Announce Type: new Abstract: Access to diverse, well-annotated medical images with interactive learning tools is fundamental for training practitioners in medicine and related field

applicationsarxiv-cs-cv

7 May 2026

A-CODE: Fully Atomic Protein Co-Design with Unified Multimodal Diffusion

arXiv:2605.03360v1 Announce Type: cross Abstract: We present A-CODE, a fully atomic unified one-stage protein co-design model that simultaneously refines discrete atom types and continuous atom coordi

researcharxiv-cs-lg

6 May 2026

RoboAlign-R1: Distilled Multimodal Reward Alignment for Robot Video World Models

model-releasesarxiv-cs-ro

arXiv:2605.03821v1 Announce Type: new Abstract: Existing robot video world models are typically trained with low-level objectives such as reconstruction and perceptual similarity, which are poorly ali

6 May 2026

Bolek: A Multimodal Language Model for Molecular Reasoning

arXiv:2605.02745v1 Announce Type: new Abstract: Molecular property models increasingly support high-stakes drug-discovery decisions, but their outputs are often difficult to audit: classical predictor

safetyarxiv-cs-lg

5 May 2026

Applications

Multimodal Confidence Modeling in Audio-Visual Quality Assessment

arXiv:2605.01219v1 Announce Type: cross Abstract: Audio-visual quality assessment (AVQA) is essential for streaming, teleconferencing, and immersive media. In realistic streaming scenarios, distortion

applicationsarxiv-cs-cv

5 May 2026

Self-Supervised Learning for Multimodal Non-Rigid 3D Shape Matching

arXiv:2303.10971v2 Announce Type: replace Abstract: The matching of 3D shapes has been extensively studied for shapes represented as surface meshes, as well as for shapes represented as point clouds.

5 May 2026

Hypergraph and Latent ODE Learning for Multimodal Root Cause Localization in Microservices

model-releasesarxiv-cs-lg

arXiv:2605.00351v1 Announce Type: new Abstract: Root cause localization in cloud native microservice systems requires modeling complex service dependencies, irregular temporal dynamics, and heterogene

4 May 2026

Introducing WARM-VR: Benchmark Dataset for Multimodal Wearable Affect Recognition in Virtual Reality

model-releasesarxiv-cs-lg

arXiv:2605.00184v1 Announce Type: new Abstract: With the growing integration of human-computer interaction into everyday life, advances in machine learning have enabled systems to better perceive and

4 May 2026

Structural Prognostic Event Modeling for Multimodal Cancer Survival Analysis

arXiv:2512.01116v3 Announce Type: replace Abstract: The integration of histology images and gene profiles has shown great promise for improving survival prediction in cancer. However, current approach

4 May 2026

GuideDog: A Real-World Egocentric Multimodal Dataset for Blind and Low-Vision Accessibility-Aware Guidance

arXiv:2503.12844v2 Announce Type: replace Abstract: For people affected by blindness and low vision (BLV), safe and independent navigation remains a major challenge, impacting over 2.2 billion individ

Iterative Multimodal Retrieval-Augmented Generation for Medical Question Answering

arXiv:2604.27724v1 Announce Type: new Abstract: Medical retrieval-augmented generation (RAG) systems typically operate on text chunks extracted from biomedical literature, discarding the rich visual c

Learning Generalizable Multimodal Representations for Software Vulnerability Detection

arXiv:2604.25711v2 Announce Type: replace-cross Abstract: Source code and its accompanying comments are complementary yet naturally aligned modalities-code encodes structural logic while comments capt

Linguistically Informed Multimodal Fusion for Vietnamese Scene-Text Image Captioning: Dataset, Graph Framework, and Phonological Attention

arXiv:2604.27712v1 Announce Type: cross Abstract: Scene-text image captioning requires fusing three information streams -- visual features, OCR-detected text, and linguistic knowledge -- to generate d

safetyarxiv-cs-cl

VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking

arXiv:2601.08611v2 Announce Type: replace-cross Abstract: The growing scale of online misinformation urgently demands Automated Fact-Checking (AFC). Existing benchmarks for evaluating AFC systems, how

A Multimodal Pre-trained Network for Integrated EEG-Video Seizure Detection

arXiv:2604.26379v1 Announce Type: new Abstract: Reliable seizure detection in mouse models is essential for preclinical epilepsy research, yet manual review of synchronized video-EEG recordings is lab

safetyarxiv-cs-cv

30 Apr 2026

FASH-iCNN: Making Editorial Fashion Identity Inspectable Through Multimodal CNN Probing

arXiv:2604.26186v1 Announce Type: new Abstract: Fashion AI systems routinely encode the aesthetic logic of specific houses, editors, and historical moments without disclosing it. We present FASH-iCNN,

30 Apr 2026

Curriculum-guided multimodal representation learning enables generalizable prediction of nanomaterial-protein interactions

arXiv:2507.14245v2 Announce Type: replace Abstract: Nanomaterial-protein interactions (NPI) are pivotal to realizing the therapeutic and diagnostic potential of nanomaterials. Although AI promises to

researcharxiv-cs-lg

29 Apr 2026

Multimodal Contextualized Support for Enhancing Video Retrieval System

arXiv:2412.07584v2 Announce Type: replace Abstract: Current video retrieval systems, especially those used in competitions, primarily focus on querying individual keyframes or images rather than encod

29 Apr 2026

The Thinking Pixel: Recursive Sparse Reasoning in Multimodal Diffusion Latents

arXiv:2604.25299v1 Announce Type: new Abstract: Diffusion models have achieved success in high-fidelity data synthesis, yet their capacity for more complex, structured reasoning like text following ta

29 Apr 2026

DeepTaxon: An Interpretable Retrieval-Augmented Multimodal Framework for Unified Species Identification and Discovery

arXiv:2604.24029v1 Announce Type: cross Abstract: Identifying species in biology among tens of thousands of visually similar taxa while discovering unknown species in open-world environments remains a

28 Apr 2026

Hierarchical Prototype-based Domain Priors for Multiple Instance Learning in Multimodal Histopathology Analysis

arXiv:2604.23982v1 Announce Type: new Abstract: Digital pathology has fundamentally altered diagnostic workflows by enabling the computational analysis of gigapixel Whole Slide Images (WSIs), yet effe

safetyarxiv-cs-cv

28 Apr 2026

One Identity, Many Roles: Multimodal Entity Coreference for Enhanced Video Situation Recognition

arXiv:2604.23173v1 Announce Type: new Abstract: Video Situation Recognition (VidSitu) addresses the challenging problem of 'who did what to whom, with what, how, and where' in a video. It tests thorou

28 Apr 2026

AromaGen: Interactive Generation of Rich Olfactory Experiences with Multimodal Language Models

arXiv:2604.01650v2 Announce Type: replace-cross Abstract: Smell's deep connection with food, memory, and social experience has long motivated researchers to bring olfaction into interactive systems. Y

MTT-Bench: Predicting Social Dominance in Mice via Multimodal Large Language Models

arXiv:2604.22492v1 Announce Type: cross Abstract: Understanding social dominance in animal behavior is critical for neuroscience and behavioral studies. In this work, we explore the capability of Mult

Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Models

arXiv:2510.07632v2 Announce Type: replace Abstract: Frontier AI models have achieved remarkable progress, yet recent studies suggest they struggle with compositional reasoning, often performing at or

UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents

arXiv:2602.07038v2 Announce Type: replace-cross Abstract: Key Information Extraction (KIE) from real-world documents remains challenging due to substantial variations in layout structures, visual qual

Focus on What Matters: Fisher-Guided Adaptive Multimodal Fusion for Vulnerability Detection

arXiv:2601.02438v3 Announce Type: replace-cross Abstract: Software vulnerability detection can be formulated as a binary classification problem that determines whether a given code snippet contains se

researcharxiv-cs-ai

24 Apr 2026

Learning Physics from Pretrained Video Models: A Multimodal Continuous and Sequential World Interaction Models for Robotic Manipulation

arXiv:2603.00110v2 Announce Type: replace Abstract: The scarcity of large-scale robotic data has motivated the repurposing of foundation models from other modalities for policy learning. In this work,

safetyarxiv-cs-ro

24 Apr 2026

OmniGen2: Towards Instruction-Aligned Multimodal Generation

arXiv:2506.18871v4 Announce Type: replace-cross Abstract: In this work, we introduce OmniGen2, a versatile and open-source generative model designed to provide a unified solution for diverse generatio

22 Apr 2026

FLiP: Towards understanding and interpreting multimodal multilingual sentence embeddings

arXiv:2604.18109v1 Announce Type: new Abstract: This paper presents factorized linear projection (FLiP) models for understanding pretrained sentence embedding spaces. We train FLiP models to recover t

Multimodal In-context Learning for ASR of Low-resource Languages

arXiv:2601.05707v2 Announce Type: replace Abstract: Automatic speech recognition (ASR) still covers only a small fraction of the world's languages, mainly due to supervised data scarcity. In-context l

SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models

model-releasesarxiv-cs-lg

arXiv:2604.16883v1 Announce Type: new Abstract: In long-context decoding for LLMs and LMMs, attention becomes increasingly memory-bound because each decoding step must load a large amount of KV-cache

STEP-PD: Stage-Aware and Explainable Parkinson's Disease Severity Classification Using Multimodal Clinical Assessments

arXiv:2604.17611v1 Announce Type: new Abstract: Parkinson's disease (PD) is a progressive disorder in which symptom burden and functional impairment evolve over time, making severity staging essential

researcharxiv-cs-lg

CAVERS: Multimodal SLAM Data from a Natural Karstic Cave with Ground Truth Motion Capture

model-releasesarxiv-cs-ro

arXiv:2604.15052v1 Announce Type: new Abstract: Autonomous robots operating in natural karstic caves face perception and navigation challenges that are qualitatively distinct from those encountered in

17 Apr 2026

A Multimodal Clinically Informed Coarse-to-Fine Framework for Longitudinal CT Registration in Proton Therapy

arXiv:2604.13397v1 Announce Type: new Abstract: Proton therapy offers superior organ-at-risk sparing but is highly sensitive to anatomical changes, making accurate deformable image registration (DIR)

16 Apr 2026

AIM-CoT: Active Information-driven Multimodal Chain-of-Thought for Vision-Language Reasoning

arXiv:2509.25699v2 Announce Type: replace Abstract: Interleaved-Modal Chain-of-Thought (I-MCoT) advances vision-language reasoning, such as Visual Question Answering (VQA). This paradigm integrates sp

16 Apr 2026

Agents

M2HRI: An LLM-Driven Multimodal Multi-Agent Framework for Personalized Human-Robot Interaction

arXiv:2604.11975v1 Announce Type: new Abstract: Multi-robot systems hold significant promise for social environments such as homes and hospitals, yet existing multi-robot works treat robots as functio

agentsarxiv-cs-ro

15 Apr 2026

ReefMapGS: Enabling Large-Scale Underwater Reconstruction by Closing the Loop Between Multimodal SLAM and Gaussian Splatting

arXiv:2604.11992v1 Announce Type: cross Abstract: 3D Gaussian Splatting is a powerful visual representation, providing high-quality and efficient 3D scene reconstruction, but it is crucially dependent

15 Apr 2026

A Mamba-Based Multimodal Network for Multiscale Blast-Induced Rapid Structural Damage Assessment

arXiv:2604.11709v1 Announce Type: new Abstract: Accurate and rapid structural damage assessment (SDA) is crucial for post-disaster management, helping responders prioritise resources, plan rescues, an

safetyarxiv-cs-ai

Bidirectional Cross-Attention Fusion of High-Res RGB and Low-Res HSI for Multimodal Automated Waste Sorting

arXiv:2603.13941v2 Announce Type: replace Abstract: Growing waste streams and the transition to a circular economy require efficient automated waste sorting. In industrial settings, materials move on

Hardware

Can LLMs Reason About Attention? Towards Zero-Shot Analysis of Multimodal Classroom Behavior

arXiv:2604.03401v2 Announce Type: replace-cross Abstract: Understanding student engagement usually requires time-consuming manual observation or invasive recording that raises privacy concerns. We pre

hardwarearxiv-cs-ai

VisText-Mosquito: A Unified Multimodal Dataset for Visual Detection, Segmentation, and Textual Explanation on Mosquito Breeding Sites

arXiv:2506.14629v3 Announce Type: replace-cross Abstract: Mosquito-borne diseases pose a major global health risk, requiring early detection and proactive control of breeding sites to prevent outbreak

researcharxiv-cs-cl

Multimodal Anomaly Detection for Human-Robot Interaction