Search: “multimodal”

GuideDog: A Real-World Egocentric Multimodal Dataset for Blind and Low-Vision Accessibility-Aware Guidance

arXiv:2503.12844v2 Announce Type: replace Abstract: For people affected by blindness and low vision (BLV), safe and independent navigation remains a major challenge, impacting over 2.2 billion individ

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Iterative Multimodal Retrieval-Augmented Generation for Medical Question Answering

DGX agent

arXiv:2604.27724v1 Announce Type: new Abstract: Medical retrieval-augmented generation (RAG) systems typically operate on text chunks extracted from biomedical literature, discarding the rich visual c

model-releasesarxiv-cs-ai

1 May 2026

Learning Generalizable Multimodal Representations for Software Vulnerability Detection

arXiv:2604.25711v2 Announce Type: replace-cross Abstract: Source code and its accompanying comments are complementary yet naturally aligned modalities-code encodes structural logic while comments capt

Linguistically Informed Multimodal Fusion for Vietnamese Scene-Text Image Captioning: Dataset, Graph Framework, and Phonological Attention

arXiv:2604.27712v1 Announce Type: cross Abstract: Scene-text image captioning requires fusing three information streams -- visual features, OCR-detected text, and linguistic knowledge -- to generate d

safetyarxiv-cs-cl

VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking

arXiv:2601.08611v2 Announce Type: replace-cross Abstract: The growing scale of online misinformation urgently demands Automated Fact-Checking (AFC). Existing benchmarks for evaluating AFC systems, how

A Multimodal Pre-trained Network for Integrated EEG-Video Seizure Detection

arXiv:2604.26379v1 Announce Type: new Abstract: Reliable seizure detection in mouse models is essential for preclinical epilepsy research, yet manual review of synchronized video-EEG recordings is lab

safetyarxiv-cs-cv

30 Apr 2026

FASH-iCNN: Making Editorial Fashion Identity Inspectable Through Multimodal CNN Probing

arXiv:2604.26186v1 Announce Type: new Abstract: Fashion AI systems routinely encode the aesthetic logic of specific houses, editors, and historical moments without disclosing it. We present FASH-iCNN,

30 Apr 2026

Curriculum-guided multimodal representation learning enables generalizable prediction of nanomaterial-protein interactions

arXiv:2507.14245v2 Announce Type: replace Abstract: Nanomaterial-protein interactions (NPI) are pivotal to realizing the therapeutic and diagnostic potential of nanomaterials. Although AI promises to

researcharxiv-cs-lg

29 Apr 2026

Multimodal Contextualized Support for Enhancing Video Retrieval System

arXiv:2412.07584v2 Announce Type: replace Abstract: Current video retrieval systems, especially those used in competitions, primarily focus on querying individual keyframes or images rather than encod

29 Apr 2026

The Thinking Pixel: Recursive Sparse Reasoning in Multimodal Diffusion Latents

arXiv:2604.25299v1 Announce Type: new Abstract: Diffusion models have achieved success in high-fidelity data synthesis, yet their capacity for more complex, structured reasoning like text following ta

29 Apr 2026

DeepTaxon: An Interpretable Retrieval-Augmented Multimodal Framework for Unified Species Identification and Discovery

arXiv:2604.24029v1 Announce Type: cross Abstract: Identifying species in biology among tens of thousands of visually similar taxa while discovering unknown species in open-world environments remains a

28 Apr 2026

Hierarchical Prototype-based Domain Priors for Multiple Instance Learning in Multimodal Histopathology Analysis

arXiv:2604.23982v1 Announce Type: new Abstract: Digital pathology has fundamentally altered diagnostic workflows by enabling the computational analysis of gigapixel Whole Slide Images (WSIs), yet effe

safetyarxiv-cs-cv

28 Apr 2026

One Identity, Many Roles: Multimodal Entity Coreference for Enhanced Video Situation Recognition

arXiv:2604.23173v1 Announce Type: new Abstract: Video Situation Recognition (VidSitu) addresses the challenging problem of 'who did what to whom, with what, how, and where' in a video. It tests thorou

28 Apr 2026

AromaGen: Interactive Generation of Rich Olfactory Experiences with Multimodal Language Models

arXiv:2604.01650v2 Announce Type: replace-cross Abstract: Smell's deep connection with food, memory, and social experience has long motivated researchers to bring olfaction into interactive systems. Y

MTT-Bench: Predicting Social Dominance in Mice via Multimodal Large Language Models

arXiv:2604.22492v1 Announce Type: cross Abstract: Understanding social dominance in animal behavior is critical for neuroscience and behavioral studies. In this work, we explore the capability of Mult

Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Models

arXiv:2510.07632v2 Announce Type: replace Abstract: Frontier AI models have achieved remarkable progress, yet recent studies suggest they struggle with compositional reasoning, often performing at or

UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents

arXiv:2602.07038v2 Announce Type: replace-cross Abstract: Key Information Extraction (KIE) from real-world documents remains challenging due to substantial variations in layout structures, visual qual

Focus on What Matters: Fisher-Guided Adaptive Multimodal Fusion for Vulnerability Detection

arXiv:2601.02438v3 Announce Type: replace-cross Abstract: Software vulnerability detection can be formulated as a binary classification problem that determines whether a given code snippet contains se

researcharxiv-cs-ai

24 Apr 2026

Learning Physics from Pretrained Video Models: A Multimodal Continuous and Sequential World Interaction Models for Robotic Manipulation

arXiv:2603.00110v2 Announce Type: replace Abstract: The scarcity of large-scale robotic data has motivated the repurposing of foundation models from other modalities for policy learning. In this work,

safetyarxiv-cs-ro

24 Apr 2026

OmniGen2: Towards Instruction-Aligned Multimodal Generation

arXiv:2506.18871v4 Announce Type: replace-cross Abstract: In this work, we introduce OmniGen2, a versatile and open-source generative model designed to provide a unified solution for diverse generatio

22 Apr 2026

FLiP: Towards understanding and interpreting multimodal multilingual sentence embeddings

arXiv:2604.18109v1 Announce Type: new Abstract: This paper presents factorized linear projection (FLiP) models for understanding pretrained sentence embedding spaces. We train FLiP models to recover t

Multimodal In-context Learning for ASR of Low-resource Languages

arXiv:2601.05707v2 Announce Type: replace Abstract: Automatic speech recognition (ASR) still covers only a small fraction of the world's languages, mainly due to supervised data scarcity. In-context l

SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models

model-releasesarxiv-cs-lg

arXiv:2604.16883v1 Announce Type: new Abstract: In long-context decoding for LLMs and LMMs, attention becomes increasingly memory-bound because each decoding step must load a large amount of KV-cache

STEP-PD: Stage-Aware and Explainable Parkinson's Disease Severity Classification Using Multimodal Clinical Assessments

arXiv:2604.17611v1 Announce Type: new Abstract: Parkinson's disease (PD) is a progressive disorder in which symptom burden and functional impairment evolve over time, making severity staging essential

researcharxiv-cs-lg

CAVERS: Multimodal SLAM Data from a Natural Karstic Cave with Ground Truth Motion Capture

model-releasesarxiv-cs-ro

arXiv:2604.15052v1 Announce Type: new Abstract: Autonomous robots operating in natural karstic caves face perception and navigation challenges that are qualitatively distinct from those encountered in

17 Apr 2026

A Multimodal Clinically Informed Coarse-to-Fine Framework for Longitudinal CT Registration in Proton Therapy

arXiv:2604.13397v1 Announce Type: new Abstract: Proton therapy offers superior organ-at-risk sparing but is highly sensitive to anatomical changes, making accurate deformable image registration (DIR)

16 Apr 2026

AIM-CoT: Active Information-driven Multimodal Chain-of-Thought for Vision-Language Reasoning

arXiv:2509.25699v2 Announce Type: replace Abstract: Interleaved-Modal Chain-of-Thought (I-MCoT) advances vision-language reasoning, such as Visual Question Answering (VQA). This paradigm integrates sp

16 Apr 2026

Agents

M2HRI: An LLM-Driven Multimodal Multi-Agent Framework for Personalized Human-Robot Interaction

arXiv:2604.11975v1 Announce Type: new Abstract: Multi-robot systems hold significant promise for social environments such as homes and hospitals, yet existing multi-robot works treat robots as functio

agentsarxiv-cs-ro

15 Apr 2026

ReefMapGS: Enabling Large-Scale Underwater Reconstruction by Closing the Loop Between Multimodal SLAM and Gaussian Splatting

arXiv:2604.11992v1 Announce Type: cross Abstract: 3D Gaussian Splatting is a powerful visual representation, providing high-quality and efficient 3D scene reconstruction, but it is crucially dependent

15 Apr 2026

A Mamba-Based Multimodal Network for Multiscale Blast-Induced Rapid Structural Damage Assessment

arXiv:2604.11709v1 Announce Type: new Abstract: Accurate and rapid structural damage assessment (SDA) is crucial for post-disaster management, helping responders prioritise resources, plan rescues, an

safetyarxiv-cs-ai

Bidirectional Cross-Attention Fusion of High-Res RGB and Low-Res HSI for Multimodal Automated Waste Sorting

arXiv:2603.13941v2 Announce Type: replace Abstract: Growing waste streams and the transition to a circular economy require efficient automated waste sorting. In industrial settings, materials move on

Hardware

Can LLMs Reason About Attention? Towards Zero-Shot Analysis of Multimodal Classroom Behavior

arXiv:2604.03401v2 Announce Type: replace-cross Abstract: Understanding student engagement usually requires time-consuming manual observation or invasive recording that raises privacy concerns. We pre

hardwarearxiv-cs-ai

VisText-Mosquito: A Unified Multimodal Dataset for Visual Detection, Segmentation, and Textual Explanation on Mosquito Breeding Sites

arXiv:2506.14629v3 Announce Type: replace-cross Abstract: Mosquito-borne diseases pose a major global health risk, requiring early detection and proactive control of breeding sites to prevent outbreak

researcharxiv-cs-cl

Multimodal Anomaly Detection for Human-Robot Interaction

arXiv:2604.09326v1 Announce Type: cross Abstract: Ensuring safety and reliability in human-robot interaction (HRI) requires the timely detection of unexpected events that could lead to system failures

safetyarxiv-cs-cv

13 Apr 2026

On the Spectral Geometry of Cross-Modal Representations: A Functional Map Diagnostic for Multimodal Alignment

arXiv:2604.08579v1 Announce Type: cross Abstract: We study cross-modal alignment between independently pretrained vision (DINOv2) and language (all-MiniLM-L6-v2) encoders using the functional map fram

safetyarxiv-cs-ai

13 Apr 2026

ChangeBridge: Spatiotemporal Image Generation with Multimodal Controls for Remote Sensing

arXiv:2507.04678v3 Announce Type: replace Abstract: Spatiotemporal image generation is a highly meaningful task, which can generate future scenes conditioned on given observations. However, existing c

Local Ai

Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models

arXiv:2604.06912v1 Announce Type: cross Abstract: MLLMs require high-resolution visual inputs for fine-grained tasks like document understanding and dense scene perception. However, current global res

local-aiarxiv-cs-ai

SyncBreaker:Stage-Aware Multimodal Adversarial Attacks on Audio-Driven Talking Head Generation

arXiv:2604.08405v1 Announce Type: new Abstract: Diffusion-based audio-driven talking-head generation enables realistic portrait animation, but also introduces risks of misuse, such as fraud and misinf

What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoning

arXiv:2604.06995v1 Announce Type: new Abstract: Existing Graphical User Interface (GUI) reasoning tasks remain challenging, particularly in UI understanding. Current methods typically rely on direct s

Zatom-1: A Multimodal Flow Foundation Model for 3D Molecules and Materials

arXiv:2602.22251v3 Announce Type: replace-cross Abstract: General-purpose 3D chemical modeling encompasses molecules and materials, requiring both generative and predictive capabilities. However, most

researcharxiv-cs-ai

Beyond Generation: Using Multimodal LLMs as Detectors of LLM-Generated Deepfake Facial Images

arXiv:2610.21760v1 Announce Type: new Abstract: As image generation models become increasingly capable of producing highly realistic content, traditional forensic techniques are struggling to keep up.

Agents

CoWorld Reasoner: Agentic Multimodal Reasoning with the Evolving Video World Model-Driven Harness

arXiv:2610.21910v1 Announce Type: new Abstract: As Vision-Language Models (VLMs) advance, they are increasingly expected to perform spatiotemporal reasoning over motion, actions, and view changes beyo

agentsarxiv-cs-ai

FuseAdapt: Parameter-Efficient Multimodal Fusion for Semantic Segmentation with Missing Modalities

arXiv:2610.21779v1 Announce Type: new Abstract: Multi-modal semantic segmentation benefits from complementary sensors, but this benefit is fragile when some modalities are degraded or missing at test

Multimodal AI Detection for Math-Heavy Scientific Text

arXiv:2610.21062v1 Announce Type: new Abstract: As large language models become increasingly capable of generating, and assisting with, scientific writing, there is a need for greater transparency and

Agents

FACT: A Multimodal Force-Action-Track Foundation Model for Human-to-Robot Dexterity

arXiv:2610.19155v1 Announce Type: new Abstract: Human demonstrations offer a scalable alternative to costly robot data collection for dexterous manipulation, but most human datasets lack force measure

agentsarxiv-cs-ro

Instruction Mining From Images: Constructing a Synthetic Dataset for Multimodal Learning ILSP DCVLR Competition Submission

arXiv:2610.19932v1 Announce Type: new Abstract: This work presents a compact and reproducible pipeline for synthesizing and curating image-based instruction-response pairs for DCVLR. We draw from the

Multimodal Medical Image Parsing with Task-Specific Low-Rank Adapters

arXiv:2610.19901v1 Announce Type: new Abstract: Adapting a single vision-language model to medical image parsing requires training choices that reflect the demands of each task. We study task-specific

RetailAgent: Structured Adverse Timing in Self-Conditioned Multimodal LLM Trading Agents

arXiv:2608.28399v2 Announce Type: replace Abstract: In financial markets, a sequential policy that reacts systematically to price movements may become predictable to other market participants. This pa

safetyarxiv-cs-ai