Search: “multimodal”

MISID: A Multimodal Multi-turn Dataset for Complex Intent Recognition in Strategic Deception Games

arXiv:2604.12700v1 Announce Type: new Abstract: Understanding human intent in complex multi-turn interactions remains a fundamental challenge in human-computer interaction and behavioral analysis. Whi

15 Apr 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generation

DGX agent

arXiv:2604.11804v1 Announce Type: new Abstract: In this work, we study Human-Object Interaction Video Generation (HOIVG), which aims to synthesize high-quality human-object interaction videos conditio

model-releasesarxiv-cs-cv

14 Apr 2026

OmniUMI: Towards Physically Grounded Robot Learning via Human-Aligned Multimodal Interaction

arXiv:2604.10647v1 Announce Type: new Abstract: UMI-style interfaces enable scalable robot learning, but existing systems remain largely visuomotor, relying primarily on RGB observations and trajector

safetyarxiv-cs-ro

14 Apr 2026

All You Need for Object Detection: From Pixels, Points, and Prompts to Next-Gen Fusion and Multimodal LLMs/VLMs in Autonomous Vehicles

arXiv:2510.26641v4 Announce Type: replace Abstract: Autonomous Vehicles (AVs) are transforming the future of transportation through advances in intelligent perception, decision-making, and control sys

agentsarxiv-cs-cv

13 Apr 2026

Tutorials

Degradation-Robust Fusion: An Efficient Degradation-Aware Diffusion Framework for Multimodal Image Fusion in Arbitrary Degradation Scenarios

arXiv:2604.08922v1 Announce Type: new Abstract: Complex degradations like noise, blur, and low resolution are typical challenges in real world image fusion tasks, limiting the performance and practica

tutorialsarxiv-cs-cv

13 Apr 2026

Detection of Hate and Threat in Digital Forensics: A Case-Driven Multimodal Approach

arXiv:2604.08609v1 Announce Type: cross Abstract: Digital forensic investigations increasingly rely on heterogeneous evidence such as images, scanned documents, and contextual reports. These artifacts

13 Apr 2026

Tutorials

Analyzing Multimodal Interaction Strategies for LLM-Assisted Manipulation of 3D Scenes

arXiv:2410.22177v2 Announce Type: replace-cross Abstract: As more applications of large language models (LLMs) for 3D content for immersive environments emerge, it is crucial to study user behaviour t

tutorialsarxiv-cs-ai

10 Apr 2026

Interpretable Tau-PET Synthesis from Multimodal T1-Weighted and FLAIR MRI Using Partial Information Decomposition Guided Disentangled Quantized Half-UNet

arXiv:2602.22545v2 Announce Type: replace Abstract: Tau positron emission tomography (tau-PET) is an important in vivo biomarker of Alzheimer's disease, but its cost, limited availability, and acquisi

10 Apr 2026

A Flow-Theoretic Multimodal Control Framework Fusing Mental and Physical Engagement for Upper-Limb Rehabilitation

arXiv:2610.21590v1 Announce Type: new Abstract: While Assist-as-Needed (AAN) control aims to maximize user participation in robot-assisted training, existing methods relying solely on physical metrics

researcharxiv-cs-ro

Local Ai

CogDrift-4D: Hierarchical Multimodal Modeling of Temporal Behavioral Decay for MCI Detection

arXiv:2610.20877v1 Announce Type: cross Abstract: Early detection of mild cognitive impairment remains a critical bottleneck in dementia care, demanding scalable tools capable of capturing subtle beha

local-aiarxiv-cs-cl

Decoding Naturalistic Speech from fMRI with Temporal Modeling and Multimodal Alignment

arXiv:2610.22049v1 Announce Type: cross Abstract: Reconstructing what a person perceives from their brain activity is a stringent test of whether neural recordings carry meaningful information. Natura

safetyarxiv-cs-ai

Intern-Decision: Efficient Multimodal Decision Making with Autoregressive Masked Language Modeling

arXiv:2610.22131v1 Announce Type: new Abstract: Interactive agents repeatedly choose among a finite set of actions, labels, or ordinal outcomes. Generating a free-form response for each choice adds la

agentsarxiv-cs-cl

ChemArgus: Automated Scientific Diagnostics for Lab-Grade Multimodal Chemistry Reasoning: A Frontier Benchmark

arXiv:2610.20012v1 Announce Type: new Abstract: Chemistry evaluation has moved past multiple-choice probes toward open-ended expert problems, yet the benchmarks that define the current frontier still

15 Oct 2026

On the Limits of Representation-Based Test-Time Adaptation for Multimodal LLMs

arXiv:2610.19892v1 Announce Type: cross Abstract: Vision-language models suffer significant accuracy drops under common corruptions. While test-time adaptation (TTA) successfully mitigates these shift

safetyarxiv-cs-cv

15 Oct 2026

A 35B multimodal model running on a phone with ~3GB memory? MiniCPM-V 4.7 demo is pretty wild

model-releasesr-localllama

Just saw this demo from OpenBMB for MiniCPM-V 4.7, and the on-device capability is pretty impressive. A 35B-A3B MoE vision-language model running locally on mobile devices is already interesting, but

Interpretable Multimodal EEG-ECG Framework for Neonatal Seizure Prediction and Patient-Adaptive Transfer Learning

arXiv:2511.14110v2 Announce Type: replace-cross Abstract: Neonatal seizures carry substantial risks of acute brain injury and long-term neurodevelopmental impairment, motivating the need for timely id

Let the Mind Conceive, Let the Hand Achieve: Following Multimodal Instructions for Long-Horizon Robotic Manipulation

model-releasesarxiv-cs-ro

arXiv:2610.18352v1 Announce Type: new Abstract: A robot following a multi-step instruction needs to act on the right objects in the right order, even as the scene changes. It must also maintain awaren

Merit: Making every modality earn its contribution in multimodal attention decoding

arXiv:2610.17507v1 Announce Type: new Abstract: More than 1.5 billion people have hearing loss. When several people talk at once, hearing aids amplify every voice alike. Auditory attention decoding (A

Neuro-Symbolic Graph Transformers for Hierarchical Intention Recognition from Multimodal Human Activity Sequences

arXiv:2610.18176v1 Announce Type: cross Abstract: Proactive human-robot collaboration requires autonomous systems to infer not only immediate actions but also the underlying hierarchical intentions go

agentsarxiv-cs-lg

Toward a Multimodal Foundation Model for Plasma State Reconstruction

arXiv:2610.15076v1 Announce Type: new Abstract: Plasma experiments observe coupled physical states through sparse, heterogeneous, and incomplete diagnostics. We present a proof of concept toward a mul

SeoulMMOD: A Large-Scale Multimodal Origin-Destination Flow Benchmark

arXiv:2610.13730v1 Announce Type: new Abstract: Origin-destination (OD) flow forecasting supports urban mobility decisions by predicting the volume and patterns of human movement across a city. Most e

12 Oct 2026

Unsupervised Multimodal Reward Modeling

arXiv:2610.12707v1 Announce Type: cross Abstract: Vision-language reward models are trained on annotated preference pairs, which are expensive to collect, can disagree, and are ranked by benchmark acc

12 Oct 2026

ActiveMedAgent: Cost-Aware Trajectory Learning for Multimodal Medical Diagnosis

arXiv:2610.11140v1 Announce Type: cross Abstract: Clinical diagnosis is inherently sequential: clinicians escalate from cheap to costly tests only when additional evidence is expected to resolve diagn

safetyarxiv-cs-ai

9 Oct 2026

Humanity's Sixth Sense: Benchmarking Intuitive Visual Reasoning in Multimodal Models

arXiv:2610.08966v2 Announce Type: replace Abstract: Humans perceive far more in a scene than what is explicitly depicted: a single glance captures past causes and future trajectories; a quick peek det

9 Oct 2026

Improving Image-Based Nutrition Estimation Through Multimodal Food-Item Verification and Recovery

arXiv:2610.11144v1 Announce Type: cross Abstract: Single-image nutrition estimation can fail silently when visible foods are missed. Even when a food is correctly identified, its proposed region may n

9 Oct 2026

DiTS: Multimodal Diffusion Transformers Are Time Series Forecasters

arXiv:2602.06597v2 Announce Type: replace Abstract: While generative modeling facilitates probabilistic time series forecasting, incorporating heterogeneous exogenous information remains challenging.

safetyarxiv-cs-lg

Applications

GRC-Net: Global Representation Consistency Network for Unsupervised Multimodal Anomaly Detection

arXiv:2610.09329v1 Announce Type: new Abstract: Automated quality inspection is essential for ensuring product reliability in manufacturing.While image-based methods effectively capture appearance-rel

applicationsarxiv-cs-cv

HarnessIR: Harnessing Multimodal Foundation Models for Universal Real-World Image Restoration

arXiv:2610.10133v1 Announce Type: new Abstract: Real-world low-quality images suffer from complex mixed degradations, including but not limited to noise, blur, atmospheric effects, etc. Recent agentic

MultiFly: A Real-World Multimodal Aerial Dataset with Annotation-Efficient Label Transfer and Cross-Modal Semantic Consistency

arXiv:2610.10359v1 Announce Type: cross Abstract: We introduce MultiFly, a real-world, low-altitude UAV dataset for semantic perception across RGB, thermal, LiDAR, and radar modalities. MultiFly provi

Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models

arXiv:2501.07972v2 Announce Type: replace-cross Abstract: The target of video moment retrieval (VMR) is predicting temporal spans within a video that semantically match a given linguistic query. Exist

A Swarm-Coordinated Multi-Robot System for Early Stress Detection in Agricultural Rows Using Multimodal Leaf Sensing

arXiv:2610.08603v1 Announce Type: cross Abstract: Early stress detection in crops is a necessity today to improve efficiency and reduce waste of time, money, and effort. However, most modern technique

agentsarxiv-cs-ai

Local Ai

Attention from Above: A Multimodal Model for Drone-Based Object Localization

arXiv:2607.17669v2 Announce Type: replace Abstract: Drone-based object detection technology has advanced rapidly, becoming increasingly sophisticated and efficient. Recently, research trends have expa

local-aiarxiv-cs-cv

DSV-Mem: Evaluating Multimodal Memory in Professional Workflows for MLLM Agents

arXiv:2610.08102v1 Announce Type: new Abstract: Conversational MLLM agents are increasingly expected to assist in professional workflows, from AI research and engineering design to product management

ESP: Energy-Score Policy for One-Step Multimodal Action Generation

arXiv:2610.07696v1 Announce Type: new Abstract: Generative action models based on diffusion and flow matching have been increasingly adopted in vision-language-action (VLA) policies for their ability

safetyarxiv-cs-ro

Real-Time Generation of Game Video Commentary with Multimodal LLMs: Pause-Aware Decoding Approaches

arXiv:2603.02655v2 Announce Type: replace-cross Abstract: Real-time video commentary generation provides textual descriptions of ongoing events in videos. It supports accessibility and engagement in d

Speak to a Protein: An Interactive Multimodal Co-Scientist

arXiv:2510.17826v2 Announce Type: replace-cross Abstract: Building a working mental model of a protein typically requires weeks of reading, cross-referencing crystal and predicted structures, and insp

Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection

arXiv:2610.06394v1 Announce Type: new Abstract: Human-object interaction (HOI) detection aims to localize human-object pairs and recognize their interactions. Traditional supervised methods perform st

6 Oct 2026

PerCoV2: Ultra-Low Bit-Rate Perceptual Image Compression via Query-Based 1D Multimodal Image Tokens

arXiv:2503.09368v2 Announce Type: replace Abstract: Despite recent progress in learned image compression, current image codecs still struggle to maintain realistic reconstructions at low bit-rates, of

6 Oct 2026

How Robust Is Multimodal Claim Verification to LLM Rewriting?

arXiv:2610.02841v1 Announce Type: new Abstract: LLMs are known to introduce stylistic changes into generated text, yet how these stylistic shifts affect model decisions on scientific tasks remains und

researcharxiv-cs-cl

5 Oct 2026

MiDShip: Multimodal Dataset of Ship Cargo Hold Structures for Engineering Design

arXiv:2610.02214v1 Announce Type: cross Abstract: Ship structures govern vessel strength, safety, and manufacturability, but their design must satisfy hundreds of classification society requirements,

5 Oct 2026

Multimodal reasoning for broadly neutralizing antibody discovery from label-free human B cell repertoires across virus families

arXiv:2610.03160v1 Announce Type: cross Abstract: Discovering broadly neutralizing antibodies (bnAbs) from human natural immune repertoires remains a fundamental challenge in immunology, hindered by:

5 Oct 2026

Beyond Unimodal Bases: Pullback Geometry for Multimodal Data

arXiv:2610.00708v1 Announce Type: new Abstract: Data-driven Riemannian geometry provides nonlinear interpolation and geometric representations of high-dimensional data. For these operations to be stat

researcharxiv-cs-lg

2 Oct 2026

Paying for Too Many Tokens? Valid and Cost-Efficient Multimodal LLM Annotation with Simple Heuristics

arXiv:2610.00809v1 Announce Type: new Abstract: Vision-Language Models (VLMs) enable video annotation at scale, but costs accumulate quickly: processing a typical 60-second short-form video at one fra

2 Oct 2026

Applications

TouchTherm: Building Multimodal Digital Twins of Objects for Tactile and Thermal Rendering

arXiv:2610.01943v1 Announce Type: new Abstract: Robotic simulation and virtual reality increasingly require object assets that capture not only visual geometry but also the physical cues underlying ta

applicationsarxiv-cs-ro

2 Oct 2026

SimTrace: Grounded Multimodal User Trajectories Generation for Online User Modeling

arXiv:2609.38397v1 Announce Type: new Abstract: Virtual clients offer a cost-effective approach to support applications such as A/B testing, recommender system development, and interface evaluation. H

agentsarxiv-cs-ai

1 Oct 2026

Calibration-First Cross-Cohort Multimodal Temporal Learning for Transferable Asthma-Risk Forecasting

arXiv:2609.35795v1 Announce Type: cross Abstract: Asthma deterioration forecasting must remain reli- able when patient populations, sensor ecosystems, and available modalities change across cohorts. E

30 Sept 2026

Evaluating the Evaluators: Diagnosing Large Multimodal Models for AI-Generated Image Assessment

arXiv:2609.37576v1 Announce Type: new Abstract: With the rapid advancement of text-to-image (T2I) generation, robust evaluation becomes critical yet challenging, as traditional metrics fail to capture

30 Sept 2026

InsightMap: Structured Spatial Modeling for Embodied Multimodal Reasoning

arXiv:2609.37187v1 Announce Type: new Abstract: Language-guided navigation requires connecting partial observations to a persistent spatial reference and learning how actions change that representatio