Search: “multimodal”

Your Multimodal Speech Model Says I Have a Face for Radio

model-releasesarxiv-cs-cl

arXiv:2605.30472v1 Announce Type: new Abstract: As large neural models have become better at language tasks, researchers are increasingly building multi- and omnimodal models that handle more modaliti

1 Jun 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Archon: A Unified Multimodal Model for Holistic Digital Human Generation

DGX agent

arXiv:2605.30311v1 Announce Type: cross Abstract: Digital humans are fundamental to immersive interaction, yet creating a unified model for holistic modalities, including text, audio, motion, and visu

researcharxiv-cs-ai

29 May 2026

Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance

arXiv:2411.14279v2 Announce Type: replace-cross Abstract: Large vision-language models (LVLMs) have achieved impressive results in various vision-language tasks. However, despite showing promising per

safetyarxiv-cs-cl

OmniCD: A Foundational Framework for Remote Sensing Image Change Detection Guided by Multimodal Semantics

arXiv:2605.30168v1 Announce Type: new Abstract: Change detection (CD) in remote sensing is vital for applications such as urban monitoring and disaster assessment, yet traditional methods struggle wit

Local Ai

UniNote: A Unified Embedding Model for Multimodal Representation and Ranking

arXiv:2605.29287v1 Announce Type: cross Abstract: Item-to-Item (I2I) retrieval is a fundamental part of modern content platforms, supporting critical industrial workflows from recommendation engines t

local-aiarxiv-cs-cv

VFEAgent: A Multimodal Agent Framework for End-to-End Automated Finite Element Analysis

arXiv:2605.28978v1 Announce Type: new Abstract: Finite Element Analysis (FEA) serves as the cornerstone of modern engineering design. However, its workflow is inherently complex and relies heavily on

agentsarxiv-cs-ai

From Affect to Complex Behavior: Advancing Multimodal Human-Centered AI at the 10th ABAW Workshop & Competition

arXiv:2605.27451v1 Announce Type: new Abstract: The 10th Affective & Behavior Analysis in-the-Wild (ABAW) Workshop and Competition, held at CVPR 2026, continues to advance research on modelling, analy

28 May 2026

MeniOmni: A Structured Multimodal Benchmark for Holistic Meniscus Injury Assessment

arXiv:2605.28161v1 Announce Type: new Abstract: Clinical diagnosis of meniscus injuries requires radiologists to integrate volumetric MRI evidence with patient context (e.g., sex, age, BMI) and to pro

28 May 2026

Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models

arXiv:2605.27243v1 Announce Type: new Abstract: Large vision-language models increasingly rely on long-context modeling to reason over documents, hour-level videos, and long-horizon agent trajectories

agentsarxiv-cs-cv

27 May 2026

Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos

arXiv:2605.26232v1 Announce Type: new Abstract: Pre-trained video large language models excel at visual reasoning. However, they struggle when videos arrive with auxiliary streams, such as audio, dept

27 May 2026

PolyFusionAgent: A Multimodal Foundation Model and Autonomous AI Assistant for Polymer Property Prediction and Inverse Design

arXiv:2605.26543v1 Announce Type: new Abstract: Polymer discovery is central to fields ranging from energy storage to biomedicine, but it is hindered by an astronomically large chemical design space a

agentsarxiv-cs-ai

27 May 2026

Dynamic Optimization and Safety Indicator Injection for Jailbreaking Text-to-Image Models with Multimodal Safety Filters

arXiv:2505.18979v2 Announce Type: replace Abstract: Text-to-image (T2I) models can generate not-safe-for-work (NSFW) content, motivating multi-stage safety pipelines with both text and image filters.

safetyarxiv-cs-lg

26 May 2026

Multimodal Alignment and Preference Optimization for Zero-Shot Conditional RNA Generation

arXiv:2605.23961v1 Announce Type: cross Abstract: The design of RNA molecules that interact with specific proteins is a critical challenge in experimental and computational biology. Despite recent pro

26 May 2026

RAG4Outcome: A Retrieval-Augmented Multimodal Framework for Prognostic Prediction in Chronic Osteomyelitis

arXiv:2605.22833v1 Announce Type: cross Abstract: Chronic osteomyelitis presents substantial prognostic challenges due to its high recurrence risk and complex postoperative recovery trajectories. Trad

25 May 2026

Case-Aware Medical Image Classification with Multimodal Knowledge Graphs and Reliability-Guided Refinement

arXiv:2605.22547v1 Announce Type: new Abstract: Deep learning has brought significant progress to medical image classification, yet most existing methods still rely on isolated visual evidence and can

22 May 2026

Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction

arXiv:2605.21869v1 Announce Type: new Abstract: We present our submission to the Hume-ABAW10 Emotional Mimicry Intensity (EMI) Challenge, which aims to predict six continuous emotion intensity dimensi

22 May 2026

ArPoMeme: An Annotated Arabic Multimodal Dataset for Political Ideology and Polarization

arXiv:2605.20967v1 Announce Type: new Abstract: Memes have become a prominent medium of political communication in the Arab world, reflecting how humor, imagery, and text interact to express ideologic

researcharxiv-cs-cl

21 May 2026

QwenSafe: Multimodal Content Rating Description Identification via Preference-Aligned VLMs

arXiv:2605.20584v1 Announce Type: new Abstract: Mobile app marketplaces require developers to disclose standardized content rating descriptors (CRDs) to inform users about potentially sensitive or res

21 May 2026

Identifiable Multimodal Causal Representation Learning under Partial Latent Sharing

arXiv:2605.19135v1 Announce Type: new Abstract: Causal representation learning (CRL) seeks to uncover meaningful latent variables and their corresponding causal structure from high-dimensional observa

researcharxiv-cs-lg

20 May 2026

Towards Trust Calibration in Socially Interactive Agents: Investigating Gendered Multimodal Behaviors Generation with LLMs

model-releasesarxiv-cs-cl

arXiv:2605.19798v1 Announce Type: new Abstract: As Socially Interactive Agents (SIAs) become increasingly integrated into daily life, the ability to calibrate user trust to an agent's actual capabilit

20 May 2026

CatalyticMLLM: A Graph-Text Multimodal Large Language Model for Catalytic Materials

arXiv:2605.17254v1 Announce Type: new Abstract: Property prediction and inverse structural design of catalytic materials are typically modeled as two independent tasks: the former predicts target prop

Cross-modal Affinity-aligned Multimodal Learning Analytics for Predicting Student Collaboration Satisfaction in Game-Based Learning

arXiv:2605.16806v1 Announce Type: cross Abstract: Collaborative game-based learning environments offer rich opportunities for small-group knowledge construction, yet automatically predicting student c

Local Ai

How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A

arXiv:2605.16359v1 Announce Type: cross Abstract: Vision-language models improve perception by feeding increasingly long visual token sequences into language backbones, but the resulting inference cos

local-aiarxiv-cs-ai

MUSE: Multimodal Uncertainty Quantification of State Estimation

arXiv:2605.17421v1 Announce Type: new Abstract: Accurate visual state estimation has been a central topic in robotics with a wide range of applications in robot navigation, autonomous driving, and aut

agentsarxiv-cs-ro

SkyNative: A Native Multimodal Framework for Remote Sensing Visual Evidence Reasoning

arXiv:2605.17949v1 Announce Type: new Abstract: Remote sensing vision-language models commonly rely on pretrained visual encoders to convert images into semantic features before language-model reasoni

Missing-Data-Induced Phase Transitions in Spectral PLS for Multimodal Learning

arXiv:2601.21294v2 Announce Type: replace Abstract: Partial Least Squares (PLS) learns shared structure from paired data via the top singular vectors of the empirical cross-covariance (PLS-SVD), but m

safetyarxiv-cs-lg

18 May 2026

Multimodal Object Detection Under Sparse Forest-Canopy Occlusion

arXiv:2605.15326v1 Announce Type: new Abstract: Reliable detection of humans beneath forest canopy remains a difficult remote-sensing challenge due to sparse, structured, and viewpoint-dependent occlu

18 May 2026

Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models

arXiv:2605.16165v1 Announce Type: cross Abstract: Autoregressive next-token training offers a unified formulation for image generation and text understanding, but it also creates strong modality compe

18 May 2026

Learning Multimodal Embeddings for Traffic Accident Prediction and Causal Estimation

arXiv:2512.02920v3 Announce Type: replace-cross Abstract: We consider analyzing traffic accident patterns using both road network data and satellite images aligned to road graph nodes. Previous work f

15 May 2026

Text Knows What, Tables Know When: Clinical Timeline Reconstruction via Retrieval-Augmented Multimodal Alignment

arXiv:2605.15168v1 Announce Type: cross Abstract: Reconstructing precise clinical timelines is essential for modeling patient trajectories and forecasting risk in complex, heterogeneous conditions lik

15 May 2026

Tutorials

Multimodal Graph-based Classification of Esophageal Motility Disorders

arXiv:2605.13623v1 Announce Type: new Abstract: Diagnosing esophageal motility disorders pose significant challenges due to the complexity of high-resolution impedance manometry (HRIM) data and variab

tutorialsarxiv-cs-lg

14 May 2026

Multimodal Hidden Markov Models for Persistent Emotional State Tracking

arXiv:2605.12838v1 Announce Type: new Abstract: Tracking an interpretable emotional arc of a conversation via the sentiment of individual utterances processed as a whole is central to both understandi

researcharxiv-cs-ai

14 May 2026

Leveraging Multimodal Large Language Models for All-in-One Image Restoration via a Mixture of Frequency Experts

arXiv:2605.11444v1 Announce Type: new Abstract: All-in-one image restoration seeks to recover clean images from inputs affected by diverse and unknown degradations using a unified framework. Recent me

13 May 2026

Looking and Listening Inside and Outside: Multimodal Artificial Intelligence Systems for Driver Safety Assessment and Intelligent Vehicle Decision-Making

arXiv:2602.07668v2 Announce Type: replace Abstract: The looking-in-looking-out (LILO) framework has enabled intelligent vehicle applications that understand both the outside scene and the driver state

13 May 2026

Ace-Skill: Bootstrapping Multimodal Agents with Prioritized and Clustered Evolution

arXiv:2605.08887v1 Announce Type: new Abstract: Self-evolving agents present a promising path toward continual adaptation by distilling task interactions into reusable knowledge artifacts. In practice

agentsarxiv-cs-ai

BEACON: A Multimodal Dataset for Learning Behavioral Fingerprints from Gameplay Data

arXiv:2605.10867v1 Announce Type: cross Abstract: Continuous authentication in high-stakes digital environments requires datasets with fine-grained behavioral signals under realistic cognitive and mot

CADBench: A Multimodal Benchmark for AI-Assisted CAD Program Generation

arXiv:2605.10873v1 Announce Type: cross Abstract: Recovering editable CAD programs from images or 3D observations is central to AI-assisted design, but progress is difficult to measure because existin

Tutorials

Dual Tuning for Reasoning Efficacy-Driven Data Curation in Multimodal LLM Training

arXiv:2603.04415v2 Announce Type: replace Abstract: Reasoning post-training improves Large Language Models (LLMs) on complex tasks such as mathematics and coding, but its benefits across diverse multi

tutorialsarxiv-cs-cl

HOME-KGQA: A Benchmark Dataset for Multimodal Knowledge Graph Question Answering on Household Daily Activities

arXiv:2605.09348v1 Announce Type: cross Abstract: Large Language Models (LLMs) provide flexible natural language processing capabilities, while knowledge graphs (KGs) offer explicit and structured kno

MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image

model-releasesarxiv-cs-cl

arXiv:2605.10616v1 Announce Type: cross Abstract: Tabular Foundation Models have recently established the state of the art in supervised tabular learning, by leveraging pretraining to learn generaliza

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Planner

arXiv:2506.01301v2 Announce Type: replace Abstract: Theory-of-Mind (ToM) enables humans to infer mental states-such as beliefs, desires, and intentions-forming the foundation of social cognition. Howe

researcharxiv-cs-ai

PIDNet: Progressive Implicit Decouple Network for Multimodal Action Quality Assessment

arXiv:2605.08945v1 Announce Type: new Abstract: Action quality assessment (AQA) aims to automatically quantify the execution quality of human actions in videos and is valuable for applications such as

SeePhys Pro: Diagnosing Modality Transfer and Blind-Training Effects in Multimodal RLVR for Physics Reasoning

arXiv:2605.09266v1 Announce Type: new Abstract: We introduce SeePhys Pro, a fine-grained modality transfer benchmark that studies whether models preserve the same reasoning capability when critical in

Self-Captioning Multimodal Interaction Tuning: Amplifying Exploitable Redundancies for Robust Vision Language Models

arXiv:2605.08145v1 Announce Type: cross Abstract: Current vision language models face hallucination and robustness issues against ambiguous or corrupted modalities. We hypothesize that these issues ca

researcharxiv-cs-ai

UniShield: Unified Face Attack Detection via KG-Informed Multimodal Reasoning

arXiv:2605.08709v1 Announce Type: new Abstract: Unified face attack detection (UAD) requires recognizing physical spoofing and digital forgery within a shared decision space, yet existing discriminati

Applications

AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models

arXiv:2605.07478v1 Announce Type: new Abstract: Speech-driven facial animation requires accurate correspondence between acoustic signals and facial motion, especially for articulation-related mouth mo

applicationsarxiv-cs-cv

11 May 2026

Fidel-TS: A High-Fidelity Multimodal Benchmark for Time Series Forecasting

model-releasesarxiv-cs-lg

arXiv:2509.24789v4 Announce Type: replace Abstract: The evaluation of time series forecasting models is hindered by a lack of high-quality benchmarks, leading to overestimated assessments of progress.

11 May 2026

Applications

MIST: Multimodal Interactive Speech-based Tool-calling Conversational Assistants for Smart Homes