Search: “multimodal”

Autonomous Driving with Priority-Ordered STL Specifications Under Multimodal Uncertainty

arXiv:2606.20336v2 Announce Type: replace Abstract: Autonomous vehicles must plan trajectories that satisfy multiple requirements, such as safety, traffic-rule compliance, and passenger comfort. Howev

safetyarxiv-cs-ro

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

Deep Multimodal Wearable Sensor Fusion for Detection of Body-Focused Repetitive Behaviors

DGX agent

arXiv:2608.09830v1 Announce Type: new Abstract: Body-focused repetitive behaviors, such as hair pulling and skin picking, are compulsive motor actions commonly associated with obsessive-compulsive and

researcharxiv-cs-lg

11 Aug 2026

Med-CRAFT: An Information System for Explainable and Configurable Construction of Multimodal Medical QA Datasets

arXiv:2512.01045v2 Announce Type: replace Abstract: Data-intensive artificial intelligence applications increasingly rely on large-scale, high-quality, explainable, and reproducible datasets, yet the

Multimodal Skin Lesion Classification with Swin Transformer and Clinical Metadata Fusion

arXiv:2608.07574v1 Announce Type: new Abstract: Skin lesion classification plays an important role in supporting the early diagnosis of skin cancer. However, automated analysis remains challenging due

SafeSceneReason: A Multimodal Reasoning Benchmark Connecting Industrial Hazards with Accident Knowledge

arXiv:2608.09230v1 Announce Type: new Abstract: Industrial-safety understanding requires more than detecting workers, equipment, and personal protective equipment. Models must also assess compliance,

When One Modality Is Not Enough: Multimodal Sex and Life-Stage Classification of Red Deer from Aerial RGB-Thermal Video

arXiv:2608.06973v1 Announce Type: new Abstract: Aerial drone surveys increasingly support wildlife population estimation, yet a useful census is more than a count: population dynamics are defined by s

10 Aug 2026

CSGen: A Multi-Domain Curvilinear Structure Generation Model via Hierarchical Multimodal Diffusion

arXiv:2608.04655v1 Announce Type: cross Abstract: Curvilinear structure analysis is an important and fundamental task in multimedia. However, the controllable generation of images with precise curvili

6 Aug 2026

MOON3.0: Reasoning-aware Multimodal Representation Learning for E-commerce Product Understanding

arXiv:2604.00513v3 Announce Type: replace-cross Abstract: With the rapid growth of e-commerce, exploring general representations rather than task-specific ones has attracted increasing attention. Alth

6 Aug 2026

Visual Anchoring in Diffusion: Multimodal Zero-Shot Skeleton Action Recognition

arXiv:2608.04623v1 Announce Type: new Abstract: Zero-shot Skeleton Action Recognition (ZSAR) remains ambiguous when unseen actions share similar skeleton joint dynamics but differ in objects or scene

6 Aug 2026

AquaJEPA: Action-Conditioned Multimodal Predictive Representations for Underwater Robot Dynamics

arXiv:2607.29393v1 Announce Type: new Abstract: Underwater robots combine complementary sensors whose reliability changes abruptly with water visibility, viewpoint, and vehicle motion. We introduce Aq

researcharxiv-cs-ro

CALM-AH: An ABAW11-Calibrated Multimodal Ensemble with Reliability-Gated Multi-Expert Consensus for Video-Level Ambivalence and Hesitancy Recognition

arXiv:2607.29310v1 Announce Type: new Abstract: Ambivalence and hesitancy (A/H) are subtle behavioural states that may be expressed through language, voice, facial activity, and other non-verbal cues.

Distance-aware Soft Prompt Guidance for Multimodal Valence-Arousal Estimation

arXiv:2603.13415v2 Announce Type: replace Abstract: Valence-arousal (VA) estimation is crucial for capturing the nuanced nature of human emotions in naturalistic environments. While pre-trained vision

safetyarxiv-cs-cv

JoVA: Unified Multimodal Learning for Joint Video-Audio Generation and Editing

arXiv:2512.13677v2 Announce Type: replace Abstract: In this paper, we present JoVA, a streamlined framework that unifies joint video-audio generation and editing. While existing methods often rely on

safetyarxiv-cs-cv

MMShopBench: A Real-Log Benchmark for Multimodal, Multi-Turn Shopping Agents

arXiv:2607.29002v1 Announce Type: new Abstract: Online shoppers increasingly turn to AI shopping assistants, using images and multi-turn dialogue to express and refine product needs that are difficult

The Theoretical Foundation of Socratic Tests: Dynamic, Multimodal, Conversational Examinations

arXiv:2607.29624v1 Announce Type: cross Abstract: Traditional static assessments rely on a subtractive, deficit-based grading model that often penalizes ambition and obscures diagnostic feedback. Conv

safetyarxiv-cs-ai

ClinLens: Towards Long-Horizon Coding Agents for Longitudinal Multimodal Clinical Data Science

arXiv:2607.26155v1 Announce Type: new Abstract: Clinical data-science agents must transform heterogeneous longitudinal records into auditable analyses, yet existing benchmarks largely isolate medical

31 Jul 2026

Symphony of Bias: Exploring Gender Associations with Musical Instruments in Multimodal LLMs

model-releasesarxiv-cs-cl

arXiv:2607.26355v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly embedded in everyday life and widely used for information seeking, raising concerns about their potential

30 Jul 2026

A Cost-Effective Multimodal LLM Reasoning Framework for Question Answering over Irregular Clinical Time Series

arXiv:2607.25947v1 Announce Type: new Abstract: Question answering (QA) over irregular clinical time series (ICTS) plays a pivotal role in a wide range of healthcare applications. Although recent mult

Knowledge-Guided Multimodal Reasoning over Interacting Streams for Video-Level Ambivalence and Hesitancy Recognition

arXiv:2607.25961v1 Announce Type: cross Abstract: Ambivalence and hesitancy (A/H) are conflicting affective states that precede the delay or abandonment of health behaviour change. Recognition of A/H

Multimodal Hybrid Retrieval-Augmented Generation for Scientific Document Understanding using Open-Source SLMs

arXiv:2607.24799v1 Announce Type: cross Abstract: Large Language Models tend to hallucinate when answering domain-specific ques tions from scientific documents without prior fine-tuning. Currently, me

Neurai-VN Benchmark: Standardized Machine Learning Models for Multimodal Digital Phenotyping in Mental Health Classification

model-releasesarxiv-cs-lg

arXiv:2607.25232v1 Announce Type: new Abstract: Digital phenotyping (DP) using smartphones and wearable devices has shown considerable potential for mental health monitoring. However, progress remains

BATON: A Multimodal Benchmark for Bidirectional Automation Transition Observation in Naturalistic Driving

arXiv:2604.07263v2 Announce Type: replace-cross Abstract: Existing driving automation (DA) systems on production vehicles rely on human drivers to decide when to engage DA while requiring them to rema

28 Jul 2026

Do Visual Features Improve Other-Initiated Repair Detection? A Dyadic Multimodal Approach

arXiv:2607.23845v1 Announce Type: new Abstract: Other-initiated Self-repair, or in short Other-initiated Repair (OIR), is an essential mechanism in conversational interaction, whereby a recipient sign

28 Jul 2026

Environment-Aware Channel Inference via Cross-Modal Flow: From Multimodal Sensing to Wireless Channels

arXiv:2512.04966v2 Announce Type: replace-cross Abstract: Accurate channel state information (CSI) underpins reliable and efficient wireless communication. However, acquiring CSI via pilot estimation

safetyarxiv-cs-lg

Tutorials

M^3-Gen: Interpretable Multimodal Generation of Gene Expression Profiles Using Clinical and Imaging Data

arXiv:2607.21343v1 Announce Type: cross Abstract: Integrating heterogeneous biomedical data, including clinical metadata, histopathology images, and molecular profiles, is crucial for comprehensive di

tutorialsarxiv-cs-ai

Monkey King Bang: A Unified Scientific Multimodal Foundation Model

arXiv:2607.20557v1 Announce Type: cross Abstract: Scientific discovery is increasingly shifting from isolated disciplines to multi-domain reasoning, and AI for science faces a similar transition. Exis

Multimodal Pretraining for Generalizable EEG Representation Learning

arXiv:2607.21384v1 Announce Type: new Abstract: Electroencephalography (EEG) models used for epilepsy are often limited to specific datasets and tasks. This limited approach can make it challenging to

Local Ai

Appearance Pointers -- Multimodal Region Control of Diffusion Transformers

arXiv:2607.19344v1 Announce Type: new Abstract: Controllable image generation remains challenging for creative professionals, who often require precise regional control over materials, object identiti

local-aiarxiv-cs-cv

Deep Shape Regression for Planar Curves with Multimodal Covariates

arXiv:2607.19600v1 Announce Type: cross Abstract: The shape of a planar curve is the geometric information that remains once translation, rotation, scale and reparametrisation are removed and is of in

safetyarxiv-cs-lg

Hypothesis-and-Refinement Learning of Organic Structures from Multimodal Spectroscopic Data

model-releasesarxiv-cs-lg

arXiv:2607.19816v1 Announce Type: cross Abstract: Determining molecular structures from spectroscopic data remains fundamentally challenging because the inverse problem is intrinsically underdetermine

Trustworthy Privacy-Preserving Multimodal Federated Learning for Personalised Breast Cancer Prediction

arXiv:2607.19532v1 Announce Type: cross Abstract: Federated learning has emerged as a potential solution to privacy concerns associated with using sensitive health data for training predictive models,

safetyarxiv-cs-ai

Agents

Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild

arXiv:2607.13881v1 Announce Type: cross Abstract: Human-object interaction detection (HOID) has traditionally been formulated as a supervised detection problem over predefined interaction categories.

agentsarxiv-cs-ai

16 Jul 2026

MAG: A Web-Agent Benchmark and Harness for Multimodal Action and Guide Generation

arXiv:2607.10079v2 Announce Type: replace Abstract: Digital Adoption Platforms (DAPs) are embedded overlays widely used on web systems to guide users through operations inside a page, helping them get

15 Jul 2026

Applications

ReflectWorld-MM: An Entity-Oriented Multimodal Memory System for Open-Ended Video Streams

arXiv:2607.09759v2 Announce Type: replace-cross Abstract: Building assistants that can continually watch the world, remember what they see, and reason over their accumulated experience is a long-stand

applicationsarxiv-cs-ai

15 Jul 2026

VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness

arXiv:2607.08112v1 Announce Type: new Abstract: We introduce VSRo-200, the first large-scale dataset for visual speech recognition (lip reading) in Romanian, comprising 200 hours of real-world podcast

10 Jul 2026

HunyuanVideo-HOMA: Generic Human-Object Interaction in Multimodal Driven Human Animation

arXiv:2506.08797v2 Announce Type: replace Abstract: To address key limitations in human-object interaction (HOI) video generation -- specifically the reliance on curated motion data, limited generaliz

9 Jul 2026

Multimodal Voice Activity Projection for Turn-Taking in Social Robots with Voice-Activity-Related Pretrained Encoders

arXiv:2607.07294v1 Announce Type: cross Abstract: Turn-taking prediction is a key requirement for social robots involved in human-human interaction, particularly in mediator settings, where the robot

9 Jul 2026

Multimodal Analytics of Cybersecurity Crisis Preparation Exercises: What Predicts Success?

arXiv:2603.28553v2 Announce Type: replace-cross Abstract: Instructional alignment, the match between intended cognition and enacted activity, is central to effective instruction but hard to operationa

safetyarxiv-cs-lg

Multimodal Molecular Representation Learning with Graph Neural Networks, Deep & Cross Networks, and SMILES Embeddings

model-releasesarxiv-cs-lg

arXiv:2607.05736v1 Announce Type: new Abstract: Molecular property prediction often relies on isolated data modalities, where continuous 3D graph neural networks (GNNs) struggle to efficiently capture

NAMD: Virtual Follow-up Computed Tomography Synthesis via Nodule-Aligned Multimodal Diffusion Models for Early Lung Cancer Diagnosis

arXiv:2603.15932v2 Announce Type: replace Abstract: Lung cancer remains the leading cause of cancer-related mortality worldwide, with survival outcomes critically dependent on early and accurate detec

Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliability

model-releasesarxiv-cs-cl

arXiv:2607.06196v1 Announce Type: new Abstract: Current AI safety evaluation and benchmarking frameworks predominantly rely on Western-centric culture-agnostic defaults that mask critical regional law

Local Ai

Enhancing Large Multimodal Models in Key Information Extraction via Scene-Aware Document Synthesis

arXiv:2607.04636v1 Announce Type: new Abstract: Key Information Extraction (KIE) converts visually rich documents into structured data, but practical deployment remains challenging: strong performance

local-aiarxiv-cs-cv

GestaltMML: Enhancing Rare Genetic Disease Diagnosis through Multimodal Machine Learning Combining Facial Images and Clinical Text

arXiv:2312.15320v3 Announce Type: replace-cross Abstract: Individuals with suspected rare genetic disorders often undergo multiple clinical evaluations, imaging studies, laboratory tests, and genetic

H-OPD: Confidence Aware Heterogeneous Multi-Teacher Multimodal On-policy Distillation

arXiv:2607.02592v1 Announce Type: new Abstract: On-policy distillation (OPD) has recently emerged as an effective post-training paradigm by providing supervision on student-generated trajectories. How

safetyarxiv-cs-cv

Paired Uterine Whole-Slide Images and Pathology Reports for Multimodal Computational Pathology

arXiv:2607.04020v1 Announce Type: new Abstract: Uterine diseases represent an important category of gynecologic pathology and require accurate histopathological assessment for diagnosis and treatment

PhenoNEST: A Neuro-Symbolic Framework for Ontology-Aware Multimodal Plant Phenotyping and Trait Discovery

arXiv:2607.03245v1 Announce Type: new Abstract: High-throughput plant phenotyping generates valuable data that often remains trapped in unstructured text and isolated RGB images. To bridge this semant

researcharxiv-cs-lg

Triple-Phase Multimodal Knowledge Aggregation Framework for Microbial Keratitis Subtype Diagnosis on Slit-Lamp Photography

arXiv:2607.03740v1 Announce Type: cross Abstract: Microbial keratitis requires rapid pathogen identification to guide treatment, but culture- and PCR-based diagnostics are slow and resource-intensive.

UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy

arXiv:2603.24690v2 Announce Type: replace Abstract: In-context learning (ICL) enables fast task adaptation from demonstrations without per-task parameter updates but remains highly sensitive to exampl