Search: “multimodal”

Joint Training Is Not Enough: Conditioned Cross-Granularity Training for Multimodal Document Understanding

arXiv:2609.00756v1 Announce Type: new Abstract: The Mutual Reinforcement Effect (MRE) asks whether a fine, span-level and a coarse, document-level task help each other when one model handles both. We

researcharxiv-cs-cl

2 Sept 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

Anchoring Speech with Semantics: A Multimodal Adapter Mechanism for Automatic Speech Recognition in Low-Resource Languages

DGX agent

arXiv:2608.29239v1 Announce Type: new Abstract: Low-resource ASR remains difficult because scarce transcripts provide limited supervised evidence for target-side generation. To address this gap, we pr

researcharxiv-cs-cl

1 Sept 2026

Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence

arXiv:2606.15932v3 Announce Type: replace Abstract: While Large Language Models (LLMs) have substantially advanced text-to-code generation, many real programming tasks specify intent through visual ar

agentsarxiv-cs-cl

1 Sept 2026

Do large language models scrutinise what they review? A multimodal audit of scoring calibration, error detection, and author-identity effects

arXiv:2608.28626v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used to generate peer reviews, prompting examination of their capacity for critical evaluation. This stu

1 Sept 2026

Multimodal Large Language Models Predict Urban Safety Perception but Encode Non-Neutral Demographic Priors

arXiv:2503.00610v2 Announce Type: replace-cross Abstract: Understanding how people perceive urban environments is essential for inclusive planning, yet conventional surveys are costly and difficult to

safetyarxiv-cs-ai

1 Sept 2026

uScenes: A Multimodal RGB and 3D Sonar Dataset for Underwater Robot Perception

arXiv:2608.27795v1 Announce Type: new Abstract: Robust perception is essential for the deployment of autonomous underwater robots. However, optical cameras become unreliable under poor illumination an

agentsarxiv-cs-cv

31 Aug 2026

Local Ai

A Safety-Gated Multimodal AI Backend for Mental-Health Support: Hierarchical State Representation, Conservative Risk Fusion, and Controlled Generation in Anian

arXiv:2608.26162v1 Announce Type: new Abstract: Safety-critical mental-health support systems must distinguish when supportive conversation is appropriate from when free-form generation should be bloc

local-aiarxiv-cs-ai

28 Aug 2026

Case2Flow: Bridging Patient Cases and Guideline Flowcharts through Multimodal Retrieval

arXiv:2608.26414v1 Announce Type: new Abstract: Medical guidelines encode rich, evidence-based decision logic, yet the specific decision artifact a clinician needs is hard to locate within a guideline

safetyarxiv-cs-cl

28 Aug 2026

Applications

Drift-Aware Multimodal User Representation Learning via Multi-Scale Temporal Modeling and Sparse Mixture-of-Experts

arXiv:2608.25773v1 Announce Type: new Abstract: Understanding user preferences from noisy and temporally evolving social media behaviors is fundamentally challenging due to interest drift, where user

applicationsarxiv-cs-lg

27 Aug 2026

PlanSightRAG: A Visual-First Multimodal RAG for Automating Question Answering and Compliance Checking for Civil Standard Plans

model-releasesarxiv-cs-cl

arXiv:2608.26091v1 Announce Type: cross Abstract: Civil infrastructure compliance checking has long relied on engineers manually reading legacy 2D plans; however, OCR-based automation strips away the

27 Aug 2026

StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models

model-releasesarxiv-cs-cv

arXiv:2608.26067v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models have demonstrated effectiveness in robot manipulation, yet state-of-the-art models such as pi0.5 operate under a sin

27 Aug 2026

MolEmb: Multimodal Large Language Models Can Be Strong Molecular Embedding Models

arXiv:2608.23646v1 Announce Type: new Abstract: Molecular embedding models can serve as foundational infrastructure for computational chemistry and drug discovery, where reusable vector representation

26 Aug 2026

Applications

AI-based worker guidance in assembly and disassembly operations using multimodal ego/exo-centric data capture and structured task knowledge

arXiv:2608.22617v1 Announce Type: cross Abstract: Assembly and disassembly processes rely on expert knowledge that is difficult to document, reuse, and transfer. This paper presents a data-centric app

applicationsarxiv-cs-ai

DesignAgent3D: Interactive 3D Scene Editing via Designer-like Multimodal Reasoning

arXiv:2608.21438v1 Announce Type: new Abstract: Text guided 3D scene editing provides an intuitive interface for modifying reconstructed environments, but remains difficult because natural language de

safetyarxiv-cs-cv

DF-MoE: Generalizable Deepfake Detection via Multimodal Sparse Mixture-of-Experts

arXiv:2608.23363v1 Announce Type: cross Abstract: Audio-visual deepfake detection is an actively studied topic, where one of the main challenges is to develop detectors able to generalize across deepf

From Multimodal Observation to Interpretable Suggestions: Counterfactual Time-Expanded Relational Modeling of Surgical Teams

arXiv:2608.23254v1 Announce Type: new Abstract: In surgery, patient safety is threatened not only by technical issues but also by poor teamwork. However, existing surgical AI-based solutions focus mai

UniMem: Unifying Multimodal Memory and Control for Vision-Language-Action Models

arXiv:2608.22869v1 Announce Type: cross Abstract: While Vision-Language-Action (VLA) models have leveraged internet-scale pretraining and task-focused finetuning to achieve strong performance on long-

safetyarxiv-cs-cv

Recognition-Conditioned Reasoning: A Training-Free Multimodal-LLM Pipeline for Fine-Grained Micro-Action Understanding

arXiv:2608.21022v1 Announce Type: new Abstract: Micro-actions are subtle, short, low-amplitude body movements, such as a fidgeting hand or a slight head tilt, that humans perform with little conscious

24 Aug 2026

Representation Affects Retrieval: A Case Study of Skill Discovery and Routing in a Multimodal Agent Harness

arXiv:2608.20389v1 Announce Type: new Abstract: A production agent harness must discover and rank, from a growing library of skills, the one most appropriate for a user's task. At small scale this sel

24 Aug 2026

Inter-X++: A Comprehensive Benchmark for Multimodal Human-Human Interaction Analysis

model-releasesarxiv-cs-cv

arXiv:2608.20312v1 Announce Type: new Abstract: The capability to perceive and synthesize human-human interactions is fundamental to developing intelligent digital human systems. However, existing dat

21 Aug 2026

Can a Lightweight Multimodal Model Estimate LLM Reasoning Performance? A Study for Compute-Optimal Document Inference

arXiv:2608.18591v1 Announce Type: new Abstract: Uniformly allocating inference reasoning budgets to LLMs is expensive and prone to over-thinking penalties; especially in document tasks where visual la

DentAgent: Evidence-Centric Multi-Agent Coordination for Multimodal Dental Reasoning

arXiv:2608.18878v1 Announce Type: new Abstract: Oral diseases affect billions of people worldwide, underscoring a pressing need for accurate and reliable dental assessment that integrates heterogeneou

agentsarxiv-cs-ai

Eyes on the Image: Gaze Supervised Multimodal Learning for Chest X-ray Diagnosis and Report Generation

model-releasesarxiv-cs-cv

arXiv:2508.13068v2 Announce Type: replace Abstract: Medical vision-language models still struggle to match radiologists' attention and to verbalize findings with explicit spatial grounding. We address

Multimodal Rapport Estimation in Real-World HRI

model-releasesarxiv-cs-cl

arXiv:2608.18401v1 Announce Type: cross Abstract: Evaluating interaction quality in real-world HRI is an important challenge. If interaction quality can be estimated reliably, the results can be used

Auditing Exposure to Harmful Content on TikTok using Multimodal Language Models: A Cross-National, Age-Stratified Study

model-releasesarxiv-cs-cl

arXiv:2608.17583v1 Announce Type: new Abstract: Online video platforms can expose young users to harmful content, but independent audits remain difficult because video annotation is costly and moderat

19 Aug 2026

Closing the Affective Loop: Multimodal Speaker-Listener Emotion-Dynamics-Aware Empathetic Social Robots

arXiv:2608.16686v1 Announce Type: cross Abstract: Empathetic social robots should respond not only to what users say, but also to how their emotions dynamically evolve during interaction. However, exi

safetyarxiv-cs-cl

Lesioned Multimodal Language Models Reproduce Aphasic Picture-Naming Patterns

arXiv:2607.11621v2 Announce Type: replace Abstract: Aphasia following stroke commonly produces systematic naming errors with characteristic profiles, but whether general-purpose language models not de

NARRATE: A Multimodal Real-World Australian Driving Dataset for Human-Centred Explanations in Automated Driving

arXiv:2608.14767v1 Announce Type: cross Abstract: Automated vehicles must explain their decisions in ways that passengers can understand, monitor, and trust. Existing language-annotated driving datase

PathFinder: Joint Decompositions of Linked Multimodal Datasets

arXiv:2608.14951v1 Announce Type: new Abstract: Low-rank matrix decompositions can uncover patterns and structure in data and have a number of different applications across many disciplines. Extension

researcharxiv-cs-lg

Sequential Multimodal Evidence Optimization for Product Media Ranking in E-Commerce

arXiv:2608.15662v1 Announce Type: new Abstract: On modern e-commerce stores, customers consume ordered slates of heterogeneous product media, such as images, videos, and 3D renders, before making purc

Tutorials

Generalizable Operating Room Expert with Multimodal Enhancement

arXiv:2508.08199v2 Announce Type: replace Abstract: Precise spatial modeling in the operating room (OR) is essential for intraoperative awareness, hazard avoidance, and surgical decision-making. Altho

tutorialsarxiv-cs-cv

14 Aug 2026

How effective are VLMs in assisting humans in inferring the quality of mental models from Multimodal short answers?

arXiv:2603.00056v2 Announce Type: replace-cross Abstract: STEM Mental models can play a critical role in assessing students' conceptual understanding of a topic. They not only offer insights into what

13 Aug 2026

Local Ai

Embodied Multimodal Grounding for Open-Vocabulary Mobile Manipulation via Semantic 3D Gaussian Splatting

arXiv:2608.10756v1 Announce Type: cross Abstract: Embodied mobile manipulation requires language, visual observations, three-dimensional scene structure, and action feasibility to be aligned before ex

local-aiarxiv-cs-cv

12 Aug 2026

Unlocking the Power of Medical Tabular Data via Semantic-Aware Multimodal Pre-training

arXiv:2608.10522v1 Announce Type: cross Abstract: While vision-language models dominate medical representation learning, unstructured text lacks the dense, quantitative diagnostic phenotypes inherent

12 Aug 2026

Auditing Instruction-Trajectory Mismatches in Multimodal Robot Demonstrations

arXiv:2608.07895v1 Announce Type: cross Abstract: Robot demonstration datasets used to train vision-language-action policies can contain a subtle but harmful failure mode: trajectories that are behavi

From Visual to Multimodal: Systematic Ablation of Encoders and Fusion Strategies in Animal Identification

arXiv:2603.02270v2 Announce Type: replace Abstract: Automated animal identification is a practical task for reuniting lost pets with their owners, yet current systems often struggle due to limited dat

Jagle: Building a Large-Scale Japanese Multimodal Post-Training Dataset for Vision-Language Models

arXiv:2604.02048v2 Announce Type: replace Abstract: Developing vision-language models (VLMs) that generalize across diverse tasks requires large-scale training datasets with diverse content. In Englis

MateInfoUB: A Real-World Benchmark for Testing LLMs in Competitive, Multilingual, and Multimodal Educational Tasks

arXiv:2507.03162v2 Announce Type: replace-cross Abstract: The rapid advancement of Large Language Models (LLMs) has transformed various domains, particularly computer science (CS) education. These mod

Transformers for Multimodal Brain State Decoding: Integrating Functional Magnetic Resonance Imaging Data and Medical Metadata

arXiv:2512.08462v2 Announce Type: replace Abstract: Decoding brain states from functional magnetic resonance imaging (fMRI) data is vital for advancing neuroscience and clinical applications. While tr

researcharxiv-cs-lg

Applications

Visual Distortion Detection in UGC Images Using Large Multimodal Models

arXiv:2608.09122v1 Announce Type: cross Abstract: The localized depiction of perceptual quality has long been a crucial, yet underexplored, challenge in image quality assessment (IQA). Existing approa

applicationsarxiv-cs-ai

Walkable to Whom? Capturing Subjective Variability in Walkability Perception Using Multimodal Deep Learning

arXiv:2608.06934v1 Announce Type: cross Abstract: Visual perception of walkability varies substantially across individuals, reflecting differences in personal characteristics, experiences, and prefere

10 Aug 2026

KILVO: Kinematic-Inertial-LiDAR-Visual Odometry with Robust Multimodal Adaptation for Humanoid Robots

model-releasesarxiv-cs-ro

arXiv:2608.05647v1 Announce Type: new Abstract: This article presents a kinematic-inertial-LiDAR-visual odometry for humanoid robots, called KILVO. Tailored to the platform features, requirements, and

7 Aug 2026

SciQNet: Two-Stage Multimodal Adaptation for Scientific Image Quality Assessment

arXiv:2608.05691v1 Announce Type: new Abstract: Scientific images are essential for communicating experimental observations, quantitative evidence and conceptual knowledge. Unlike natural images, thei

7 Aug 2026

Vorch-IR: Long-Form Unified Multimodal Identity Replacement Video Generation

arXiv:2608.05648v1 Announce Type: new Abstract: Video identity replacement seeks to transfer the identities of one or more subjects while preserving the motion, expressions, and temporal structure of

7 Aug 2026

AutoProteinEngine: A Large Language Model Driven Agent Framework for Multimodal AutoML in Protein Engineering

arXiv:2411.04440v1 Announce Type: cross Abstract: Protein engineering is important for biomedical applications, but conventional approaches are often inefficient and resource-intensive. While deep lea

agentsarxiv-cs-ai

Multimodal Alignment Through Joint Kernel Entropic Gromov--Wasserstein Optimal Transport

arXiv:2608.04234v1 Announce Type: cross Abstract: We study the problem of aligning data from multiple modalities into a shared representation space, focusing on settings where strong pretrained unimod

Prototype-based Self-Supervised Multimodal Learning for PPG and Accelerometry Signals

arXiv:2510.09764v2 Announce Type: replace Abstract: Modeling multi-modal time-series data is critical for capturing system-level dynamics, particularly in biosignals where modalities such as ECG, PPG,

researcharxiv-cs-lg

ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation

arXiv:2608.04436v1 Announce Type: new Abstract: Text-to-image (T2I) models can produce visually compelling images, yet they remain limited on open-world tasks that require complex semantic understandi

safetyarxiv-cs-cv