Search: “multimodal”

Transformers for Multimodal Brain State Decoding: Integrating Functional Magnetic Resonance Imaging Data and Medical Metadata

arXiv:2512.08462v2 Announce Type: replace Abstract: Decoding brain states from functional magnetic resonance imaging (fMRI) data is vital for advancing neuroscience and clinical applications. While tr

researcharxiv-cs-lg

11 Aug 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Applications

Visual Distortion Detection in UGC Images Using Large Multimodal Models

DGX agent

arXiv:2608.09122v1 Announce Type: cross Abstract: The localized depiction of perceptual quality has long been a crucial, yet underexplored, challenge in image quality assessment (IQA). Existing approa

applicationsarxiv-cs-ai

11 Aug 2026

Walkable to Whom? Capturing Subjective Variability in Walkability Perception Using Multimodal Deep Learning

arXiv:2608.06934v1 Announce Type: cross Abstract: Visual perception of walkability varies substantially across individuals, reflecting differences in personal characteristics, experiences, and prefere

10 Aug 2026

KILVO: Kinematic-Inertial-LiDAR-Visual Odometry with Robust Multimodal Adaptation for Humanoid Robots

model-releasesarxiv-cs-ro

arXiv:2608.05647v1 Announce Type: new Abstract: This article presents a kinematic-inertial-LiDAR-visual odometry for humanoid robots, called KILVO. Tailored to the platform features, requirements, and

7 Aug 2026

SciQNet: Two-Stage Multimodal Adaptation for Scientific Image Quality Assessment

arXiv:2608.05691v1 Announce Type: new Abstract: Scientific images are essential for communicating experimental observations, quantitative evidence and conceptual knowledge. Unlike natural images, thei

7 Aug 2026

Vorch-IR: Long-Form Unified Multimodal Identity Replacement Video Generation

arXiv:2608.05648v1 Announce Type: new Abstract: Video identity replacement seeks to transfer the identities of one or more subjects while preserving the motion, expressions, and temporal structure of

7 Aug 2026

Agents

AutoProteinEngine: A Large Language Model Driven Agent Framework for Multimodal AutoML in Protein Engineering

arXiv:2411.04440v1 Announce Type: cross Abstract: Protein engineering is important for biomedical applications, but conventional approaches are often inefficient and resource-intensive. While deep lea

agentsarxiv-cs-ai

Multimodal Alignment Through Joint Kernel Entropic Gromov--Wasserstein Optimal Transport

arXiv:2608.04234v1 Announce Type: cross Abstract: We study the problem of aligning data from multiple modalities into a shared representation space, focusing on settings where strong pretrained unimod

safetyarxiv-cs-lg

Prototype-based Self-Supervised Multimodal Learning for PPG and Accelerometry Signals

arXiv:2510.09764v2 Announce Type: replace Abstract: Modeling multi-modal time-series data is critical for capturing system-level dynamics, particularly in biosignals where modalities such as ECG, PPG,

researcharxiv-cs-lg

ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation

arXiv:2608.04436v1 Announce Type: new Abstract: Text-to-image (T2I) models can produce visually compelling images, yet they remain limited on open-world tasks that require complex semantic understandi

safetyarxiv-cs-cv

ArtECulture: Benchmarking Culture-Conditioned Visual Emotion Understanding in Multimodal Large Language Models

model-releasesarxiv-cs-cl

arXiv:2608.03358v1 Announce Type: new Abstract: Existing visual emotion understanding methods typically ignore cultural variations in emotional perception. We introduce culture-conditioned visual emot

Evidence-Grounded Multimodal Knowledge Graph Construction for Multi-Lecture Educational Reasoning

arXiv:2608.03161v1 Announce Type: new Abstract: Lecture videos distribute knowledge across speech, slide text, diagrams, equations, and presentation order, which transcript-only retrieval does not ful

LocAnyMed: Vision-Language Grounding for Multimodal Medical Images

arXiv:2608.03322v1 Announce Type: new Abstract: Medical visual grounding connects free-form clinical queries to spatial evidence in medical images and is an important component of interpretable medica

Multimodal Auto-regressive Transformer Surrogate for Modeling Variable Operations and Quantifying Uncertainty in Geological Carbon Storage

arXiv:2608.02629v1 Announce Type: cross Abstract: The use of variable well perforation and injection strategies can improve the efficiency of geological carbon storage operations. We develop a new mul

Multimodal Plant Root Phenotyping with Integration of 3D Skeleton Extraction and Language Analysis

arXiv:2608.03109v1 Announce Type: new Abstract: Plant root phenotyping is fundamental to understanding below-ground structures, optimizing crop management, and improving agricultural sustainability. T

Applications

Implicit Neural Representations for Multimodal Longitudinal Image Imputation and Interpolation

arXiv:2608.02324v1 Announce Type: new Abstract: Longitudinal multiparametric MRI is central to follow-up imaging in oncology, yet real-world clinical data are characterised by missing sequences, heter

applicationsarxiv-cs-cv

4 Aug 2026

New York Smells: A Large Multimodal Dataset for Olfaction

arXiv:2511.20544v2 Announce Type: replace Abstract: While olfaction is central to how animals perceive the world, this rich chemical sensory modality remains largely inaccessible to machines. One key

4 Aug 2026

SPARC-Rad: A Multimodal Benchmark Dataset and Evaluation Pipeline for Spatial and Anatomical Reasoning in Radiology Vision-Language Models

arXiv:2608.00100v1 Announce Type: new Abstract: Vision-language models (VLMs) are increasingly being evaluated for medical imaging, but many available benchmarks emphasize disease classification, repo

4 Aug 2026

Digital Harf: A Clinically Integrated Multimodal AI System for Pervasive Arabic Speech and Language Therapy

arXiv:2607.27212v1 Announce Type: cross Abstract: Children with Autism Spectrum Disorder in Arabic-speaking countries face compounded barriers to effective speech and language therapy: a shortage of q

safetyarxiv-cs-cl

31 Jul 2026

A Closer Look at Dynamic Scene Graph Generation In the Era of Multimodal Large Language Models

arXiv:2503.15846v2 Announce Type: replace Abstract: Dynamic Scene Graph Generation (DSGG) aims to capture objects and their evolving relations in videos. Despite recent progress, the practicality and

30 Jul 2026

OmniAD: Detect and Understand Industrial Anomaly via Multimodal Reasoning

arXiv:2505.22039v2 Announce Type: replace Abstract: While anomaly detection has made significant progress, generating detailed analyses that incorporate industrial knowledge remains a challenge. To ad

30 Jul 2026

What Can Latent World Models Know? Physical Parameter Identifiability in Multimodal Predictive Representations

model-releasesarxiv-cs-lg

arXiv:2607.27017v1 Announce Type: new Abstract: A central premise of latent world models is that predicting the future forces a representation to internalize the physics of its environment. Which phys

30 Jul 2026

DisasterTD: Disaster Toponym Disambiguation Using Multimodal LLMs and Cross-View Geolocalization

model-releasesarxiv-cs-ai

arXiv:2607.24856v1 Announce Type: cross Abstract: Social media imagery (SMI) provides timely and fine-grained ground perspectives that are valuable for situational awareness and emergency response. Un

FinAbstain: Uncertainty-Calibrated Multimodal RAG for Selective Financial Forecasting

arXiv:2607.24875v1 Announce Type: new Abstract: Large language models (LLMs) can synthesize financial narratives but may express high confidence when evidence is sparse, stale, or contradictory. This

researcharxiv-cs-lg

Food Image Segmentation with LLM-Derived Ingredient Labels and Multimodal Fusion

arXiv:2607.25820v1 Announce Type: new Abstract: Food image segmentation plays a vital role in health-related applications such as nutrition tracking and personalized health monitoring. However, existi

LaP-Forensics: Latent-Pixel Consistency Guided Multimodal Reasoning for Deepfake Detection

arXiv:2607.25962v1 Announce Type: new Abstract: Recent generative models can produce images with few obvious visual artifacts, weakening detectors and explanations that rely only on surface appearance

Towards Reliable Stain Transfer: An Iterative Data-Model Co-Optimization Framework Based on Multimodal Expert-Guided Assessment

arXiv:2607.25393v1 Announce Type: new Abstract: Histopathological examination primarily relies on hematoxylin and eosin (H&E) and immunohistochemistry (IHC) staining. Although IHC provides critical mo

DriveDNA: A Large-Scale Multimodal Naturalistic Driving Dataset and Benchmark for Driving Style Identification

model-releasesarxiv-cs-lg

arXiv:2607.23822v1 Announce Type: new Abstract: Driving style captures stable, driver-specific patterns in how a vehicle is driven. In naturalistic data, however, this signal is hard to isolate becaus

GaitFace: A Multimodal Dataset for Long-Range Person Identification

arXiv:2607.23542v1 Announce Type: new Abstract: Efficient border control is becoming a significant global challenge, mainly due to severe congestion and extended passenger waiting times. To mitigate t

Local Ai

Harnessing X-ray Absorption Spectroscopy Data through Multimodal Mining of Battery Literature

arXiv:2607.23886v1 Announce Type: cross Abstract: X-ray absorption spectroscopy (XAS) is central to understanding the local electronic and atomic structure of materials, yet most published spectra rem

local-aiarxiv-cs-ai

Tutorials

How to deal with text only vector search across multimodal embedding space? [D]

tutorialsr-machinelearning

My data set is a list of images, each equipped with a a couple sentences of text. A user would search primarily with text only. My default approach is using BM25, but how would I facilitate searching

Autoregressive EHR Foundation Models with Multimodal Inputs

arXiv:2607.22264v1 Announce Type: new Abstract: Autoregressive foundation models trained on tokenized electronic health records (EHRs) can support zero-shot clinical prediction, yet most operate on st

safetyarxiv-cs-lg

27 Jul 2026

Benchmarking Fine-tuning and Retrieval Strategies for a Multimodal Language Model on the NRC Reactor Operator Licensing Examination

model-releasesarxiv-cs-cl

arXiv:2607.22067v1 Announce Type: new Abstract: The integration of large language models (LLMs) into the nuclear power industry requires outputs grounded in domain-specific knowledge. This study evalu

27 Jul 2026

Local Multimodal Music Alignment from Global Supervision

arXiv:2607.10023v2 Announce Type: replace-cross Abstract: Understanding music requires understanding localized relationships across data modalities, e.g., how time in performance audio maps onto posit

safetyarxiv-cs-lg

27 Jul 2026

Human-Inspired Framework for Robotic Craniotomy: Integrating Multimodal Fusion and Adaptive Trajectory Adjustment

arXiv:2607.21058v1 Announce Type: new Abstract: Manual craniotomy is a high-risk, skill-dependent procedure associated with surgeon fatigue and potential dural injury. While robotic approaches have im

safetyarxiv-cs-ro

Multimodal Learning for Arcing Detection in Pantograph-Catenary Systems

arXiv:2602.08792v2 Announce Type: replace-cross Abstract: The pantograph-catenary interface is essential for ensuring uninterrupted and reliable power delivery in electrified rail systems. However, el

RL-MACRO: A Cybernetic Closed-Loop Intelligence Framework for Multimodal Adaptive Robotic Craniotomy

arXiv:2607.21113v1 Announce Type: new Abstract: Autonomous robotic craniotomy requires continuous regulation of tool-tissue interactions to mitigate mechanical overload and thermal damage while mainta

safetyarxiv-cs-ro

Unlearning Under Imbalance: Benchmarking Fairness in Multimodal LLM Unlearning

model-releasesarxiv-cs-ai

arXiv:2607.21300v1 Announce Type: cross Abstract: Machine unlearning has emerged as a tool for removing personal data from trained models to comply with recent AI regulations. To evaluate unlearning e

HOMIE: Human-object Centric Video Personalization via Multimodal Intelligent Enhancement

arXiv:2607.18217v2 Announce Type: replace Abstract: Human-object centric video personalization (HOCVP) is a core task within subject-driven video generation. However, existing methods suffer from two

safetyarxiv-cs-cv

In-Context Learning for Wound Classification with Small Multimodal Language Models

arXiv:2607.18819v1 Announce Type: new Abstract: Wound image classification is often treated as a task-specific supervised learning problem, requiring substantial amounts of manually labelled data and

Leveraging ChatGPT's Multimodal Vision Capabilities to Rank Satellite Images by Poverty Level: Advancing Tools for Social Science Research

arXiv:2501.14546v2 Announce Type: replace-cross Abstract: This paper investigates the novel application of Large Language Models (LLMs) with vision capabilities to analyze satellite imagery for villag

Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos

arXiv:2607.19857v1 Announce Type: cross Abstract: Language-guided aerial perception aims to understand user-specified tiny targets in complex unmanned aerial vehicle (UAV) scenes. In real UAV deployme

A novel unsupervised machine learning strategy to handle multimodal cardiac PET/MRI data

arXiv:2607.13936v1 Announce Type: new Abstract: Arrhythmogenic left ventricular cardiomyopathy is a genetic myocardial disease difficult to diagnose due to the lack of gold standard criteria. Simultan

Kepler-Encoder-v0.1: Towards a Multimodal Embedding Model for Robots

arXiv:2607.13522v1 Announce Type: cross Abstract: A robot must understand the state of its own body, but a camera sees only part of it. Force and contact leave almost no trace in a single frame, and r

Agents

M^ext{4}World: A Multi-view Multimodal Driving World Model for Interactive Object Manipulation and Minute-long Streaming

arXiv:2607.14005v1 Announce Type: new Abstract: Driving-world generation has emerged as a core capability for scalable autonomous-driving simulation, yet existing methods remain limited in object-leve

agentsarxiv-cs-cv

Local Ai

Multimodal Assessment of Pancreatic Cancer Resectability Using Deep Learning

arXiv:2607.13826v1 Announce Type: cross Abstract: Accurate determination of pancreatic ductal adenocarcinoma (PDAC) resectability relies on evaluating how the tumor interacts with major peripancreatic

local-aiarxiv-cs-ai

Auditing Data Leakage in Whole-Slide Image Multimodal Benchmarks

arXiv:2607.12278v1 Announce Type: new Abstract: Recent vision-language models (VLMs) for computational pathology report striking zero-shot performance on whole-slide image (WSI) visual question answer

15 Jul 2026

GemNav: Discrete-Token Visual Robot Navigation using a Multimodal Large Language Model