Search: “multimodal”

On the Spectral Geometry of Cross-Modal Representations: A Functional Map Diagnostic for Multimodal Alignment

arXiv:2604.08579v1 Announce Type: cross Abstract: We study cross-modal alignment between independently pretrained vision (DINOv2) and language (all-MiniLM-L6-v2) encoders using the functional map fram

safetyarxiv-cs-ai

13 Apr 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

ChangeBridge: Spatiotemporal Image Generation with Multimodal Controls for Remote Sensing

DGX agent

arXiv:2507.04678v3 Announce Type: replace Abstract: Spatiotemporal image generation is a highly meaningful task, which can generate future scenes conditioned on given observations. However, existing c

researcharxiv-cs-cv

10 Apr 2026

Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models

arXiv:2604.06912v1 Announce Type: cross Abstract: MLLMs require high-resolution visual inputs for fine-grained tasks like document understanding and dense scene perception. However, current global res

local-aiarxiv-cs-ai

SyncBreaker:Stage-Aware Multimodal Adversarial Attacks on Audio-Driven Talking Head Generation

arXiv:2604.08405v1 Announce Type: new Abstract: Diffusion-based audio-driven talking-head generation enables realistic portrait animation, but also introduces risks of misuse, such as fraud and misinf

What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoning

arXiv:2604.06995v1 Announce Type: new Abstract: Existing Graphical User Interface (GUI) reasoning tasks remain challenging, particularly in UI understanding. Current methods typically rely on direct s

Zatom-1: A Multimodal Flow Foundation Model for 3D Molecules and Materials

arXiv:2602.22251v3 Announce Type: replace-cross Abstract: General-purpose 3D chemical modeling encompasses molecules and materials, requiring both generative and predictive capabilities. However, most

researcharxiv-cs-ai

Beyond Generation: Using Multimodal LLMs as Detectors of LLM-Generated Deepfake Facial Images

arXiv:2610.21760v1 Announce Type: new Abstract: As image generation models become increasingly capable of producing highly realistic content, traditional forensic techniques are struggling to keep up.

CoWorld Reasoner: Agentic Multimodal Reasoning with the Evolving Video World Model-Driven Harness

arXiv:2610.21910v1 Announce Type: new Abstract: As Vision-Language Models (VLMs) advance, they are increasingly expected to perform spatiotemporal reasoning over motion, actions, and view changes beyo

agentsarxiv-cs-ai

FuseAdapt: Parameter-Efficient Multimodal Fusion for Semantic Segmentation with Missing Modalities

arXiv:2610.21779v1 Announce Type: new Abstract: Multi-modal semantic segmentation benefits from complementary sensors, but this benefit is fragile when some modalities are degraded or missing at test

Multimodal AI Detection for Math-Heavy Scientific Text

model-releasesarxiv-cs-cl

arXiv:2610.21062v1 Announce Type: new Abstract: As large language models become increasingly capable of generating, and assisting with, scientific writing, there is a need for greater transparency and

FACT: A Multimodal Force-Action-Track Foundation Model for Human-to-Robot Dexterity

arXiv:2610.19155v1 Announce Type: new Abstract: Human demonstrations offer a scalable alternative to costly robot data collection for dexterous manipulation, but most human datasets lack force measure

agentsarxiv-cs-ro

Instruction Mining From Images: Constructing a Synthetic Dataset for Multimodal Learning ILSP DCVLR Competition Submission

arXiv:2610.19932v1 Announce Type: new Abstract: This work presents a compact and reproducible pipeline for synthesizing and curating image-based instruction-response pairs for DCVLR. We draw from the

Multimodal Medical Image Parsing with Task-Specific Low-Rank Adapters

arXiv:2610.19901v1 Announce Type: new Abstract: Adapting a single vision-language model to medical image parsing requires training choices that reflect the demands of each task. We study task-specific

RetailAgent: Structured Adverse Timing in Self-Conditioned Multimodal LLM Trading Agents

arXiv:2608.28399v2 Announce Type: replace Abstract: In financial markets, a sequential policy that reacts systematically to price movements may become predictable to other market participants. This pa

safetyarxiv-cs-ai

Efficient Hybrid Hard--Soft Signal Temporal Logic MPC for Driving under Multimodal Uncertainty

arXiv:2610.15934v1 Announce Type: new Abstract: Motion planning for automated vehicles in interactive traffic has to simultaneously ensure safety, respect traffic rules, and maintain comfortable progr

safetyarxiv-cs-ro

Multimodal Fusion of RGB and Complementary Modalities for Semantic Segmentation

arXiv:2610.17800v1 Announce Type: new Abstract: Multi-modal semantic segmentation adds an auxiliary sensing stream to RGB images (RGB-X), such as thermal, LiDAR, depth, event, polarization, or light f

Multimodal Sensor-Gated Mid-Air Launch and Vision-Based Recovery of a Child UAV on a Hovering Mothership Drone

arXiv:2610.15740v1 Announce Type: new Abstract: Launching a micro unmanned aerial vehicle (UAV) from a hovering carrier and recovering it in flight requires reliable confirmation that the child is sec

agentsarxiv-cs-ro

Parameter-Efficient Multimodal Brain Tumor Segmentation With Stage-Wise Shared Adapters

arXiv:2610.18571v1 Announce Type: new Abstract: Different MRI modalities reveal different parts of a brain tumor. Segmentation models combine information from these modalities to outline tumor regions

Decision 3.0: open, multimodal decision models from 0.6B to 27B. Ranks #1 on both Jev Decision Index boards

model-releasesr-localllama

With the vLLM Semantic Router team we released Decision 3.0 (d3), our new SOTA decision model family ranging from 0.6B to 27B, Apache-2.0, all reading text, images and video. How it works - Built on Q

12 Oct 2026

Model-free Bayesian Clustering and Multimodal Posterior Summarisation

arXiv:2610.13397v1 Announce Type: cross Abstract: We develop a model-free, noise-aware Bayesian clustering framework that constructs a martingale posterior over clusterings from a predictive Bayesian

researcharxiv-cs-lg

12 Oct 2026

DataVista: Diagnosing Multimodal LLMs on Data Video Understanding

arXiv:2610.11993v1 Announce Type: cross Abstract: Data video is a media form that integrates data visualization with video narrative, widely adopted in news reporting and business analysis. Compared w

Deflating the Hessian: Rank-4 W4A4 Quantization for Multimodal Diffusion Transformers

arXiv:2610.11315v1 Announce Type: cross Abstract: In diffusion transformers, low-rank branches can mitigate 4-bit weight--activation (W4A4) post-training quantization (PTQ) loss by decomposing each we

Applications

EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams

arXiv:2610.12248v1 Announce Type: new Abstract: Wearable augmented reality (AR) assistants are moving toward continuous real-world interaction, where they perceive the user's activity through first-pe

applicationsarxiv-cs-cl

OneSearch-VL: Unified Multimodal Deep Research Agent for Image and Video

arXiv:2610.12419v1 Announce Type: new Abstract: Single-image, multi-image, and video deep research require different visual operations but share a workflow of visual grounding, external retrieval, and

agentsarxiv-cs-cv

Immiscible Diffusion Policy: Preserving Multimodal Robot Actions through Label-Free Noise Assignment

arXiv:2610.09369v1 Announce Type: cross Abstract: When diffusion policies were first introduced, they were expected to recover multi-modal action distributions. However, we find this expectation does

safetyarxiv-cs-lg

Tutorials

Multimodal LLMs Can Learn to Read Brain Signals: A Vision--Language Model for Unified Multi-Task EEG Decoding

arXiv:2610.09355v1 Announce Type: cross Abstract: Learning EEG representations that generalize across cognitive tasks, subjects, and recording conditions remains a key challenge in electroencephalogra

tutorialsarxiv-cs-cv

PrismSSL: One Interface, Many Modalities; A Single-Interface Library for Multimodal Self-Supervised Learning

model-releasesarxiv-cs-lg

arXiv:2511.17776v2 Announce Type: replace Abstract: We present PrismSSL, a Python library that unifies state-of-the-art self-supervised learning (SSL) methods across audio, vision, graphs, and cross-m

RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments

arXiv:2610.10409v1 Announce Type: cross Abstract: General-purpose agents increasingly write code, use tools, and complete complex digital tasks, raising the question of how far these capabilities carr

researcharxiv-cs-lg

BrainTRACE: Tracing Longitudinal, Multimodal, and Volumetric Evidence in Brain MRI Clinical Reasoning

arXiv:2610.06571v1 Announce Type: new Abstract: Brain MRI interpretation is a longitudinal clinical reasoning problem: radiologists compare serial studies, integrate information across MRI sequences,

6 Oct 2026

Conditional Flow Matching for Single-Neuron Electrophysiology: Capturing Multimodal Responses Across Stimuli

arXiv:2610.06520v1 Announce Type: new Abstract: Neurons of the brain exhibit a rich repertoire of electrophysiology dynamics with the same repeated stimulus eliciting very different voltage responses

researcharxiv-cs-lg

6 Oct 2026

v0.40.0-rc6: model: add multimodal embeddings (#18820)

Implements the EmbeddingGemma2Model architecture on the MLX runner: 24-layer bidirectional text encoder with PLE, shared gemma4 vision/audio towers, mean-pool + L2 output. /api/embed accepts per-item

local-aiollama-releases

6 Oct 2026

Align Then Reason: A Multimodal Lip-Sync Judge for Dubbing

arXiv:2610.00825v1 Announce Type: new Abstract: Dubbing quality control requires a reference-free judge that can determine whether a candidate text line matches a speaker's visible articulation in bot

2 Oct 2026

Same Reward, Different Skills: When Multimodal RL Learns to Look

model-releasesarxiv-cs-lg

arXiv:2610.01908v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) improves vision-language benchmark scores even without visual information during training. With im

2 Oct 2026

Spherical Interpolation for Backward-Compatible Multimodal Representations

arXiv:2609.39836v1 Announce Type: new Abstract: Contrastive vision-language models map visual and textual representations into a shared normalized embedding space, making cosine similarity the natural

local-aiarxiv-cs-cv

1 Oct 2026

FedSocket: Recipient-Executable Knowledge Exchange for Heterogeneous Multimodal Federated Learning

arXiv:2609.37582v1 Announce Type: cross Abstract: Federated knowledge must remain usable by recipients with different modalities, private architectures, and tasks. We present FedSocket, which makes re

local-aiarxiv-cs-ai

Tutorials

Multimodal Detection of Higher-Order Behavioral Constructs: Self-Compassion in Structured Reflective Interaction

arXiv:2609.37148v1 Announce Type: cross Abstract: Many of the qualities that matter most in how people learn and grow, how someone regulates their emotions, reflects on a setback, or stays aware of ot

tutorialsarxiv-cs-cl

OmniVCBench: Benchmarking Evidence-Grounded Multimodal Reasoning Towards AI Virtual Cells

arXiv:2609.37773v1 Announce Type: new Abstract: Artificial Intelligence Virtual Cells (AIVCs) are envisioned as scientific agents that simulate cellular responses, explain underlying mechanisms, and s

Salt++: Context-Aligned Post-Training for Few-Step Streaming Multimodal Generation

arXiv:2609.36995v1 Announce Type: cross Abstract: Few-step streaming audio--video generation requires both causal modeling and step distillation, yet standard training recipes face two context-related

safetyarxiv-cs-ai

VISTA: Value-Informed Event Appraisal for Multimodal Emotion Conflict

arXiv:2609.37324v1 Announce Type: new Abstract: Conflicting emotional cues can be individually valid: a subdued voice may reflect a blocked goal while a smile satisfies a social obligation. Their inte

researcharxiv-cs-ai

AMBIT: Anticipatory Multimodal Body Recruitment for Bimanual Tracking on a Humanoid

arXiv:2609.33484v1 Announce Type: new Abstract: A humanoid with 5-DoF arms cannot track generic bimanual end-effector trajectories with its arms alone; pelvis and waist motion must be recruited, but w

researcharxiv-cs-ro

29 Sept 2026

Refinement Symmetry in Multimodal Transformers

arXiv:2609.32669v1 Announce Type: cross Abstract: Attention weights depend on token counts, which change with the representation of a signal. We study refinement symmetry: splitting a representation w

29 Sept 2026

The Text Beside the Image: Detection, Utility and Leakage for Trustworthy Multimodal Medical Data and Beyond

arXiv:2609.33280v1 Announce Type: cross Abstract: Medical images are released with the reports that describe them, and protecting the image does not protect the report. This paper measures the text co

29 Sept 2026

DepthEvidence: Unifying Metric Depth Prediction and Geometric Reasoning in Multimodal Language Models

arXiv:2609.31103v1 Announce Type: cross Abstract: Spatial reasoning with metric constraints requires linking objects to geometric measurements and preserving their numerical content during language re

28 Sept 2026

Tutorials

Accelerate multimodal RL training with SkyRL on Amazon SageMaker HyperPod

Learn how to run SkyRL, an open-source reinforcement learning framework, on Amazon SageMaker HyperPod to post-train a Qwen3-VL-8B vision-language model with GRPO. This walkthrough covers building the

tutorialsaws-ml-blog

Multimodal Routing and Region Refinement for Language-Guided Medical Image Segmentation

arXiv:2609.28860v1 Announce Type: new Abstract: Textual descriptions can reduce ambiguity in medical image segmentation by specifying the finding and location to be delineated. Existing text-guided me

Multimodal Thinking with Renderable Programs

model-releasesarxiv-cs-cl

arXiv:2609.30130v1 Announce Type: cross Abstract: Current vision-language models (VLMs) excel at visual content understanding and text-based reasoning, yet their structure limits the advancement of in

OncoVision: Integrating Mammography and Clinical Data through Attention-Driven Multimodal AI for Enhanced Breast Cancer Diagnosis

arXiv:2511.19667v2 Announce Type: replace Abstract: OncoVision is a privileged-information training framework that uses mammography images and clinical features during training and performs inference

WebArxiv: A Reproducible Benchmark for Evaluating Multimodal Web Agents on arXiv Tasks

arXiv:2507.00938v4 Announce Type: replace-cross Abstract: Foundation models now enable autonomous agents to interact with real-world websites, but existing benchmarks emphasize general-purpose browsin