Search: “multimodal”

RL makes MLLMs see better than SFT

arXiv:2510.16333v2 Announce Type: replace Abstract: A dominant assumption in Multimodal Language Model (MLLM) research is that its performance is largely inherited from the LLM backbone, given its imm

Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Grounding

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

DGX agent

arXiv:2604.11244v1 Announce Type: new Abstract: Advances in Multimodal Large Language Models (MLLMs) are transforming video captioning from a descriptive endpoint into a semantic interface for both vi

model-releasesarxiv-cs-cv

14 Apr 2026

Semantic-Geometric Dual Compression: Training-Free Visual Token Reduction for Ultra-High-Resolution Remote Sensing Understanding

arXiv:2604.11122v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have demonstrated immense potential in Earth observation. However, the massive visual tokens generated when p

Sign Language Recognition in the Age of LLMs

model-releasesarxiv-cs-cl

arXiv:2604.11225v1 Announce Type: cross Abstract: Recent Vision Language Models (VLMs) have demonstrated strong performance across a wide range of multimodal reasoning tasks. This raises the question

Tutorials

TacMan-Turbo: Proactive Tactile Control for Robust and Efficient Articulated Object Manipulation

arXiv:2508.02204v4 Announce Type: replace Abstract: Adept manipulation of articulated objects is essential for robots to operate successfully in human environments. Such manipulation requires both eff

tutorialsarxiv-cs-ro

UNIGEOCLIP: Unified Geospatial Contrastive Learning

arXiv:2604.11668v1 Announce Type: new Abstract: The growing availability of co-located geospatial data spanning aerial imagery, street-level views, elevation models, text, and geographic coordinates o

VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments

arXiv:2506.02387v3 Announce Type: replace Abstract: Recent advancements in Vision Language Models (VLMs) have expanded their capabilities to interactive agent tasks, yet existing benchmarks remain lim

Enhancing the Safety of Medical Vision-Language Models by Synthetic Demonstrations

arXiv:2506.09067v2 Announce Type: replace-cross Abstract: Generative medical vision-language models~(Med-VLMs) are primarily designed to generate complex textual information~(e.g., diagnostic reports)

Growing a Multi-head Twig via Distillation and Reinforcement Learning to Accelerate Large Vision-Language Models

arXiv:2503.14075v3 Announce Type: replace-cross Abstract: Large vision-language models (VLMs) have demonstrated remarkable capabilities in open-world multimodal understanding, yet their high computati

researcharxiv-cs-cl

Learning Vision-Language-Action World Models for Autonomous Driving

arXiv:2604.09059v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models have recently achieved notable progress in end-to-end autonomous driving by integrating perception, reasoning, and

Tutorials

Physically Grounded 3D Generative Reconstruction under Hand Occlusion using Proprioception and Multi-Contact Touch

arXiv:2604.09100v1 Announce Type: new Abstract: We propose a multimodal, physically grounded approach for metric-scale amodal object reconstruction and pose estimation under severe hand occlusion. Unl

tutorialsarxiv-cs-cv

PinpointQA: A Dataset and Benchmark for Small Object-Centric Spatial Understanding in Indoor Videos

arXiv:2604.08991v1 Announce Type: cross Abstract: Small object-centric spatial understanding in indoor videos remains a significant challenge for multimodal large language models (MLLMs), despite its

Streaming Video Instruction Tuning

arXiv:2512.21334v2 Announce Type: replace Abstract: We present Streamo, a real-time streaming video LLM that serves as a general-purpose interactive assistant. Unlike existing online video models that

researcharxiv-cs-cv

Tora3: Trajectory-Guided Audio-Video Generation with Physical Coherence

arXiv:2604.09057v1 Announce Type: new Abstract: Audio-video (AV) generation has recently made strong progress in perceptual quality and multimodal coherence, yet generating content with plausible moti

Asking like Socrates: Socrates helps VLMs understand remote sensing images

arXiv:2511.22396v2 Announce Type: replace-cross Abstract: Recent multimodal reasoning models, inspired by DeepSeek-R1, have significantly advanced vision-language systems. However, in remote sensing (

DiffVC: A Non-autoregressive Framework Based on Diffusion Model for Video Captioning

arXiv:2604.08084v1 Announce Type: new Abstract: Current video captioning methods usually use an encoder-decoder structure to generate text autoregressively. However, autoregressive methods have inhere

researcharxiv-cs-cv

Agents

MedRoute: RL-Based Dynamic Specialist Routing in Multi-Agent Medical Diagnosis

arXiv:2604.06180v1 Announce Type: cross Abstract: Medical diagnosis using Large Multimodal Models (LMMs) has gained increasing attention due to capability of these models in providing precise diagnose

agentsarxiv-cs-lg

PEER: Unified Process-Outcome Reinforcement Learning for Structured Empathetic Reasoning

arXiv:2508.09521v2 Announce Type: replace Abstract: Emotional support conversations require more than fluent responses. Supporters need to understand the seeker's situation and emotions, adopt an appr

safetyarxiv-cs-cl

Semantic Noise Reduction via Teacher-Guided Dual-Path Audio-Visual Representation Learning

arXiv:2604.08147v1 Announce Type: cross Abstract: Recent advances in audio-visual representation learning have shown the value of combining contrastive alignment with masked reconstruction. However, j

Agents

UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding

arXiv:2507.22025v4 Announce Type: replace Abstract: The emergence of Multimodal Large Language Models (MLLMs) has driven significant advances in Graphical User Interface (GUI) agent capabilities. Neve

agentsarxiv-cs-ai

UniversalVTG: A Universal and Lightweight Foundation Model for Video Temporal Grounding

arXiv:2604.08522v1 Announce Type: new Abstract: Video temporal grounding (VTG) is typically tackled with dataset-specific models that transfer poorly across domains and query styles. Recent efforts to

APRiL: An Open-ended Benchmark for Chinese Social Awareness in Large Language Models

model-releasesarxiv-cs-cl

arXiv:2610.20785v1 Announce Type: new Abstract: Large language models are increasingly used in social interactions, where appropriate responses require recognizing implicit cues, inferring mental stat

Cross-Embodiment Causal Transfer from Human Videos to Robot Manipulation

arXiv:2610.21798v1 Announce Type: cross Abstract: Scaling robot manipulation models requires large amounts of interaction data, yet collecting diverse real-robot experience remains costly. Human manip

local-aiarxiv-cs-ai

MOSS: Mode-Oriented Source Shaping for Robot Flow Matching Policies

arXiv:2610.22193v1 Announce Type: new Abstract: In robotic manipulation, the same observation often admits multiple feasible actions. Generative policies such as diffusion and flow matching naturally

local-aiarxiv-cs-ro

OmicsBench: a long-horizon multi-turn benchmark of AI agents on research-study-scale bioinformatics

arXiv:2610.21074v1 Announce Type: new Abstract: Computational biologists increasingly delegate substantial analyses to AI agents in long-horizon, multi-turn sessions, yet most benchmarks of agents' bi

Optimal Hydra: Model Heterogeneous Federated Learning for LLMs using Data-Informed Hydra-Nets

arXiv:2610.20753v1 Announce Type: cross Abstract: Large language models (LLMs) trained on vast amounts of data have demonstrated impressive performance over a range of tasks, but future performance ga

local-aiarxiv-cs-ai

Twice Upon a Time: Complementary Representations for Time-Series Reasoning

arXiv:2610.22327v1 Announce Type: cross Abstract: Time Series Language Models (TSLMs) enable language-based understanding and reasoning over temporal data, yet a fundamental question remains: what inf

AesCode: Aesthetic Code Generation with Decoupled Cross-Modal Rewards

arXiv:2610.18860v1 Announce Type: new Abstract: Generating information-rich visual artifacts such as slides, posters, and dashboards as code is appealing because the outputs are structured, editable,

AlignAudio: A Unified Dual-Alignment Framework for Noise-Robust Speech-Conditioned Generation

arXiv:2610.14697v1 Announce Type: cross Abstract: Speech-conditioned generation maps speech signals to target audio outputs. Prior end-to-end speech-conditioned generative models achieve high fidelity

Applications

An Initial Study of Physiological Markers for Deepfake Speech Detection

arXiv:2610.18934v1 Announce Type: cross Abstract: Conventional deepfake speech detectors primarily target synthesis artifacts and often degrade under unseen attacks or recording conditions. Human spee

applicationsarxiv-cs-cv

Articulation Upper Bounds for Language Models

arXiv:2610.19721v1 Announce Type: cross Abstract: Language captures only a subset of the perceptions, intuitions and intents that form meaning between speakers. What, then, are the limits of language?

researcharxiv-cs-cl

Certifying Almost-Sure Reachability in Unknown MDPs

arXiv:2610.18119v1 Announce Type: cross Abstract: {em Specification-Guided Reinforcement Learning (RL)} has largely concerned itself with learning an optimal policy against a formal specification in a

Constructed regional photovoltaic generation fields improve ultra-short-term distributed solar forecasting

arXiv:2610.16110v1 Announce Type: new Abstract: Ultra-short-term forecasting of densely distributed photovoltaic (PV) systems remains challenging because external meteorological and satellite products

researcharxiv-cs-ai

Frame Evidence:A Benchmark for Reference Frames and Visual Evidence in Multi-Image Spatial Reasoning

arXiv:2610.19564v1 Announce Type: new Abstract: A spatial answer depends on the reference frame specified in language and its supporting visual evidence. Item accuracy alone obscures whether correctne

GenState-AI: State-Aware Dataset for Text-to-Video Retrieval on AI-Generated Videos

arXiv:2603.14426v2 Announce Type: replace Abstract: Existing text-to-video retrieval benchmarks often permit matching from static appearance, making it difficult to determine whether models capture st

Applications

Horizon Scans can be accelerated using novel information retrieval and artificial intelligence tools

arXiv:2504.01627v2 Announce Type: replace-cross Abstract: Introduction: Horizon scanning in healthcare assesses early signals of innovation, crucial for timely adoption. Current horizon scanning faces

applicationsarxiv-cs-ai

Agents

How Should a Camera-First AI Agent Work?

arXiv:2610.14772v2 Announce Type: new Abstract: Camera-first agents must act on the visual world as it unfolds. We argue that the central obstacle for future camera-first agents is not recognition abi

agentsarxiv-cs-ai

HyWe: Unifying Search and Operation for Real-World Hybrid Web Tasks

arXiv:2610.16319v1 Announce Type: new Abstract: Real-world hybrid web tasks need both information gathering and page interaction, yet search-based and operation-based agents are developed and evaluate

Imaging-anchored Multiomics in Cardiovascular Disease: Integrating Cardiac Imaging, Bulk, Single-cell, and Spatial Transcriptomics

arXiv:2601.07871v2 Announce Type: replace-cross Abstract: Cardiovascular disease arises from interactions between inherited risk, molecular programmes, and tissue-scale remodelling that are observed c

Hardware

Into the LidarVerse: A Real-Time Unified LiDAR World Model Across Tasks and Sensors

arXiv:2610.19201v1 Announce Type: new Abstract: We present LidarVerse, a real-time unified LiDAR world model for sensor simulation across tasks and sensors. Existing LiDAR simulators largely follow on

hardwarearxiv-cs-cv

LED-BENCH: A Benchmark for Large-format Engineering Drawing Understanding

arXiv:2610.19571v1 Announce Type: new Abstract: Real-world engineering drawings are often large-format and information-dense. Answering practical engineering questions therefore requires both fine-gra

Loose Lips: Understanding Privacy Leakage in Agentic Dialogue

arXiv:2610.14071v1 Announce Type: cross Abstract: As LLM agents are increasingly deployed in high-stakes enterprise settings, understanding how they handle sensitive information in agentic dialogue ha

Lost in Translation: Why Lower Bias Scores Don't Mean Fairer Models

arXiv:2610.20479v1 Announce Type: new Abstract: Bias audits of multilingual language models usually compare a model's average stereotype score in English with its average in another language and read

safetyarxiv-cs-cl

MakSafe: A Spatiotemporal Semantic Framework for Formal Specification and Compliance Checking of Road Vehicle Safety Rules

model-releasesarxiv-cs-cl

arXiv:2610.19735v1 Announce Type: new Abstract: Scenario-based validation is essential for automated road vehicles but cannot cover the full variability of openworld traffic. We present MakSafe, a for

MedHarnessBench: A Unified Benchmark for Healthcare Agent Evaluation

arXiv:2610.15714v1 Announce Type: new Abstract: Healthcare agents must transform longitudinal clinical evidence into reliable actions through heterogeneous tools and stateful workflows, conditions und

Mitigating Cross-Modal Hallucinations by Correcting Modality Misrouting in Omnimodal LLMs

arXiv:2610.18141v1 Announce Type: new Abstract: Omnimodal large language models (OLLMs) are susceptible to cross-modal hallucinations under audio-visual interference. We find that, for a substantial f

researcharxiv-cs-ai

MobiGuide: Vocabulary-Guided Online Recovery for Mobile Manipulation

arXiv:2604.01567v2 Announce Type: replace Abstract: A central challenge in mobile manipulation is preserving multiple plausible action modes while remaining reactive during execution. For example, gra

local-aiarxiv-cs-ro

Applications

Nudge the Judge: Uncovering Program Decision Hijacking in Real-World Jev Applications

arXiv:2610.18612v1 Announce Type: cross Abstract: System One models such as Jev change how software makes decisions by mapping open-ended semantic contexts into structured judgments that directly driv

applicationsarxiv-cs-ai