Search: “multimodal”

MPerS: Dynamic MLLM MixExperts Perception-Guided Remote Sensing Scene Segmentation

arXiv:2605.10769v1 Announce Type: cross Abstract: The multimodal fusion of images and scene captions has been extensively explored and applied in various fields. However, when dealing with complex rem

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

PPU-Bench:Real World Benchmark for Personalized Partial Unlearning in Vision Language Models

DGX agent

arXiv:2605.08800v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) may memorize sensitive cross-modal information during pretraining. However, existing MLLM unlearning benchmar

model-releasesarxiv-cs-ai

12 May 2026

Applications

Rethinking Gating Mechanism in Sparse MoE: Handling Arbitrary Modality Inputs with Confidence-Guided Gate

arXiv:2505.19525v3 Announce Type: replace-cross Abstract: Effectively managing missing modalities is a fundamental challenge in real-world multimodal learning scenarios, where data incompleteness ofte

applicationsarxiv-cs-ai

SleepWalk: A Three-Tier Benchmark for Stress-Testing Instruction-Guided Vision-Language Navigation

arXiv:2605.10376v1 Announce Type: new Abstract: Vision-Language Models (VLMs) have advanced rapidly in multimodal perception and language understanding, yet it remains unclear whether they can reliabl

Spatial Priming Outperforms Semantic Prompting: A Grid-Based Approach to Improving LLM Accuracy on Chart Data Extraction

arXiv:2605.08220v1 Announce Type: new Abstract: The automated extraction of data from scientific charts is a critical task for large-scale literature analysis. While multimodal Large Language Models (

researcharxiv-cs-ai

Masks Can Talk: Extracting Structured Text Information from Single-Modal Images for Remote Sensing Change Detection

arXiv:2605.07178v1 Announce Type: new Abstract: Remote sensing change detection is pivotal for urban monitoring, disaster assessment, and environmental resource management. Yet, unimodal deep learning

11 May 2026

Pan-FM: A Pan-Organ Foundation Model with Saliency-Guided Masking for Missing Robustness

arXiv:2605.07055v1 Announce Type: cross Abstract: Foundation models (FMs) have shown great promise in medical imaging, but most FMs are trained on unimodal data within isolated domains, such as brain

safetyarxiv-cs-ai

11 May 2026

VIMCAN: Visual-Inertial 3D Human Pose Estimation with Hybrid Mamba-Cross-Attention Network

arXiv:2605.07552v1 Announce Type: new Abstract: The rapid advances in deep learning have significantly enhanced the accuracy of multimodal 3D human pose estimation (HPE). However, the state-of-the-art

11 May 2026

ReasonAudio: A Benchmark for Evaluating Reasoning Beyond Matching in Text-Audio Retrieval

arXiv:2605.03361v2 Announce Type: new Abstract: As multimodal content continues to expand at a rapid pace, audio retrieval has emerged as a key enabling technology for media search, content organizati

7 May 2026

Vision-EKIPL: External Knowledge-Infused Policy Learning for Visual Reasoning

arXiv:2506.06856v3 Announce Type: replace Abstract: Visual reasoning is crucial for understanding complex multimodal data and advancing Artificial General Intelligence. Existing methods enhance the re

7 May 2026

Chart-FR1: Visual Focus-Driven Fine-Grained Reasoning on Dense Charts

arXiv:2605.01882v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have shown considerable potential in chart understanding and reasoning tasks. However, they still struggle with

5 May 2026

IPS: In-Prompt Process Supervision for Short Video Content Moderation

arXiv:2412.15251v3 Announce Type: replace Abstract: Multimodal large language models (MLLMs) are effective at capturing the semantics of short video content; however, they often fail to attend to the

5 May 2026

VGR: Visual Grounded Reasoning

arXiv:2506.11991v3 Announce Type: replace-cross Abstract: In the field of multimodal chain-of-thought (CoT) reasoning, existing approaches predominantly rely on reasoning on pure language space, which

4 May 2026

Judge, Then Drive: A Critic-Centric Vision Language Action Framework for Autonomous Driving

arXiv:2604.27366v1 Announce Type: new Abstract: Recent advances in vision language action (VLA) models have shown remarkable potential for autonomous driving by directly mapping multimodal inputs to c

1 May 2026

ReGATE: Learning Faster and Better with Fewer Tokens in MLLMs

arXiv:2507.21420v3 Announce Type: replace-cross Abstract: The computational cost of training multimodal large language models (MLLMs) grows rapidly with the number of processed tokens. Existing effici

researcharxiv-cs-cl

30 Apr 2026

MMLANDMARKS: a Cross-View Instance-Level Benchmark for Geo-Spatial Understanding

arXiv:2512.17492v2 Announce Type: replace Abstract: Geo-spatial analysis of our world benefits from a multimodal approach, as every single geographic location can be described in numerous ways (images

29 Apr 2026

Tutorials

Towards Seamless Lunar Mosaics: Deep Radiometric Normalization for Cross-Sensor Orbital Imagery Using Chandrayaan-2 TMC Data

arXiv:2604.25208v1 Announce Type: new Abstract: Radiometric inconsistencies remain a major challenge in generating seamless lunar mosaics from multi-mission orbital imagery due to variability in illum

tutorialsarxiv-cs-cv

29 Apr 2026

CrossGuard: Safeguarding MLLMs against Joint-Modal Implicit Malicious Attacks

arXiv:2510.17687v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) achieve strong reasoning and perception capabilities but are increasingly vulnerable to jailbreak att

LinguDistill: Recovering Linguistic Ability in Vision-Language Models via Selective Cross-Modal Distillation

arXiv:2604.00829v3 Announce Type: replace-cross Abstract: Adapting pretrained language models (LMs) into vision-language models (VLMs) can degrade their native linguistic capability due to representat

Local Ai

Meta is about to release a pixel space model (Tuna-2)

local-air-stablediffusion

Tuna-2 is a unified multimodal model that performs visual understanding and generation directly based on pixel embeddings, employing simple patch embedding layers to encode visual input without a VAE

Nemotron-3-Nano-Omni-30B-A3B-Reasoning, New model?

model-releasesr-localllama

NVIDIA Nemotron 3 Nano Omni is a multimodal large language model that unifies video, audio, image, and text understanding for enterprise Q&A, summarization, transcription, and document intelligence, w

SIV-Bench: A Video Benchmark for Social Interaction Understanding and Reasoning

arXiv:2506.05425v2 Announce Type: replace-cross Abstract: Understanding social interaction, which encompasses perceiving numerous and subtle multimodal cues, inferring unobservable mental states and r

CNSL-bench: Benchmarking the Sign Language Understanding Capabilities of MLLMs on Chinese National Sign Language

arXiv:2604.22367v1 Announce Type: cross Abstract: Sign language research has achieved significant progress due to the advances in large language models (LLMs). However, the intrinsic ability of LLMs t

Agents

Source-Modality Monitoring in Vision-Language Models

arXiv:2604.22038v1 Announce Type: new Abstract: We define and investigate source-modality monitoring -- the ability of multimodal models to track and communicate the input source from which pieces of

agentsarxiv-cs-cl

Tutorials

Structure-Guided Diffusion Model for EEG-Based Visual Cognition Reconstruction

arXiv:2604.22649v1 Announce Type: cross Abstract: Objective: Decoding visual information from electroencephalography (EEG) is an important problem in neuroscience and brain-computer interface (BCI) re

tutorialsarxiv-cs-cv

System-Mediated Attention Imbalances Make Vision-Language Models Say Yes

arXiv:2601.12430v2 Announce Type: replace Abstract: Vision-language model (VLM) hallucination is commonly linked to imbalanced allocation of attention across input modalities: system, image and text.

Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities

arXiv:2604.22177v1 Announce Type: new Abstract: Multimodal MRI offers complementary information for brain tumor segmentation, but clinical scans often lack one or more modalities, which degrades segme

ARFBench: Benchmarking Time Series Question Answering Ability for Software Incident Response

arXiv:2604.21199v1 Announce Type: cross Abstract: Time series question-answering (TSQA), in which we ask natural language questions to infer and reason about properties of time series, is a promising

24 Apr 2026

R-CoV: Region-Aware Chain-of-Verification for Alleviating Object Hallucinations in LVLMs

arXiv:2604.20696v1 Announce Type: new Abstract: Large vision-language models (LVLMs) have demonstrated impressive performance in various multimodal understanding and reasoning tasks. However, they sti

23 Apr 2026

Tutorials

Render-in-the-Loop: Vector Graphics Generation via Visual Self-Feedback

arXiv:2604.20730v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) have shown promising capabilities in generating Scalable Vector Graphics (SVG) via direct code synthesis. Howev

tutorialsarxiv-cs-cv

23 Apr 2026

Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs

arXiv:2512.08923v2 Announce Type: replace Abstract: We introduce two new benchmarks REST and REST+ (Render-Equivalence Stress Tests) to enable systematic evaluation of cross-modal inconsistency in mul

researcharxiv-cs-ai

23 Apr 2026

AD-Copilot: A Vision-Language Assistant for Industrial Anomaly Detection via Visual In-context Comparison

arXiv:2603.13779v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) have achieved impressive success in natural visual understanding, yet they consistently underperform

22 Apr 2026

Air-Know: Arbiter-Calibrated Knowledge-Internalizing Robust Network for Composed Image Retrieval

arXiv:2604.19386v1 Announce Type: new Abstract: Composed Image Retrieval (CIR) has attracted significant attention due to its flexible multimodal query method, yet its development is severely constrai

22 Apr 2026

ORCA: An Agentic Reasoning Framework for Hallucination and Adversarial Robustness in Vision-Language Models

arXiv:2509.15435v2 Announce Type: replace-cross Abstract: Large Vision-Language Models (LVLMs) exhibit strong multimodal capabilities but remain vulnerable to hallucinations from intrinsic errors and

22 Apr 2026

EmoVerse: A MLLMs-Driven Emotion Representation Dataset for Interpretable Visual Emotion Analysis

arXiv:2511.12554v2 Announce Type: replace Abstract: Visual Emotion Analysis (VEA) aims to bridge the affective gap between visual content and human emotional responses. Despite its promise, progress i

IMA-MoE: An Interpretable Modality-Aware Mixture-of-Experts Framework for Characterizing the Neurobiological Signatures of Binge Eating Disorder

arXiv:2604.17028v1 Announce Type: new Abstract: Binge eating disorder (BED) is the most prevalent eating disorder. However, current diagnostic frameworks remain largely grounded in symptom-based crite

MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge

model-releasesarxiv-cs-cl

arXiv:2604.18164v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) have been increasingly used as automatic evaluators-a paradigm known as MLLM-as-a-Judge. However, their reliabi

ORSIFlow: Saliency-Guided Rectified Flow for Optical Remote Sensing Salient Object Detection

arXiv:2603.28584v2 Announce Type: replace Abstract: Optical Remote Sensing Image Salient Object Detection (ORSI-SOD) remains challenging due to complex backgrounds, low contrast, irregular object shap

Local Ai

Topology-Aware Layer Pruning for Large Vision-Language Models

arXiv:2604.16502v1 Announce Type: new Abstract: Large Language Models (LLMs) have demonstrated strong capabilities in natural language understanding and reasoning, while recent extensions that incorpo

local-aiarxiv-cs-cv

UniCon: Unified Framework for Efficient Contrastive Alignment via Kernels

arXiv:2604.16678v1 Announce Type: new Abstract: Contrastive objectives power state-of-the-art multimodal models, but their training remains slow, relying on long stochastic optimization. We propose a

safetyarxiv-cs-lg

Weakly-Supervised Referring Video Object Segmentation through Text Supervision

arXiv:2604.17797v1 Announce Type: new Abstract: Referring video object segmentation (RVOS) aims to segment the target instance in a video, referred by a text expression. Conventional approaches are mo

Find, Fix, Reason: Context Repair for Video Reasoning

arXiv:2604.16243v1 Announce Type: new Abstract: Reinforcement learning has advanced video reasoning in large multi-modal models, yet dominant pipelines either rely on on-policy self-exploration, which

20 Apr 2026

Tools

Building a Fast Multilingual OCR Model with Synthetic Data

This article describes techniques for developing an efficient optical character recognition (OCR) model capable of processing multiple languages, leveraging synthetic data generation to reduce annotat

toolshugging-face

Fall into a Pit, Gain in a Wit: Cognitive-Guided Harmful Meme Detection via Misjudgment Risk Pattern Retrieval

model-releasesarxiv-cs-lg

arXiv:2510.15946v3 Announce Type: replace Abstract: Internet memes have emerged as a popular multimodal medium, yet they are increasingly weaponized to convey harmful opinions through subtle rhetorica

IROSA: Interactive Robot Skill Adaptation using Natural Language

arXiv:2603.03897v3 Announce Type: replace-cross Abstract: Foundation models have demonstrated impressive capabilities across diverse domains, while imitation learning provides principled methods for r

Local Ai

One RL to See Them All: Visual Triple Unified Reinforcement Learning

arXiv:2505.18129v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) is becoming an important direction for post-training vision-language models (VLMs), but public training methodolog

local-aiarxiv-cs-cl

CausalDisenSeg: A Causality-Guided Disentanglement Framework with Counterfactual Reasoning for Robust Brain Tumor Segmentation Under Missing Modalities

arXiv:2604.13409v1 Announce Type: new Abstract: In clinical practice, the robustness of deep learning models for multimodal brain tumor segmentation is severely compromised by incomplete MRI data. Thi

16 Apr 2026

Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA

arXiv:2604.13731v1 Announce Type: new Abstract: Multi-page Document Visual Question Answering requires reasoning over semantics, layouts, and visual elements in long, visually dense documents. Existin