Search: “multimodal”

Composed Vision-Language Retrieval for Skin Cancer Case Search via Joint Alignment of Global and Local Representations

arXiv:2603.09108v2 Announce Type: replace Abstract: Medical image retrieval aims to identify clinically relevant lesion cases to support diagnostic decision making, education, and quality control. In

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping

DGX agent

arXiv:2510.09741v3 Announce Type: replace Abstract: Multimodal large language models (MLLMs) often miss small details and spatial relations in cluttered scenes, leading to errors in fine-grained perce

model-releasesarxiv-cs-cv

21 Apr 2026

EgoSound: Benchmarking Sound Understanding in Egocentric Videos

arXiv:2602.14122v2 Announce Type: replace Abstract: Multimodal Large Language Models (MLLMs) have recently achieved remarkable progress in vision-language understanding. Yet, human perception is inher

From Attribution to Abstention: Training-Free Attention-Based Auditing for Clinical Summarization

arXiv:2601.16397v2 Announce Type: replace Abstract: Deploying multimodal large language models (MLLMs) for clinical summarization demands not only fluent generation but also transparency about where e

researcharxiv-cs-cl

From Handwriting to Structured Data: Benchmarking AI Digitisation of Handwritten Forms

arXiv:2604.16504v1 Announce Type: new Abstract: Manual digitisation of structured handwritten documents is slow and costly. We benchmark 17 leading frontier multi-modal large language models and open-

GaLa: Hypergraph-Guided Visual Language Models for Procedural Planning

arXiv:2604.17241v1 Announce Type: new Abstract: Implicit spatial relations and deep semantic structures encoded in object attributes are crucial for procedural planning in embodied AI systems. However

researcharxiv-cs-ro

HABIT: Chrono-Synergia Robust Progressive Learning Framework for Composed Image Retrieval

arXiv:2604.18037v1 Announce Type: new Abstract: Composed Image Retrieval (CIR) is a flexible image retrieval paradigm that enables users to accurately locate the target image through a multimodal quer

How Much Data is Enough? The Zeta Law of Discoverability in Biomedical Data, featuring the enigmatic Riemann zeta function

arXiv:2604.17581v1 Announce Type: new Abstract: How much data is enough to make a scientific discovery? As biomedical datasets scale to millions of samples and AI models grow in capacity, progress inc

researcharxiv-cs-lg

INTENT: Invariance and Discrimination-aware Noise Mitigation for Robust Composed Image Retrieval

arXiv:2604.18051v1 Announce Type: new Abstract: Composed Image Retrieval (CIR) is a challenging image retrieval paradigm that enables to retrieve target images based on multimodal queries consisting o

MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problems

arXiv:2503.16549v2 Announce Type: replace Abstract: Despite strong results on many tasks, multimodal large language models (MLLMs) still underperform on visual mathematical problem solving, especially

PBSBench: A Multi-Level Vision-Language Framework and Benchmark for Hematopathology Whole Slide Image Interpretation

arXiv:2604.17570v1 Announce Type: new Abstract: Peripheral Blood Smear (PBS) is a critical microscopic examination in hematopathology that yields whole-slide imaging (WSI). Unlike solid tissue patholo

SaFeR-Steer: Evolving Multi-Turn MLLMs via Synthetic Bootstrapping and Feedback Dynamics

arXiv:2604.16358v1 Announce Type: cross Abstract: MLLMs are increasingly deployed in multi-turn settings, where attackers can escalate unsafe intent through the evolving visual-text history and exploi

safetyarxiv-cs-cl

SpatialImaginer: Towards Adaptive Visual Imagination for Spatial Reasoning

arXiv:2604.17385v1 Announce Type: new Abstract: Spatial intelligence, which refers to the ability to reason about geometric and physical structure from visual observations, remains a core challenge fo

TextTIGER: Text-based Intelligent Generation with Entity Prompt Refinement for Text-to-Image Generation

arXiv:2504.18269v2 Announce Type: replace Abstract: When generating images from prompts that include specific entities, the model must retain as much entity-specific knowledge as possible. However, th

researcharxiv-cs-cl

Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training

arXiv:2603.23885v3 Announce Type: replace Abstract: Document parsing has recently advanced with multimodal large language models (MLLMs) that directly map document images to structured outputs. Tradit

VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysis

arXiv:2509.16538v3 Announce Type: replace-cross Abstract: We propose VC-Inspector, a lightweight, open-source large multimodal model (LMM) for reference-free evaluation of video captions, with a focus

researcharxiv-cs-cl

VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction

model-releasesarxiv-cs-cl

arXiv:2505.20279v4 Announce Type: replace-cross Abstract: The rapid advancement of Large Multimodal Models (LMMs) for 2D images and videos has motivated extending these models to understand 3D scenes,

Waking Up Blind: Cold-Start Optimization of Supervision-Free Agentic Trajectories for Grounded Visual Perception

arXiv:2604.17475v1 Announce Type: cross Abstract: Small Vision-Language Models (SVLMs) are efficient task controllers but often suffer from visual brittleness and poor tool orchestration. They typical

safetyarxiv-cs-cl

When Text Hijacks Vision: Benchmarking and Mitigating Text Overlay-Induced Hallucination in Vision Language Models

arXiv:2604.17375v1 Announce Type: new Abstract: Recent advances in Vision-Language Models (VLMs) have substantially enhanced their ability across multimodal video understanding benchmarks spanning tem

CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval

model-releasesarxiv-cs-ai

arXiv:2604.15663v1 Announce Type: cross Abstract: Code search, framed as information retrieval (IR), underpins modern software engineering and increasingly powers retrieval-augmented generation (RAG),

Tutorials

Cross-modal learning for plankton recognition

arXiv:2603.16427v2 Announce Type: replace Abstract: This paper considers self-supervised cross-modal coordination as a strategy enabling utilization of multiple modalities and large volumes of unlabel

tutorialsarxiv-cs-cv

FineSteer: A Unified Framework for Fine-Grained Inference-Time Steering in Large Language Models

arXiv:2604.15488v1 Announce Type: cross Abstract: Large language models (LLMs) often exhibit undesirable behaviors, such as safety violations and hallucinations. Although inference-time steering offer

safetyarxiv-cs-ai

Hierarchical Codec Diffusion for Video-to-Speech Generation

arXiv:2604.15923v1 Announce Type: cross Abstract: Video-to-Speech (VTS) generation aims to synthesize speech from a silent video without auditory signals. However, existing VTS methods disregard the h

Agents

Integrating Graphs, Large Language Models, and Agents: Reasoning and Retrieval

arXiv:2604.15951v1 Announce Type: new Abstract: Generative AI, particularly Large Language Models, increasingly integrates graph-based representations to enhance reasoning, retrieval, and structured d

agentsarxiv-cs-ai

Applications

MRGEN: A Conceptual Framework for LLM-Powered Mixed Reality Authoring Tools for Education

arXiv:2604.15341v1 Announce Type: cross Abstract: Mixed Reality (MR) offers immersive and multimodal opportunities for education but remains difficult for teachers to author without technical expertis

applicationsarxiv-cs-ai

ReactBench: A Benchmark for Topological Reasoning in MLLMs on Chemical Reaction Diagrams

model-releasesarxiv-cs-ai

arXiv:2604.15994v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) excel at recognizing individual visual elements and reasoning over simple linear diagrams. However, when faced

SIMMER: Cross-Modal Food Image--Recipe Retrieval via MLLM-Based Embedding

arXiv:2604.15628v1 Announce Type: cross Abstract: Cross-modal retrieval between food images and recipe texts is an important task with applications in nutritional management, dietary logging, and cook

safetyarxiv-cs-cl

VIB-Probe: Detecting and Mitigating Hallucinations in Vision-Language Models via Variational Information Bottleneck

arXiv:2601.05547v2 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) have demonstrated remarkable progress in multimodal tasks, but remain susceptible to hallucinations, where gener

researcharxiv-cs-ai

Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Tools

arXiv:2510.08480v2 Announce Type: replace Abstract: Multimodal large language models (MLLMs) have demonstrated remarkable potential in bridging visual and textual reasoning, yet their reliance on text

Watching Movies Like a Human: Egocentric Emotion Understanding for Embodied Companions

arXiv:2604.15823v1 Announce Type: new Abstract: Embodied robotic agents often perceive movies through an egocentric screen-view interface rather than native cinematic footage, introducing domain shift

Bird-SR: Bidirectional Reward-Guided Diffusion for Real-World Image Super-Resolution

arXiv:2602.07069v2 Announce Type: replace Abstract: Powered by multimodal text-to-image priors, diffusion-based super-resolution excels at synthesizing intricate details; however, models trained on sy

Tutorials

Feature Extraction in the Remote Sensing Data Value Chain: A Systematic Review of Methods and Applications

arXiv:2510.18935v3 Announce Type: replace Abstract: Earth observation involves collecting, analyzing, and processing an ever-growing mass of data. This planetary data is crucial for addressing relevan

tutorialsarxiv-cs-cv

Industry

Grok 4.3 can Take in Video and Extract Audio files

Grok 4.3, an AI model by xAI, has been updated with multimodal capabilities enabling it to process video input and extract audio from video files. This enhancement expands Grok's functionality beyond

industryelon-musk--x

Industry

Grok 4.3 is still an early beta that will improve almost every day, but try it out! We will publish release notes as we fix bugs and add fun…

Grok 4.3 is still an early beta that will improve almost every day, but try it out! We will publish release notes as we fix bugs and add functionality. Grok 4.3 beta is natively multimodal, and the fr

industryelon-musk--x

Hardware

HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding

arXiv:2601.14724v3 Announce Type: replace-cross Abstract: Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated significant improvement in offline video understanding. Howe

hardwarearxiv-cs-cl

RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Framework

arXiv:2604.15308v1 Announce Type: new Abstract: High-level autonomous driving requires motion planners capable of modeling multimodal future uncertainties while remaining robust in closed-loop interac

Tutorials

ReSS: Learning Reasoning Models for Tabular Data Prediction via Symbolic Scaffold

arXiv:2604.13392v1 Announce Type: new Abstract: Tabular data remains prevalent in high-stakes domains such as healthcare and finance, where predictive models are expected to provide both high accuracy

tutorialsarxiv-cs-ai

To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs

arXiv:2603.18373v2 Announce Type: replace Abstract: When VLMs answer correctly, do they genuinely rely on visual information or exploit language shortcuts? We introduce the Tri-Layer Diagnostic Framew

Towards Design Compositing

arXiv:2604.14605v1 Announce Type: new Abstract: Graphic design creation involves harmoniously assembling multimodal components such as images, text, logos, and other visual assets collected from diver

Cracking the Code of Juxtaposition: Can AI Models Understand the Humorous Contradictions

model-releasesarxiv-cs-cl

arXiv:2405.19088v3 Announce Type: replace Abstract: Recent advancements in large multimodal language models have demonstrated remarkable proficiency across a wide range of tasks. Yet, these models sti

Cyclic 2.5D Perceptual Loss for Cross-Modal 3D Medical Image Synthesis: T1w MRI to Tau PET

arXiv:2406.12632v3 Announce Type: replace-cross Abstract: Positron emission tomography (PET) provides molecular biomarkers for Alzheimer's disease and related dementias (ADRD) and is increasingly used

fMRI-LM: Towards a Universal Foundation Model for Language-Aligned fMRI Understanding

model-releasesarxiv-cs-cl

arXiv:2511.21760v3 Announce Type: replace Abstract: Recent advances in multimodal large language models (LLMs) have enabled unified reasoning across images, audio, and video, but extending such capabi

Hardware

Hydra: Unifying Document Retrieval and Generation in a Single Vision-Language Model

arXiv:2603.28554v2 Announce Type: replace Abstract: Visual document understanding typically requires separate retrieval and generation models, doubling memory and system complexity. We present Hydra,

hardwarearxiv-cs-cv

Hardware

MaMe & MaRe: Matrix-Based Token Merging and Restoration for Efficient Visual Perception and Synthesis

arXiv:2604.13432v1 Announce Type: new Abstract: Token compression is crucial for mitigating the quadratic complexity of self-attention mechanisms in Vision Transformers (ViTs), which often involve num

hardwarearxiv-cs-cv

Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges

model-releasesarxiv-cs-lg

arXiv:2604.13602v1 Announce Type: new Abstract: Reinforcement Learning from Human Feedback (RLHF) and related alignment paradigms have become central to steering large language models (LLMs) and multi

See&Say: Vision Language Guided Safe Zone Detection for Autonomous Package Delivery Drones

arXiv:2604.13292v1 Announce Type: new Abstract: Autonomous drone delivery systems are rapidly advancing, but ensuring safe and reliable package drop-offs remains highly challenging in cluttered urban

SLQ: Bridging Modalities via Shared Latent Queries for Retrieval with Frozen MLLMs

arXiv:2604.13710v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) exhibit strong reasoning and world knowledge, yet adapting them for retrieval remains challenging. Existing app

Local Ai

UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding

arXiv:2604.14113v1 Announce Type: cross Abstract: GUI grounding, which localizes interface elements from screenshots given natural language queries, remains challenging for small icons and dense layou

local-aiarxiv-cs-cl