Search: “multimodal”

Towards Camera-Robust 3D Localization: Equation-Anchored Tool-Use for MLLMs

arXiv:2605.19528v1 Announce Type: new Abstract: 3D localization in Multimodal Large Language Models (MLLMs), including 3D object detection and 3D visual grounding, is fundamentally limited by camera i

20 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Safety

Universal Skeleton Understanding via Differentiable Rendering and MLLMs

DGX agent

arXiv:2603.18003v4 Announce Type: replace Abstract: Multimodal large language models (MLLMs) exhibit strong visual-language reasoning, yet cannot process structured, non-visual data such as human skel

safetyarxiv-cs-cv

20 May 2026

Applications

VCR: Learning Valid Contextual Representation for Incomplete Wearable Signals

arXiv:2605.18837v1 Announce Type: cross Abstract: Wearable devices enable continuous health monitoring from multimodal signals, but real-world deployment is hindered by limited labeled data and pervas

applicationsarxiv-cs-ai

20 May 2026

AgroCoT: A Chain-of-Thought Benchmark for Evaluating Reasoning in Vision-Language Models for Agriculture

arXiv:2511.23253v3 Announce Type: replace Abstract: Recent advancements in Vision-Language Models (VLMs) have significantly impacted various industries. In agriculture, these multimodal capabilities h

Safety

Alignment and Safety of Diffusion Models via Reinforcement Learning and Reward Modeling: A Survey

arXiv:2505.17352v2 Announce Type: replace Abstract: Diffusion models have become a central paradigm for image and multimodal generation, yet their deployment raises persistent questions about alignmen

safetyarxiv-cs-cv

ANVIL: Analogies and Videos for Lecturers

arXiv:2605.16295v1 Announce Type: cross Abstract: We present ANVIL, a multimodal generative system that automates the production of analogy-based instructional animations for computer science topics.

tutorialsarxiv-cs-ai

By now, you've probably heard about Gemini Omni, our new model designed to create anything from any input, starting with video. But... what'…

model-releasesgoogle-ai--x

Google AI announced Gemini Omni, a new multimodal model capable of generating diverse content types from various input formats, with initial focus on video generation capabilities. The model represent

CATA: Continual Machine Unlearning via Conflict-Averse Task Arithmetic

arXiv:2605.18610v1 Announce Type: cross Abstract: Vision-language models (VLMs) have shown remarkable ability in aligning visual and textual representations, enabling a wide range of multimodal applic

Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting

arXiv:2508.04227v2 Announce Type: replace Abstract: Vision-language models (VLMs) and the recent surge of Multimodal Large Language Models (MLLMs) have revolutionized artificial intelligence with unpr

CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Models

arXiv:2605.17826v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) excel at multimodal reasoning, yet it remains unclear whether their answers are grounded in visual evidence or driven by

local-aiarxiv-cs-ai

Cross-Source Supervision for Bone Infection Segmentation in Dual-Modality PET-CT

arXiv:2605.16373v1 Announce Type: cross Abstract: Early and accurate diagnosis and lesion localization of bone infections are crucial for clinical treatment. PET-CT integrates anatomical information f

local-aiarxiv-cs-ai

CrossView Suite: Harnessing Cross-view Spatial Intelligence of MLLMs with Dataset, Model and Benchmark

arXiv:2605.18621v1 Announce Type: cross Abstract: Spatial intelligence requires multimodal large language models (MLLMs) to move beyond single-view perception and reason consistently about objects, vi

EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning

arXiv:2605.17262v1 Announce Type: new Abstract: Despite extensive efforts on egocentric video datasets and benchmarks, understanding users' internal states, which is crucial for enabling seamless AI a

Evaluating Cognitive Age Alignment in Interactive AI Agents

arXiv:2605.17894v1 Announce Type: new Abstract: While agentic AI and its core multimodal large language models (MLLMs) have demonstrated remarkable promise in language and visual reasoning across doma

Explicit Logic Channel for Validation and Enhancement of MLLMs on Zero-Shot Tasks

arXiv:2603.11689v2 Announce Type: replace Abstract: Frontier Multimodal Large Language Models (MLLMs) exhibit remarkable capabilities in Visual-Language Comprehension (VLC) tasks. However, they are of

From Prediction to Intervention: The Evolution of AI in Biomedicine

arXiv:2605.16293v1 Announce Type: cross Abstract: Artificial intelligence has advanced rapidly in biomedicine through large-scale multimodal data integration, enabling increasingly accurate prediction

tutorialsarxiv-cs-ai

Google targets AI agents and video generation with Gemini 3.5 Flash and Omni

model-releasessiliconangle

Google LLC today introduced two new generative artificial intelligence models that push its Gemini family further into AI agents and multimodal creation: Gemini 3.5 Flash, a fast reasoning model desig

HierEdit: Region-Aware Hierarchical Diffusion for Efficient High-Resolution Editing

arXiv:2605.17294v1 Announce Type: new Abstract: High-resolution image editing is essential for professional and creative applications, yet existing multimodal diffusion-based editors remain computatio

local-aiarxiv-cs-cv

Hilbert-Geo: Solving Solid Geometric Problems by Neural-Symbolic Reasoning

arXiv:2605.16385v1 Announce Type: cross Abstract: Geometric problem solving, as a typical multimodal reasoning problem, has attracted much attention and made great progress recently, however most of w

Safety

Latent Action Control for Reasoning-Guided Unified Image Generation

arXiv:2605.16961v1 Announce Type: cross Abstract: Unified multimodal models can encode visual understanding and image generation within a shared backbone, yet understanding does not automatically tran

safetyarxiv-cs-ai

Leveraging Latent Visual Reasoning in Silence

arXiv:2605.18641v1 Announce Type: new Abstract: Latent visual reasoning involves visual evidence more directly in multimodal reasoning by inserting continuous latent tokens before textual generation.

tutorialsarxiv-cs-cv

Machine Learning Enabled Graph Analysis of Particulate Composites: Application to Solid-state Battery Cathodes

arXiv:2512.16085v2 Announce Type: replace-cross Abstract: Particulate composites underpin many solid-state chemical and electrochemical systems, where microstructural features such as multiphase bound

tutorialsarxiv-cs-cv

OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models

arXiv:2605.18041v1 Announce Type: new Abstract: Omnimodal large language models (OmniLLMs) have recently gained increasing attention for unified audio-video understanding. However, processing long mul

Rethinking Point Clouds as Sequences: A Causal Next-Token Predictive Learning Framework

arXiv:2605.17566v1 Announce Type: new Abstract: With the rapid progress of multimodal foundation models and predictive pre-training, an important open question is how to equip 3D point clouds with a p

local-aiarxiv-cs-cv

Some fun Gemini Omni use cases from the community👇🧵 (We’ll keep updating this thread throughout the day)

model-releasesgoogle-ai--x

This X thread from Google AI showcases practical and creative applications of Gemini Omni, Google's multimodal AI model, as demonstrated and shared by the user community. The thread appears to be a cu

SPIKE: An Adaptive Dual Controller Framework for Cost-Efficient Long-Horizon Game Agents

arXiv:2605.18636v1 Announce Type: new Abstract: Long-horizon multimodal agents in open-world games must stay goal-directed across many low-level interactions under tight token and latency budgets. Exi

StrLoRA: Towards Streaming Continual Visual Instruction Tuning for MLLMs

arXiv:2605.16353v1 Announce Type: cross Abstract: Continual Visual Instruction Tuning (CVIT) enables Multimodal Large Language Models to incrementally acquire new abilities. However, existing CVIT met

TIGER-FG: Text-Guided Implicit Fine-Grained Grounding for E-commerce Retrieval

arXiv:2605.18434v1 Announce Type: cross Abstract: E-commerce image search often takes a cropped image as the query, while each candidate is represented by full item images and structured text. This im

Agents

Trajectory-Aware Adaptive Inference in Object Detection Models

arXiv:2605.16397v1 Announce Type: cross Abstract: The increasing integration of sensors in autonomous maritime navigation has led to large-scale multimodal datasets, raising challenges in achieving ef

agentsarxiv-cs-ai

UniPPTBench: A Unified Benchmark for Presentation Generation Across Diverse Input Settings

arXiv:2605.17356v1 Announce Type: new Abstract: Existing works typically focus on presentation generation under isolated input settings, whereas real-world use cases span diverse scenarios, including

What Matters for Grocery Product Retrieval with Open Source Vision Language Models

arXiv:2605.18029v1 Announce Type: new Abstract: Multimodal product retrieval (MPR) underpins checkout-free retail and automated inventory systems, yet it demands fine-grained SKU discrimination that s

WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments

arXiv:2605.16402v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) have revolutionized GUI automation, yet their efficacy is largely established on idealized, single-layer interf

CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models

arXiv:2601.21798v2 Announce Type: replace Abstract: Large Language Models(LLMs) have revolutionized text generation and multimodal perception,but their capabilities in 3D content generation remain und

Constrained latent state modeling: A unifying perspective on representation learning under competing constraints

arXiv:2605.15995v1 Announce Type: cross Abstract: Learning latent representations from complex data is central to modern machine learning, spanning temporal, multimodal, and partially observed systems

tutorialsarxiv-cs-ai

DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding

arXiv:2605.15542v1 Announce Type: new Abstract: GUI agents powered by Multimodal Large Language Models (MLLMs) have demonstrated impressive capability in understanding and executing user instructions.

GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions

arXiv:2605.15764v1 Announce Type: cross Abstract: Understanding social interactions requires reasoning over subtle non-verbal cues, yet current multimodal large language models (MLLMs) often fail to i

Safety

LRCP: Low-Rank Compressibility Guided Visual Token Pruning for Efficient LVLMs

arXiv:2605.15621v1 Announce Type: new Abstract: Large vision-language models (LVLMs) achieve strong multimodal understanding, but their inference cost grows rapidly with the number of visual tokens, e

safetyarxiv-cs-cv

On RGB-TIR Stereo Calibration under Extreme Resolution Asymmetry

arXiv:2605.15860v1 Announce Type: new Abstract: Accurate geometric calibration of RGB-thermal infrared (TIR) stereo camera systems is essential for multimodal building envelope analysis, yet remains c

Sound Sparks Motion: Audio and Text Tuning for Video Editing

arXiv:2605.15307v1 Announce Type: cross Abstract: Motion-centric video editing remains difficult for large generative video models, which often respond well to appearance changes but struggle to produ

local-aiarxiv-cs-cv

VLMs Trace Without Tracking: Diagnosing Failures in Visual Path Following

arXiv:2605.15672v1 Announce Type: cross Abstract: Vision-language models (VLMs) achieve strong performance on multimodal benchmarks, but may still lack robust control over basic visual operations. We

local-aiarxiv-cs-ai

Introducing Gemini Omni

model-releasesgoogle-deepmind

Gemini Omni is a multimodal AI model that allows users to create content from any input and edit naturally using conversational language . Users can combine images, audio, video, and text as input to

17 May 2026

b9169

local-aillama-cpp-releases

Release b9169 of llama.cpp includes updates to multi-token multimodal decoding (mtmd) functionality, adding chunks and fixing preprocessing for Qwen3A models . The changes include attention mask imple

Agents

DermAgent: A Self-Reflective Agentic System for Dermatological Image Analysis with Multi-Tool Reasoning and Traceable Decision-Making

arXiv:2605.14403v1 Announce Type: new Abstract: Dermatological diagnosis requires integrating fine-grained visual perception with expert clinical knowledge. Although Multimodal Large Language Models (

agentsarxiv-cs-cv

Do We Really Need External Tools to Mitigate Hallucinations? SIRA: Shared-Prefix Internal Reconstruction of Attribution

arXiv:2605.14621v1 Announce Type: cross Abstract: Large vision-language models (LVLMs) often hallucinate when language priors dominate weak or ambiguous visual evidence. Existing contrastive decoding

IntentVLA: Short-Horizon Intent Modeling for Aliased Robot Manipulation

arXiv:2605.14712v1 Announce Type: cross Abstract: Robot imitation data are often multimodal: similar visual-language observations may be followed by different action chunks because human demonstrators

LoRA in LoRA: Towards Parameter-Efficient Architecture Expansion for Continual Visual Instruction Tuning

arXiv:2508.06202v2 Announce Type: replace-cross Abstract: Continual Visual Instruction Tuning (CVIT) enables Multimodal Large Language Models (MLLMs) to incrementally learn new tasks over time. Howeve

MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs

arXiv:2605.14966v1 Announce Type: cross Abstract: Large vision-language models (LVLMs) have achieved remarkable performance across diverse multimodal tasks, yet they continue to suffer from hallucinat

MultiEmo-Bench: Multi-label Visual Emotion Analysis for Multi-modal Large Language Models

arXiv:2605.14635v1 Announce Type: cross Abstract: This paper introduces a multi-label visual emotion analysis benchmark dataset for comprehensively evaluating the ability of multimodal large language