Search: “multimodal”

Beyond APIs: Probing the Limits of MLLMs in Physical Tool Use

arXiv:2606.10803v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) excel at utilizing digital APIs and increasingly serve as the 'brain' of embodied AI, instructing robots to i

10 Jun 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

CoCoSI: Collaborative Cognitive Map Construction for Spatial Intelligence

DGX agent

arXiv:2606.10401v1 Announce Type: new Abstract: Spatial intelligence is a key frontier for multimodal large language models (MLLMs), enabling them to reason about the physical world from visual experi

model-releasesarxiv-cs-cv

10 Jun 2026

P3D-Bench: Benchmarking MLLMs for Parametric 3D Generation and Structural Reasoning

arXiv:2606.11152v1 Announce Type: new Abstract: Multimodal large language models can write code to produce complex programs as well as use programs to do 3D modeling, which opens up a new avenue for 3

10 Jun 2026

SPACE: Source-free Proxy Anchor Concept Erasure for MLLMs

arXiv:2606.09868v1 Announce Type: cross Abstract: As Multimodal Large Language Models (MLLMs) face growing privacy risks and regulatory constraints, machine unlearning (MU) has emerged as a crucial so

10 Jun 2026

A Multi-modal Agentic Co-pilot for Evidence Grounded Computational Pathology

arXiv:2606.08093v1 Announce Type: new Abstract: Pathology is the cornerstone of modern medicine, where accurate decision-making relies heavily on evidence-based practices. While artificial intelligenc

Agents

AlloSpatial: Agentic Harness Framework for Spatial Reasoning in Foundation Models

arXiv:2606.08952v1 Announce Type: new Abstract: Multimodal Foundation Models (MFMs) have made substantial progress, yet remain fragile in spatial reasoning over the physical world. A key bottleneck li

agentsarxiv-cs-ai

Are Reasoning Vision-Language Models Robust to Semantic Visual Distractions?

arXiv:2606.08894v1 Announce Type: new Abstract: Reasoning Vision-Language Models (VLMs) achieve strong performance on complex multimodal tasks, but reliable real-world application requires handling vi

Artificial Intelligence for Mathematical Reasoning: An Integrated Survey of Language Models, Neuro-symbolic Systems, and Verified Discovery

arXiv:2606.08728v1 Announce Type: new Abstract: Mathematical reasoning has long served as a stringent test of machine intelligence; over the past decade, it has moved from a niche problem within NLP t

AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMs

arXiv:2606.07643v1 Announce Type: cross Abstract: Recent advances in Omni-Multimodal Large Language Models (Omni-MLLMs) have enabled strong integration of vision, audio, and language. However, their a

Constraint-Aware Optimization for Robust Protein Stability Prediction

arXiv:2606.08100v1 Announce Type: new Abstract: Multimodal DeltaDelta G predictors integrating protein language models with inverse-folding representations achieve strong in-distribution accuracy on t

safetyarxiv-cs-lg

Local Ai

Counterfactual Reasoning for Fine-Grained Evidence Disentanglement in VideoQA

arXiv:2606.09181v1 Announce Type: new Abstract: Recent advances in video multimodal models have significantly improved VideoQA performance. However, these systems often rely on spurious statistical co

local-aiarxiv-cs-cv

CRANE: Knowledge Editing for Reasoning MLLMs

arXiv:2606.09033v1 Announce Type: new Abstract: The emergence of reasoning multimodal large language models (MLLMs), which generate explicit chain-of-thought (CoT) reasoning before producing answers,

DyCo-RL: Dynamic Cross-Modal Coordination for Visual Reasoning

arXiv:2606.08035v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a leading paradigm for enhancing visual reasoning in Multimodal Large Language Mode

safetyarxiv-cs-cv

Applications

FMplex: Model Virtualization for Serving Extensible Foundation Models

arXiv:2606.09643v1 Announce Type: cross Abstract: Foundation models (FMs) are increasingly used as backbones for downstream tasks across language, vision, time-series, and multimodal applications. Yet

applicationsarxiv-cs-ai

Guided Discovery of New Behaviors using Diffusion Policies

arXiv:2606.08743v1 Announce Type: new Abstract: Diffusion models have become a powerful tool for generative modeling in robotics, with diffusion policies excelling at modeling multimodal action-trajec

safetyarxiv-cs-ro

SSAFE: Simple and Strong AI-Generated Image Detection via Frozen Vision Encoders

arXiv:2606.08634v1 Announce Type: new Abstract: The rapid advancement of generative models has blurred the boundary between synthetic and real imagery, creating an urgent need for reliable deepfake de

Agents

Syll: Open-Source Personal Automation with Cross-Surface Execution

arXiv:2606.07594v1 Announce Type: new Abstract: Personal AI agents must increasingly operate across APIs, shells, web surfaces, and desktop GUIs, yet many systems remain tuned to a single interface an

agentsarxiv-cs-ai

The Last Visible Pixel: Probing Fine-Scale Perception in Vision-Language Models

arXiv:2606.07861v1 Announce Type: cross Abstract: Recent vision-language models (VLMs) excel at multimodal understanding and reasoning, yet their fine-grained visual perception remains underexplored.

Where Does the Answer Come From? Benchmarking View-Level Visual Evidence Identification in Multi-View MLLMs for Autonomous Driving

arXiv:2606.09644v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) achieve strong results on visual reasoning benchmarks, but answer accuracy alone does not indicate whether a

Audio-Visual World Models: Grounding Multisensory Imagination for Embodied Agents

arXiv:2512.00883v3 Announce Type: replace-cross Abstract: World models simulate environmental dynamics to enable agents to plan and reason about future states. While existing approaches have primarily

Tutorials

Dreaming when Necessary: Advancing World Action Models with Adaptive Multi-Modal Reasoning

arXiv:2606.07089v1 Announce Type: new Abstract: World Action Models (WAMs) offer a promising approach to embodied intelligence, yet existing methods rely heavily on video prediction as action priors a

tutorialsarxiv-cs-ro

EVA: Evolving Semantic Adversaries for Red-Teaming GUI Agents Against Environmental Injection Attacks

arXiv:2505.14289v2 Announce Type: replace Abstract: Graphical User Interface (GUI) agents powered by Multimodal Large Language Models (MLLMs) are increasingly deployed yet vulnerable to Environmental

safetyarxiv-cs-ai

GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratios

arXiv:2606.06967v1 Announce Type: new Abstract: Generative policies provide expressive and multimodal action distributions, making them attractive for reinforcement learning (RL) in complex continuous

safetyarxiv-cs-lg

HybridCodec: Fast Dual-Stream, Semantically Enhanced Neural Audio Codec

arXiv:2606.06743v1 Announce Type: cross Abstract: The popularity of neural audio codecs as speech tokenizers has surged with the advent of Multimodal Large Language Models. New codec architectures wit

researcharxiv-cs-ai

MoDA: Modulation Adapter for Fine-Grained Visual Grounding in Instructional MLLMs

arXiv:2506.01850v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) have achieved remarkable success in instruction-following tasks by integrating pretrained visual enco

researcharxiv-cs-ai

mtmd : add video input support by ngxson · Pull Request #24269 · ggml-org/llama.cpp

model-releasesr-localllama

PR #24269 added native video input to llama.cpp's multimodal (mtmd) system, merging on June 8, 2026. The implementation uses FFmpeg as a subprocess to decode video frames and expands a single video ma

OpenGlass: Open-Source Smart Glasses for On-Device Event-Based Gesture Recognition

arXiv:2606.07431v1 Announce Type: new Abstract: Smart eyewear enables unobtrusive, context-aware interaction through multimodal sensors and on-device intelligence, but is severely limited by power, me

Seeing Without Exposing: Adaptive Privacy Control for Open-World, Context-Hungry MLLMs

arXiv:2606.07175v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have raised new privacy challenges. On the data side, user-provided inputs often include unpredictable sensitiv

Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors

arXiv:2606.06891v1 Announce Type: new Abstract: Despite advances in 3D scene understanding, existing 3D Large Multimodal Models operate in offline settings, requiring complete scene observations or pr

The Dual Mechanisms of Spatial Variable Binding in Vision-Language Models

arXiv:2603.22278v2 Announce Type: replace Abstract: Many multimodal tasks, such as image captioning and visual question answering, require vision-language models (VLMs) to bind objects with their prop

GIPO: Gaussian Importance Sampling Policy Optimization

arXiv:2603.03955v2 Announce Type: replace-cross Abstract: Post-training with reinforcement learning (RL) has recently shown strong promise for advancing multimodal agents beyond supervised imitation.

safetyarxiv-cs-ai

6 Jun 2026

I Know What You Meme, Even If it Emerged Today: Understanding Evolving Memes through Open-World Knowledge Acquisition

arXiv:2606.05316v1 Announce Type: new Abstract: Multimodal memes are dynamic and often require up to date background knowledge for interpretation. Existing methods often overlook such knowledge or rel

6 Jun 2026

ChartAttack: Testing the Vulnerability of LLMs to Malicious Prompting in Chart Generation

arXiv:2601.12983v3 Announce Type: replace Abstract: Multimodal large language models (MLLMs) are increasingly used to automate chart generation from data tables, improving analysis and reporting effic

researcharxiv-cs-cl

Disentangled Fine-Grained Prototype Learning for Incomplete Image-Tabular Classification

arXiv:2606.05455v1 Announce Type: new Abstract: The missing-modality problem poses a significant challenge in image-tabular multimodal learning across a wide range of multimedia applications, includin

safetyarxiv-cs-cv

Harmonious Parameter Adaptation in Continual Visual Instruction Tuning for Safety-Aligned MLLMs

arXiv:2511.20158v2 Announce Type: replace Abstract: While continual visual instruction tuning (CVIT) has shown promise in adapting multimodal large language models (MLLMs), existing studies predominan

Latent Implicit Visual Reasoning

arXiv:2512.21218v2 Announce Type: replace Abstract: While Large Multimodal Models (LMMs) have made significant progress, they remain largely text-centric, relying on language as their core reasoning m

Learning Visual Spatial Planning from Symbolic State via Modality-Gap-Aware Self-Distillation

arXiv:2606.06076v1 Announce Type: cross Abstract: While vision-language models excel at general multimodal understanding, they still struggle with visual spatial planning. We attribute this to a perce

safetyarxiv-cs-cv

LLM-Conditioned Synthesis of Pathological Gaits via Structured Gait-Language Representations

arXiv:2606.06048v1 Announce Type: new Abstract: Pathological gait datasets remain scarce due to privacy, recruitment, cost, and movement variability. Our work presents a multimodal LLM-guided framewor

LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video

model-releasesarxiv-cs-cl

arXiv:2606.05677v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have advanced image and video understanding and can increasingly handle longer visual inputs. Long-horizon ta

PAR3D: A Unified 3D-MLLM with Part-Aware Representation for Scene Understanding

arXiv:2606.06485v1 Announce Type: new Abstract: Recent advances in 3D multimodal large language models (3D-MLLMs) have enabled unified solutions for 3D scene understanding tasks, including visual ques

The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?

model-releasesarxiv-cs-cl

arXiv:2504.10020v4 Announce Type: replace Abstract: Contrastive decoding strategies are widely used to reduce object hallucinations in multimodal large language models (MLLMs). These methods work by c

UltraVR: A Diagnostic Ultra-Resolution Image-VQA Benchmark for Evidence-Grounded Reasoning

arXiv:2606.05576v1 Announce Type: new Abstract: Vision-language models (VLMs) excel on visual question answering and multimodal reasoning benchmarks. Yet their capability on ultra-resolution images -

Local Ai

b9503

local-aillama-cpp-releases

Release b9503 of llama.cpp fixes multimodal (mtmd) support by handling Gemma 4 audio projector embedding size , specifically removing the projection_dim from clip_n_mmproj_embd. This build includes pr

Agents

Building the AI factory for self-improving agents: What’s new in Arize AX

Arize AX is adding managed agents, full-agent experimentation, expanded multimodal support, and Harness-as-a-Judge to help teams observe, evaluate, and improve production agents. The post Building the

agentsarize-ai

NVIDIA Nemotron 3.5 Content Safety Now Available on Vultr

Deploy NVIDIA Nemotron 3.5 Content Safety on Vultr Cloud GPU with Day Zero support for multimodal AI moderation, custom policy enforcement, multilingual safety workflows, and scalable enterprise AI go

model-releasesvultr

VCIFBench: Evaluating Complex Instruction Following for Video Understanding

model-releasesarxiv-cs-cl

arXiv:2606.04588v1 Announce Type: new Abstract: Multimodal large language models have made rapid progress in video understanding, yet existing benchmarks largely rely on simple prompts and provide lim

Local Ai

Where's gemma4:12b?

Gemma 4 12B is the first medium-sized, encoder-free multimodal model capable of natively ingesting audio and video , recently released by Google. The model is available on Ollama with 11.7M downloads

local-air-ollama

A Fast Screening Approach for High-dimensional Outcomes and High-dimensional Predictors