Search: “multimodal”

Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning

arXiv:2610.13215v1 Announce Type: new Abstract: Learning new tasks in multimodal continual instruction tuning can cause catastrophic forgetting of previously acquired knowledge. Although multi-expert

12 Oct 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Industry

Introducing Decision 3.0: towards open multimodal foundation decision models ⚡ Open and state-of-the-art from 0.6B to 27B: #1 on both Jev De…

DGX agent

Introducing Decision 3.0: towards open multimodal foundation decision models ⚡ Open and state-of-the-art from 0.6B to 27B: #1 on both Jev Decision Index boards in our evaluation, on a new Pareto front

industryclem-delangue--x

11 Oct 2026

Leakage-Controlled Multimodal Learning for Diagnosis and Progression Prediction in Alzheimer's Disease Research

arXiv:2610.10648v1 Announce Type: new Abstract: Alzheimer's disease prediction involves irregular visits, heterogeneous measurements and incomplete modalities. This study presents a multimodal multita

Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning

arXiv:2610.12060v1 Announce Type: new Abstract: Processing long visual token sequences from high-resolution images makes multi-step reasoning computationally expensive for multimodal Large Language Mo

MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface

arXiv:2610.11316v1 Announce Type: cross Abstract: System One models output constrained decisions and probability distributions rather than free-form text generation. While prevailing paradigms rely on

Applications

Multimodal Large Language Models as Image Classifiers

arXiv:2603.06578v3 Announce Type: replace Abstract: Multimodal Large Language Model (MLLM) classification performance depends critically on evaluation protocol and ground truth quality. Studies compar

applicationsarxiv-cs-cv

Agents

OA-MAP: Evidence-Grounded Multi-Agent Multimodal Framework for Interpretable Knee Osteoarthritis Progression

arXiv:2610.12134v1 Announce Type: new Abstract: Knee osteoarthritis (KOA) progression prediction can support patient monitoring, requiring the integration of multimodal data and multidomain expertise.

agentsarxiv-cs-ai

WOVEN: Weaving Visual World Modeling into Multimodal LLMs

model-releasesarxiv-cs-cl

arXiv:2610.12417v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) struggle with spatial, embodied, physical, and temporal reasoning. We hypothesize that these failures reflect

Beyond Perturbation Magnitude: Direction-Dependent Responses in Multimodal Geometric Representations

arXiv:2610.08533v1 Announce Type: new Abstract: Geometric alignment scores based on Gram determinants provide a compact way to model higher-order consistency among modalities, yet how such scores resp

7 Oct 2026

CueRator: Agentic Search for Symbolic Rules to Adapt Frozen Multimodal Encoders

arXiv:2610.07868v1 Announce Type: new Abstract: Large language model agents have been used to search over symbolic structures such as programs and equations. We propose CueRator, an agentic framework

7 Oct 2026

Multimodal Knowledge Distillation for Gastric Adenocarcinoma Classification from Whole-Slide Images

arXiv:2610.07913v1 Announce Type: cross Abstract: Gastric adenocarcinoma (GA) is a leading cause of cancer-related mortality worldwide, and accurate histopathological subtype classification from whole

7 Oct 2026

A Strong Baseline for Evaluating Vision Encoders in Multimodal Large Language Models

arXiv:2610.05413v1 Announce Type: new Abstract: Evaluating vision encoders requires metrics that reliably predict their downstream performance in multimodal large language models (MLLMs). Although rec

CleanMDM: Clean Motion Diffusion Model for Multimodal Motion Cleanup

arXiv:2610.05411v1 Announce Type: new Abstract: Motion capture data is rarely directly usable, as they typically exhibit missing segments, jitter, drift and contact artifacts. Traditionally, corrupted

Meet Mistral Large 4, aka Le Chonk. • 1T parameters, natively multimodal. 49B active. It is the best open weights model from US or Europe on…

model-releasesclem-delangue--x

Meet Mistral Large 4, aka Le Chonk. • 1T parameters, natively multimodal. 49B active. It is the best open weights model from US or Europe on aggregated benchmarks. • State-of-the-art on critical workl

On the Geometry of Multimodal Saturation: Riemannian VICReg

arXiv:2610.06096v1 Announce Type: new Abstract: In self-supervised learning, a third modality should improve, or at least preserve, performance. Across nine image-text-tabular datasets, we show that i

safetyarxiv-cs-lg

pplx-decider-v1.1-27b, our updated open weights multimodal decision model, is now available. It scores the highest on the new @huggingface D…

model-releasesperplexity--x

pplx-decider-v1.1-27b, our updated open weights multimodal decision model, is now available. It scores the highest on the new @huggingface Decision Index 0.3 benchmark. pplx-decider-v1.1-27b costs hal

Local Ai

Residual Visual Credit Optimization: Conserved Evidence Routing for Multimodal Reinforcement Learning

arXiv:2610.04918v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards scales multimodal reasoning, but an outcome reward says how much a trajectory is worth, not how that valu

local-aiarxiv-cs-lg

Agents

deep agents just shipped a round of multimodal upgrades! 1. broader file support on OpenAI Responses: docx, pptx, xlsx, csv, json, markdown,…

deep agents just shipped a round of multimodal upgrades! 1. broader file support on OpenAI Responses: docx, pptx, xlsx, csv, json, markdown, not just pdfs 2. opt-in binary offloading to keep big files

agentsharrison-chase--x

5 Oct 2026

TasteBench: Multimodal Benchmark for Sensory Prediction, from Molecules to Sustainable Foods

arXiv:2610.02599v1 Announce Type: new Abstract: Sustainable protein discovery lacks the fast computational proxies, analogous to molecular docking or density functional theory, that accelerate drug an

5 Oct 2026

ZeroMAG: Zero-Shot Multimodal Adapter Generation for Plug-and-Play EEG Foundation Models

arXiv:2610.03546v1 Announce Type: new Abstract: EEG foundation models (EFMs) capture reusable knowledge from large-scale EEG data, while many EEG recordings also include companion physiological signal

5 Oct 2026

Local Ai

MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation

arXiv:2601.09879v2 Announce Type: replace-cross Abstract: Recent progress in medical vision-language models (VLMs) has achieved strong performance on image-level text-centric tasks such as report gene

local-aiarxiv-cs-ai

OmniMed-Jev: Calibrating LVLM Confidence for Trustworthy Medical Multimodal Decisions via System One

arXiv:2610.00381v1 Announce Type: new Abstract: Medical models are judged not only on correctness, but on whether reported confidence matches actual accuracy. Generalist multimodal medical models have

Local Ai

ReCast: Contract-Preserving Protection for Fixed-Interface Multimodal Reasoning

arXiv:2610.01184v1 Announce Type: cross Abstract: Remote multimodal models offer strong numerical reasoning capabilities over charts and speech, but sending private inputs risks exposing sensitive con

local-aiarxiv-cs-ai

Wasserstein Gradient Flows and Forward-Only Diffusion Are Not Enough for Multimodal Sampling

arXiv:2610.02081v1 Announce Type: cross Abstract: There has been a proliferation of sampling algorithms based on Wasserstein gradient flows (WGF) and forward-only diffusion processes (FODP), often acc

CAMOS: Coupled Oscillatory State-Space Model for Multimodal Clinical Time-Series

arXiv:2609.39484v1 Announce Type: cross Abstract: Longitudinal clinical cohorts are multimodal, irregularly sampled and pervasively incomplete: in ADNI, positron emission tomography and cerebrospinal

researcharxiv-cs-ai

1 Oct 2026

AdaKerNet: Neural Kernel Decoding for Task-Adaptive Prediction with Multimodal Large Models

arXiv:2609.36368v1 Announce Type: cross Abstract: Large foundation models have been introduced with the promise of efficient adaptation to downstream tasks. Yet, under limited supervision, MLLMs, an i

Mutually Adversarial Self-Training with Evolving Data for Unified Multimodal Models

arXiv:2609.36224v1 Announce Type: cross Abstract: Unified multimodal models (UMMs) combine image generation and visual understanding in a shared backbone. Since generation and understanding are invers

researcharxiv-cs-ai

Tutorials

PreviewDiff: Multimodal Critic-Guided Search over Diffusion Latents

arXiv:2609.36199v1 Announce Type: cross Abstract: Diffusion models can produce striking images and videos, but they still struggle with the compositional details that make a generation faithful to a p

tutorialsarxiv-cs-ai

RAEGNet: Relation-Aware Evidence Graph Network for Harm-Aware Multimodal Fake News Detection

arXiv:2609.36902v1 Announce Type: new Abstract: Existing multimodal fake news detection methods often introduce external information to assist detection. However, most of them rely on entity-level ret

researcharxiv-cs-cl

A Multimodal Feature Distillation with Mamba-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities

arXiv:2404.14019v3 Announce Type: replace Abstract: Existing brain tumor segmentation methods usually utilize multiple Magnetic Resonance Imaging (MRI) modalities in brain tumor images for segmentatio

AnesTRACE: Benchmarking Intraoperative Anesthesia from Multimodal Perception to Multi-step Decision-Making

arXiv:2609.32740v1 Announce Type: new Abstract: Intraoperative anesthesia requires systems to interpret evolving multimodal evidence, recommend timely management, and revise decisions as patient state

Are We Making Progress in Multimodal Domain Generalization? A Comprehensive Benchmark Study

model-releasesarxiv-cs-cv

arXiv:2605.06643v2 Announce Type: replace Abstract: Despite the growing popularity of Multimodal Domain Generalization (MMDG) for enhancing model robustness, it remains unclear whether reported perfor

Energy Vision--Language--Action: A Controlled Multimodal Benchmark for Intent-Conditioned Residential Energy Management

model-releasesarxiv-cs-lg

arXiv:2609.31648v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models are studied mainly in robotics, where visual observations and language instructions are mapped to physical actions.

GraphSelect for Budgeted Representation Selection in Multimodal Graph Inference

arXiv:2609.33561v1 Announce Type: new Abstract: Multimodal graph predictors combine text, images, and relations to classify connected entities. How much of this input is needed to preserve their predi

Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models

arXiv:2609.34972v1 Announce Type: new Abstract: Long visual token sequences often account for a substantial fraction of the computational overhead in multimodal large language models~(MLLMs). Existing

MAESTRO: a Multimodal Auditory-attention Egocentric Speech-TRacking Open corpus

model-releasesarxiv-cs-lg

arXiv:2609.31898v1 Announce Type: cross Abstract: Humans rely on gaze, head movements, and visual cues to attend to speakers in noisy environments, yet auditory attention decoding (AAD) has been studi

MM-VeriRec: Failure-Guided Fusion for Verifiable Agentic Multimodal Recommendation

model-releasesarxiv-cs-cv

arXiv:2609.31718v1 Announce Type: cross Abstract: Images often carry recommendation constraints that text metadata only hints at. A movie may need to look dark, a product may need a minimal style, and

Agents

Recent Advances in Agentic Agri-Robotic Phenotyping: A Perspective Review from Fragmented Multimodal Sensing to Unified PhenoAgent Intelligence

arXiv:2609.34567v1 Announce Type: new Abstract: This review examines the evolution of plant phenotyping from conventional manual trait measurement to high-throughput, robotic, and artificial intellige

agentsarxiv-cs-cv

Resolution as a First-Class Decision: Task-Conditioned Routing for Efficient Multimodal Large Language Models

arXiv:2609.34942v1 Announce Type: new Abstract: The inference efficiency of Multimodal Large Language Models (MLLMs) is severely constrained by massive visual token sequences induced by high-resolutio

SPIDER: Multi-Layer Semantic Token Pruning and Adaptive Sub-Layer Skipping in Multimodal Large Language Models

arXiv:2609.34977v1 Announce Type: new Abstract: Multimodal Large Language Models face significant efficiency challenges that stem from two distinct yet coupled sources: data redundancy and computation

Tutorials

WorldTS: World Modeling for Multimodal Covariate-aware Time Series Forecasting

arXiv:2609.31162v1 Announce Type: new Abstract: Time series forecasting is typically framed as learning a direct mapping from historical to future observations in the observation space. However, seque

tutorialsarxiv-cs-lg

28 Sept 2026

Is Reasoning Always Useful? Rethinking Reasoning Utility in Universal Multimodal Embeddings

arXiv:2609.29560v1 Announce Type: new Abstract: Reasoning-enhanced universal multimodal embeddings (UME) improve heterogeneous retrieval, but plausible rationales do not necessarily produce discrimina

safetyarxiv-cs-ai

KathDB-FAO: Synthesized Query Plans in a Multimodal DBMS

arXiv:2609.28761v1 Announce Type: cross Abstract: We design, implement, and evaluate KathDB-FAO, a new query evaluation subsystem for our KathDB multimodal DBMS. KathDB-FAO takes as input a query in n

researcharxiv-cs-ai

Multimodal AI predicts clinical outcomes of drug combinations from preclinical data

arXiv:2503.02781v3 Announce Type: replace-cross Abstract: Predicting clinical outcomes from preclinical data is essential for selecting safe and effective drug combinations and for reducing late-stage

safetyarxiv-cs-ai

The Alignment Illusion in Multimodal Large Language Models

arXiv:2609.30210v1 Announce Type: new Abstract: Layer-wise visual-text similarity in Multimodal Large Language Models (MLLMs) is widely interpreted as evidence that the language model progressively in

TraceGuard: Adaptive Multimodal Poison Filtering through Cross-Feature Rank Agreement

arXiv:2609.29099v1 Announce Type: cross Abstract: Multimodal training relies on image-text corpora collected from external sources, creating opportunities for attackers to poison the data. Stealthy at

AgroBench: A Reproducible Multimodal Benchmark for Weakly Supervised Crop Yield Learning from County Statistics and Pixel Observations

model-releasesarxiv-cs-cv

arXiv:2609.26809v1 Announce Type: new Abstract: Reliable agricultural yield statistics are typically reported at coarse administrative scales, whereas modern geospatial machine learning methods requir

24 Sept 2026

Pose-Aware Multimodal Automatic Tagging for Greek Traditional Music

arXiv:2609.27094v1 Announce Type: cross Abstract: Automatic tagging is a core task in Music Information Retrieval (MIR), yet most tagging systems exploit only audio. Live music performance is inherent