Search: “benchmark”

Detecting and Evaluating Medical Hallucinations in Large Vision Language Models

arXiv:2406.10185v2 Announce Type: replace Abstract: Large Vision Language Models (LVLMs) are increasingly integral to healthcare applications, including medical visual question answering and imaging r

28 Apr 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Safety

Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning

DGX agent

arXiv:2604.23036v1 Announce Type: cross Abstract: Despite MoE models leading many benchmarks, supervised fine-tuning (SFT) for the MoE architectures remains difficult because its router layers are fra

safetyarxiv-cs-cl

28 Apr 2026

StratRAG: A Multi-Hop Retrieval Evaluation Dataset for Retrieval-Augmented Generation Systems

arXiv:2604.22757v1 Announce Type: cross Abstract: We introduce StratRAG, an open-source retrieval evaluation dataset for benchmarking Retrieval-Augmented Generation (RAG) systems on multi-hop reasonin

28 Apr 2026

Holo360D: A Large-Scale Real-World Dataset with Continuous Trajectories for Advancing Panoramic 3D Reconstruction and Beyond

arXiv:2604.22482v1 Announce Type: new Abstract: While feed-forward 3D reconstruction models have advanced rapidly, they still exhibit degraded performance on panoramas due to spherical distortions. Mo

Navigating Large-Scale Document Collections: MuDABench for Multi-Document Analytical QA

arXiv:2604.22239v1 Announce Type: cross Abstract: This paper introduces the task of analytical question answering over large, semi-structured document collections. We present MuDABench, a benchmark fo

TRACE: Topology-aware Reconstruction of Accidents in CARLA for AV Evaluation

model-releasesarxiv-cs-ro

arXiv:2604.22068v1 Announce Type: cross Abstract: Validating Autonomous Vehicles (AVs) requires exposure to rare, safety-critical scenarios, infrequent in routine driving data. Existing benchmarks add

Wiggle and Go! System Identification for Zero-Shot Dynamic Rope Manipulation

arXiv:2604.22102v1 Announce Type: cross Abstract: Many robotic tasks are unforgiving; a single mistake in a dynamic throw can lead to unacceptable delays or unrecoverable failure. To mitigate this, we

AUDITA: A New Dataset to Audit Humans vs. AI Skill at Audio QA

arXiv:2604.21766v1 Announce Type: new Abstract: Existing audio question answering benchmarks largely emphasize sound event classification or caption-grounded queries, often enabling models to succeed

Can MLLMs 'Read' What is Missing?

arXiv:2604.21277v1 Announce Type: new Abstract: We introduce MMTR-Bench, a benchmark designed to evaluate the intrinsic ability of Multimodal Large Language Models (MLLMs) to reconstruct masked text d

Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMs

arXiv:2509.21361v2 Announce Type: replace-cross Abstract: Large language model (LLM) providers boast big numbers for maximum context window sizes. To test the real world use of context windows, we 1)

DAVIS: OOD Detection via Dominant Activations and Variance for Increased Separation

arXiv:2601.22703v2 Announce Type: replace Abstract: Detecting out-of-distribution (OOD) inputs is a critical safeguard for deploying machine learning models in the real world. However, most post-hoc d

Learning to Communicate: Toward End-to-End Optimization of Multi-Agent Language Systems

arXiv:2604.21794v1 Announce Type: new Abstract: Multi-agent systems built on large language models have shown strong performance on complex reasoning tasks, yet most work focuses on agent roles and or

Multilingual and Domain-Agnostic Tip-of-the-Tongue Query Generation for Simulated Evaluation

arXiv:2604.21096v1 Announce Type: cross Abstract: Tip-of-the-Tongue (ToT) retrieval benchmarks have largely focused on English, limiting their applicability to multilingual information access. In this

Symbolic Grounding Reveals Representational Bottlenecks in Abstract Visual Reasoning

arXiv:2604.21346v1 Announce Type: new Abstract: Vision--language models (VLMs) often fail on abstract visual reasoning benchmarks such as Bongard problems, raising the question of whether the main bot

TRAVELFRAUDBENCH: A Configurable Evaluation Framework for GNN Fraud Ring Detection in Travel Networks

arXiv:2604.21093v1 Announce Type: cross Abstract: We introduce TravelFraudBench (TFG), a configurable benchmark for evaluating graph neural networks (GNNs) on fraud ring detection in travel platform g

Wiring the 'Why': A Unified Taxonomy and Survey of Abductive Reasoning in LLMs

arXiv:2604.08016v2 Announce Type: replace Abstract: Regardless of its foundational role in human discovery and sense-making, abductive reasoning--the inference of the most plausible explanation for an

Measuring the Machine: Evaluating Generative AI as Pluralist Sociotechical Systems

arXiv:2604.20545v1 Announce Type: new Abstract: In measurement theory, instruments do not simply record reality; they help constitute what is observed. The same holds for generative AI evaluation: ben

23 Apr 2026

Research

Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs

arXiv:2512.08923v2 Announce Type: replace Abstract: We introduce two new benchmarks REST and REST+ (Render-Equivalence Stress Tests) to enable systematic evaluation of cross-modal inconsistency in mul

researcharxiv-cs-ai

23 Apr 2026

Tokenised Flow Matching for Hierarchical Simulation Based Inference

arXiv:2604.20723v1 Announce Type: cross Abstract: The cost of simulator evaluations is a key practical bottleneck for Simulation Based Inference (SBI). In hierarchical settings with shared global para

23 Apr 2026

AutomationBench

arXiv:2604.18934v1 Announce Type: new Abstract: Existing AI benchmarks for software automation rarely combine cross-application coordination, autonomous API discovery, and policy adherence. Real busin

Compile to Compress: Boosting Formal Theorem Provers by Compiler Outputs

arXiv:2604.18587v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated significant potential in formal theorem proving, yet state-of-the-art performance often necessitates pr

LiveVLN: Breaking the Stop-and-Go Loop in Vision-Language Navigation

model-releasesarxiv-cs-ro

arXiv:2604.19536v1 Announce Type: new Abstract: Recent navigation systems achieve strong benchmark results, yet real-world deployment often remains visibly stop-and-go. This bottleneck arises because

LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models

arXiv:2604.18803v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) are increasingly deployed in settings where reliable visual grounding carries operational consequences, yet their behavi

Lost in Translation: Do LVLM Judges Generalize Across Languages?

arXiv:2604.19405v1 Announce Type: new Abstract: Automatic evaluators such as reward models play a central role in the alignment and evaluation of large vision-language models (LVLMs). Despite their gr

Mango: Multi-Agent Web Navigation via Global-View Optimization

arXiv:2604.18779v1 Announce Type: new Abstract: Existing web agents typically initiate exploration from the root URL, which is inefficient for complex websites with deep hierarchical structures. Witho

Multi-modal Test-time Adaptation via Adaptive Probabilistic Gaussian Calibration

arXiv:2604.19093v1 Announce Type: cross Abstract: Multi-modal test-time adaptation (TTA) enhances the resilience of benchmark multi-modal models against distribution shifts by leveraging the unlabeled

RepIt: Steering Language Models with Concept-Specific Refusal Vectors

arXiv:2509.13281v5 Announce Type: replace Abstract: Current safety evaluations of language models rely on benchmark-based assessments that may miss localized vulnerabilities. We present RepIt, a simpl

Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments

arXiv:2604.19528v1 Announce Type: cross Abstract: This technical note revisits the relationship between RaBitQ and TurboQuant under a unified comparison framework. We compare the two methods in terms

Towards Optimal Agentic Architectures for Offensive Security Tasks

arXiv:2604.18718v1 Announce Type: cross Abstract: Agentic security systems increasingly audit live targets with tool-using LLMs, but prior systems fix a single coordination topology, leaving unclear w

Xpertbench: Expert Level Tasks with Rubrics-Based Evaluation

arXiv:2604.02368v4 Announce Type: replace Abstract: As Large Language Models (LLMs) exhibit plateauing performance on conventional benchmarks, a pivotal challenge persists: evaluating their proficienc

An Integrated Deep-Learning Framework for Peptide-Protein Interaction Prediction and Target-Conditioned Peptide Generation with ConGA-PePPI and TC-PepGen

model-releasesarxiv-cs-lg

arXiv:2604.18467v1 Announce Type: new Abstract: Motivation: Peptide-protein interactions (PepPIs) are central to cellular regulation and peptide therapeutics, but experimental characterization remains

Research

BhashaSutra: A Task-Centric Unified Survey of Indian NLP Datasets, Corpora, and Resources

arXiv:2604.18423v1 Announce Type: new Abstract: India's linguistic landscape, spanning 22 scheduled languages and hundreds of marginalized dialects, has driven rapid growth in NLP datasets, benchmarks

researcharxiv-cs-cl

Applications

Geometric and Quantum Kernel Methods for Predicting Skeletal Muscle Outcomes in chronic obstructive pulmonary disease

arXiv:2601.00921v2 Announce Type: replace Abstract: Quantum methods are increasingly proposed for healthcare, but translational biomarker studies demand transparent benchmarking and robust small-datas

applicationsarxiv-cs-lg

JudgeMeNot: Personalizing Large Language Models to Emulate Judicial Reasoning in Hebrew

arXiv:2604.18041v1 Announce Type: new Abstract: Despite significant advances in large language models, personalizing them for individual decision-makers remains an open problem. Here, we introduce a s

Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length

arXiv:2512.04677v5 Announce Type: replace Abstract: Audio-driven avatar interaction demands real-time, streaming, and infinite-length generation -- capabilities fundamentally at odds with the sequenti

MeasHalu: Mitigation of Scientific Measurement Hallucinations for Large Language Models with Enhanced Reasoning

arXiv:2604.16929v1 Announce Type: new Abstract: The accurate extraction of scientific measurements from literature is a critical yet challenging task in AI4Science, enabling large-scale analysis and i

Applications

Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval

arXiv:2604.18360v1 Announce Type: cross Abstract: Audio-text retrieval systems based on Contrastive Language-Audio Pretraining (CLAP) achieve strong performance on traditional benchmarks; however, the

applicationsarxiv-cs-cl

OptunaHub: A Platform for Black-Box Optimization

model-releasesarxiv-cs-lg

arXiv:2510.02798v2 Announce Type: replace Abstract: Black-box optimization (BBO) underpins advances in domains such as AutoML and Materials Informatics, yet implementations of algorithms and benchmark

PDDL-Mind: Large Language Models are Capable on Belief Reasoning with Reliable State Tracking

arXiv:2604.17819v1 Announce Type: new Abstract: Large language models (LLMs) perform substantially below human level on existing theory-of-mind (ToM) benchmarks, even when augmented with chain-of-thou

ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering

arXiv:2510.09351v2 Announce Type: replace Abstract: While Small Language Models (SLMs) have demonstrated promising performance on an increasingly wide array of commonsense reasoning benchmarks, curren

SpeechMedAssist: Efficiently and Effectively Adapting Speech Language Models for Medical Consultation

arXiv:2601.04638v2 Announce Type: replace Abstract: Medical consultations are intrinsically speech-centric. However, most prior works focus on long-text-based interactions, which are cumbersome and pa

ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence

arXiv:2603.24621v2 Announce Type: replace Abstract: We introduce ARC-AGI-3, an interactive benchmark for studying agentic intelligence through novel, abstract, turn-based environments in which agents

Agents

Capture the Flags: Family-Based Evaluation of Agentic LLMs via Semantics-Preserving Transformations

arXiv:2602.05523v2 Announce Type: replace-cross Abstract: Agentic large language models (LLMs) are increasingly evaluated on cybersecurity tasks using capture-the-flag (CTF) benchmarks, yet existing p

agentsarxiv-cs-ai

Applications

Information Router for Mitigating Modality Dominance in Vision-Language Models

arXiv:2604.16264v1 Announce Type: new Abstract: Vision Language models (VLMs) have demonstrated strong performance across a wide range of benchmarks, yet they often suffer from modality dominance, whe

applicationsarxiv-cs-cv

MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models

arXiv:2511.10262v3 Announce Type: replace-cross Abstract: Full-Duplex Speech Language Models (FD-SLMs) enable real-time, overlapping conversational interactions, offering a more dynamic user experienc

Neurosymbolic Repo-level Code Localization

arXiv:2604.16021v1 Announce Type: cross Abstract: Code localization is a cornerstone of autonomous software engineering. Recent advancements have achieved impressive performance on real-world issue be

Optimizing Korean-Centric LLMs via Token Pruning

arXiv:2604.16235v1 Announce Type: new Abstract: This paper presents a systematic benchmark of state-of-the-art multilingual large language models (LLMs) adapted via token pruning - a compression techn

Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidence

arXiv:2603.13091v2 Announce Type: replace Abstract: The growing interest in embodied agents increases the demand for spatiotemporal video understanding, yet existing benchmarks largely emphasize extra