Search: “benchmark”

TravelEval: A Comprehensive Benchmarking Framework for Evaluating LLM-Powered Travel Planning Agents

arXiv:2606.01046v1 Announce Type: new Abstract: The development of Large Language Models (LLMs) has significantly improved travel planning applications, yet evaluating such models is limited by existi

2 Jun 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesis

DGX agent

arXiv:2606.01869v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly asked not only to write static interfaces, but to construct executable interactive worlds from natural lan

model-releasesarxiv-cs-ai

2 Jun 2026

A Visually Impaired Assistance Benchmark for VLM-as-a-Judge Evaluation

arXiv:2605.31351v1 Announce Type: new Abstract: AI-based Visually Impaired Assistance (VIA) remains challenging, largely due to the high cost of human evaluation. The VLM-as-a-Judge paradigm may offer

AMNESIA: A Large Scale Medical Unlearning Benchmark Suite with Disease-Informed Analysis

arXiv:2605.30599v1 Announce Type: cross Abstract: Medical knowledge is continuously evolving. This creates a need to update or selectively forget information encoded in already-trained medical LLMs. M

Benchmarking and Enhancing Text-to-Image Models for Generating Visual Representations in Early Arithmetic Education

arXiv:2605.31212v1 Announce Type: cross Abstract: AI systems are increasingly used to support educational content creation, yet it remains unclear whether they can generate outputs that faithfully rep

BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMs

arXiv:2605.30900v1 Announce Type: new Abstract: Current multimodal models handle static image recognition well, but intuitive physical reasoning remains a weakness. Predicting how objects will move an

EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision

arXiv:2605.31557v1 Announce Type: new Abstract: Continuous episodic memory is a core capability for autonomous agents operating in dynamic, real-world environments, yet current streaming video benchma

ERGeoBench:A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models

arXiv:2605.31251v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) have shown strong potential as embodied agents, yet embodied geo-localization remains underexplored due to th

HypoSpace: A Diagnostic Benchmark for Set-Valued Hypothesis Generation under Underdetermination and Sublinear Coverage Bounds

arXiv:2510.15614v3 Announce Type: replace Abstract: Many scientific problems are underdetermined: multiple distinct hypotheses are equally consistent with the same observations. In such settings, effe

On the Robustness of Multilingual Text Embedding Rankings Across Learning Tasks, Languages, and Benchmark Datasets

arXiv:2605.31142v1 Announce Type: cross Abstract: Large-scale multilingual text embedding models play crucial role in both research and industry, yet their behavior in language-specific, multi-task se

ReTabAD: A Benchmark for Restoring Semantic Context in Tabular Anomaly Detection

arXiv:2510.02060v2 Announce Type: replace Abstract: In tabular anomaly detection (AD), textual semantics often carry critical signals, as the definition of an anomaly is closely tied to domain-specifi

SimulCost: A Cost-Aware Benchmark and Toolkit for Automating Physics Simulations with LLMs

arXiv:2603.20253v2 Announce Type: replace-cross Abstract: Evaluating LLM agents for scientific tasks has focused on token costs while ignoring tool-use costs like simulation time and experimental reso

SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models

arXiv:2605.31597v1 Announce Type: new Abstract: Measuring structured object understanding in vision foundation models remains challenging due to inconsistent evaluation protocols and limited part-leve

SPM-Bench: Benchmarking Large Language Models for Scanning Probe Microscopy

arXiv:2602.22971v2 Announce Type: replace Abstract: As LLMs achieved breakthroughs in general reasoning, their proficiency in specialized scientific domains reveals pronounced gaps in existing benchma

TeachObs: A Human-Validated Benchmark for Multimodal Teaching Observation and Model Evaluation

arXiv:2605.30673v1 Announce Type: new Abstract: Classroom videos contain observable teaching practices, but their pedagogical and visual signals are rarely organized in forms suitable for model evalua

AttuneBench: A Conversation-Based Benchmark for LLM Emotional Intelligence

arXiv:2605.21739v2 Announce Type: replace Abstract: Emotional intelligence (EI), the ability to perceive, understand, and respond appropriately to others' emotional states, is central to human communi

Benchmarking Open-Source Safety Guard Models: A Comprehensive Evaluation

arXiv:2605.28830v1 Announce Type: cross Abstract: As Large Language Models (LLMs) are increasingly deployed in safety-critical applications, robust content moderation becomes essential. We present a c

Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinese

arXiv:2605.29667v1 Announce Type: new Abstract: When Large Language Models (LLMs) are deployed in Chinese-language settings, a troubling pattern emerges: safety systems that work well in English break

InsightEval: An Expert-Curated Benchmark for Assessing Insight Discovery in LLM-Driven Data Agents

arXiv:2511.22884v2 Announce Type: replace Abstract: Data analysis has become an indispensable part of scientific research. To discover the latent knowledge and insights hidden within massive datasets,

MAGA-Bench: Machine-Augment-Generated Text via Alignment Detection Benchmark

arXiv:2601.04633v2 Announce Type: replace Abstract: Machine-Generated Text (MGT) is becoming increasingly difficult to distinguish from Human-Written Text (HWT). This trend has exacerbated malicious a

Research

MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinically Realistic EHR Settings

arXiv:2605.30295v1 Announce Type: cross Abstract: Large language models (LLMs) show promise for clinical reasoning and decision support, but evaluation in realistic, electronic health record-congruent

researcharxiv-cs-ai

MusTBENCH: Benchmarking and Advancing Temporal Grounding in Music LLMs

arXiv:2605.29300v1 Announce Type: cross Abstract: Recent Large Audio-Language Models (LALMs) have demonstrated promising abilities in understanding musical content. However, whether their responses ar

NICE: A Theory-Grounded Diagnostic Benchmark for Social Intelligence of LLMs

arXiv:2605.29685v1 Announce Type: new Abstract: As large language models (LLMs) are increasingly applied in social contexts such as emotional companionship and customer service, measuring their social

OmniMatBench: A Human-Calibrated Multimodal Reasoning Benchmark Across 19 Materials Science Subfields

arXiv:2605.29833v1 Announce Type: new Abstract: As multimodal language models play an increasingly important role in scientific research, materials science offers a critical testbed due to its interdi

Personalized Turn-Level User Conversation Satisfaction Benchmark

arXiv:2605.29711v1 Announce Type: cross Abstract: User satisfaction with AI assistants is highly personalized: the same response may satisfy one user but disappoint another depending on what each user

Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages

arXiv:2510.05291v2 Announce Type: replace Abstract: As Large Language Models (LLMs) develop stronger multilingual capabilities, their sensitivity to culturally diverse entities becomes increasingly im

DynaSchedBench: Calibrated Dynamic Scheduling Benchmarks and Observability Paradox in LLM-based Scheduling Agents

arXiv:2605.27566v1 Announce Type: new Abstract: Progress in neural combinatorial optimization for Dynamic Flexible Job Shop Scheduling Problem (DFJSP) is currently hindered by a methodological tension

Graph Neural Networks for Source Detection: A Review and Benchmark Study

arXiv:2512.20657v2 Announce Type: replace-cross Abstract: The source detection problem arises when an epidemic process unfolds over a contact network, and the objective is to identify its point of ori

KSAFE-MM: A Multimodal Safety Benchmark via Localized Contextualization for Korean Cultural Risks

arXiv:2605.28013v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) exacerbate safety risks by introducing vulnerabilities across multiple modalities, such as language and vision.

MeniOmni: A Structured Multimodal Benchmark for Holistic Meniscus Injury Assessment

arXiv:2605.28161v1 Announce Type: new Abstract: Clinical diagnosis of meniscus injuries requires radiologists to integrate volumetric MRI evidence with patient context (e.g., sex, age, BMI) and to pro

MMTABREAL: Real-World Benchmark for Multimodal Table Understanding

arXiv:2505.21771v2 Announce Type: replace-cross Abstract: Multimodal tables i.e. tabular layouts interleaved with charts, maps, icons, and color encodings are ubiquitous in real applications yet remai

Plug-and-Play Benchmarking of Reinforcement Learning Algorithms for Large-Scale Flow Control

arXiv:2601.15015v2 Announce Type: replace Abstract: Reinforcement learning (RL) has shown promising results in active flow control (AFC), yet progress in the field remains difficult to assess as exist

POINav: Benchmarking and Enhancing Final-Meters Arrival in Real-World Vision-Language Navigation

arXiv:2605.28237v1 Announce Type: cross Abstract: Real-world navigation is fundamentally driven by Points of Interest (POIs), yet reaching a precise POI remains a critical 'final-meters' challenge. Ex

SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding

arXiv:2601.21666v2 Announce Type: replace Abstract: Multimodal Large Language Models (MLLMs) are a major focus of recent AI research. However, most prior work focuses on static image understanding, wh

Benchmark Leakage Trap: Can We Trust LLM-based Recommendation?

arXiv:2602.13626v3 Announce Type: replace Abstract: The expanding integration of Large Language Models (LLMs) into recommender systems poses critical challenges to evaluation reliability. This paper i

BESPOKE: Benchmark for Search-Augmented Large Language Model Personalization via Diagnostic Feedback

arXiv:2509.21106v2 Announce Type: replace Abstract: Search-augmented large language models (LLMs) have advanced information-seeking tasks by integrating retrieval into generation, reducing users' cogn

Beyond Holistic Models: Systematic Component-level Benchmarking of Deep Multivariate Time-Series Forecasting

arXiv:2605.26562v1 Announce Type: new Abstract: While previous research in multivariate time series forecasting has focused on developing complex holistic models, this work advocates for a shift towar

Drive-P2D: A Progressive Perception-to-Decision Benchmark for VLMs in Autonomous Driving

arXiv:2601.14702v2 Announce Type: replace Abstract: Autonomous driving requires reliable perception and safe decision-making in complex scenarios. Recent vision-language models (VLMs) demonstrate reas

EconCausal: A Context-Aware Economic Reasoning Benchmark for Large Language Models

arXiv:2510.07231v4 Announce Type: replace-cross Abstract: Socio-economic causal effects depend heavily on their institutional and environmental contexts. The same intervention can produce different, e

EpiQAL: Benchmarking Large Language Models in Epidemiological Question Answering and Reasoning

arXiv:2601.03471v3 Announce Type: replace-cross Abstract: Reliable epidemiological reasoning requires synthesizing study evidence to infer disease burden, transmission dynamics, and intervention effec

FAB-Bench: A Framework for Adaptive RAG Benchmarking in Semiconductor Manufacturing

arXiv:2605.26476v1 Announce Type: new Abstract: Retrieval-Augmented Generation (RAG) has become critical for knowledge-intensive applications, yet evaluating its performance in vertical domains remain

JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errors

arXiv:2605.26955v1 Announce Type: cross Abstract: As large language models (LLMs) are increasingly deployed to users around the world, they are integrated into everyday tasks across diverse cultural c

Memory Architectures for Multi-Turn Text-to-SQL: A Benchmark and Empirical Study

arXiv:2605.26394v1 Announce Type: new Abstract: Multi-turn Text-to-SQL is central to enterprise analytics yet remains predominantly evaluated in single-turn settings. We introduce EnterpriseMem-Bench,

ODOV: Benchmark the Open-Domain Open-Vocabulary Object Detection

arXiv:2508.01253v2 Announce Type: replace Abstract: Existing studies typically investigate domain shift and category shift as independent problems, however, in real-world scenarios, the two types of s

ORLoopBench: Solver-in-the-Loop Benchmarks for Self-Correction and Behavioral Rationality in Operations Research

arXiv:2601.21008v3 Announce Type: replace-cross Abstract: Operations Research practitioners debug infeasible models through an iterative process: inspecting Irreducible Infeasible Subsystems ( IIS), i

PRISM: A Multi-Dimensional Benchmark for Evaluating LLM Peer Reviewers

arXiv:2605.26730v1 Announce Type: new Abstract: The rapid growth in submissions to machine learning venues has strained the scientific peer-review system and intensified interest in LLM-based automate

RoadGIE: Towards A Global-Scale Aerial Benchmark for Generalizable Interactive Road Extraction

arXiv:2605.26862v1 Announce Type: new Abstract: Accurate road segmentation from aerial imagery is fundamental to many geospatial applications. However, existing datasets often suffer from limited scen

TowerMind: A Tower Defence Game Learning Environment and Benchmark for LLM as Agents

arXiv:2601.05899v2 Announce Type: replace Abstract: Recent breakthroughs in Large Language Models (LLMs) have positioned them as a promising paradigm for agents, with long-term planning and decision-m