Search: “evaluation”

Depth over Fidelity in Fixed-Budget Noisy Evolution Strategies

arXiv:2606.06555v1 Announce Type: cross Abstract: Noisy evolution strategies under fixed evaluation budgets face a depth-fidelity trade-off: spending evaluations to denoise intra-generation rankings r

safetyarxiv-cs-lg

8 Jun 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

OpenHalDet: A Unified Benchmark for Hallucination Detection across Diverse Generation Scenarios

DGX agent

arXiv:2606.06959v1 Announce Type: cross Abstract: Hallucination detection is essential for the reliable deployment of large language models (LLMs). However, existing evaluations face two core challeng

model-releasesarxiv-cs-ai

8 Jun 2026

UrduMMLU: A Massive Multitask Benchmark for Urdu Language Understanding

arXiv:2606.07167v1 Announce Type: cross Abstract: Meaningful multilingual evaluation must test models in the target language and educational context. Urdu, spoken by more than 230 million people, lack

8 Jun 2026

Benchmark Everything Everywhere All at Once

arXiv:2606.06462v1 Announce Type: new Abstract: Benchmarks are fundamental for evaluating and advancing LLMs and MLLMs by providing standardized and explicit measures of performance. However, their co

6 Jun 2026

Applications

Ouvia: A User-centered Framework for Measuring Usability of Speech Translation in Real-World Communication Scenarios

arXiv:2606.06177v1 Announce Type: new Abstract: Speech translation (ST) is increasingly adopted in user applications, yet its evaluation largely focuses on decontextualized testbeds and holistic quali

applicationsarxiv-cs-cl

5 Jun 2026

PEFT of SLM for Telecommunications Customer Support: A Comparative Study of LoRA Configurations with Energy Consumption Analysis

arXiv:2606.05176v1 Announce Type: new Abstract: While large language models (LLMs) show strong performance in natural language understanding and generation, their evaluation and adaptation to domain-s

5 Jun 2026

CodegenBench: Can LLMs Write Efficient Code Across Architectures?

arXiv:2606.04023v1 Announce Type: cross Abstract: While large language models (LLMs) have been extensively evaluated on code generation tasks for general-purpose programming and GPU-accelerated enviro

4 Jun 2026

Research

How Users Understand Robot Foundation Model Performance through Task Success Rates and Beyond

arXiv:2602.03920v2 Announce Type: replace Abstract: Robot Foundation Models (RFMs) represent a promising approach to developing general-purpose home robots. Given the broad capabilities of RFMs, users

researcharxiv-cs-ro

4 Jun 2026

NextMotionQA: Benchmarking and Judging Human Motion Understanding with Vision-Language Models

arXiv:2606.04773v1 Announce Type: cross Abstract: Reliable evaluation of human motion understanding is fundamental to advancing embodied AI, robotics, and animation. However, existing benchmarks suffe

4 Jun 2026

Research

AI Rater Discrimination Depends on Scoring Protocol in Complex Clinical Decision-Making

arXiv:2606.03198v1 Announce Type: cross Abstract: Clinical AI evaluation increasingly delegates scoring to large language models (LLMs) acting as AI raters, yet their scoring behavior across evaluatio

researcharxiv-cs-ai

Black-box, Adaptive, Efficient, Transferable, Harmful, Applicable... Attacks Are All You Need to Break LLMs

arXiv:2606.03647v1 Announce Type: cross Abstract: Accurately evaluating adversarial robustness is a longstanding challenge. A flawed attack design can inflate robustness estimates, making deployment r

Applications

Building Trust in Black-box Optimization: A Comprehensive Framework for Explainability

arXiv:2410.14573v2 Announce Type: replace-cross Abstract: Optimizing costly black-box functions within a constrained evaluation budget presents significant challenges in many real-world applications.

applicationsarxiv-cs-ai

PieArena: Ranking and Profiling Language Agents in Realistic Negotiation Scenarios

arXiv:2602.05302v3 Announce Type: replace Abstract: We present an in-depth evaluation of LLMs' ability to negotiate, a central business task requiring strategic reasoning, theory of mind, and economic

SCOPE: Real-Time Natural Language Camera Agent at the Edge

arXiv:2606.02951v1 Announce Type: cross Abstract: Deploying language-driven agents in robotics requires evaluations that reflect real-world task demands: natural-language instructions with reproducibl

ADRA-Bank: A Modular Benchmark for Academic Deep Research Agents

arXiv:2512.00986v3 Announce Type: replace Abstract: A surge in academic publications calls for automated deep research (DR) systems, but accurately evaluating them is still an open problem. First, exi

ChartArena: Benchmarking Chart Parsing across Languages, Scenarios, and Formats

model-releasesarxiv-cs-cv

arXiv:2606.01348v1 Announce Type: new Abstract: Charts are a primary medium for conveying quantitative and relational information, yet systematically evaluating chart parsing models remains difficult.

HypothesisMed: Inference-Time Answer Fusion and Structured Hypothesis-Space Reporting for Biomedical Question Answering

arXiv:2606.00971v1 Announce Type: new Abstract: Biomedical question answering with large language models is commonly evaluated using answer accuracy, but answer accuracy alone does not indicate whethe

Agents

Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation

arXiv:2602.03619v2 Announce Type: replace Abstract: Nowadays, developing reliable DeepResearch-style long-form report generation remains challenging, as training and evaluation lack verifiable reward

agentsarxiv-cs-cl

RoboTrustBench: Benchmarking the Trustworthiness of Video World Models for Robotic Manipulation

arXiv:2606.01600v1 Announce Type: cross Abstract: Video world models are increasingly used in robotic manipulation, yet existing benchmarks mostly evaluate them under valid, feasible, and safe instruc

RoleCDE:Benchmarking and Mitigating Role-Alignment Trade-offs in Role-Playing Agents

arXiv:2606.01552v1 Announce Type: new Abstract: Role-playing agents(RPAs) are widely used to steer large language models(LLMs) toward role-consistent behavior, yet existing benchmarks mainly evaluate

Reassessing Extractive QA Datasets at Scale: LLM-as-a-Judge and In-Depth Analyses

arXiv:2504.11972v3 Announce Type: replace Abstract: Extractive QA tasks are commonly evaluated using Exact Match (EM) and F1-score, but these metrics often fail to reflect true model performance. Rece

safetyarxiv-cs-cl

1 Jun 2026

SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models

model-releasesarxiv-cs-cv

arXiv:2605.31597v1 Announce Type: new Abstract: Measuring structured object understanding in vision foundation models remains challenging due to inconsistent evaluation protocols and limited part-leve

1 Jun 2026

'Be My Cheese?': Cultural Nuance Benchmarking for Machine Translation in Multilingual LLMs

arXiv:2602.04729v2 Announce Type: replace Abstract: We present a large-scale human evaluation benchmark for assessing cultural localisation in machine translation produced by state-of-the-art multilin

Personalized Turn-Level User Conversation Satisfaction Benchmark

arXiv:2605.29711v1 Announce Type: cross Abstract: User satisfaction with AI assistants is highly personalized: the same response may satisfy one user but disappoint another depending on what each user

PhAIL: A Real-Robot VLA Benchmark and Distributional Methodology

model-releasesarxiv-cs-ro

arXiv:2605.29710v1 Announce Type: new Abstract: Real-world evaluation of vision-language-action (VLA) policies still rests on binary success rate at a fixed timeout with N le 25 rollouts per condition

SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers

arXiv:2605.29059v1 Announce Type: cross Abstract: Smart contract decompilation aims to recover high-level source code from bytecode, but evaluating decompilers remains difficult because existing studi

World Models in Words: Auditing Physical State-Transition Commitments in Vision-Language Models

arXiv:2605.29585v1 Announce Type: new Abstract: Vision-language models (VLMs) are increasingly used to answer questions about physical scenes, yet most evaluations reduce performance to a final answer

Auditing Stance Asymmetry in Generative Explanations

arXiv:2605.27988v1 Announce Type: new Abstract: Bias evaluation for language models has made substantial progress on bounded comparisons, such as overt derogation, stereotype association, or label-sen

safetyarxiv-cs-cl

Benchmarking AI for low-resource contexts: Thinking beyond leaderboards

arXiv:2605.28508v1 Announce Type: new Abstract: Existing AI evaluation practices often fail to capture how systems actually perform in low-resource environments, where operational constraints shape us

Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems

arXiv:2605.27492v1 Announce Type: cross Abstract: LLM agents are rapidly evolving from coding assistants into autonomous software engineering systems. However, existing evaluation methodologies remain

From paper to benchmark: agentic, framework-based reproduction of under-specified methods in machine health intelligence

arXiv:2605.28371v1 Announce Type: new Abstract: Industrial Prognostics and Health Management (PHM) provides a representative case study for a broader challenge in applied machine learning: translating

KSAFE-MM: A Multimodal Safety Benchmark via Localized Contextualization for Korean Cultural Risks

arXiv:2605.28013v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) exacerbate safety risks by introducing vulnerabilities across multiple modalities, such as language and vision.

Relevant Is Not Warranted: Evidence-Force Calibration for Cited RAG

arXiv:2605.28044v1 Announce Type: new Abstract: Cited RAG evaluation often treats visible sources as a grounding signal, but a real, topically relevant citation can still under-warrant the attached wo

The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness

arXiv:2605.28190v1 Announce Type: new Abstract: Embedding benchmarks like MTEB report a single score per model, implicitly treating robustness as a static, scalar property. We argue that embedding rob

VibeSearchBench: Benchmarking Long-horizon Proactive Search in the Wild

arXiv:2605.27882v1 Announce Type: cross Abstract: LLM-based agents score well on search benchmarks, yet real users consistently find results unsatisfying, revealing a persistent evaluation-experience

Jailbreak susceptibility prediction and mitigation via the behavioral geometry of models

arXiv:2605.26409v1 Announce Type: cross Abstract: Evaluating and mitigating a generative system's susceptibility to jailbreak attacks is critical to its safe deployment. Given the number of deployable

Learning to Predict Future-Aligned Research Proposals with Language Models

arXiv:2603.27146v3 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used to assist ideation in research, but evaluating the quality of LLM-generated research proposals re

Memory Architectures for Multi-Turn Text-to-SQL: A Benchmark and Empirical Study

arXiv:2605.26394v1 Announce Type: new Abstract: Multi-turn Text-to-SQL is central to enterprise analytics yet remains predominantly evaluated in single-turn settings. We introduce EnterpriseMem-Bench,

OmniToM: Benchmarking Theory of Mind in LLMs via Explicit Belief Modeling

arXiv:2605.26322v1 Announce Type: new Abstract: Theory of Mind (ToM), the ability to infer others' knowledge, intentions, and emotions, is commonly evaluated in large language models (LLMs) using end-

SpatialBench: Is Your Spatial Foundation Model an All-Round Player?

model-releasesarxiv-cs-cv

arXiv:2605.27367v1 Announce Type: new Abstract: While spatial foundation models have demonstrated impressive performance on standard datasets, a critical question remains: are they truly all-round pla

TowerMind: A Tower Defence Game Learning Environment and Benchmark for LLM as Agents

arXiv:2601.05899v2 Announce Type: replace Abstract: Recent breakthroughs in Large Language Models (LLMs) have positioned them as a promising paradigm for agents, with long-term planning and decision-m

A Large-Scale Dataset and Benchmark: Do Protein-Ligand Models Learn Binding Sites or Just Binding Likelihood?

arXiv:2605.24045v1 Announce Type: cross Abstract: Protein-ligand modeling underpins computational drug discovery and molecular design. Existing protein-ligand benchmarks typically evaluate whether a p

GroupTravelBench: Benchmarking LLM Agents on Multi-Person Travel Planning

arXiv:2605.25200v1 Announce Type: new Abstract: Travel planning is a realistic task for evaluating the planning and tool-use abilities of LLM agents. However, existing benchmarks typically assume only

Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks

arXiv:2605.24217v1 Announce Type: new Abstract: As Large Language Models (LLMs) transition from research environments to production deployments, evaluating their performance against strict Service Lev

Agents

Proper Scoring Rules for Agentic Uncertainty Quantification

arXiv:2605.24756v1 Announce Type: new Abstract: Language-model agents increasingly emit uncertainty signals throughout a trajectory, but existing agentic UQ evaluations often conflate ranking usefulne

agentsarxiv-cs-ai

Side-by-side Comparison Amplifies Dialect Bias in Language Models

arXiv:2605.24384v1 Announce Type: cross Abstract: Language models (LMs) can exhibit systematic biases against speakers based on variations in their dialects, even in the absence of a dialect label, a

FATHOMS-RAG: A Framework for the Assessment of Thinking and Observation in Multimodal Systems that use Retrieval Augmented Generation

arXiv:2510.08945v3 Announce Type: replace Abstract: Retrieval-augmented generation (RAG) has emerged as a promising paradigm for improving factual accuracy in large language models (LLMs). We introduc

25 May 2026

Research

Worse than Random: The Importance of a Baseline for Unsupervised Feature Selection