Search: “evaluation”

arXiv:2610.17641v1 Announce Type: new Abstract: Bias scores in language-model evaluations often conflate what a model prefers with whether it answers at all. We identify this confound in extsc{BBQ}, w

arXiv:2610.04561v2 Announce Type: replace Abstract: Differentiable programming connects scientific computation with gradient-based inference, learning and design. Extending these capabilities across a

arXiv:2610.18308v1 Announce Type: cross Abstract: Conversational Recommender Systems (CRSs) require reliable user simulators for cost-effective evaluation. LLM-based simulators have made realistic sim

arXiv:2610.14603v1 Announce Type: new Abstract: Due to the high cost of hardware experiments, robot policies are predominantly evaluated by comparing their empirical performance on a limited number of

NVIDIA NeMo Gym is an open‑source evaluation environment that diagnoses failure modes and tracks error propagation across multistage chip‑design workflows. In Siemens’ Fuse EDA case, the Gym was used

arXiv:2610.14332v1 Announce Type: new Abstract: Accurate sampling from continuous-time generative models can require many model evaluations. Low-step sampling reduces cost, but coarse updates introduc

arXiv:2610.14731v1 Announce Type: new Abstract: Long-term memory benchmarks for language agents primarily evaluate whether information from earlier interactions can be retrieved and used when relevant

arXiv:2610.12768v1 Announce Type: cross Abstract: Semantic query engines evaluate natural-language filters with a large language model, and every evaluation is a costly oracle call. Optimizers that re

Live results, coding benchmarks included, agentic benchmarks included, domains and other classifications filters included. (Spoiler: DeepSeek-V4-Vision-Exp rules, but other models have their rule area

arXiv:2610.11638v1 Announce Type: new Abstract: Evaluating user experience (UX) with automated computational methods has gained increasing attention, supported by empirical evidence from UXBench. Howe

arXiv:2610.09726v1 Announce Type: new Abstract: Estimating remaining shelf life (RSL) from images could provide affordable decision support for perishable produce, but evaluation protocols can substan

arXiv:2610.09671v1 Announce Type: new Abstract: Recent advances in reasoning-oriented large language models (LLMs) have motivated increasing evaluation of their ability to perform professional decisio

arXiv:2610.07677v1 Announce Type: new Abstract: In this paper, we show that standard evaluations of high-resolution Model Inversion Attacks (MIAs) significantly underestimate training-data privacy lea

arXiv:2610.05094v1 Announce Type: cross Abstract: Researchers increasingly use Large Language Models as judges (LLM-as-a-judge) to evaluate model outputs. Yet there are no standards for how to design

arXiv:2610.08215v1 Announce Type: new Abstract: Learning from experience is essential for LLM agents to adapt to unfamiliar and dynmaic environments. Evaluating this ability is therefore important for

arXiv:2609.33678v2 Announce Type: replace Abstract: We introduce SWE-Game, a benchmark of 247 tasks grounded in 41 executable reference Godot games spanning 13 gameplay categories in 2D and 3D. Five t

arXiv:2610.04558v1 Announce Type: cross Abstract: We present ClimateBench v2, a standardized protocol for evaluating climate models on diagnostics expected to be informative for their skill in project

arXiv:2610.03928v1 Announce Type: new Abstract: Screen localization and pointing estimation are key to low-cost interactive devices. Yet developing and evaluating these algorithms requires realistic d

arXiv:2610.02827v1 Announce Type: new Abstract: Modern AI systems are designed to refuse hazardous requests. A jailbreak is a prompt crafted to bypass those safeguards and elicit outputs that the syst

arXiv:2610.02982v1 Announce Type: new Abstract: Programmable logic controllers (PLCs) coordinate industrial equipment by reading sensor inputs and issuing control commands. Evaluating whether large la

We’re launching ReviewBench, a benchmark for code review agents built on representative GitHub pull requests, multi-source ground truth, calibrated evaluation, and production-aligned metrics. The post

arXiv:2610.00677v1 Announce Type: new Abstract: Advancements in augmented reality (AR) continue to foster innovative solutions, facilitating novel methodologies within educational systems, healthcare

arXiv:2510.17932v5 Announce Type: replace-cross Abstract: We introduce Chart2Code, a new benchmark for evaluating the chart understanding and code generation capabilities of large multimodal models (L

arXiv:2609.38355v1 Announce Type: new Abstract: We present HalluScoring 2026, a shared task for evaluating hallucination detection and factual verification in Arabic question answering under challengi

arXiv:2609.39673v1 Announce Type: new Abstract: Can a pretrained graph model replace training and tuning a separate predictor for each dataset? Answering this requires evaluating prediction quality al

arXiv:2609.38792v1 Announce Type: new Abstract: We study training LLM judges from natural language feedback, especially for subjective tasks where the verdict depends strongly on which evaluation crit

arXiv:2609.38137v1 Announce Type: new Abstract: Language-model (LM) harnesses enable LMs to operate effectively over long contexts using additional compute. However, existing long-context evaluations

arXiv:2609.34281v1 Announce Type: new Abstract: High-dimensional Bayesian optimization (HDBO) seeks sample-efficient optimization when the number of variables is large relative to the evaluation budge

arXiv:2609.35257v1 Announce Type: new Abstract: Zeroth-order (ZO) optimization offers a memory-efficient alternative for LLM fine-tuning by estimating updates only from forward evaluations of perturbe

arXiv:2609.33202v1 Announce Type: new Abstract: Flow-matching models generate high-quality images and videos, but repeated neural network evaluations make sampling expensive. Skipping evaluations redu

arXiv:2609.30514v1 Announce Type: cross Abstract: Voice agents are increasingly deployed in workflows where failed interactions can affect transactions, access, and other consequential outcomes, creat

arXiv:2609.29251v1 Announce Type: new Abstract: CAR-bench evaluates whether tool-using agents stay reliable under real-world uncertainty, executing every tool inside the evaluator so that each tool-re

arXiv:2609.27891v1 Announce Type: cross Abstract: Repository-level coding benchmarks have become the standard for evaluating coding agents, yet they inherently suffer from data leakage because they ar

arXiv:2609.22219v1 Announce Type: new Abstract: Evaluating an aligned language model by reading its answers assumes the answers carry the distinction the evaluator cares about. We introduce LLM endogn

arXiv:2609.18748v1 Announce Type: cross Abstract: Telecom fraud scripts evolve rapidly and are often designed to resemble routine service conversations, creating two key requirements for audio-based t

arXiv:2609.18156v1 Announce Type: new Abstract: Teochew has a substantial speaker community and exhibits distinctive lexical, syntactic, and pragmatic features, yet textual resources for evaluating la

arXiv:2609.16582v1 Announce Type: cross Abstract: Spoken sarcasm detectors may exploit lexical content, prosody, or their interaction, yet conventional evaluation cannot reveal which cues drive their

arXiv:2609.16816v1 Announce Type: cross Abstract: Language model-generated rubrics are increasingly used as reward signals for rubric-based reinforcement learning, LLM-as-a-judge evaluation, and autom

arXiv:2609.17496v1 Announce Type: new Abstract: LLM assistants are widely used for daily social advice, yet evaluating their social reasoning in such consultation settings remains challenging since (i

arXiv:2602.13576v2 Announce Type: replace-cross Abstract: Evaluation and alignment pipelines for large language models increasingly rely on LLM-based judges, whose behavior is guided by natural-langua

arXiv:2510.18368v2 Announce Type: replace Abstract: We present extbf{Korean SimpleQA (KoSimpleQA)}, a benchmark for evaluating factuality in large language models (LLMs) with a focus on Korean cultura

arXiv:2609.12475v1 Announce Type: new Abstract: Comprehensive benchmark suites are essential for improving large language models (LLMs), but many widely used benchmarks are redundant, making evaluatio

arXiv:2609.11185v1 Announce Type: new Abstract: Evidence-based medicine demands strict logical consistency, yet current evaluations of large language models (LLMs) prioritize superficial label matchin

arXiv:2609.09428v1 Announce Type: new Abstract: Evaluating the quality of explanations produced by explainable AI (XAI) methods remains challenging because existing approaches often rely on subjective

arXiv:2609.05587v1 Announce Type: new Abstract: Existing evaluations of tool-using agents primarily measure whether an agent can successfully complete diverse tasks with tools. These evaluations gener

arXiv:2609.07353v1 Announce Type: new Abstract: Large language models (LLMs) are used to simulate human participants in psychological research. We asked whether LLMs that reproduce human evaluations o