Search: “evaluation”

arXiv:2602.24119v2 Announce Type: replace Abstract: Purpose: This study evaluates the quality of commercial large language model (LLM) machine translation (MT) for Ancient Greek technical prose and be

arXiv:2604.13882v1 Announce Type: new Abstract: The evaluation of supervised machine learning models is a critical stage in the development of reliable predictive systems. Despite the widespread avail

arXiv:2604.11632v1 Announce Type: new Abstract: We introduce CARTBENCH, a museum-grounded benchmark for evaluating vision-language models (VLMs) on Chinese artworks beyond short-form recognition and Q

arXiv:2604.08947v1 Announce Type: cross Abstract: As Large Language Models (LLMs) become increasingly prevalent in text simplification, systematically evaluating their outputs across diverse prompting

arXiv:2604.05172v2 Announce Type: replace Abstract: Large language model (LLM) agents are increasingly deployed to automate productivity tasks (e.g., email, scheduling, document management), but evalu

arXiv:2604.06367v1 Announce Type: cross Abstract: Web agents automate browser tasks, ranging from simple form completion to complex workflows like ordering groceries. While current benchmarks evaluate

arXiv:2610.19003v1 Announce Type: new Abstract: Abstractive summarization remains challenging across languages, where linguistic quality and factual consistency may diverge, particularly in low-resour

arXiv:2610.14346v2 Announce Type: replace Abstract: Enterprise customer-support agents increasingly rely on Large Language Models (LLMs) to execute bounded, tool-using workflows. We study this problem

arXiv:2610.16152v1 Announce Type: new Abstract: Although post-training significantly enhances model performance, the use of massive undisclosed data raises concerns about benchmark contamination and c

arXiv:2610.12766v1 Announce Type: new Abstract: Machine-learning models for perovskite solar cells are typically evaluated using random data splits, which we show systematically leak information throu

arXiv:2610.11586v1 Announce Type: new Abstract: Existing evaluation methods for healthcare LLMs primarily assess factual correctness,safety, and fluency, while providing limited insight into whether g

arXiv:2610.10619v1 Announce Type: cross Abstract: Large language model (LLM) agents are rapidly reshaping software engineering, accompanied by an explosion of new code benchmarks. Yet nearly all exist

arXiv:2610.09152v1 Announce Type: new Abstract: Multilingual LLM benchmarks omit Slovak, a morphologically rich West Slavic language of five million speakers, or cover it only by machine translation.

arXiv:2610.05413v1 Announce Type: new Abstract: Evaluating vision encoders requires metrics that reliably predict their downstream performance in multimodal large language models (MLLMs). Although rec

arXiv:2606.19788v2 Announce Type: replace Abstract: We present CombEval, a dynamic benchmark for evaluating combinatorial counting in large language models. CombEval represents each problem as a typed

arXiv:2609.38357v1 Announce Type: new Abstract: Conversational safety evaluations often test language models with a single harmful prompt, even though real-world systems interact with users through lo

arXiv:2609.39344v1 Announce Type: cross Abstract: Speech transcripts used as long-term memory must preserve both words and stable speaker identities. Existing meeting-transcription metrics either igno

arXiv:2609.36218v1 Announce Type: cross Abstract: Large language models are increasingly evaluated in specialized domains such as law, medicine, software engineering, and cybersecurity, yet film remai

arXiv:2609.37317v1 Announce Type: new Abstract: Omnimodal evaluation should go beyond independent text, image, and speech production: individually plausible outputs may not express a coherent shared e

arXiv:2609.32632v1 Announce Type: cross Abstract: Multiobjective Bayesian optimization (MOBO) with Gaussian process (GP) surrogates is a sample efficient approach to solving multiobjective optimizatio

arXiv:2601.01842v2 Announce Type: replace Abstract: Dictionary definitions are an essential resource for learning word senses, but manually creating them is costly. We thus study dictionary definition

arXiv:2609.29001v1 Announce Type: new Abstract: Despite strong performance on standard benchmarks, it remains unclear whether large language models (LLMs) evaluate social pragmatics in ways that align

arXiv:2609.27385v1 Announce Type: cross Abstract: Time-series foundation models (TSFMs) provide forecasts for operational decisions, but accuracy alone does not determine their value. Evaluating agent

arXiv:2609.24662v1 Announce Type: new Abstract: LLM-based agents increasingly operate in environments where they interact with users, tools, and external systems. Yet most security evaluations assume

arXiv:2609.22599v1 Announce Type: new Abstract: Large language model (LLM) judges provide a flexible and scalable method for evaluating model and agent outputs, but their verdicts can be sensitive to

arXiv:2411.10163v3 Announce Type: replace Abstract: Large language models (LLMs) demonstrate remarkable performance across various tasks, prompting researchers to develop diverse evaluation benchmarks

arXiv:2609.22700v1 Announce Type: new Abstract: Step-level reasoning evaluators are commonly based on autoregressive language models, whose causal attention restricts each step representation to the p

Agent evaluation is moving from scoring individual function calls to judging full task completion within executable environments that track state over multi‑step tool use. Scoring occurs at two levels

arXiv:2609.21223v1 Announce Type: new Abstract: Vision-language-conditioned robot policies integrate perception, language understanding, and control for general-purpose manipulation. However, existing

arXiv:2609.20684v1 Announce Type: new Abstract: Large language models are increasingly used in healthcare communication, yet most evaluations emphasize response quality while assuming that the user's

arXiv:2609.20794v1 Announce Type: new Abstract: Generative models are increasingly used to solve scientific inverse problems, but existing evaluations still focus primarily on whether a method can pro

GoBench evaluates LLMs on 9x9 Go games against a ladder of KataGo opponents, from random to superhuman. It measures general reasoning ability, strongly correlates with ARC-AGI 2 (r=0.83 correlation),

the independence of AI evaluators is going to matter more... so i made it a 'benchmark': http://evaluatorbench.com (research preview) source-linked directory of third-party evaluators of frontier AI.

arXiv:2510.09302v2 Announce Type: replace-cross Abstract: While Multimodal Large Language Models (MLLMs) have achieved remarkable success in difficult purely textual mathematical reasoning tasks, even

arXiv:2609.14654v1 Announce Type: new Abstract: Convolutional neural networks (CNNs) are typically evaluated using held-out classification accuracy, an approach that presupposes predictions are based

Brendan Bordelon / Politico: OpenAI backs a provision of the FRONTIER Act, a bipartisan House proposal, that would require AI companies to embed outside evaluators to ensure model safety — Artificial

arXiv:2609.15017v1 Announce Type: cross Abstract: Prompt-injection detectors are typically evaluated using aggregate F1 on in-distribution test data, which offers limited insight into behavior under d

arXiv:2609.13150v1 Announce Type: cross Abstract: Reference-free quality predictors such as UTMOS, DNSMOS and SCOREQ are the de facto automatic evaluators for text-to-speech (TTS) and are increasingly

arXiv:2609.11172v1 Announce Type: new Abstract: Recent visual generators produce high-fidelity images yet often violate physical consistency under ego-motion, limiting their use for spatial reasoning

arXiv:2609.09664v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed as personalized assistants that interact with users over extended periods of time. As conversatio

arXiv:2609.02054v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly deployed in interactive systems where understanding user intent precisely is paramount. A key capability f

arXiv:2505.23713v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly deployed in socially grounded applications, where success requires interpreting context, inferring oth

arXiv:2609.00103v1 Announce Type: cross Abstract: Memory is widely viewed as an important unsolved problem for LLMs and VLMs, and current benchmarks typically evaluate it by testing accuracy over long

arXiv:2609.00578v1 Announce Type: new Abstract: Large Language Models (LLMs) can solve complex problems, but their misuse in high-risk domains can lead to severe consequences. Model providers therefor

arXiv:2608.29919v1 Announce Type: cross Abstract: The rapid proliferation of LLMs has further heightened the need to develop dependable AI-generated text detection, especially beyond English. Neverthe

arXiv:2608.30065v1 Announce Type: cross Abstract: Large language models (LLMs) demonstrate strong Helpfulness, Harmlessness, and Honesty (3H) alignment in English-centric settings, but these gains tra