Search: “evaluation”

arXiv:2509.26100v2 Announce Type: replace Abstract: The rapid integration of Large Language Models (LLMs) into high-stakes domains necessitates reliable safety and compliance evaluation. However, exis

arXiv:2605.14167v1 Announce Type: new Abstract: Every AI benchmark operationalizes theoretical assumptions about the capability it claims to assess. When assumptions function as unexamined commitments

arXiv:2605.12395v1 Announce Type: new Abstract: Background: Many different approaches to controlled text generation (CTG) have been proposed over recent years, but it is difficult to get a clear pictu

arXiv:2605.11479v1 Announce Type: new Abstract: Policy evaluation is a fundamental component of the development and deployment pipeline for robotic policies. In modern manipulation systems, this probl

arXiv:2509.08031v3 Announce Type: replace-cross Abstract: Large Audio Language Models (LALMs) are rapidly advancing, but evaluating them remains challenging due to inefficient and non-standardized too

arXiv:2605.07096v1 Announce Type: cross Abstract: Evaluating a new model on an existing benchmark is often necessary to understand its behavior before deployment. For modern evaluation frameworks, gen

arXiv:2605.07102v1 Announce Type: new Abstract: Evaluating literary quality requires assessing interpretive dimensions such as cultural representation, emotional depth, and philosophical sophisticatio

arXiv:2605.04458v1 Announce Type: new Abstract: Evaluation of long-form, citation-backed reports has lately received significant attention due to the wide-scale adoption of retrieval-augmented generat

Our evaluation of OpenAI's GPT-5.5 cyber capabilities The UK's AI Security Institute previously evaluated Claude Mythos: now they've evaluated GPT-5.5 for finding security vulnerability and found it t

arXiv:2604.25665v1 Announce Type: new Abstract: Reliable evaluation of large language model (LLM)-generated summaries remains an open challenge, particularly across heterogeneous domains and document

arXiv:2604.24710v1 Announce Type: new Abstract: Objective. Clinical AI documentation systems require evaluation methodologies that are clinically valid, economically viable, and sensitive to iterative

arXiv:2604.17200v1 Announce Type: new Abstract: Recent advances in summary evaluation are based on model-based metrics to assess quality dimensions, such as completeness, conciseness, and faithfulness

arXiv:2601.02933v3 Announce Type: replace Abstract: Human evaluation is the gold standard for multilingual NLP, but is often skipped in practice and substituted with automatic metrics because it is no

arXiv:2601.11374v2 Announce Type: replace Abstract: Scientific writing is an expert-domain task that demands deep domain knowledge, task-specific requirements and reasoning capabilities that leverage

arXiv:2604.12988v1 Announce Type: cross Abstract: Execution Accuracy (EX), the widely used metric for evaluating the effectiveness of Natural Language to SQL (NL2SQL) solutions, is becoming increasing

arXiv:2604.09497v1 Announce Type: cross Abstract: Accurate evaluation is central to the large language model (LLM) ecosystem, guiding model selection and downstream adoption across diverse use cases.

arXiv:2604.08970v1 Announce Type: cross Abstract: We study predictive multilingual evaluation: estimating how well a model will perform on a task in a target language when direct benchmark results are

arXiv:2610.13627v1 Announce Type: new Abstract: Evaluating large language models (LLMs) on AMLworld's smallest test splits is costly because each of over a million targets requires a separate graph pr

arXiv:2603.18048v3 Announce Type: replace Abstract: Recent Audio Multimodal Large Language Models (Audio MLLMs) demonstrate impressive performance on speech benchmarks, yet it remains unclear whether

arXiv:2610.13678v1 Announce Type: new Abstract: Prior work suggests fraudulent scientists write their articles with higher rates of linguistic obfuscation than genuine scientists, with a possible mech

arXiv:2610.13440v1 Announce Type: cross Abstract: Recent accent classification systems can achieve high accuracy in standard in-corpus evaluations. However, when speakers overlap between the training

arXiv:2607.10826v2 Announce Type: replace-cross Abstract: Automated evaluation is essential for scaling generative 3D systems, where exhaustive human review is costly and slow. Yet the reliability of

arXiv:2610.07781v1 Announce Type: new Abstract: Post-training quantization reduces the cost of deploying language-model agents, but its effect on recovery from temporary tool failures can depend on ho

arXiv:2610.03826v1 Announce Type: new Abstract: Benchmark scores license claims about the capabilities being evaluated. We audit the inference licensed by an evaluation protocol, rather than the predi

arXiv:2610.05387v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated strong performance on coding and reasoning benchmarks; however, their ability to solve graph-structured m

arXiv:2610.04366v1 Announce Type: new Abstract: Reliable safety evaluation of autonomous vehicles (AVs) is essential to improving road safety, yet it depends critically on realistic simulation of rare

arXiv:2610.03315v1 Announce Type: new Abstract: Trajectory evaluation is essential for improving the reliability of LLM-based agents, but production use makes it expensive to run repeatedly. Modern ag

arXiv:2610.00366v1 Announce Type: new Abstract: A persistent agent must decide both what to retain as information arrives and what to surface once a query appears, yet memory evaluations can confound

arXiv:2609.32230v1 Announce Type: new Abstract: Surrogate models are commonly evaluated by how often they agree with their teacher model over an evaluation set. Local variation in this agreement is we

arXiv:2609.30471v1 Announce Type: cross Abstract: Reference-based LLM-as-a-judge evaluation assumes the reference answer is the target. In deployed agentic systems that operate over dynamic entities (

AI Security Institute: In simulations, GPT-6 Astra conducted unsanctioned supply-chain attacks more often than earlier OpenAI models when prompted only to perform a cyber evaluation — Our new evaluati

arXiv:2609.30670v1 Announce Type: new Abstract: Streaming video understanding requires models to interpret evidence as it arrives, yet current evaluations often report task scores without specifying w

arXiv:2609.26081v1 Announce Type: new Abstract: Fixed-budget robustness evaluation can select the wrong frozen vision encoder. An encoder that survives a shallow attack may lose most of that robustnes

arXiv:2609.23307v1 Announce Type: cross Abstract: This paper presents a comparative evaluation of dense embedding models for semantic candidate-job matching in high-volume staffing workflows. Incoming

arXiv:2609.22127v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to draft contractual language, yet conventional accuracy or preference-based evaluations are poorly m

arXiv:2609.22478v1 Announce Type: cross Abstract: Behavioural evaluations of hosted language models can vary because the evaluated service, the measurement instrument, or both differ across runs. We s

arXiv:2609.18649v1 Announce Type: new Abstract: Warning: This paper contains examples of stereotypes and social bias. LLMs are increasingly used in interactive settings by the general public, making t

My name is Chris Painter, and I'm the President of METR (Model Evaluation and Threat Research). I know we've made a lot of new friends on the internet the last couple of days, so I thought I'd take th

arXiv:2609.14637v1 Announce Type: new Abstract: Large language model agents are increasingly deployed for long-horizon task execution. However, current evaluation paradigms face three major limitation

arXiv:2609.14795v1 Announce Type: cross Abstract: There is a tradeoff in machine translation meta-evaluation between prioritizing alignment with adequacy versus fluency. The balance depends on the com

arXiv:2609.12761v1 Announce Type: new Abstract: Vision encoders for Cell Painting are typically ranked by a single evaluation, commonly replicate mean average precision (mAP). We introduce CP-BG-Bench

arXiv:2609.11028v1 Announce Type: cross Abstract: LM-agent benchmarks increasingly function as interactive evaluation infrastructure. Agents observe state, call tools, modify workspaces, submit artifa

Multi-turn agents fail in ways single-turn evaluation misses: one early mistake corrupts every later turn. This post introduces the Agent Evaluation Metric (AEM), a decomposable, turn-level way to mea

arXiv:2609.06783v1 Announce Type: cross Abstract: LLM agents operate in workflows where unsafe actions can have real consequences. Existing safety evaluations often reduce behavior to a single score,

arXiv:2609.08950v1 Announce Type: cross Abstract: Text-to-SQL systems translate natural language queries into executable SQL, democratizing access to structured data. Despite recent advances driven by

arXiv:2609.08805v1 Announce Type: new Abstract: Preoperative evaluation of trigeminal neuralgia (TN) often requires joint interpretation of structural MRI, which depicts the trigeminal nerve and surro