Search: “evaluation”

arXiv:2609.09672v1 Announce Type: new Abstract: The rapid advancement of audio and multimodal large language models has unlocked transformative speech understanding capabilities, yet evaluation framew

arXiv:2609.08149v1 Announce Type: new Abstract: SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analysi

arXiv:2609.05658v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly being explored for automating SystemVerilog Assertion (SVA) generation, yet most evaluations report correc

arXiv:2609.05140v1 Announce Type: cross Abstract: Background and Objective: External evaluation of medical-imaging AI is often collapsed into discrimination. We evaluated a computational protocol that

One thing that has bothered me about LLM benchmarks for a while is that most of them are essentially snapshots. A model is evaluated, a score is published, and we tend to talk about that score as if i

arXiv:2609.03331v1 Announce Type: new Abstract: Vision-language models (VLMs) are increasingly deployed in multi-turn settings where users may describe visual content with incorrect assumptions. Yet e

arXiv:2609.03781v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used in multilingual settings, yet their safety is still evaluated primarily in English. This limits our

arXiv:2609.04167v1 Announce Type: cross Abstract: Repository-level software engineering benchmarks have significantly advanced the evaluation of coding agents, but existing benchmarks primarily measur

arXiv:2603.19539v2 Announce Type: replace-cross Abstract: We introduce an expert curated, real-world benchmark for evaluating document-grounded question-answering (QA) motivated by generic drug assess

arXiv:2609.00909v1 Announce Type: new Abstract: Reliable evaluation of automated coronary computed tomography angiography (CCTA) report generation requires standardized multicentre benchmarks and clin

arXiv:2609.00608v1 Announce Type: new Abstract: LLM-based user simulators are increasingly used to evaluate autonomous agents at scale, in place of costly human evaluations. Despite this promise, thes

arXiv:2609.00063v1 Announce Type: cross Abstract: The growing use of large language models (LLMs) for search and information retrieval underscores the need to evaluate their reliability in high-stakes

arXiv:2609.01073v1 Announce Type: new Abstract: The LLM-as-a-judge (LLMaJ) framework offers a cost-effective and reproducible solution for automatic evaluation. However, current evaluation practices t

arXiv:2609.00038v1 Announce Type: cross Abstract: Outcome-only evaluation is the production default for LLM agents: show a judge the request and the final reply and ask whether it was handled well. Th

arXiv:2601.16690v2 Announce Type: replace Abstract: We introduce EMemBench, a programmatic benchmark generator for evaluating long-term episodic memory of agents through interactive games. Rather than

arXiv:2608.29458v1 Announce Type: cross Abstract: Sandbagging, in which a model deliberately underperforms on an evaluation despite retaining the underlying capability, threatens the safety evaluation

arXiv:2608.30916v1 Announce Type: new Abstract: Agent evaluations often use one benchmark to choose a workflow and then search for task types where its advantage weakens, so both conclusions are selec

arXiv:2606.16011v2 Announce Type: replace Abstract: Standard accuracy benchmarks evaluate whether large language models (LLMs) reach correct answers. However, they do not test whether models maintain

arXiv:2608.27844v1 Announce Type: new Abstract: Existing evaluations of harmful content detection rely predominantly on static benchmarks, which struggle to reflect the interactive adversarial ecosyst

arXiv:2512.12066v3 Announce Type: replace-cross Abstract: Current safety evaluations of large language models rely on single-shot testing, implicitly assuming that model responses are deterministic an

arXiv:2608.26379v1 Announce Type: cross Abstract: Retrieval evaluation for retrieval-augmented generation (RAG) is increasingly designed around whether retrieved passages contain evidence that can sup

arXiv:2509.02855v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used to produce open-ended, interpretive annotations, yet there is no validated, scalable measure of i

arXiv:2608.24858v1 Announce Type: new Abstract: Marginalized importance weighting evaluates a target policy by reweighting offline state-action samples with its discounted occupancy ratio, characteriz

Most users never rate a bad interaction. They just stop. Perceived Error is LangSmith's new managed judge that reads thread evidence and flags where your agent went sideways. Trained in-house, outperf

arXiv:2608.23766v1 Announce Type: cross Abstract: Between AI-assisted item generation and expert review sits a computational evaluator whose decisions are usually treated as technical preliminaries. Y

arXiv:2602.01718v2 Announce Type: replace Abstract: Predicting generalization from quantities available before target-test evaluation remains a central challenge in deep learning. The systematic bench

arXiv:2608.18300v1 Announce Type: new Abstract: LLM-as-a-Judge, which leverages a large language model to evaluate natural language generated by another AI application or model, has become a standard,

arXiv:2608.17810v1 Announce Type: cross Abstract: The evaluation of large language models (LLMs) relies heavily on human-designed assessments, implicitly assuming that AI and humans employ similar und

arXiv:2604.02118v2 Announce Type: replace Abstract: Natural language explanations of time series data are increasingly produced by foundation models in high stakes domains, making factual correctness

Perceived Error is one of the clearest signals that your agent is giving users a great experience. We tuned the model and prompt around signals that tend to carry across domains, like a user correctin

Written by: Alex Tselevich, Michael Maturi Introduction Adversarial misuse of AI has increased the risk of data theft and extortion events, because when proprietary source code is exposed, defenders m

arXiv:2608.14329v1 Announce Type: cross Abstract: Principle-based regulation, with evaluative standards such as 'fair, clear, and not misleading' or 'deliver good outcomes', cannot be reduced to binar

arXiv:2606.13172v3 Announce Type: replace Abstract: Learned representations are central to modern machine learning and are typically evaluated through predictive performance, robustness, uncertainty e

arXiv:2608.09298v1 Announce Type: cross Abstract: Action-conditioned world models (ACWMs) promise to provide embodied AI with scalable predictive simulators for planning, policy evaluation, and data g

arXiv:2608.02617v1 Announce Type: cross Abstract: We evaluate whether clinician pairwise preferences provide a reliable signal of clinical safety in large language model (LLM) evaluation using expert

arXiv:2608.00205v1 Announce Type: new Abstract: Evaluation of faithfulness of text summarization treats a model generated summary as faithful only if every of its sentences is supported by the source

arXiv:2607.27404v1 Announce Type: new Abstract: Existing benchmarks for electrocardiogram foundation models primarily evaluate downstream predictive performance, providing limited insight into whether

arXiv:2607.28008v1 Announce Type: new Abstract: Representation engineering reads and steers capability directions in large language models, yet methods are typically evaluated on paper-specific synthe

arXiv:2607.26314v1 Announce Type: cross Abstract: Stealth, the discipline of achieving an objective without revealing your presence, capabilities, or collected intelligence, is what separates sophisti

arXiv:2607.26200v1 Announce Type: new Abstract: Content-moderation classifiers are usually evaluated in isolation, but deployment requires choosing where to intervene and what follows a flag. We evalu

arXiv:2607.24758v1 Announce Type: new Abstract: Large language models are capable of recognizing evaluation contexts and altering their behavior to reflect evaluator expectations rather than typical d

arXiv:2507.11548v3 Announce Type: replace-cross Abstract: The use of publicly available generative AI systems for resume evaluation is often justified by the assumption that these tools reduce bias re

arXiv:2607.24647v1 Announce Type: new Abstract: AI-driven autonomous research (AR) systems are becoming increasingly effective across a broad range of tasks. Their performance, however, is still evalu

arXiv:2607.22182v1 Announce Type: new Abstract: Large language model (LLM) evaluation spans diverse tasks and benchmarks, yet evidence remains organized around tasks rather than the capabilities they

arXiv:2607.22456v1 Announce Type: new Abstract: Existing lexical distance, similarity, and evaluation metrics operate on Unicode code points, which can misrepresent errors in writing systems where a s

arXiv:2603.12572v5 Announce Type: replace Abstract: Memory embeddings are crucial for memory-augmented systems, such as OpenClaw, but their evaluation is underexplored in current text embedding benchm