Search: “evaluation”

arXiv:2602.13249v2 Announce Type: replace-cross Abstract: Small-molecule foundation models are typically pretrained on standalone molecular data, unlike vision and language models that often benefit f

Streaming vision-language models (VLMs) continuously generate responses given an instruction prompt and an online stream of input frames. This is a core mechanism for real-time visual assistants. Exis

arXiv:2605.20712v1 Announce Type: new Abstract: Automatic speech recognition replaces typing only when correction costs less than manual entry, a threshold determined by error types, not counts: fixin

arXiv:2605.18798v1 Announce Type: new Abstract: We propose non-parametric estimators for the average run length (ARL) and average detection delay (ADD) in quickest changepoint detection (QCD) under fi

arXiv:2411.10959v4 Announce Type: replace-cross Abstract: We study causal inference in experiments and quasi-experiments, where the economic outcome is imperfectly measured by a remotely sensed variab

arXiv:2603.11045v2 Announce Type: replace-cross Abstract: Inverse problems for stiff parabolic partial differential equations (PDEs), such as the inverse heat conduction problem (IHCP), are severely i

arXiv:2506.12090v2 Announce Type: replace Abstract: This paper introduces ChatbotManip, a novel dataset for studying manipulation in Chatbots. It contains simulated generated conversations between a c

arXiv:2605.10278v1 Announce Type: new Abstract: Background: Radiogenomics allows identification of radiological biomarkers for genomic phenotypes. In glioblastoma, these biomarkers could potentially c

arXiv:2601.17644v3 Announce Type: replace-cross Abstract: The growing adoption of multimodal Retrieval-Augmented Generation (mRAG) pipelines for vision-centric tasks (e.g., visual QA) introduces impor

arXiv:2605.04895v1 Announce Type: new Abstract: Published transfer-BO comparisons often estimate an average treatment effect of acquisition choice over hidden regime variables, while practitioners nee

arXiv:2604.27819v1 Announce Type: new Abstract: Multi-server MCP agents create an information-flow control problem: faithful tool composition can turn individually benign read/write permissions into c

arXiv:2604.27279v1 Announce Type: cross Abstract: Audio-based stuttering systems to date have been trained for detection -- what disfluency is present now -- leaving prediction, the capability needed

arXiv:2604.23887v1 Announce Type: cross Abstract: LLM-powered applications routinely embed secrets in system prompts, yet models can be tricked into revealing them. We built an adaptive attacker that

arXiv:2604.22068v1 Announce Type: cross Abstract: Validating Autonomous Vehicles (AVs) requires exposure to rare, safety-critical scenarios, infrequent in routine driving data. Existing benchmarks add

arXiv:2604.21131v1 Announce Type: cross Abstract: AI-agent guardrails are memoryless: each message is judged in isolation, so an adversary who spreads a single attack across dozens of sessions slips p

arXiv:2510.18914v4 Announce Type: replace-cross Abstract: Large language models often display undesirable behaviors embedded in their internal representations, undermining fairness, inconsistency drif

arXiv:2604.21096v1 Announce Type: cross Abstract: Tip-of-the-Tongue (ToT) retrieval benchmarks have largely focused on English, limiting their applicability to multilingual information access. In this

arXiv:2604.19792v1 Announce Type: new Abstract: This paper presents OpenCLAW-P2P v6.0, a comprehensive evolution of the decentralized collective-intelligence platform in which autonomous AI agents pub

arXiv:2509.21267v3 Announce Type: replace Abstract: Large language models often generate homogeneous outputs, but whether this is problematic depends on the specific task. For objective math tasks, re

arXiv:2510.09671v2 Announce Type: replace Abstract: Table Question Answering (TQA) aims to answer natural language questions about tabular data, often accompanied by additional contexts such as text p

arXiv:2604.15940v1 Announce Type: new Abstract: Fixed Radius Near Neighbor (FRNN) search is an alternative to the widely used k Nearest Neighbors (kNN) search. Unlike kNN, FRNN determines a label or a

arXiv:2604.12743v1 Announce Type: new Abstract: While recent research has explored AI tools' ability to classify the quality of mathematical tasks (arXiv:2603.03512), little is known about their capac

arXiv:2604.12047v1 Announce Type: new Abstract: PDF files are primarily intended for human reading rather than automated processing. In addition, the heterogeneous content of PDFs, such as text, table

arXiv:2512.17425v2 Announce Type: replace Abstract: Exoskeletons modulate human movement across diverse applications, from performance augmentation to daily-life assistance. These systems often enforc

arXiv:2604.02360v1 Announce Type: cross Abstract: The transformative potential of large language models (LLMs) in education, such as improving accessibility and personalized learning, is being eclipse

arXiv:2404.00727v3 Announce Type: replace Abstract: All state-of-the-art coreference resolution (CR) models involve finetuning a pretrained language model. Whether the superior performance of one CR m

arXiv:2610.19185v1 Announce Type: new Abstract: Evaluation of large language models (LLMs) remains largely pre-scientific: it involves many design decisions, with little guidance on how to make them.

arXiv:2605.16615v1 Announce Type: new Abstract: In many applications, human and LLM evaluators use assessments of relevant criteria to create an overall evaluation for an item or individual. For examp

arXiv:2609.32462v1 Announce Type: new Abstract: Motion-language models are typically scored by motion-language evaluators, but how well these evaluators ground language structure remains unclear. Here

arXiv:2503.17039v3 Announce Type: replace-cross Abstract: Automatic text summarization relies on automatic evaluation to quickly determine the quality of summarization models via automatic metrics and

🚀Today we launched LangSmith Tuned Evaluators, starting with Perceived Error. Tuned Evaluators run on production traces to catch undesirable agent behavior and attach feedback that you can use in your

🧑‍⚖️Evaluating Deep Agents with LangSmith on AWS Great deep dive blog with our friends at AWS on evaluating DeepAgents with LangSmith Covers datapoint and evaluator design for longer horizon agents ht

arXiv:2609.37576v1 Announce Type: new Abstract: With the rapid advancement of text-to-image (T2I) generation, robust evaluation becomes critical yet challenging, as traditional metrics fail to capture

arXiv:2609.09895v1 Announce Type: new Abstract: Video-language models and video agents can produce hallucinations that conflict with spatiotemporal evidence. Existing benchmarks mainly evaluate model

arXiv:2604.11543v1 Announce Type: cross Abstract: Novelty is a core requirement in academic publishing and a central focus of peer review, yet the growing volume of submissions has placed increasing p

arXiv:2610.19166v1 Announce Type: new Abstract: Meta-evaluation of text generation metrics and LLM-judges is primarily conducted using metric-human correlations. However, prior work has shown that cor

arXiv:2607.00297v1 Announce Type: cross Abstract: When LLM agents use evaluator feedback to adapt their behavior in closed loops, evaluator biases propagate through the agent's strategy distribution -

arXiv:2606.29719v1 Announce Type: cross Abstract: Measurements of proprietary LLM evaluators can become invalid within weeks -- we document one case and provide the diagnostic framework to detect it.

arXiv:2606.23403v2 Announce Type: replace Abstract: As agentic LLM systems move from prototypes to deployment, evaluating them has become more important and more difficult. Evaluation goals are often

arXiv:2505.00759v3 Announce Type: replace-cross Abstract: The steady improvements of text-to-image (T2I) generative models lead to slow deprecation of automatic evaluation benchmarks that rely on stat

arXiv:2609.10594v1 Announce Type: cross Abstract: Expert evaluation of jailbreak responses is costly and difficult to scale, so the community increasingly relies on automated evaluators to determine w

arXiv:2609.06611v1 Announce Type: cross Abstract: Assessing the literary quality of narratives requires evaluating interpretive dimensions (cultural representation, emotional depth, and philosophical

arXiv:2609.02168v1 Announce Type: new Abstract: Fragmented safety evaluation undermines the governance of dangerous AI capabilities. We present a modular framework that evaluates each model through th

arXiv:2608.26827v1 Announce Type: new Abstract: Patient-conditioned acquisition policies for ECG lead-channel selection can outperform population-wide fixed protocols by tailoring the channel budget t

arXiv:2608.25452v1 Announce Type: new Abstract: We introduce VGA-BenchV2, an extended human-aligned benchmark and optimization framework for jointly evaluating and improving video generation quality a

LangSmith’s new Tuned Evaluators look pretty interesting. Perceived Error can flag agent mistakes in production by picking up on user corrections and unresolved conversations. What’s interesting is it