Search: “evaluation”

arXiv:2609.21133v1 Announce Type: cross Abstract: SQL has been augmented with AI operators, enabling modern data analytics platforms to derive insights from both structured and unstructured data. We o

arXiv:2609.19524v1 Announce Type: new Abstract: Benchmark scores alone provide an incomplete basis for assessing the trustworthiness of modern artificial intelligence systems. Large language models (L

arXiv:2609.20538v1 Announce Type: new Abstract: An AI evaluation can be perfectly reproducible and still support the wrong claim. This risk is acute in closed-loop systems: policy determines visited s

arXiv:2609.19654v1 Announce Type: new Abstract: Travel-planning agents generate itineraries that may become infeasible after acceptance because of flight cancellations, hotel unavailability, or attrac

arXiv:2609.17731v1 Announce Type: new Abstract: High-resolution land use and land cover (LULC) products derived from Sentinel-2 imagery are widely used for environmental monitoring and land management

arXiv:2609.17913v1 Announce Type: new Abstract: Face--voice association models may rely on language or gender cues in the voice rather than on speaker-specific voice characteristics, which can lead to

arXiv:2609.19101v1 Announce Type: new Abstract: As models scale, reward hacking becomes more frequent, more sophisticated, and more consequential. Does it leave a telltale signature in model represent

Things to try with Jev right now: 1. LLM-as-a-Judge evaluation 2. Routing for agent harnesses harness 3. Scaling agent orchestration by enabling smarter subagent creation with SOTA classification capa

arXiv:2609.18769v1 Announce Type: new Abstract: Correctly answering a question grounded in normative documents often depends on information outside any single passage: whether the retrieved document i

arXiv:2609.16822v1 Announce Type: new Abstract: This study covers foundational concepts for enhancing plan-execution flexibility, including partial-order planning, the producer-consumer-threat formali

arXiv:2609.15055v1 Announce Type: cross Abstract: Deep learning has led to numerous proposed architectures for Automated Emotion Recognition (AER) from electrocardiogram (ECG) data, but inconsistencie

How often do AI agents cheat? We’re releasing CheatBench, a reward gaming evaluation spanning math, coding, knowledge work, visual tasks, and more. After Hugging Face, AI companies tried to address th

arXiv:2609.13279v1 Announce Type: new Abstract: Fashion attribute extraction is evaluated inconsistently: results are reported as single aggregate numbers across image types that pose different proble

arXiv:2609.15964v1 Announce Type: new Abstract: Large language models (LLMs) have been widely adopted for clinical question answering (QA). Current systems can attach citations to their answers, but t

Great paper from Amazon. In discusses when not to trust LLM judges for agent evaluation. (bookmark it) A common way to compare task agents is to have an LLM user simulator talk to each one and an LLM

arXiv:2609.11489v1 Announce Type: new Abstract: Cooperative AI agents are evaluated against other AIs, yet human cooperation relies on implicit conventions---shared protocols for reading meaning beyon

arXiv:2609.10132v1 Announce Type: cross Abstract: The development of generative artificial intelligence resources enables opportunities of speeding up systems and engineering design work. This contrib

arXiv:2512.09417v2 Announce Type: replace Abstract: Head swapping replaces an entire head while preserving pose, expression, body motion, and scene. Progress is limited by the lack of cross-identity p

arXiv:2609.11242v1 Announce Type: new Abstract: Video generation has advanced to produce visually compelling and temporally coherent results. Yet, whether these models can genuinely think with video--

arXiv:2609.09898v1 Announce Type: new Abstract: Large Language Models (LLMs) have shown promise for translating Natural Language (NL) planning descriptions into PDDL problem instances. However, standa

Multi-agent systems fail in ways traditional monitoring misses. This post presents a dual-layer approach to monitoring production agents: Amazon Bedrock AgentCore Evaluations for continuous quality sc

arXiv:2609.09257v1 Announce Type: new Abstract: Sensor-based AI systems are rarely operated under the conditions under which they were trained: devices, personnel and recording epochs change, and each

arXiv:2609.05762v1 Announce Type: new Abstract: UAV image collections contain spatially and temporally related frames, yet semantic-segmentation benchmarks commonly split them at image level. Such spl

arXiv:2609.05535v1 Announce Type: new Abstract: Verdict-only evaluation does not reveal whether a vision-language model (VLM) used the visual evidence that should support its decision. We study this p

arXiv:2603.00039v2 Announce Type: replace Abstract: LLM-as-a-judge ensembles are the standard paradigm for scalable evaluation, but their aggregation mechanisms suffer from a fundamental flaw: they im

arXiv:2609.06107v1 Announce Type: new Abstract: Data policies for reinforcement learning with verifiable rewards (RLVR) determine which rollouts are used, how strongly they are weighted, and which dom

arXiv:2609.07145v1 Announce Type: new Abstract: Planning six-degree-of-freedom (6-DoF) grasps for unseen objects in cluttered tabletop scenes from a single-view depth image requires accurate and effic

arXiv:2609.06190v1 Announce Type: new Abstract: Behavioral evaluations perturb an input and read the induced change in the output in order to certify that a system uses that input. We show that the nu

arXiv:2609.07605v1 Announce Type: new Abstract: Agentic systems can interpret user requests, search the live web, and use external tools, but their ability to transform retrieved web content into a us

arXiv:2609.08901v1 Announce Type: new Abstract: Approximate machine unlearning aims to remove the influence of specific training data from a trained model without retraining from scratch. We identify

Wire Amazon Bedrock AgentCore Evaluations into a GitHub Actions pipeline: deploy an AI agent and an OAuth-protected MCP server to AgentCore runtime, invoke the agent with test prompts, score the respo

arXiv:2609.04292v1 Announce Type: new Abstract: Human mobility predictability concerns the best prediction performance attainable from a given target and input information, but its ground truth is not

arXiv:2609.04898v1 Announce Type: cross Abstract: Repository-scale refactoring requires coding agents to propagate a single change across many interdependent files without altering program behavior, y

@openai: In response to the “wiki incident”, OpenAI says it is working on a framework for reporting misalignment incidents during training, evaluation, and deployment — How we think about the “wiki in

arXiv:2609.03052v1 Announce Type: new Abstract: As services move online, trust institutions such as banks, lenders, and governments must verify the identity of remote users. Fraud detection tools are

arXiv:2505.00284v3 Announce Type: replace-cross Abstract: Rapid advances in vision-language models (VLMs) have generated growing interest in their application to autonomous driving. A prevailing assum

arXiv:2609.01992v1 Announce Type: new Abstract: Agent evaluations face two distinct evidentiary questions: whether a reported claim is recomputable from retained evidence (sufficiency), and whether th

arXiv:2609.02813v1 Announce Type: new Abstract: Text-to-image models excel at natural image synthesis but struggle with graphic design, where success depends on satisfying precise constraints on typog

arXiv:2609.02396v1 Announce Type: new Abstract: Radiology reports are written primarily for clinicians, and their specialized terminology often makes them difficult for patients to interpret. As a res

arXiv:2609.01985v1 Announce Type: new Abstract: As LLM coding agents increasingly perform end-to-end engineering work, we lack empirical characterization of how they behave on systems-level requiremen

arXiv:2609.01470v1 Announce Type: new Abstract: As AI systems are increasingly used to draft radiology reports, reliably evaluating their clinical quality remains a critical challenge. Large language

arXiv:2609.01202v1 Announce Type: cross Abstract: Clinical trials are essential for advancing cancer care and drug development, but many fail because of insufficient patient enrollment. While there is

arXiv:2608.28802v1 Announce Type: cross Abstract: Facial appearance editing powers popular applications like FaceApp and Photoshop. Generative Adversarial Networks (GANs) and 3D Morphable Models (3DMM

arXiv:2608.29408v1 Announce Type: new Abstract: This study introduces a Visual Question Answering model designed specifically for nondestructive evaluation applications. VQA models allow inspectors to

arXiv:2608.29100v1 Announce Type: new Abstract: Agricultural-robot development requires simulation environments that can jointly support realistic scene construction, virtual sensing, autonomous navig

arXiv:2608.30110v1 Announce Type: cross Abstract: Nowcasting headline macroeconomic indicators, i.e., estimating an indicator's value for the current reference period before its official release, is c