Search: “evaluation”

arXiv:2501.03624v2 Announce Type: replace-cross Abstract: Large language models (LLMs) excel on many NLP benchmarks, but their behavior on real-world, semi-structured prediction remains underexplored.

arXiv:2604.20225v1 Announce Type: new Abstract: Evaluating the multilingual and multicultural capabilities of Large Language Models (LLMs) is essential for their global utility. However, current bench

arXiv:2601.15755v3 Announce Type: replace Abstract: Large language models are increasingly used to represent human opinions, values, or beliefs, and their steerability towards these ideals is an activ

arXiv:2604.18943v1 Announce Type: new Abstract: With the rise in capabilities of large language models (LLMs) and their deployment in real-world tasks, evaluating LLM alignment with human preferences

arXiv:2604.16706v1 Announce Type: cross Abstract: Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, but this assumption has rarely been validated a

arXiv:2604.16976v1 Announce Type: new Abstract: Point cloud denoising is a fundamental and crucial challenge in real-world point cloud applications. Existing quantitative evaluation metrics for point

arXiv:2504.04927v2 Announce Type: replace-cross Abstract: As generative AI (GenAI) is increasingly applied in persona development to represent real users, understanding the implications and limitation

arXiv:2602.14517v2 Announce Type: replace Abstract: Large language models (LLMs) have achieved strong results in mathematical reasoning, and are increasingly deployed as tutoring and learning support

arXiv:2604.14188v1 Announce Type: cross Abstract: Large language models have demonstrated impressive performance across many domains of mathematics and physics. One natural question is whether such mo

arXiv:2604.11996v1 Announce Type: cross Abstract: Should we trust Large Language Models (LLMs) with high accuracy? LLMs achieve high accuracy on reasoning benchmarks, but correctness alone does not re

arXiv:2604.12994v1 Announce Type: cross Abstract: Logical vulnerabilities in software stem from flaws in program logic rather than memory safety, which can lead to critical security failures. Although

arXiv:2604.12040v1 Announce Type: cross Abstract: We present SIR-Bench, a benchmark of 794 test cases for evaluating autonomous security incident response agents that distinguishes genuine forensic in

arXiv:2603.25620v2 Announce Type: replace Abstract: Large language model (LLM)-based persona agents are rapidly being adopted as scalable proxies for human participants across diverse domains. Yet the

arXiv:2509.20868v2 Announce Type: replace-cross Abstract: Structured reasoning can improve the inference performance of large language models (LLMs), but it also introduces computational cost and cont

ParseBench is here!📊 We’ve just released ParseBench, an open benchmark + dataset for evaluating document parsing at scale. It includes: • 2,000+ human-reviewed enterprise documents • 167,000 evaluatio

arXiv:2603.18019v2 Announce Type: replace Abstract: Do language model benchmarks actually measure what practitioners intend them to ? High-level metadata is too coarse to convey the granular reality o

arXiv:2604.06742v1 Announce Type: cross Abstract: Large Language Models (LLMs) are driving a shift towards intent-driven development, where agents build complete software from scratch. However, existi

arXiv:2610.20836v1 Announce Type: new Abstract: Evaluator-guided reasoning presents a local control problem at every step: trust the current answer, return a fallback, or spend more inference. We form

arXiv:2610.21604v1 Announce Type: new Abstract: Scientific inquiry involves iterative cycles of exploration, hypothesis generation, and reasoning over observations to uncover relationships in complex

arXiv:2610.16838v1 Announce Type: new Abstract: As language-model agents write code, move money and change records on behalf of users, deciding whether an action is authorized has become a central saf

arXiv:2610.14016v1 Announce Type: new Abstract: Most existing sports datasets are individual-centric or ball-centric, focusing on individual player actions while ignoring interactions among teammates.

arXiv:2610.18030v1 Announce Type: new Abstract: Closed-loop evaluation of end-to-end (E2E) driving systems typically relies on generating sensor observations and re-processing them through the percept

Aditya Soni / Reuters: Sources: Anthropic and OpenAI have pushed for substantial control over what external AI evaluators can publish, out of concern that trade secrets could leak — Anthropic and Open

arXiv:2610.13534v1 Announce Type: new Abstract: Long-horizon embodied agents must maintain task-relevant information, coordinate heterogeneous robots, and adapt their decisions as execution unfolds. E

arXiv:2610.12596v1 Announce Type: new Abstract: Agentic large language models now tackle complex tasks, yet strategic reasoning over long horizons is under-studied: most agentic benchmarks are short a

arXiv:2610.12341v1 Announce Type: new Abstract: Adversarial games have driven advances from heuristic search to reinforcement learning, yet learning and adapting strategies from limited samples remain

arXiv:2610.11482v1 Announce Type: cross Abstract: Context: Large language model (LLM) agents are increasingly used as software and data-engineering assistants, yet evidence about locally deployable op

arXiv:2610.12023v1 Announce Type: new Abstract: Increasingly, AI interviewers are being developed to elicit open-ended responses in applications like market research, public polling, preference elicit

arXiv:2610.10066v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in both visual understanding and code generation. However, existing b

🧬 Training and evaluating AI scientists on drug target discovery is hard. We introduce DrugTargetWorld: Synthetic Biomedical Worlds for Training and Benchmarking AI Scientists The environment turns en

arXiv:2610.07403v1 Announce Type: new Abstract: Long-term memory allows Large Language Models (LLMs) to maintain personalized context across interactions, but retrieved user history can induce memory-

arXiv:2610.07669v1 Announce Type: cross Abstract: We assessed the value of two live human-AI interactions in a precision disease control project in California vineyards. The project tested whether 202

arXiv:2610.07563v1 Announce Type: new Abstract: Large language models (LLMs) have the potential to meet a key goal in economics: a quantitative model of household decision making, across a variety of

arXiv:2610.06511v1 Announce Type: new Abstract: To evaluate the frontier, we must measure models not by what they say, but by what they can engineer and build in grounded physical environments. We int

arXiv:2610.05590v1 Announce Type: new Abstract: Cold-start drug-drug interaction (DDI) prediction tests whether models can identify clinically significant interactions for drugs without training-time

arXiv:2610.06306v1 Announce Type: new Abstract: General purpose language models are increasingly able to control robotic hardware. Understanding the capabilities and safety of these models when embodi

arXiv:2610.05810v1 Announce Type: cross Abstract: Machine learning (ML)-powered poisoning attack vectors are adversarial maneuvers whereby an attacker intentionally inserts, corrupts, or alters traini

arXiv:2610.02877v1 Announce Type: new Abstract: Large language models (LLMs) are widely used as automatic judges, with validity typically assessed via alignment with human scores. However, aggregate a

arXiv:2610.02878v1 Announce Type: new Abstract: We evaluate the performance of Vision Language Models in Visual Question Answering (VQA) when questions violate Grice's maxims. To do this, we use VLMs

arXiv:2610.02122v1 Announce Type: cross Abstract: Real-world enterprise data science and analytics workflows require reasoning across dozens of tables, performing statistical analyses, and acting on t

arXiv:2610.00554v1 Announce Type: new Abstract: We evaluate hybrid quantum-classical machine learning for the reduced-order prediction of spatiotemporal brain deformation fields. To mitigate the compu

arXiv:2609.38954v1 Announce Type: cross Abstract: Large language model (LLM) agents could help security operations centers (SOCs) investigate advanced persistent threats (APTs) by turning weak leads i

arXiv:2609.39863v1 Announce Type: new Abstract: Large language models frequently fail to balance staying truthful with being supportive. They often exhibit sycophancy in responses to users, agreeing w

arXiv:2609.36488v1 Announce Type: new Abstract: Large language model (LLM) agents have demonstrated strong performance on complex web navigation tasks, yet they remain brittle in real-world settings w

arXiv:2603.08064v3 Announce Type: replace Abstract: Most evaluations of generative models rely on feature-distribution metrics such as FID, which operate on continuous recognition features that are ex

arXiv:2609.33401v2 Announce Type: replace-cross Abstract: Model-based judges support agent security by detecting prompt injections, assessing interaction risks, and screening harmful requests. System