Search: “evaluation”

arXiv:2603.23160v2 Announce Type: replace Abstract: Benchmarking large language models (LLMs) and agents in multi-turn interactive scenarios is essential for understanding their practical capabilities

arXiv:2605.29976v1 Announce Type: cross Abstract: We evaluate the climate simulation capabilities of ArchesWeather and ArchesWeatherGen, two machine learning models originally trained for weather fore

arXiv:2605.30284v1 Announce Type: new Abstract: Scientific discovery is an inherently creative and uncertain process, requiring reasoning beyond the recall of known knowledge. While many benchmarks ha

arXiv:2603.02097v5 Announce Type: replace Abstract: Open-ended medical LLM evaluation remains weakly grounded in physician-calibrated coverage of clinically relevant response criteria, especially in l

arXiv:2605.27766v1 Announce Type: new Abstract: LLM safety evaluations predominantly test models in isolation, yet deployed AI agents increasingly operate within persistent social environments alongsi

arXiv:2601.01627v3 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) are increasingly deployed in healthcare field, it becomes essential to carefully evaluate their medical safety

arXiv:2605.23965v1 Announce Type: new Abstract: Large Language Models (LLMs) achieve strong performance on logical reasoning benchmarks, yet their reliability remains uncertain. Existing evaluations r

Learn how to design, calibrate, and run LLM-as-a-judge evaluators with fixed labels, human agreement checks, trace context, and Phoenix Evals. The post How to build LLM-as-a-Judge evaluators that hold

arXiv:2605.17758v1 Announce Type: new Abstract: Synthetic data is widely used in healthcare to create datasets that are similar to original data but without the privacy concerns. Generating and evalua

arXiv:2602.17831v2 Announce Type: replace Abstract: Evaluating the reasoning capabilities of Large Language Models is increasingly challenging as models improve. Human curation of hard questions is hi

arXiv:2603.16011v2 Announce Type: replace-cross Abstract: Large language model (LLM) coding agents increasingly operate at the repository level, motivating benchmarks that evaluate their ability to op

arXiv:2601.03627v3 Announce Type: replace Abstract: We introduce EPAG, a benchmark dataset and framework designed for Evaluating the Pre-consultation Ability of LLMs using diagnostic Guidelines. LLMs

arXiv:2602.09514v3 Announce Type: replace-cross Abstract: Long-horizon planning is widely recognized as a core capability of autonomous LLM-based agents; however, current evaluation frameworks suffer

arXiv:2605.08503v1 Announce Type: new Abstract: Interactive narrative tasks require LLMs to sustain a coherent, evolving story while adapting to a user over multiple turns. However, suitable benchmark

arXiv:2605.00420v1 Announce Type: cross Abstract: Evaluating the true forecasting ability of AI agents requires environments resistant to overfitting, free from centralized trust, and grounded in ince

arXiv:2506.19089v5 Announce Type: replace-cross Abstract: We introduce StorySim, a programmable framework for synthetically generating stories to evaluate the theory of mind (ToM) and world modeling (

arXiv:2604.24071v1 Announce Type: new Abstract: The increasing scale and variability of peer review in scholarly venues has created an urgent need for systematic, interpretable, and extensible tools t

arXiv:2603.29928v2 Announce Type: replace Abstract: Tabular foundation models such as TabPFN and TabICL already produce full predictive distributions, yet prevailing regression benchmarks evaluate the

arXiv:2604.20389v1 Announce Type: cross Abstract: The rapid evolution and use of Large Language Models (LLMs) in professional workflows require an evaluation of their domain-specific knowledge against

arXiv:2508.06226v2 Announce Type: replace Abstract: Geometry problem solving (GPS) poses significant challenges for Multimodal Large Language Models (MLLMs) in diagram comprehension, knowledge applica

arXiv:2604.19697v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have shown promising reasoning abilities, yet evaluating their performance in specialized domains remains chall

arXiv:2604.16980v1 Announce Type: new Abstract: Background: Large language models (LLMs) are increasingly proposed for diagnostic support, but few evaluations use real-world multimodal inpatient data,

arXiv:2604.17153v1 Announce Type: new Abstract: Transforming legal text into executable decision logic is a longstanding challenge in legal informatics. With the rise of LLMs, this task has gained ren

arXiv:2604.15961v1 Announce Type: new Abstract: There is no consensus in the field of synthetic data on concise metrics for quality evaluations or benchmarks on large health datasets, such as historic

arXiv:2604.15038v1 Announce Type: cross Abstract: The evaluation of fairness in machine learning systems has become a central concern in high-stakes applications, including biometric recognition, heal

We comprehensively benchmarked Opus 4.7 on document understanding. We evaluated it through ParseBench - our comprehensive OCR benchmark for enterprise documents where we evaluate tables, text, charts,

arXiv:2604.09606v1 Announce Type: new Abstract: Traditional benchmarks for large language models (LLMs), such as HELM and AIR-BENCH, primarily assess safety risk through breadth-oriented evaluation ac

arXiv:2604.11502v1 Announce Type: cross Abstract: Contextual causal reasoning is a critical yet challenging capability for Large Language Models (LLMs). Existing benchmarks, however, often evaluate th

arXiv:2604.09678v1 Announce Type: cross Abstract: As agentic network management gains popularity, there is a critical need for evaluation frameworks that transcend static, one-shot testing. To address

arXiv:2604.10866v1 Announce Type: new Abstract: AI agents are expected to perform professional work across hundreds of occupational domains (from emergency department triage to nuclear reactor safety

arXiv:2604.06173v1 Announce Type: cross Abstract: Legal QA benchmarks have predominantly focused on case law, overlooking the unique challenges of statute-centric regulatory reasoning. In statutory do

arXiv:2604.06793v1 Announce Type: cross Abstract: Software documentation is crucial for repository comprehension. While Large Language Models (LLMs) advance documentation generation from code snippets

arXiv:2511.03295v3 Announce Type: replace-cross Abstract: Automatic evaluation of ST systems is typically performed by comparing translation hypotheses with one or more reference translations. While e

arXiv:2509.22750v3 Announce Type: replace Abstract: Real-world multi-hop QA is naturally linked with ambiguity, where a single query can trigger multiple reasoning paths that require independent resol

arXiv:2604.07788v1 Announce Type: cross Abstract: We introduce PeReGrINE, a benchmark and evaluation framework for personalized review generation grounded in graph-structured user--item evidence. PeRe

arXiv:2610.21767v1 Announce Type: new Abstract: Can AI agents organize scientific investigations that lead to well-supported conclusions? To evaluate this capability, we introduce PhysClosureBench, a

arXiv:2610.18449v1 Announce Type: new Abstract: AI agents for autonomous scientific discovery must decide which experiments to run under a limited budget. Most benchmarks for science agents provide fi

arXiv:2610.20472v1 Announce Type: new Abstract: Cultural attitudes and social norms evolve over time, yet most cultural benchmarks for large language models (LLMs) treat culture as a static attribute

arXiv:2610.14539v1 Announce Type: new Abstract: Real-world business requests often leave key requirements unstated because frontline practitioners struggle to articulate rules they know from experienc

arXiv:2610.18745v1 Announce Type: cross Abstract: Large language models increasingly generate Solidity smart contracts that, once deployed, can never be patched, making a single latent fault permanent

arXiv:2610.17456v1 Announce Type: cross Abstract: Language models are often evaluated alone but increasingly deployed together, routed between, made to debate, ensembled, or collaboratively trained. F

arXiv:2610.18455v1 Announce Type: new Abstract: Rapid advances in artificial intelligence-generated content (AIGC) have significantly promoted using text-to-image (T2I) models to generate images with

arXiv:2610.17354v1 Announce Type: new Abstract: Current evaluation protocols for medical image segmentation remain largely anchored in sample-wise comparisons, e.g., Dice and Hausdorff distance, which

arXiv:2610.16347v1 Announce Type: new Abstract: Forward-Forward learning replaces end-to-end backpropagation with local, layer-wise training objectives. Its variants are evaluated primarily by classif

arXiv:2610.13021v1 Announce Type: new Abstract: Continual learning requires models to accumulate past experience and transfer it to new tasks. Despite substantial research on continual learning, exist

arXiv:2610.12360v1 Announce Type: new Abstract: When retrieved evidence contradicts an agent's prior beliefs, does it revise its answer, acknowledge uncertainty, or persist with an incorrect conclusio