Search: “evaluation”

FACT-E: Causality-Inspired Evaluation for Trustworthy Chain-of-Thought Reasoning

arXiv:2604.10693v1 Announce Type: new Abstract: Chain-of-Thought (CoT) prompting has improved LLM reasoning, but models often generate explanations that appear coherent while containing unfaithful int

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

FinTrace: Holistic Trajectory-Level Evaluation of LLM Tool Calling for Long-Horizon Financial Tasks

DGX agent

arXiv:2604.10015v1 Announce Type: new Abstract: Recent studies demonstrate that tool-calling capability enables large language models (LLMs) to interact with external environments for long-horizon fin

model-releasesarxiv-cs-ai

14 Apr 2026

Hidden Failures in Robustness: Why Supervised Uncertainty Quantification Needs Better Evaluation

arXiv:2604.11662v1 Announce Type: new Abstract: Recent work has shown that the hidden states of large language models contain signals useful for uncertainty estimation and hallucination detection, mot

researcharxiv-cs-cl

Persona Non Grata: Single-Method Safety Evaluation Is Incomplete for Persona-Imbued LLMs

arXiv:2604.11120v1 Announce Type: new Abstract: Personality imbuing customizes LLM behavior, but safety evaluations almost always study prompt-based personas alone. We show this is incomplete: prompti

UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agents

arXiv:2604.11557v1 Announce Type: new Abstract: Tool-use capability is a fundamental component of LLM agents, enabling them to interact with external systems through structured function calls. However

Utilizing and Calibrating Hindsight Process Rewards via Reinforcement with Mutual Information Self-Evaluation

arXiv:2604.11611v1 Announce Type: new Abstract: To overcome the sparse reward challenge in reinforcement learning (RL) for agents based on large language models (LLMs), we propose Mutual Information S

safetyarxiv-cs-cl

Local Ai

From Frames to Events: Rethinking Evaluation in Human-Centric Video Anomaly Detection

arXiv:2604.09327v1 Announce Type: new Abstract: Pose-based Video Anomaly Detection (VAD) has gained significant attention for its privacy-preserving nature and robustness to environmental variations.

local-aiarxiv-cs-cv

Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognition

arXiv:2604.09121v1 Announce Type: cross Abstract: Recent years have witnessed remarkable progress in automatic speech recognition (ASR), driven by advances in model architectures and large-scale train

agentsarxiv-cs-ai

SAGE: A Service Agent Graph-guided Evaluation Benchmark

arXiv:2604.09285v1 Announce Type: new Abstract: The development of Large Language Models (LLMs) has catalyzed automation in customer service, yet benchmarking their performance remains challenging. Ex

TaxPraBen: A Scalable Benchmark for Structured Evaluation of LLMs in Chinese Real-World Tax Practice

arXiv:2604.08948v1 Announce Type: new Abstract: While Large Language Models (LLMs) excel in various general domains, they exhibit notable gaps in the highly specialized, knowledge-intensive, and legal

Beyond Surface Judgments: Human-Grounded Risk Evaluation of LLM-Generated Disinformation

arXiv:2604.06820v1 Announce Type: new Abstract: Large language models (LLMs) can generate persuasive narratives at scale, raising concerns about their potential use in disinformation campaigns. Assess

10 Apr 2026

KBS: A Korean Benchmark Suite for Foundation Model Evaluation

arXiv:2610.21227v1 Announce Type: new Abstract: We present a Korean Benchmark Suite (KBS) comprising Ko-ARC, Ko-GSM8K, Ko-EQ-Bench, Ko-WinoGrande, Ko-LAMBADA, and Ko-IFEval, covering six capabilities

16 Oct 2026

MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environments

arXiv:2610.20939v1 Announce Type: new Abstract: Online reinforcement learning (RL) with verifiable rewards has driven rapid progress in mathematics, coding, and tool use, but scaling this success to G

agentsarxiv-cs-ai

16 Oct 2026

A Strong Harness Narrows the Model Gap: A Comparative Evaluation for Legislative Document-to-XML Conversion

arXiv:2610.19999v1 Announce Type: new Abstract: Legal-content reproduction requires matching the source exactly in section hierarchy, verbatim text, citations, and amendment spans; a dropped citation

15 Oct 2026

A Controlled Evaluation of Curvature Inspired Scalar Feature Scaling for Domain Generalization

arXiv:2610.18227v1 Announce Type: cross Abstract: Domain-specific curvature is an appealing idea for domain generalization (DG), but curvature learning can denote substantially different computations.

researcharxiv-cs-cv

Auditing Observation-Quality Shortcuts in the Evaluation of Person-Centered Motion for Video Anomaly Detection

arXiv:2610.16471v1 Announce Type: new Abstract: Video anomaly detection (VAD) identifies deviations from normal patterns, but such deviations do not necessarily correspond to safety-critical events. W

safetyarxiv-cs-cv

Beyond a Better Score: Long-Horizon Agentic ML Development and Evaluation Protocol for Physics Time Series

arXiv:2610.14024v1 Announce Type: new Abstract: Physics time-series data are central to fundamental discoveries involving dark matter, neutrinos, and gravitational waves. Machine learning has shown st

Deep Learning-based Semantic Segmentation of Dental Radiographs: Comparative Evaluation of PSPNet, DeepLabv3, UNet, and UNet++

arXiv:2610.15061v1 Announce Type: new Abstract: Accurate and efficient analysis of dental radiographs has become increasingly important for disease diagnosis and clinical decision making. In particula

researcharxiv-cs-cv

GIFT-Eval-CTX: A Benchmark For Contextual Forecasting Model Evaluation

arXiv:2610.17794v1 Announce Type: new Abstract: Contextual forecasting seeks to improve time series forecasting by incorporating forward-looking information, such as planned events, external condition

Applications

LLM Evaluation is a Forecasting Problem

arXiv:2610.17018v1 Announce Type: new Abstract: Standard procedure in machine learning is to randomly split train and test data, emulating IID draws from one underlying data distribution. In practice,

applicationsarxiv-cs-lg

BT-Agent: Deploying Onboard Robot Agents with Behavior Trees and Multi-Fidelity Evaluation

model-releasesarxiv-cs-ro

arXiv:2610.13862v1 Announce Type: new Abstract: Deploying language-model agents onboard mobile robots requires task reasoning and responsive execution within limited compute and memory. We propose BT-

12 Oct 2026

Local Ai

Not Just an Evaluation: A Case Study of Discourse Particle Sensitivity in Language Models

arXiv:2610.13794v1 Announce Type: new Abstract: Discourse particles subtly shape the flow of discourse by reflecting conversational context and interlocutor attitudes, often leaving the truth-conditio

local-aiarxiv-cs-cl

12 Oct 2026

Low-Cost Sensor Calibration for Indoor Air Quality Monitoring: A Dataset, Evaluation Scenarios, and a Lightweight Model

arXiv:2610.11236v1 Announce Type: new Abstract: Low-cost sensors enable scalable indoor air quality monitoring but require calibration because of nonlinear distortions, noise, and temporal drift. The

researcharxiv-cs-lg

9 Oct 2026

TRACE: Diagnosing Verifier Brittleness in Agentic Evaluation

arXiv:2610.11678v1 Announce Type: new Abstract: Verifier scores now serve as both benchmark metrics and training rewards for large language model (LLM) agents, and a change in score is routinely read

9 Oct 2026

Embedded Evaluation of Task Admission Coalescing in Decentralized Multi-Robot Systems

model-releasesarxiv-cs-ro

arXiv:2610.09187v1 Announce Type: new Abstract: Multi-robot task allocators in dynamic missions commonly admit newly released tasks immediately, potentially invoking allocation for each new arrival. W

8 Oct 2026

PHRBench: A Behavioral Evaluation of Post-Hallucination Reasoning in LLMs

arXiv:2610.10455v1 Announce Type: new Abstract: Hallucinated information can propagate through multi-stage LLM systems and become part of the context for subsequent reasoning. Existing studies of post

8 Oct 2026

Applications

Memory Depth and Reconstructed Context Width: A Controlled Evaluation of Hierarchical Retrieval

arXiv:2610.08300v1 Announce Type: new Abstract: Long-term conversational memory is becoming an integral component of modern LLM systems. Proposed architectures group records by topics and events, cons

applicationsarxiv-cs-cl

7 Oct 2026

Component-Level Evaluation of Adaptive PINN Training for CFD-Oriented Crystal Growth Simulation

arXiv:2610.05127v1 Announce Type: new Abstract: Physics-informed neural network (PINN) training minimizes a weighted combination of partial differential equation (PDE), boundary-condition, and initial

6 Oct 2026

EXHIB: A Benchmark for Realistic and Diverse Evaluation of Function Similarity in the Wild

arXiv:2604.01554v2 Announce Type: replace-cross Abstract: Binary Function Similarity Detection (BFSD) is a core problem in software security, supporting tasks such as vulnerability analysis, malware c

6 Oct 2026

Probabilistic Pedestrian Forecasts from a Handheld Phone: World-Frame Heat Maps, Visual-Inertial Height Drift, and Evaluation without Ground Truth

model-releasesarxiv-cs-cv

arXiv:2610.04736v1 Announce Type: new Abstract: A pedestrian with a phone could be shown where nearby people will be in the next few seconds, if the forecast stays on the ground while the phone moves,

6 Oct 2026

Evaluation of Sampling Strategies and Physics-Informed Kolmogorov--Arnold Networks in Unbounded Domains

arXiv:2512.12074v2 Announce Type: replace Abstract: Physics-informed neural networks (PINNs) have emerged as an effective approach for solving partial differential equations (PDEs) by incorporating ph

5 Oct 2026

Forecasting from Counterfactual Simulator Rollouts: A Sim2Real Evaluation

arXiv:2610.03662v1 Announce Type: new Abstract: Deploying a new decision policy creates a cold-start problem for prediction models whose targets depend on the policy's actions: historical observations

safetyarxiv-cs-lg

5 Oct 2026

Can LLMs Reason Over Long Horizons? An Empirical Evaluation of Context Strategies for Longitudinal Clinical Reasoning

arXiv:2610.00562v1 Announce Type: cross Abstract: Longitudinal clinical reasoning requires large language models (LLMs) to identify and integrate relevant evidence distributed across extended patient

researcharxiv-cs-ai

False Floors: LLM Safety Routing Evaluations Break Under Distribution Shift

arXiv:2610.01535v1 Announce Type: cross Abstract: Safety routers send each request to one of several models and are judged against the best single model. A major routing benchmark picks that comparato

No One Architecture Fits All: A Cross-Environment Evaluation of Hierarchical Red Team Agents

arXiv:2610.00557v1 Announce Type: cross Abstract: Autonomous red team agents increasingly stress-test AI-enabled cyber defenses by planning strategy and executing multistage attacks. Reinforcement lea

When Harnesses Lose the Signal: Causal Evaluation of Recovery in LLM Agents

arXiv:2610.00372v1 Announce Type: new Abstract: Large language model agents rely on external harnesses to pass information between the model and its environment and to recover from execution errors. Y

ChartRevise: A Dataset and Evaluation Protocol for Exact Chart Editing via Code

arXiv:2609.38642v1 Announce Type: new Abstract: Chart editing requires cross-modal edit grounding, realizing a requested visual change in the code that draws it, with necessary related updates and wit

researcharxiv-cs-ai

1 Oct 2026

Local Ai

Coherence-Aware Distributional Evaluation of Open-Ended Text Generation

arXiv:2609.34240v2 Announce Type: replace-cross Abstract: Existing open-ended generation metrics measure likelihood, lexical diversity, or distributional similarity in generic representation space, ye

local-aiarxiv-cs-ai

30 Sept 2026

Evaluation Choices Shape Biomedical ML Claims: A Pediatric Pneumonia Benchmark Case Study

model-releasesarxiv-cs-cv

arXiv:2609.37848v1 Announce Type: new Abstract: Biomedical machine learning papers often compress model performance into one headline number. That number can look like a property of the model even whe

30 Sept 2026

API Secrets Should Never Become Tokens in the LLM's Vocabulary: A Threat Analysis of API Credential Handling in LLM Agent Systems and an Empirical Evaluation of a Vault-Mediated Execution Boundary

arXiv:2609.33371v1 Announce Type: cross Abstract: Tool-using large language model (LLM) agents turn credential hygiene from a storage problem into an execution-security problem. A key pasted into a pr

agentsarxiv-cs-lg

HeroFrame-Bench: Reference-Anchored Evaluation via Rubric--Ranking Co-Evolution for Movie Hero Frame Selection

arXiv:2609.32280v1 Announce Type: new Abstract: Hero frames are in-film stills used as source imagery for theatrical posters, streaming cover art, film database listings, and other promotional placeme

safetyarxiv-cs-cv

Large Language Models for Automated Cross-Domain Machine Learning Task Type Identification: A Benchmark Dataset and Evaluation

arXiv:2609.35335v1 Announce Type: new Abstract: Machine learning task type identification is essential for constructing valid ML pipelines, yet in practice it is typically specified manually. We inves

Typed Decision Models: An Early Evidence Audit and Evaluation Checklist

arXiv:2609.32160v1 Announce Type: cross Abstract: Typed decision models (TDMs) return probability distributions over caller-defined options without generating text. TypeSafe released Jev, a commercial

What Can a Leaderboard Certify? Compositional Controllability for Fair Evaluation and Training of Biomedical Literature-Review Agents

arXiv:2609.32318v1 Announce Type: new Abstract: Leaderboards rank long-horizon agents by their final outputs. Yet a higher score alone does not establish whether two systems are comparable or which st

Offline Policy Evaluation as a decision support tool for designing Adaptive Experiments

arXiv:2609.30273v1 Announce Type: new Abstract: We investigate how historical data from fixed randomized experiments (A/B tests) can be used to inform the deployment of adaptive experiments based on c

safetyarxiv-cs-lg

28 Sept 2026

Beneath the Scores: Rethinking Hallucination Evaluation for Video Understanding Models

arXiv:2609.28991v1 Announce Type: cross Abstract: Video understanding is increasingly performed by multi-stage LLM agents that separate temporal grounding, visual observation, and reasoning. Yet these

agentsarxiv-cs-ai

25 Sept 2026

Can Labor Markets Function in the Age of AI? The Evaluation Bottleneck in Hiring

arXiv:2609.30058v1 Announce Type: cross Abstract: AI-assisted job-search tools have become increasingly popular by making it easier to find and apply to jobs. But by making it easier for applicants to

researcharxiv-cs-ai

25 Sept 2026

GPT-6-Astra Lights Up Embodied Navigation: Evaluation in Zero-Shot Vision-and-Language Navigation in Continuous Environments