Search: “evaluation”

Self-Preference Bias in Rubric-Based Evaluation of Large Language Models

arXiv:2604.06996v1 Announce Type: cross Abstract: LLM-as-a-judge has become the de facto approach for evaluating LLM outputs. However, judges are known to exhibit self-preference bias (SPB): they tend

10 Apr 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

AVC-Reward: Event-Centric Audio-Video Consistency Evaluation

DGX agent

arXiv:2610.22321v1 Announce Type: cross Abstract: As joint audio-video generation models become more capable, they can generate audio-visual scenes exhibiting three forms of complexity: multi-event co

model-releasesarxiv-cs-ai

16 Oct 2026

Evaluation Awareness Beyond English: Do LLMs Act Differently When Tested in Indic Languages?

arXiv:2610.22333v1 Announce Type: new Abstract: Large language models (LLMs) can often tell when they are being tested. This ability, known as evaluation awareness, can lead models to act more cautiou

16 Oct 2026

When Judges Agree and Are Wrong: Consistency-Accuracy Dissociation as a Blind Spot for Label-Free Evaluation

arXiv:2610.21128v1 Announce Type: cross Abstract: Label-free evaluation methods, such as self-consistency filtering, majority-vote aggregation, and agreement-based reliability scoring, rely on an impl

16 Oct 2026

Certify the Decision, Not Every Score: Exact Top-k Selection from Few Evaluations

arXiv:2610.14996v1 Announce Type: new Abstract: Choosing a few robust algorithms or schedules can require evaluating every candidate under many adverse scenarios, even though an agent only needs to id

Did the LLM Answer the Question? A Semantic Reconstruction Game for Jailbreak Evaluation

arXiv:2610.19306v1 Announce Type: new Abstract: Jailbreak attacks against large language models (LLMs) are evaluated using attack success rate (ASR), which depends on correctly identifying responses t

safetyarxiv-cs-lg

Equal-information evaluation of credit assignment in modular financial reinforcement learning

arXiv:2610.14750v1 Announce Type: new Abstract: Credit assignment in modular financial reinforcement learning is difficult to evaluate because an attributed signal may improve a component loss without

Open-MMUnlearning: Unifying Methods and Evaluation for MLLM Unlearning

arXiv:2610.10358v2 Announce Type: replace Abstract: As multimodal large language models (MLLMs) become more capable and widely deployed, concerns about privacy and safety have become increasingly pres

FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion Retrieval

arXiv:2610.17841v1 Announce Type: cross Abstract: Evaluating fashion retrieval systems remains challenging due to the limited availability of benchmarks that capture the fine-grained, open-ended queri

14 Oct 2026

Uncertainty Evaluation: Bridging Machine Learning and Measurement Science

arXiv:2610.18066v1 Announce Type: new Abstract: Reliable uncertainty evaluation is essential for assessing the trustworthiness of machine learning (ML) predictions, particularly in safety-critical app

safetyarxiv-cs-lg

14 Oct 2026

VisExMEM: Visual Evidence-Based Explainable Multimodal Evaluation Metric

arXiv:2610.16441v1 Announce Type: new Abstract: Current image-text evaluation often relies on a global score, which gives limited insight into which semantic details are supported by the image. Explai

14 Oct 2026

RESCUE: Is This Alarm Worth Acting On? Counterfactual Evaluation of VLA Runtime Monitors

arXiv:2610.13516v1 Announce Type: cross Abstract: Runtime failure monitors for robot policies are typically evaluated by how accurately they distinguish failed from successful executions. Yet an accur

12 Oct 2026

Perceptually Aligned Evaluation of Style Transfer

arXiv:2610.10003v1 Announce Type: new Abstract: Style transfer lacks a reliable evaluation standard: ground truth is inherently ill-defined, and existing automatic metrics often fail to reflect human

8 Oct 2026

Quad-State Safety Evaluation of Open-Weight Large Language Models on Non-Canonical Inputs

arXiv:2610.09033v1 Announce Type: new Abstract: Standard safety evaluations of large language models assess harmful requests written in canonical plain text, while models in real-world deployment rout

8 Oct 2026

Beyond the Leaderboard: Multi-Dimensional Evaluation of Dense and Mixture-of-Experts Models for Automated Program Repair

model-releasesarxiv-cs-lg

arXiv:2610.08173v1 Announce Type: cross Abstract: Automated Program Repair (APR) with language models is usually evaluated by whether a generated patch passes the test suite, which can hide difference

7 Oct 2026

Machine learning for journal entry testing: A type-aware evaluation of anomaly detectors under a review budget

arXiv:2610.06133v1 Announce Type: new Abstract: Journal entry anomaly detectors are commonly evaluated on the full population with ROC-AUC, precision and recall, ignoring the review budget and which a

researcharxiv-cs-lg

6 Oct 2026

Finding the Move Is Not Winning the Game: XiangqiBench for Closed-Loop Evaluation of LLM Agents

arXiv:2610.02425v1 Announce Type: new Abstract: Static evaluations credit a language model for naming the right move, but an agent must carry a plan through to a verified outcome while an opponent res

5 Oct 2026

Frozen Scenes, Shifting Winners: Configuration Fragility in Text-to-3D Evaluation

arXiv:2610.00447v1 Announce Type: new Abstract: Can a text-to-3D leaderboard change when every generated scene stays fixed? We audit this question for rendered-image evaluation, where camera settings

Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMs

arXiv:2610.01428v1 Announce Type: cross Abstract: Generalization in large language models (LLMs) is the ability to produce consistent and semantically stable outputs when the same input is expressed i

Measuring Asset and Scene Reconstruction Effects in Real-to-Sim Robot Evaluation

arXiv:2610.00731v1 Announce Type: new Abstract: Simulated evaluation is increasingly used alongside real-world evaluation of robot policies because it is cheaper and easier to repeat; however, its val

safetyarxiv-cs-ro

Ontology-Based Contextual AI Evaluations (OB-CAIE) Methodology

arXiv:2610.00529v1 Announce Type: new Abstract: The ontology-based contextual AI evaluation (OB-CAIE) methodology was developed to address a lack of scientific rigor that arises from unclear testing c

BITS: Rethinking Fair and Comprehensive Evaluation for Irregular Time Series Forecasting

model-releasesarxiv-cs-lg

arXiv:2609.33303v1 Announce Type: new Abstract: Despite recent progress in irregular time series forecasting, the field still lacks a unified benchmark for fair and comprehensive evaluation. Existing

29 Sept 2026

SymbolicArena: A Unified Infrastructure for Benchmark Distillation and Dynamic Evaluation in Symbolic Regression

model-releasesarxiv-cs-lg

arXiv:2609.35113v1 Announce Type: new Abstract: Symbolic regression (SR) seeks concise and interpretable mathematical expressions from data for scientific equation discovery. Existing SR benchmarks fa

29 Sept 2026

Speculative Evaluation of Stochastic LLMs

arXiv:2609.28560v1 Announce Type: cross Abstract: Evaluating a stochastic large language model is costly: benchmark scores estimate expected performance from randomized rollouts, yet uniform repetitio

25 Sept 2026

Evaluation of pre-trained models for pedagogical assessment of novel AI-assisted educational questions

arXiv:2609.27749v1 Announce Type: new Abstract: The surge in AI-assisted generation of educational materials has outpaced our capacity to validate their pedagogical quality. Automated evaluation using

24 Sept 2026

From Sentiment Classification to Actionable and Responsible Feedback: A Scoping Review and Evidence Map of NLP in Student Evaluation of Teaching, 2015-2026

arXiv:2609.27939v1 Announce Type: new Abstract: Natural language processing (NLP) applied to open-ended teaching-evaluation comments (Student Evaluation of Teaching, SET) has tracked the field's techn

safetyarxiv-cs-cl

24 Sept 2026

Optimal Sequential Annotations for Off-Policy Evaluation

arXiv:2609.26707v1 Announce Type: cross Abstract: Offline reinforcement learning and off-policy evaluation evaluates dynamic treatment rules based on retrospectively collected data prior to deployment

safetyarxiv-cs-lg

23 Sept 2026

Beyond EER: Multi-Dimensional Evaluation of Information Leakage in Speaker De-Identification

arXiv:2609.18673v1 Announce Type: cross Abstract: Speaker de-identification (SDID) aims to preserve privacy by concealing speaker identity while maintaining speech utility. However, current evaluation

17 Sept 2026

ERPBench: A State-Grounded Evaluation Paradigm for Computer-Use Agents in Enterprise Software

arXiv:2609.17885v1 Announce Type: new Abstract: Computer-use agents that operate through screenshots and simulated actions are advancing rapidly, yet their evaluation remains anchored to general deskt

17 Sept 2026

Hardware

FAME: An FPGA-Based Platform for Approximate Multipliers Evaluation with Pattern-Guided DNN Retraining

arXiv:2609.17730v1 Announce Type: cross Abstract: Approximate multipliers can reduce hardware area and energy consumption in Deep Neural Network (DNN) inference; however, they introduce computational

hardwarearxiv-cs-lg

17 Sept 2026

Beyond the Name: Demographic Leakage in De-Identified Resumes and Evaluation Artifacts in LLM Bias Audits

arXiv:2609.16501v1 Announce Type: cross Abstract: De-identified resume screening assumes that redacting explicit fields prevents ethnocultural inference; however, recent audits attribute residual leak

16 Sept 2026

Agents

Skill-based Agentic Evaluation for Real-time Data Science Tasks

arXiv:2609.16487v1 Announce Type: new Abstract: We present a framework for evaluating data-science agents on live, continuously updated data using executable ground truth and format-agnostic factoid s

agentsarxiv-cs-ai

16 Sept 2026

Don't Count the Edits, Judge by the Outcome Alone: Reward-Based Evaluation for Grammatical Error Correction

arXiv:2609.15559v1 Announce Type: cross Abstract: Grammatical error correction (GEC) evaluation has traditionally relied on reference or edit overlap, which can penalize valid rewrites that differ fro

15 Sept 2026

Measuring the Cost of Variety Conflation in Multilingual MT Evaluation: Adding Mozambican Xichangana, Nyanja and Sena to FLORES+

arXiv:2609.13847v1 Announce Type: new Abstract: In this paper, we extend FLORES+ with Portuguese-source evaluation sets for three Mozambican Bantu varieties: Xichangana, Mozambican Nyanja, and Sena. W

researcharxiv-cs-cl

15 Sept 2026

Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error

arXiv:2609.14400v1 Announce Type: new Abstract: Agent benchmarks evaluate policy compliance but assume each policy determines a unique correct action. Natural-language policies can violate this assump

15 Sept 2026

MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation

arXiv:2512.14691v3 Announce Type: replace Abstract: Modern multimodal generative models can synthesize visually compelling images and videos, but it remains unclear whether this visual fluency reflect

14 Sept 2026

UFO: Chain-of-Evaluation for Omni-Condition Alignment in Multi-Modal Image Generation

arXiv:2609.12397v1 Announce Type: new Abstract: Multi-modal image generation, particularly subject-driven customization, has garnered growing attention in recent years. Despite the rapid advancement o

14 Sept 2026

Local Ai

Beyond Word Error Rate: A Switch Aware Evaluation of ASR and Audio Language Models on English Yoruba Code-Switched Speech

arXiv:2609.11786v1 Announce Type: new Abstract: Automatic speech recognition (ASR) systems and audio language models (audio LMs) now report low error rates on monolingual benchmarks, but their behavio

local-aiarxiv-cs-cl

11 Sept 2026

Rethinking Sign Language Translation: The Impact of Signer Dependence on Model Evaluation

arXiv:2609.07965v1 Announce Type: cross Abstract: Sign Language Translation has advanced with deep learning, yet evaluations remain largely signer-dependent, with overlapping signers across train/dev/

10 Sept 2026

PhysVidBench: Language-Grounded Evaluation of Physical Commonsense in Text-to-Video Models

arXiv:2507.15824v2 Announce Type: replace Abstract: Text-to-video (T2V) models now produce striking visuals, yet they routinely violate everyday physics; objects float, tools are misused, and causal s

9 Sept 2026

AVENUE: Audio-Video EditiNg Understanding and Evaluation

arXiv:2609.04253v1 Announce Type: cross Abstract: Audio-video (AV) editing aims to modify audio and video content according to a target prompt. Unlike single-modality editing, AV editing requires mode

7 Sept 2026

MoirfEolas and CriochScore: Developing Resources for and the Evaluation of Tokenization Alignment with Irish Morphology

arXiv:2609.05022v1 Announce Type: new Abstract: This paper presents new tokenization resources for Irish and evaluation measures of alignment with the morphological boundaries of the language. We pres

safetyarxiv-cs-cl

7 Sept 2026

Local Ai

LLM Evaluation as Tensor Completion: Low Rank Structure and Semiparametric Efficiency

arXiv:2604.05460v2 Announce Type: replace-cross Abstract: Large language model (LLM) evaluation platforms increasingly rely on pairwise human judgments. These data are noisy, sparse, and non-uniform,

local-aiarxiv-cs-ai

4 Sept 2026

Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation

arXiv:2609.01603v1 Announce Type: cross Abstract: Evaluating software engineering agents on realistic benchmarks is costly, since each task may require multi-step code exploration, modification, and t

2 Sept 2026

How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation

arXiv:2609.01369v1 Announce Type: new Abstract: Reliable evaluation of open-ended question answering remains a bottleneck for measuring answer correctness of modern LLMs. Unlike multiple-choice tasks,

2 Sept 2026

Validity-Aware Jailbreak Evaluation for Large Language Models

arXiv:2609.00498v1 Announce Type: new Abstract: Jailbreak robustness has become central to large language model (LLM) safety evaluation, yet prevailing methodologies rely primarily on refusal behavior

2 Sept 2026

Beyond Semantic Accuracy: Consequence-Aware Evaluation for Safety-Critical Language Understanding

arXiv:2608.24621v2 Announce Type: replace Abstract: Can language models be trusted in safety- critical operations? In such settings, strong per- formance on semantic metrics does not guaran- tee opera

1 Sept 2026

What Does an Evaluation License? A Commit-Bound Census of Claim Replay in Inspect Evals

arXiv:2608.19269v4 Announce Type: replace-cross Abstract: Benchmarks can run without determining what their results license. We freeze a large evaluation collection and attempt to replay its historica