Search: “evaluation”

Evaluating Communicative Belief Updates in Large Language Models via Implicature Recognition and Cancellation

arXiv:2607.25094v1 Announce Type: cross Abstract: Human language is driven by unspoken beliefs and belief updates, making these critical to model for successful communication between large language mo

29 Jul 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Inspect India Evals: An Open Benchmarking Framework for Evaluating Large Language Models in the Indian Linguistic and Cultural Context

DGX agent

arXiv:2607.25375v1 Announce Type: new Abstract: India is a vast nation of over 1.4 billion people, varied by hundreds of diverse and locally specific traditions and cultures and 22 officially recogniz

model-releasesarxiv-cs-cl

29 Jul 2026

What Gets Lost When Memory Becomes Media? Evaluating AI-Generated Oral History Visualization

arXiv:2607.24756v1 Announce Type: cross Abstract: What gets lost when memory becomes media? Diaspora oral-history interviews require a double transformation; first-person recollection to third-person

agentsarxiv-cs-ai

29 Jul 2026

Applications

Evaluating the Impact of Explainable AI on Trust in AI-Assisted Code Review

arXiv:2607.24601v1 Announce Type: cross Abstract: Background: Large language models (LLMs) are increasingly used to automate code review, but the reasoning behind their decisions remains hard to under

applicationsarxiv-cs-ai

Evaluating the Impact of Reviewer Guideline Design on LLM-Based Automated Peer Review

arXiv:2607.22553v1 Announce Type: cross Abstract: Peer review is an essential process in scientific research, yet the growing workload has made its automation increasingly necessary. In this study, we

Applications

Music-Source-Separation-Training (MSST): A Unified Framework for Training and Evaluating Music Demixing Models

arXiv:2607.23395v1 Announce Type: cross Abstract: Music Source Separation (MSS), the task of recovering individual sound components (stems) from a polyphonic mixture, is central to applications rangin

applicationsarxiv-cs-lg

Plans Work in Mysterious Ways: Evaluating a Plan Mode for Spreadsheet Agents

arXiv:2607.23670v1 Announce Type: cross Abstract: Plan Modes have become standard features in agentic programming tools, allowing users to gain transparency and control by working with the agent to de

agentsarxiv-cs-ai

SWE-rebench Multilingual Update (Go, Java, Python, Rust, TS). Evaluated: GLM-5.2, DeepSeek-V4 Pro, Qwen3.6-27B and others

model-releasesr-localllama

Hi everyone! We’ve just released a major update to the leaderboard! We are expanding beyond Python with a new multilingual slice featuring real-world software engineering tasks across 5 languages. Ope

DBA-Bench: A Production-Fidelity Benchmark for LLM-Based Database Operations Agents

model-releasesarxiv-cs-cl

arXiv:2607.22165v1 Announce Type: cross Abstract: LLM-based database agents show promise, but differing task scopes, testbeds, and metrics hinder comparison. We identify four gaps between evaluation a

27 Jul 2026

Distinguishing Artificial from Authentic: Evaluating LLMs for Detecting LLM-Generated Content

arXiv:2607.20446v1 Announce Type: new Abstract: As large language models (LLMs) are increasingly used by students to generate natural language responses and program code, there is growing interest in

researcharxiv-cs-cl

24 Jul 2026

Hardware

Evaluating and Guarding Citation Faithfulness in Agentic Scientific Synthesis

arXiv:2607.20527v1 Announce Type: new Abstract: Agentic LLM systems such as OpenScholar and PaperQA2 read the scientific literature and return cited answers, and both they and their benchmarks already

hardwarearxiv-cs-ai

24 Jul 2026

Safety

Effort-Based Criticality Metrics for Evaluating 3D Perception Errors in Autonomous Driving

arXiv:2603.28029v2 Announce Type: replace-cross Abstract: Criticality metrics such as time-to-collision (TTC) quantify collision urgency but do not distinguish the operational consequences of false-po

safetyarxiv-cs-ro

23 Jul 2026

Safety

Evaluating and Mitigating Gender Bias in Pre-trained Embeddings for ML-based Recruitment

arXiv:2607.20073v1 Announce Type: new Abstract: AI-based recruitment systems that rely on machine learning models trained on historical CV data, risk perpetuating and amplifying social biases. A key c

safetyarxiv-cs-lg

23 Jul 2026

Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models

model-releasesarxiv-cs-cl

arXiv:2607.19608v1 Announce Type: new Abstract: Instruction tuning is meant to make language models follow user requests, yet it is unclear whether small models comply when an instruction conflicts wi

23 Jul 2026

How OpenAI uses human feedback to evaluate and improve LLMs

At ChatGPT scale, user frustration arrives as support tickets, ratings, social posts, and corrections buried inside conversations. OpenAI built a feedback system that can find the pattern behind a com

agentsarize-ai

21 Jul 2026

IssueBench is our internal benchmark for evaluating Engine (a continual learning agent in LangSmith) This blog by @nick_bray dives into why …

model-releasesharrison-chase--x

**IssueBench** is an internal benchmark created by LangSmith to assess the performance of *Engine*, a continual‑learning agent that scans other agents’ traces to identify, cluster, and fix issues. In

20 Jul 2026

Evaluating Vision Foundation Models for Pixel and Object Classification in Microscopy

model-releasesarxiv-cs-cv

arXiv:2603.19802v2 Announce Type: replace Abstract: Deep learning underlies most modern approaches and tools in computer vision, including biomedical imaging. However, for interactive semantic segment

16 Jul 2026

Evaluating Nonuniform Dependability Across Response Conditions: A Conditional Generalizability Framework Illustrated in Automated Essay Scoring

arXiv:2607.11981v1 Announce Type: cross Abstract: Aggregate reliability estimates can obscure heterogeneity in measurement-design burden across response conditions, so a single G- or D-study may misch

15 Jul 2026

Evaluating the Generalizability of Foundation Models for Extreme Environmental Events: Case Study of California Wildfire PM2.5

model-releasesarxiv-cs-lg

arXiv:2607.07951v1 Announce Type: new Abstract: Wildfire smoke events produce extreme PM_{2.5} concentrations that pose severe public health risks, yet forecasting rare, hazardous-level spikes remains

10 Jul 2026

Training and Evaluating Diffusion Policies with Long Context Lengths

arXiv:2606.16447v2 Announce Type: replace-cross Abstract: Imitation learning has enabled highly-dexterous robotic manipulation from RGB observations. Policies trained with these methods, however, typi

10 Jul 2026

Predicting LLM Safety Before Release by Simulating Deployment

arXiv:2607.07184v1 Announce Type: cross Abstract: Pre-deployment safety evaluations aim to inform the downstream risks of releasing a new AI model. Yet most evaluations provide limited evidence about

9 Jul 2026

An Experimental Design Approach to Evaluating Agentic AI's Autonomous Model Discovery

arXiv:2607.06413v1 Announce Type: cross Abstract: Large language model coding agents increasingly perform open-ended data modeling and analysis. These agents are stochastic and adaptive, and therefore

8 Jul 2026

Safety

A Bayesian Framework for Evaluating Scenario Compatibility in Generative Population Synthesis

arXiv:2607.03190v1 Announce Type: cross Abstract: Scenario-based transportation analysis specifies future assumptions through aggregate population targets, whereas generative population synthesis mode

safetyarxiv-cs-ai

Consistent but Miscalibrated: Evaluating LLM Limitations for Risk Communication in Natural Language

arXiv:2607.03882v1 Announce Type: cross Abstract: LLMs are increasingly deployed as post-hoc explainers of AI-generated outputs, yet it remains unclear whether they can reliably communicate probabilis

Differentiate the Evaluator, Not the Program: An Efficient Runtime Representation for Neuro-Symbolic Learning

arXiv:2607.03574v1 Announce Type: cross Abstract: AI systems increasingly propose executable scientific models whose value depends on both their symbolic structure and their fitted continuous paramete

Evaluating Time Series Foundation Models for Electricity Price Forecasting: Contamination Risk, Distributional Shifts, and Covariate Dependence

arXiv:2607.02623v1 Announce Type: new Abstract: Time series foundation models (TSFMs) have shown strong zero-shot forecasting performance, but their generalization in covariate-driven, non-stationary

researcharxiv-cs-lg

Explainable AI for Screening Abuse-Related Trauma in Bangladeshi Children: A Training-Free Multimodal Framework Evaluated on Noise-Aware Synthetic Data

arXiv:2607.04010v1 Announce Type: new Abstract: Bangladesh has an estimated 1.17 mental-health professionals per 100,000 population and only six child psychiatrists nationwide. No Bengali-language, cu

HamQASBench: A Hamiltonian-Informed Diagnostic Benchmark for Evaluating Quantum Architecture Search

arXiv:2607.04845v1 Announce Type: cross Abstract: Quantum Architecture Search (QAS) automates the design of parameterized quantum circuits for variational quantum algorithms, yet existing benchmarks o

HAS-Bench: Evaluating LLM-Based Human-Agent Systems under Configurable Human Participation

arXiv:2607.04329v1 Announce Type: new Abstract: Large language models increasingly operate in settings where humans are active collaborators rather than passive task providers. We introduce HAS-Framew

PPE-Bench: A Benchmark for Evaluating MLLM Unlearning under Private-Public Entanglement

arXiv:2607.02897v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have shown strong capabilities, but they may memorize private information from web data, raising privacy conc

RoboVista: Evaluating Vision Language Models for Diverse Robot Applications

model-releasesarxiv-cs-ro

arXiv:2607.04610v1 Announce Type: new Abstract: Diverse applications for robotics, such as industry and agriculture, require robots to operate across various embodiments, changing visual conditions, a

When Simpler Is Better: Evaluating Translation Pipelines for Medieval Latin Manuscripts

arXiv:2607.03836v1 Announce Type: cross Abstract: Despite remarkable progress in machine translation, Vision Language Models (VLMs) struggle on historical manuscripts, a domain that stresses core Natu

Beyond Skepticism: Evaluating LLMs Pedagogical Intent Reasoning with the Adaptive Pedagogical Vigilance Framework

model-releasesarxiv-cs-cl

arXiv:2607.01581v1 Announce Type: new Abstract: The capacity of Large Language Models (LLMs) to reason about pedagogical intent within instructional communication remains underexplored, particularly i

3 Jul 2026

IsoSci: A Benchmark of Isomorphic Cross-Domain Science Problems for Evaluating Reasoning versus Knowledge Retrieval in LLMs

arXiv:2607.01431v1 Announce Type: cross Abstract: We introduce ISOSCI, a benchmark of isomorphic cross-domain science problem pairs that separates reasoning ability from domain knowledge retrieval in

3 Jul 2026

How to evaluate AI agents, avoid reward hacking, and build better specs

Agent evals are repeatable tests that score whether AI agents completed a task correctly. Learn how to design rubrics, test suites, and trace-based evals that catch failures and prevent reward hacking

agentsarize-ai

2 Jul 2026

BayesBench: Evaluating LLM Belief Trajectories Under Multi-Turn Evidence Accumulation

arXiv:2606.30850v1 Announce Type: new Abstract: Large language models (LLMs) are typically deployed in multi-turn conversations, where each turn provides new evidence that should reduce epistemic unce

1 Jul 2026

Local Ai

Towards Inclusive Mobility Modeling: Characterizing and Evaluating Elderly Trajectory Patterns in Urban Systems

arXiv:2606.31207v1 Announce Type: new Abstract: The rapid advance of smart cities increasingly depends on trajectory data mining, yet underrepresented demographic groups, particularly the elderly, are

local-aiarxiv-cs-ai

1 Jul 2026

A Neuroimaging Simulation Framework for Developing and Evaluating Causal AI

arXiv:2606.28684v1 Announce Type: cross Abstract: Causally linking disease-related factors to image-derived biomarkers provides a powerful pathway to understanding disease mechanisms. Despite growing

researcharxiv-cs-lg

Animation2Code: Evaluating Temporal Visual Reasoning in Video-to-Code Generation

arXiv:2606.28593v1 Announce Type: cross Abstract: While recent vision-language models (VLMs) have achieved significant improvements on static visual-to-code tasks such as generating code for webpages,

Predicting Effects, Missing Distributions: Evaluating LLMs as Human Behavior Simulators in Operations Management

arXiv:2510.03310v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used to simulate human behavior in business, economics, and the social sciences, offering a low-

SVC-Probe: A Framework for Evaluating Perturbation Generalization in Spatial Foundation-Model Embeddings

arXiv:2606.28465v1 Announce Type: cross Abstract: This work examines perturbation generalization in spatial foundation-model embeddings derived from fluorescence microscopy images. Although these mode

SWE-Together: Evaluating Coding Agents in Interactive User Sessions

arXiv:2606.29957v1 Announce Type: cross Abstract: Most coding-agent benchmarks are static: an agent receives a complete task description up front and is judged only by its final code. Real coding assi

Tutorials

What LLMs explain is not what they believe: Evaluating explanation sufficiency under models' own input beliefs

arXiv:2606.28615v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed in high-stakes domains, where free-text explanations such as chain-of-thought and post-hoc rati

tutorialsarxiv-cs-ai

When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queries

arXiv:2606.28332v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used for medical and health-related questions, yet their safety in high-risk medical scenarios remains p

Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks

Explore how the GitHub Copilot agentic harness delivers strong results across multiple benchmarks and leading token efficiency, while maintaining flexibility to choose among more than 20 models. The p

agentsgithub-ai-blog

25 Jun 2026

MortarBench: Evaluating Mortgage Loan Origination Agents

model-releasesarxiv-cs-lg

arXiv:2606.19416v2 Announce Type: replace Abstract: Loan origination is the process by which a lender creates a new loan, from application and underwriting through approval and funding. This process s

24 Jun 2026

From Speech to Text Corpora: Evaluating ASR-Based Data Acquisition for Low-Resource Fongbe and Hausa

model-releasesarxiv-cs-lg

arXiv:2606.22274v1 Announce Type: cross Abstract: Low-resource African languages lack text corpora needed for language model training. We investigate whether ASR pipelines can extend text resources fo

23 Jun 2026

GeoFidelity-Bench: Evaluating Segment-Level Geographic Fidelity in Text-to-Image Street-View Generation