Search: “evaluation”

From Graph Retrieval to Schema Realization: Counterfactual Validation for Text-to-SPARQL over Heterogeneous Knowledge Graphs

arXiv:2508.01815v2 Announce Type: replace-cross Abstract: Text-to-SPARQL maps natural-language questions to executable SPARQL queries over RDF knowledge graphs. While standard evaluations often fix th

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

FVSpec: Real-World Property-Based Tests as Lean Challenges

DGX agent

arXiv:2606.01008v1 Announce Type: cross Abstract: We present a benchmark for evaluating AI models and agents on real-world formal software verification tasks. We first scrape 11,039 property-based tes

model-releasesarxiv-cs-ai

2 Jun 2026

Identifying High-Confidence Social Biases in LLMs for Trustworthy Conversational Tutoring Agents

arXiv:2606.01584v1 Announce Type: cross Abstract: Conversational tutoring agents have been shown to improve learning engagement and student outcomes, and large language models (LLMs) are increasingly

'I've Seen How This Goes': Characterizing Diversity via Progressive Conditional Surprise

arXiv:2606.01811v1 Announce Type: cross Abstract: Measuring the diversity of creative outputs is central to evaluating post-training mode collapse, comparing decoding strategies, and quantifying creat

LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?

arXiv:2602.16902v4 Announce Type: replace Abstract: We introduce LLM-Wikirace, a benchmark for evaluating planning, reasoning, and world knowledge in large language models (LLMs). In LLM-Wikirace, mod

Agents

PairedGTA: Generating Driving Datasets for Controlled Photometric Shift Analysis

arXiv:2606.01192v1 Announce Type: new Abstract: Evaluating the performance of visual perception systems for autonomous driving is essential to ensure reliable operation across diverse environmental sc

agentsarxiv-cs-cv

ProtStructQA: A Denotation Threshold in Protein Structural Reasoning

arXiv:2606.00451v1 Announce Type: new Abstract: Protein-language systems are often evaluated by whether they generate plausible biological text, but a structural question has a sharper semantics: it d

RealityTest: How People Probe AI Identity and Whether Models Disclose It

arXiv:2606.00168v1 Announce Type: new Abstract: AI systems are increasingly deployed in conversational settings where users may be uncertain whether they are speaking with a human or an AI. Despite mo

RescueBench: Can Embodied Agents Save Lives in the Wild ?

model-releasesarxiv-cs-cv

arXiv:2606.01848v1 Announce Type: new Abstract: Search-and-rescue (SAR) requires embodied agents to explore unfamiliar environments under multimodal uncertainty, perform multi-stage interactions, and

Revisiting Parameter-Based Knowledge Editing in Large Language Models: Theoretical Limits and Empirical Evidence

arXiv:2606.00570v1 Announce Type: cross Abstract: Parameter-based knowledge editing updates the internal knowledge of large language models (LLMs) via localized weight modifications and has attracted

SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes

arXiv:2602.09153v2 Announce Type: replace-cross Abstract: Simulation has become a key tool for training and evaluating home robots at scale, yet existing environments fail to capture the diversity and

ShaplEIG: Bayesian Experimental Design for Shapley Value Estimation

arXiv:2606.02247v1 Announce Type: cross Abstract: Shapley values are a principled attribution measure widely used in interpretable machine learning, but their exact computation scales exponentially wi

researcharxiv-cs-lg

TECCI: Tricky Edits of Collected and Curated Images

arXiv:2606.01213v1 Announce Type: cross Abstract: Despite tremendous recent progress, current text-guided image editing methods still struggle with many aspects of editing involving instruction follow

Temporal Evidence Routing with Structured Visual Evidence for TimeLogicQA

arXiv:2606.01106v1 Announce Type: new Abstract: TimeLogicQA evaluates whether video question answering systems can reason over temporal relations such as event existence, ordering, persistence, bounda

researcharxiv-cs-cv

The Right Inference Strategy Is All You Need: Nearly Training-Free Domain-Wise Inference for EgoCross Challenge

arXiv:2606.00829v1 Announce Type: new Abstract: EgoCross evaluates multimodal large language models on egocentric video question answering under substantial domain shift, where test videos come from s

researcharxiv-cs-cv

TrustLDM: Benchmarking Trustworthiness in Language Diffusion Models

arXiv:2606.00023v1 Announce Type: cross Abstract: The rapid development of Language Diffusion Models (LDMs) challenges the dominant position of auto-regressive competitors in language processing. Howe

TukaBench: A Culturally Grounded Jailbreak Benchmark for African Languages

arXiv:2606.01322v1 Announce Type: cross Abstract: Safety evaluation of Large Language Models (LLMs) remains heavily English-centric, leaving Low-Resource Languages (LRLs), particularly African ones, c

Uncovering Competency Gaps in Large Language Models and Their Benchmarks

arXiv:2512.20638v2 Announce Type: replace-cross Abstract: The evaluation of large language models relies heavily on standardized benchmarks. These benchmarks provide useful aggregated metrics, but can

When Single Answer Is Not Enough: Rethinking Single-Step Retrosynthesis Benchmarks for LLMs

arXiv:2602.03554v2 Announce Type: replace-cross Abstract: Recent progress has expanded the use of large language models (LLMs) in drug discovery, including synthesis planning. However, objective evalu

Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025

arXiv:2606.02255v1 Announce Type: cross Abstract: Human annotation is the empirical foundation of much NLP research, from dataset construction to model evaluation, but papers often leave unclear who p

researcharxiv-cs-ai

X-Stream: Exploring MLLMs as Multiplexers for Multi-Stream Understanding

model-releasesarxiv-cs-cv

arXiv:2606.02482v1 Announce Type: new Abstract: While video streaming understanding has made significant strides, real-world applications, such as live sports broadcasting, autonomous driving, and mul

Agents

A Unified and Reproducible Experimentation Framework for Speech Understanding

arXiv:2605.30899v1 Announce Type: cross Abstract: Speech foundation models and Speech LLMs have advanced speech understanding, yet deployment-oriented model selection is hindered by non-comparable eva

agentsarxiv-cs-ai

Tutorials

BAT: Better Audio Transformer Guided by Convex Gated Probing

arXiv:2602.16305v2 Announce Type: replace-cross Abstract: Probing is widely adopted in computer vision to faithfully evaluate self-supervised learning (SSL) embeddings, as finetuning may misrepresent

tutorialsarxiv-cs-lg

Beyond Classification: Dynamic Adapter Routing for Continual Multimodal Retrieval

arXiv:2605.31229v1 Announce Type: cross Abstract: While retrieval is a core function of vision-language models, continually updating these models for retrieval tasks remains critically underexplored.

researcharxiv-cs-ai

Beyond Static Dialogues: Benchmarking Realistic, Heterogeneous, and Evolving Long-Term Memory

arXiv:2605.31086v1 Announce Type: new Abstract: In existing memory benchmarks for Large Language Models (LLMs), the evaluated dialogue sessions often lack long-term semantic consistency, and the under

BlueFin: Benchmarking LLM Agents on Financial Spreadsheets

arXiv:2605.30907v1 Announce Type: cross Abstract: We present BlueFin, a benchmark that tasks large language model (LLM) agents with synthesis, manipulation, and comprehension tasks over spreadsheet wo

Counterfactual Trace Auditing of LLM Agent Skills

arXiv:2605.11946v2 Announce Type: replace Abstract: Large Language Model agents are increasingly augmented with agent skills. Current evaluation methods for skills remain limited. Most deployed benchm

Does Visual Information Play a Decisive Role in Vision-Language-Action Model Driving Behavior?

arXiv:2605.31041v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models have demonstrated promising capability in autonomous driving, highlighting the potential of unified multimodal arc

Dynamical local Frechet curve regression in manifolds

arXiv:2505.05168v3 Announce Type: replace-cross Abstract: Under mild conditions, this paper derives a least-squares local linear Frechet curve predictor for response and regressor evaluated in a separ

researcharxiv-cs-lg

Generalistic or Specific Embeddings, Which is Better? An Empirical Study on Search for Clinical Coding in Non-English Languages

arXiv:2605.30529v1 Announce Type: cross Abstract: Sentence-embedding models for semantic search are overwhelmingly developed and evaluated on English corpora. When applied to clinical retrieval in oth

'Intelegi Romaneste?'' A Recipe for Romanian Vision-Language Models

arXiv:2605.31401v1 Announce Type: new Abstract: Vision-Language Models (VLMs) largely follow the text-only LLM trajectory, excelling on English benchmarks but sharply degrading on low-resource languag

researcharxiv-cs-cl

LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis

arXiv:2605.30434v1 Announce Type: cross Abstract: Real-world data analysis is inherently iterative, yet existing benchmarks mostly evaluate isolated or short interactive tasks, leaving agents' ability

MAAT: Multi-phase Adapter-Aware Targeted Unlearning

arXiv:2605.30514v1 Announce Type: cross Abstract: Machine unlearning evaluation is structurally skewed: Why-type questions, which probe causal and relational knowledge, comprise less than 0.06% of Cou

Neuron-Level Interventions for Gendered and Gender-Neutral Generation in Language Models

arXiv:2605.30717v1 Announce Type: new Abstract: Language models (LMs) can produce gendered language and stereotypes even when given neutral prompts. Most prior work on gender bias in LMs primarily exa

safetyarxiv-cs-cl

REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge

arXiv:2603.17145v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly deployed as automated evaluators that assign numeric scores to model outputs, a paradigm known a

Reinterpreting Safety Thresholds as Neuron Spiking Thresholds

arXiv:2605.30368v1 Announce Type: cross Abstract: Surrogate Safety Measures (SSMs) are extensively utilised in the evaluation of traffic risk in automated driving contexts. However, the majority of SS

Local Ai

The Architecture of Errors: From Universal Impossibility to Patch-Local LLM Reliability

arXiv:2605.30628v1 Announce Type: cross Abstract: Universal LLM reliability is not a finite-library problem: across all possible tasks, tools, schemas, knowledge sources, and evaluator expectations, n

local-aiarxiv-cs-ai

TraceGraph: Shared Decision Landscapes for Diagnosing and Improving Agent Trajectories

arXiv:2605.31308v1 Announce Type: new Abstract: Agent benchmarks increasingly record rich interaction trajectories, yet evaluation often reduces each rollout to a pass rate or reward score. We introdu

Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classification

model-releasesarxiv-cs-lg

arXiv:2509.24901v4 Announce Type: replace-cross Abstract: Although probing frozen models has become a standard evaluation paradigm, self-supervised learning in audio defaults to fine-tuning when pursu

A Composable Multimodal Framework for cine CMR-Text-Driven Prediction of Heart Failure Outcomes

arXiv:2502.16548v3 Announce Type: replace-cross Abstract: Objective. Heart failure is one of the leading causes of death worldwide, with millions of deaths each year, according to data from the World

researcharxiv-cs-ai

Beyond Attack Success Rate: Temporal Logit Observability for LLM Safety Failures

arXiv:2605.29629v1 Announce Type: new Abstract: Attack Success Rate (ASR) evaluates each jailbreak with a single yes/no label at the end of generation, telling us whether a failure happened but not ho

BioRefusalAudit: Auditing Biosecurity Refusal Depth Using General and Domain-Fine-Tuned Sparse Autoencoders

arXiv:2605.30162v1 Announce Type: new Abstract: Biosecurity evaluations of language models typically ask whether models produce hazardous output. This paper asks a complementary question: when a model

CLUBench: A Clustering Benchmark

model-releasesarxiv-cs-lg

arXiv:2605.29933v1 Announce Type: new Abstract: Clustering is a fundamental problem in data science with a long-standing research history, yielding numerous insightful algorithms. Despite this progres

Do Physics Foundation Models Learn Generalizable Physics? A Bias-Aware Benchmark Across Physical Regimes and Distribution Shifts

arXiv:2605.29283v1 Announce Type: cross Abstract: Recent physics foundation models claim general spatiotemporal forecasting ability, yet their evaluations often collapse performance into a single aver

Local Ai

EXACT-MPPI: Exact Signed-Distance Navigation for Arbitrary-Footprint Robots from Point Clouds via Path Integral Control

arXiv:2605.29663v1 Announce Type: new Abstract: Ground robots often carry payloads, implements, or other attachments that turn their effective footprint into complex, non-convex shapes. Navigating saf

local-aiarxiv-cs-ro

Gram: Assessing sabotage propensities via automated alignment auditing

arXiv:2605.30322v1 Announce Type: cross Abstract: We introduce Gram, an automated alignment auditing framework to assess the propensity of AI agents to engage in sabotage. We evaluate Gemini models ac

Latent Performance Profiling of Large Language Models

arXiv:2605.30018v1 Announce Type: new Abstract: Large language models (LLMs) frequently achieve impressive scores on standardized benchmarks, yet accuracy alone offers a limited view of their capabili

LLMBridge: An LLM Pipeline for End-to-end Referential Bridging Resolution in English

arXiv:2605.29048v1 Announce Type: new Abstract: In this paper, we introduce LLMBridge, a new LLM based system for the task of end-to-end referential bridging resolution in English. Our bridging resolu

researcharxiv-cs-cl