Search: “evaluation”

FormalEvolve: Neuro-Symbolic Evolutionary Search for Diverse Autoformalization

arXiv:2603.19828v3 Announce Type: replace Abstract: Autoformalization aims to produce formal statements that compile and faithfully preserve the intended meaning of informal mathematics. Yet standard

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

FormInv: A Measurement Protocol for Semantic Invariance in Mathematical Reasoning Benchmarks

DGX agent

arXiv:2605.29001v1 Announce Type: cross Abstract: A paraphrase-quality audit of MathCheck (ICLR 2025) detected 4 semantically incorrect paraphrases in 129 groups (3.1%); removing them drops GPT-4o fro

model-releasesarxiv-cs-ai

29 May 2026

GEO-Bench: Benchmarking Ranking Manipulation in Generative Engine Optimization

arXiv:2605.29107v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly rank products, documents, and recommendations for user queries, which makes manipulating these rankings a gr

Inform, Coach, Relate, Listen: Auditing LLM Caregiving Support Roles

arXiv:2605.29473v1 Announce Type: cross Abstract: Language models are increasingly being deployed for conversational support in informal caregiving contexts, where interactions often extend beyond inf

Lexical categories of stem-forming roots in Mapudungun verb forms

arXiv:2502.07623v4 Announce Type: replace Abstract: After developing a computational system for morphological analysis of the Mapuche language, and evaluating it with texts from various authors and st

Metric-Dependent Annotation Saturation for Learning from Label Distributions

arXiv:2605.29797v1 Announce Type: new Abstract: When annotators disagree on a label, the disagreement itself carries signal -- and the number of annotators needed to capture it depends on the evaluati

safetyarxiv-cs-cl

Multimodal LLMs See Sentiment

arXiv:2508.16873v3 Announce Type: replace Abstract: Understanding how visual content conveys sentiment is increasingly important in a digital landscape dominated by imagery. However, sentiment percept

Tutorials

One Mask to Rule Them All: On Hidden Facts after Editing and How to Find Them

arXiv:2605.28839v1 Announce Type: new Abstract: Knowledge editing methods such as ROME and MEMIT update factual associations in transformer models by modifying MLP weights. While evaluated mainly by o

tutorialsarxiv-cs-lg

Pocket-Dentist: On-Device Dental Image Understanding via Efficient Multimodal Large Language Models

arXiv:2605.29299v1 Announce Type: cross Abstract: Evaluations of dental vision-language models remain fragmented across datasets, task definitions and metrics, and often ignore their computational cos

RAISE: RAG Design as an Architecture Search Problem

arXiv:2605.30029v1 Announce Type: new Abstract: Retrieval-augmented generation (RAG) systems expose numerous design choices spanning query rewriting, chunking, retrieval depth, reranking, and context

Rethinking FID Through the Geometry of the Reference Dataset

arXiv:2605.29335v1 Announce Type: cross Abstract: Frechet Inception Distance (FID) is widely used to evaluate image generators, yet lower FID does not always correspond to better sample quality. We sh

The Chain Holds, the Answer Folds: Trace-Answer Dissociation in Reasoning Models Under Adversarial Pressure

arXiv:2605.29087v1 Announce Type: new Abstract: Reasoning models are evaluated on single-turn benchmarks but deployed in multi-turn dialogue, where users push back on correct answers. Under sustained

Traditional machine learning vs. deep learning from dynamic graph representations of proteins' 3D folds in the task of protein structure classification

arXiv:2605.29228v1 Announce Type: new Abstract: Protein structure classification (PSC) uses supervised learning to predict a protein's CATH/SCOP(e) class from the protein's sequence or 3D structural f

researcharxiv-cs-lg

Training Deliberative Monitors for Black-Box Scheming Detection

arXiv:2605.29601v1 Announce Type: cross Abstract: As autonomous agents become more capable of performing real-world tasks, distinguishing scheming behavior from benign task pursuit may become a centra

Adaptive Temporal Gating of Longitudinal Magnetic Resonance Imaging for Alzheimer's Prediction

arXiv:2605.28397v1 Announce Type: new Abstract: Predicting conversion from Mild Cognitive Impairment (MCI) to Alzheimer's Disease (AD) is critical for early intervention. Current deep learning paradig

researcharxiv-cs-cv

AgensFlow: A Coordination-Policy Substrate for Multi-Agent Systems

arXiv:2605.27466v1 Announce Type: cross Abstract: Multi-agent systems built on large language models (LLMs) require many coordination choices that are difficult to fix a priori: which skill protocol t

safetyarxiv-cs-ai

Are Large Pre-trained Vision Language Models Effective Construction Safety Inspectors?

arXiv:2508.11011v2 Announce Type: replace Abstract: Construction safety inspections typically involve a human inspector identifying safety concerns on-site. With the rise of powerful Vision Language M

AssertLLM2: A Comprehensive LLM Benchmark for Assertion Generation from Design Specifications

arXiv:2605.27472v1 Announce Type: cross Abstract: Assertion-based verification (ABV) is a cornerstone of modern hardware design, yet manually translating design intent into formal SystemVerilog Assert

Automated Estimation of Impact Time, Impact Location, and Shuttlecock Speed in Badminton Smashes Using Event Cameras

arXiv:2605.28011v1 Announce Type: new Abstract: Quantifying impact phenomena in badminton smashes is important for evaluating both athletic performance and equipment; however, conventional measurement

safetyarxiv-cs-cv

Beyond pass@k: Redundancy-Aware RLVR for Multi-Sample Code Generation

arXiv:2605.28022v1 Announce Type: new Abstract: LLMs for code generation are commonly evaluated in repeated-sampling settings using Pass@k, where multiple candidate programs are executed against unit

CALM-IT: Generating Realistic Long-Form Motivational Interviewing Dialogues with Dual-Actor Conversational Dynamics Tracking

arXiv:2601.10085v2 Announce Type: replace Abstract: Therapeutic dialogue is not a sequence of isolated responses: client goals, motivation, resistance, and therapeutic alliance evolve over time. Yet c

Chinese Word Boundary Recovery through Character Alignment Projection

model-releasesarxiv-cs-cl

arXiv:2605.28128v1 Announce Type: new Abstract: Chinese word segmentation is especially fragile in non-standard text, where language learner errors and other character-level divergences disrupt the wo

CircuitLM: A Multi-Agent LLM-Aided Design Framework for Generating Circuit Schematics from Natural Language Prompts

arXiv:2601.04505v3 Announce Type: replace Abstract: Generating accurate circuit schematics from high-level natural language descriptions remains a persistent challenge in electronic design automation

DisasterBench: Benchmarking LLM Planning under Typed Tool Interface Constraints

model-releasesarxiv-cs-cl

arXiv:2605.27957v1 Announce Type: new Abstract: Disasters cause severe societal impacts, demanding rapid coordination of heterogeneous AI tools, from satellite analysis to flood prediction and damage

Do LLMs Build World Models From Text? A Multilingual Diagnostic of Spatial Reasoning

arXiv:2605.28277v1 Announce Type: new Abstract: Whether large language models (LLMs) construct internal spatial world models from pure-text descriptions remains contested, and whether such capabilitie

Forget to Know, Remember to Use: Context-Aware Unlearning for Large Language Models

arXiv:2510.17620v2 Announce Type: replace Abstract: Large language models may encode sensitive information or outdated knowledge that needs to be removed, to ensure responsible and compliant model res

GUI Agents for Continual Game Generation

arXiv:2605.28258v1 Announce Type: cross Abstract: Generating a game is not the same as making one that can be played. Despite advances in code generation, existing approaches treat game generation as

Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows

arXiv:2605.27922v1 Announce Type: new Abstract: LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, perform

Let the Results Speak: A Replication-First Paradigm for LLM Behavioral Benchmarking

arXiv:2605.27914v1 Announce Type: cross Abstract: Subjective evaluation of LLM behavior -- empathy, restraint, calibrated emotional tone -- is hard. Human inter-rater agreement on such qualities satur

Measuring Form and Function in Language Models

arXiv:2605.28616v1 Announce Type: cross Abstract: We introduce quantitative metrics for child language acquisition to evaluate language models. Our focus is on the formal syntactic and functional disc

MIRA: A Bilingual Benchmark for Medical Information Response Audit

arXiv:2605.28025v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to provide public-facing health information, yet existing safety evaluations overlook whether respons

MMTABREAL: Real-World Benchmark for Multimodal Table Understanding

arXiv:2505.21771v2 Announce Type: replace-cross Abstract: Multimodal tables i.e. tabular layouts interleaved with charts, maps, icons, and color encodings are ubiquitous in real applications yet remai

MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generation

arXiv:2605.28035v1 Announce Type: new Abstract: In recent years, Multi-Talker Audio-Video Generation (MTAVG) models have shown promising performance on fundamental metrics such as lip-sync and audio-v

OmniEgo-R^2: A Routed Reasoning Framework for the 1st Cross-Domain EgoCross Challenge at CVPR 2026

arXiv:2605.24481v2 Announce Type: replace Abstract: The 1st Cross-Domain EgoCross Challenge at EgoVis, CVPR 2026 evaluates whether multimodal large language models can reason over egocentric videos ac

researcharxiv-cs-cv

On the Equivariant Learning of the Q-tensor Order Parameter

arXiv:2605.27679v1 Announce Type: cross Abstract: We construct and evaluate group-equivariant neural networks for the prediction of the two-dimensional Q-tensor order parameter of nematic liquid cryst

Operational AI Deployment Assurance: Governance-State Orchestration Under Threshold-Sensitive Deployment Conditions -- A Governance Framework for High-Stakes AI Systems

arXiv:2605.27827v1 Announce Type: new Abstract: AI governance frameworks increasingly emphasize fairness, transparency, accountability, and lifecycle risk management in high-stakes domains. However, m

safetyarxiv-cs-ai

PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective

model-releasesarxiv-cs-cl

arXiv:2605.28819v1 Announce Type: cross Abstract: Parameter-efficient finetuning (PEFT) has become the standard approach for adapting large language models, yet evaluations largely emphasize downstrea

PointQ-Bench: Benchmarking Diagnostic and Interpretable Point Cloud Quality Assessment

arXiv:2605.28241v1 Announce Type: new Abstract: Point cloud quality plays a critical role in 3D acquisition, reconstruction, rendering, and perception, yet existing point cloud quality assessment (PCQ

PortBench: A Correlation-Aware, Full-Pipeline Benchmark for LLM-Driven Portfolio Management

arXiv:2605.27887v1 Announce Type: new Abstract: LLMs have shown strong performance across diverse financial tasks, yet portfolio management (PM), a critical financial decision-making task, remains poo

Applications

Principled Algorithms for Optimizing Generalized Metrics in Multi-Label Learning

arXiv:2605.28767v1 Announce Type: new Abstract: Many real-world classification tasks require predicting multiple labels per instance, necessitating the optimization of complex evaluation metrics such

applicationsarxiv-cs-lg

Securing Retrieval-Augmented Generation: A Taxonomy of Attacks, Defenses, and Future Directions

arXiv:2604.08304v2 Announce Type: replace-cross Abstract: Retrieval-augmented generation (RAG) extends large language models (LLMs) with external knowledge, but this access path also introduces securi

STAB: Specification-driven Testing for Algorithmic Bottlenecks

arXiv:2605.27981v1 Announce Type: new Abstract: Evaluating the efficiency of algorithmic code requires test cases that expose runtime bottlenecks. Previous methods generate efficiency test cases eithe

Structured Belief State and the First Precision-Aware Benchmark for LLM Memory Retrieval

arXiv:2605.11325v2 Announce Type: replace-cross Abstract: Every major benchmark for LLM memory systems, LoCoMo foremost, measures whether a model answered correctly, not whether the memory system retr

The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models

arXiv:2601.19926v2 Announce Type: replace-cross Abstract: We present a systematic review of 337 articles evaluating the syntactic abilities of Transformer-based language models (TLMs), reporting on ov

Thermodynamic properties of chemically disordered compounds via AI-driven estimation of partition function with the PULSE method

arXiv:2605.28594v1 Announce Type: cross Abstract: In this article, we present an improved version of the PULSE method (Partition function Unsupervised Learning Sampling and Evaluation) for estimating

Token Optimization Strategies for LLM-Based Oracle-to-PostgreSQL Migration

arXiv:2605.28557v1 Announce Type: cross Abstract: LLMs are increasingly used for software modernization, code translation, and database migration. However, LLM-based Oracle2PostgreSQL migration remain

UserHarness: Harnessing User Minds for Stronger Agent Theory-of-Mind

arXiv:2605.27721v1 Announce Type: cross Abstract: Understanding what a user believes and intends is central to building effective agent assistants. This ability is often evaluated through Theory-of-Mi

Verified Misguidance: Measuring Structural Citation Failures in Search-Augmented LLMs

arXiv:2605.28565v1 Announce Type: cross Abstract: Users of search-augmented LLMs rely on citations as evidence that responses are grounded in real sources, and rarely verify the cited pages themselves

safetyarxiv-cs-ai