Search: “evaluation”

DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation

model-releasesarxiv-cs-ro

arXiv:2607.08751v1 Announce Type: new Abstract: Building general-purpose dexterous manipulation policies requires benchmarks that go beyond isolated tasks to systematically evaluate policies across di

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings

DGX agent

arXiv:2508.11847v4 Announce Type: replace-cross Abstract: We propose a method for evaluating the robustness of widely used LLM ranking systems -- variants of a Bradley--Terry model -- to dropping a wo

researcharxiv-cs-lg

10 Jul 2026

Generative Action Tell-Tales: Assessing Human Motion in Synthesized Videos

arXiv:2512.01803v3 Announce Type: replace Abstract: Despite rapid advances in video generative models, robust metrics for evaluating visual and temporal correctness of complex human actions remain elu

MulTTiPop: A Multitrack Transcription Dataset for Pop Music

arXiv:2607.08756v1 Announce Type: cross Abstract: We present MulTTiPop, a dataset of pop music segments and their associated multitrack MIDI recordings for the evaluation of automatic music transcript

researcharxiv-cs-lg

PLURAL: A Global Dataset for Value Alignment

arXiv:2607.08034v1 Announce Type: cross Abstract: Large language models (LLMs) are used worldwide, yet disproportionately reflect Western values, limiting their ability to represent diverse value syst

safetyarxiv-cs-ai

Robustness Quantification for Discriminative Models: a New Robustness Metric and its Application to Dynamic Classifier Selection

arXiv:2603.23318v2 Announce Type: replace Abstract: Among the different possible strategies for evaluating the reliability of individual predictions of classifiers, robustness quantification stands ou

researcharxiv-cs-lg

Shift & Drift: A Zero-Shot Benchmark for Generalizable and Robust Autonomous Driving Motion Planning

arXiv:2607.07844v1 Announce Type: cross Abstract: While closed-loop motion planners trained on large-scale, object-level datasets, e.g., nuPlan, demonstrate strong in-distribution (ID) performance, th

SPHINX: First Explain, Then Explore

arXiv:2606.17482v2 Announce Type: replace Abstract: Generating adversarial driving scenarios is critical for evaluating and improving autonomous vehicle decision-making systems in simulation. Recent a

safetyarxiv-cs-cv

The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs

arXiv:2607.08734v1 Announce Type: new Abstract: Post-training quantization is widely used to deploy large language models in resource-constrained settings, yet its evaluation relies almost exclusively

applicationsarxiv-cs-ai

Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

arXiv:2607.08284v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them. Ho

Tutorials

Bayesian Optimization of Genetic Algorithm Hyperparameters in a Multi-Fidelity Framework for Efficient Lattice Material Design

arXiv:2607.07289v1 Announce Type: cross Abstract: This study presents a multi-fidelity framework for the systematic optimization of genetic algorithm (GA) hyperparameters. The framework integrates thr

tutorialsarxiv-cs-ai

Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering

arXiv:2607.07179v1 Announce Type: new Abstract: Document Visual Question Answering (DocVQA) presents a complex multimodal challenge, requiring models to exploit visual, textual, and layout information

LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?

arXiv:2510.09595v3 Announce Type: replace Abstract: Competitive programming problems are increasingly used to evaluate the coding capabilities of large language models (LLMs) due to their complexity a

Optimization-Embedded Active Multi-Fidelity Surrogate Learning for Multi-Condition Airfoil Shape Optimization

model-releasesarxiv-cs-lg

arXiv:2603.17057v2 Announce Type: replace-cross Abstract: Active multi-fidelity surrogate modeling is developed for multi-condition airfoil shape optimization to reduce high-fidelity CFD cost while re

Two-Stage Multi-Modal Fusion with Adaptive Alignment for Action Quality Assessment

arXiv:2607.07438v1 Announce Type: new Abstract: Action Quality Assessment (AQA) aims to evaluate how well a person performs a movement, which is essential in applications such as sports scoring, skill

Validate the Dream Before You Trust Its Verdict: Admissibility for World-Model Simulators

arXiv:2607.07196v1 Announce Type: cross Abstract: Across robotics, World Models (WMs) are increasingly used to evaluate action policies by simulating the consequences of actions in an imagined world,

safetyarxiv-cs-ai

aiAuthZ: Off-Host, Identity-Bound Authorization for AI Agents

arXiv:2607.05518v1 Announce Type: cross Abstract: AI agents issue tool calls on the basis of text they cannot verify, so any party who controls part of the context can forge the appearance of authorit

As coding models improve, evals need to become harder, fairer, and more trustworthy. Better benchmarks help the field understand real progre…

As AI coding models advance in capability, current evaluation benchmarks must evolve to remain challenging and meaningful measures of progress. OpenAI argues that improved benchmarks need to be harder

model-releasesopenai--x

Auditing of Unlearning Algorithms

model-releasesarxiv-cs-lg

arXiv:2607.05898v1 Announce Type: new Abstract: Evaluating whether unlearning algorithms truly remove training data influence remains an open challenge. We propose a practical auditor that computes da

Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents

arXiv:2607.05775v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly evaluated on their ability to use tools, plan multi-step tasks, coordinate with other agents, and ope

Agents

Point as Skeleton: Accumulated Point Cloud Enhanced Autoregressive Generation for Closed-Loop Autonomous Driving Simulation

arXiv:2607.06516v1 Announce Type: new Abstract: Evaluating end-to-end autonomous driving (E2E-AD) remains challenging, as existing driving simulation methods often trade off closed-loop interactivity

agentsarxiv-cs-cv

A Failure-Mode Benchmark for Polymorphic Sybil Poisoning in RAG

arXiv:2607.03739v1 Announce Type: cross Abstract: We release a benchmark and failure-mode-aware evaluation framework for grounded QA under coordinated retrieval poisoning. The framework partitions rea

A Large-Scale Dataset and a New Method for RemoteSensing Traffic Object Segmentation

arXiv:2607.03945v1 Announce Type: new Abstract: Remote sensing imagery plays a crucial role in evaluating regional transportation capacity. However, existing segmentation datasets often lack diversity

A Vision Based System for Guided and Collaborative Reconstruction of Fragmented Documents

arXiv:2607.03621v1 Announce Type: new Abstract: This paper presents the development and evaluation of a collaborative system for real-time reconstruction of fragmented paper documents in the context o

researcharxiv-cs-cv

Agent-driven Long-tail Simulation for Autonomous Driving

arXiv:2607.04331v1 Announce Type: cross Abstract: Evaluating autonomous driving systems in closed-loop settings requires realistic and interactive simulation, yet existing simulators largely rely on l

AgenticPD: A Stage-Aware Agentic Framework for Physical Design QoR Optimization

arXiv:2607.04758v1 Announce Type: new Abstract: Physical design quality-of-results~(QoR) optimization is hard and expensive. Choices made at one stage can help or hurt later stages. Each evaluation re

BGP route policies: Top 3 use cases by customer demand

When we first made BGP route policies for Cloud Router generally available over a year ago, our goal was to give network administrators deep, programmable control over how network paths are evaluated

safetygoogle-cloud-ai

CCFM: Collision-Constrained Flow Matching for Safety-Critical Scenario Generation

arXiv:2607.04451v1 Announce Type: new Abstract: Evaluation of autonomous vehicle (AV) planners in safety-critical closed-loop simulation is essential for real-world deployment. However, generating con

safetyarxiv-cs-cv

Classroom Behavior Monitoring with YOLO An Empirical Study in Higher Education Settings

arXiv:2607.02580v1 Announce Type: new Abstract: Classroom behavior monitoring plays a vital role in evaluating student engagement and improving teaching effectiveness. Traditional observation methods

applicationsarxiv-cs-cv

Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility

arXiv:2501.10711v5 Announce Type: replace-cross Abstract: Code-related benchmarks play a critical role in evaluating large language models (LLMs), yet their quality fundamentally shapes how the commun

Deep Learning for Semen Analysis in Male Infertility: Computer Vision, Multimodal Fusion, and Clinical Translation

arXiv:2607.05311v1 Announce Type: new Abstract: Male infertility contributes substantially to the global infertility burden, and sperm analysis remains central to diagnosis, treatment planning, and as

applicationsarxiv-cs-cv

Double Fuzzy Probabilistic Interval Linguistic Term Set and a Dynamic Fuzzy Decision Making Model based on Markov Process with tts Application in Multiple Criteria Group Decision Making

arXiv:2111.15255v2 Announce Type: replace-cross Abstract: The probabilistic linguistic term has been proposed to deal with probability distributions in provided linguistic evaluations. However, becaus

researcharxiv-cs-ai

Fun-TSG: A Function-Driven Multivariate Time Series Generator with Variable-Level Anomaly Labeling

arXiv:2604.14221v2 Announce Type: replace Abstract: Reliable evaluation of anomaly detection methods in multivariate time series remains an open challenge, largely due to the limitations of existing b

GestaltMML: Enhancing Rare Genetic Disease Diagnosis through Multimodal Machine Learning Combining Facial Images and Clinical Text

arXiv:2312.15320v3 Announce Type: replace-cross Abstract: Individuals with suspected rare genetic disorders often undergo multiple clinical evaluations, imaging studies, laboratory tests, and genetic

researcharxiv-cs-cv

Local Ai

GPU-Accelerated Polygonal Signed Distance Functions for Real-Time Collision Avoidance

arXiv:2607.04310v1 Announce Type: new Abstract: Optimization-based local planning and control require high-rate collision-avoidance constraint evaluation over a prediction horizon. In obstacle-dense e

local-aiarxiv-cs-ro

Measuring 3D Spatial Geometric Consistency in Dynamic Video Generation

arXiv:2603.19048v2 Announce Type: replace Abstract: Recent generative models can produce high-fidelity videos, yet they often exhibit 3D spatial geometric inconsistencies. Existing evaluation methods

researcharxiv-cs-cv

MTEB-PT: A Text Embedding Benchmark for Brazilian Portuguese

model-releasesarxiv-cs-cl

arXiv:2607.04581v1 Announce Type: new Abstract: Text embeddings for Portuguese have no dedicated benchmark: evaluation rests on translated corpora such as English MS MARCO or on thin multilingual cove

Agents

my hot take is that evals are the only part of agent engineering that require real thinking

Harrison Chase argues that evaluations (evals) are the most intellectually demanding aspect of agent engineering, distinguishing them from other engineering tasks that may be more routine or formulaic

agentsharrison-chase--x

Occluding the Solution Space: Planner-Agnostic Adversarial Attacks on Tolerance-Aware Manipulation

arXiv:2607.03758v1 Announce Type: new Abstract: Adversarial attacks on motion planning are crucial for evaluating and quantifying the intrinsic robustness of robotic manipulation. However, existing ap

applicationsarxiv-cs-ro

Retroactive Chain-of-Thought (RetroCoT): Forensic Reconstruction Prompts as a Safety Diagnostic Across Model Generations

arXiv:2607.04645v1 Announce Type: cross Abstract: Safety alignment in large language models is typically evaluated against direct, imperative harmful requests. We show that this alignment is highly co

Solve the Missing First Step: Can VLMs Standardize Raw Heterogeneous Medical Data?

arXiv:2607.04694v1 Announce Type: new Abstract: As vision-language models (VLMs) are increasingly applied to medical AI, existing benchmarks mainly focus on evaluating their diagnosis ability over giv

Submitted and Diagnostic Analysis of Full-Text Temporal Retrieval for LongEval-Sci

arXiv:2607.04088v1 Announce Type: cross Abstract: LongEval-Sci evaluates scientific retrieval under collection change, where a system should be effective on the current corpus and remain usable as doc

researcharxiv-cs-ai

Agents

The Language of Bargaining: Linguistic Effects in LLM Negotiations

arXiv:2601.04387v2 Announce Type: replace Abstract: Negotiation is a core component of social intelligence, requiring agents to balance strategic reasoning, cooperation, and social norms. Recent work

agentsarxiv-cs-ai

Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation

arXiv:2607.03487v1 Announce Type: cross Abstract: Mutual information (MI) estimation is a central problem in machine learning and statistics; however, existing benchmarks typically evaluate estimators

applicationsarxiv-cs-ai

TRACER: Early Failure Detection for Task-Oriented Dialogue

arXiv:2607.03974v1 Announce Type: new Abstract: Task-oriented dialogue systems often fail before the final breakdown is obvious, but most evaluation only measures failure after the conversation has al

researcharxiv-cs-cl

TrendFact: A Benchmark Towards Hotspot Perception in Automatic Fact-Checking

model-releasesarxiv-cs-cl

arXiv:2410.15135v5 Announce Type: replace Abstract: With the surge of online misinformation, Large Language Models (LLMs) and Reasoning Large Language Models (RLMs) serving as Automatic Fact-Checking

AIriskEval-edu: New Dataset for Risk Assessment in AI-mediated K-12 Educational Explanations

arXiv:2607.01934v1 Announce Type: cross Abstract: This work introduces AIriskEval-edu-db2, a new dataset designed to train and evaluate auditors based on LLMs for an explainable pedagogical risk asses

3 Jul 2026

Benchmarking Federated Learning and Knowledge Distillation for Point Cloud Classification

arXiv:2607.01272v1 Announce Type: cross Abstract: Deploying 3D point cloud analysis in privacy-sensitive, resource-constrained settings faces two barriers: data cannot be centralized, and models must