Search: “evaluation”

Beyond Outcomes: Dual-View Relational Learning for Efficient Agent Benchmarking

arXiv:2609.18909v1 Announce Type: cross Abstract: Agent benchmarks are substantially more costly to evaluate than conventional LLM benchmarks. Benchmark compression is therefore a natural solution, ye

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Do Social Patterns Hold in Synthetic Data? Analyzing Cyberbullying Dynamics in LLM-Generated and Authentic Dialogues

DGX agent

arXiv:2609.17549v1 Announce Type: new Abstract: Cyberbullying (CB) is a complex social phenomenon characterized by repeated aggression, power imbalance, and multi-party interaction. Although large lan

model-releasesarxiv-cs-cl

17 Sept 2026

GYROval: A Robust Benchmark for Cultural Value Orientation in Large Language Models

arXiv:2609.18384v1 Announce Type: cross Abstract: We present a robust benchmark for measuring cultural value orientation in large language models on the two Inglehart-Welzel axes over several domains

Made in Hungary: Comments on the performance of generative language models

arXiv:2609.18284v1 Announce Type: new Abstract: In recent years, three initiatives have emerged to develop generative language models in Hungary. The motivation behind them is the same. For Hungarian,

researcharxiv-cs-cl

ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks

arXiv:2609.18805v1 Announce Type: cross Abstract: Coding agents are typically evaluated with desired behavior specified through issues or instructions. In practical web development, however, agents ma

SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes

arXiv:2609.16233v1 Announce Type: cross Abstract: Vision-language models excel at 2D image understanding but remain limited in 3D spatial reasoning. Progress is hindered by limitations in current benc

Script Fragmentation and Format: What Drives the English-Bengali Performance Gap in Open LLMs?

arXiv:2507.23248v2 Announce Type: replace Abstract: Bengali is spoken by more than 230 million people, yet no standardized instrument evaluates large language models (LLMs) on Bengali across the task

The Neverwhere Visual Parkour Benchmark Suite

arXiv:2609.16443v1 Announce Type: cross Abstract: State-of-the-art visual locomotion controllers are increasingly capable at handling complex visual environments, making evaluating their real-world pe

Which Pretext Task Transfers? Self-Supervised Pretraining Objectives for Lung Ultrasound

arXiv:2609.16551v1 Announce Type: new Abstract: Self-supervised learning (SSL) can reduce the need for labelled medical images, but the choice of pretext objective remains unclear for lung ultrasound

researcharxiv-cs-cv

A Three-Axis Stress Test of LLM vs Classical ML for Network Intrusion Detection under Distribution Shift and Adversarial Evasion

arXiv:2609.13511v1 Announce Type: cross Abstract: Large language models are increasingly benchmarked against classical machine learning for network intrusion detection (NIDS), almost always using same

researcharxiv-cs-ai

Hardware

Accelerating HKTex without Mesh Eigensystems: Local Unfolding and Randomized Thermal Features

arXiv:2609.14105v1 Announce Type: new Abstract: Heat Kernel Textures (HKTex) represent surface appearance with intrinsic anisotropic kernels, but evaluate them using 50 global Laplace-Beltrami eigende

hardwarearxiv-cs-cv

Are LLMs Good Financial User Simulators? A Preliminary Study

arXiv:2609.15727v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as user simulators, but their ability to reproduce evolving individual financial decisions remains un

researcharxiv-cs-ai

Building a Production Greek-English Speech Recognizer

arXiv:2609.13498v1 Announce Type: cross Abstract: We report a multi-month engineering program to build Sophea, a production bilingual Greek-English automatic speech recognition system. We evaluate the

Comparing Trajectories from Positions Alone: Curvature-Based Time Alignment and Drift Error Metric

arXiv:2609.14936v1 Announce Type: new Abstract: In field robotics, acquiring independent large-scale reference trajectories more accurate than the evaluated estimates remains an open challenge. The do

safetyarxiv-cs-ro

DepthBenchCAD: When Does Deeper Auditing Yield More Reliable Conclusions?

arXiv:2609.15122v1 Announce Type: cross Abstract: Generative CAD models are expected to remain behaviorally correct after parameter edits, so increasing the number of edit checks is often treated as a

Designing UNICORN: a Unified Benchmark for Imaging in Computational Pathology, Radiology, and Natural Language

arXiv:2603.02790v2 Announce Type: replace Abstract: Foundation models are changing the way we develop medical artificial intelligence. By learning broadly generalizable features across diverse data mo

Domain-Specific Jargon in Large Language Models: A Comparative Analysis between General-Purpose and Specialist Models

arXiv:2609.13556v1 Announce Type: cross Abstract: Large Language Models (LLMs) have shown remarkable proficiency on general-purpose tasks, yet their performance often degrades in highly-specialized te

E2A-Bench: Benchmarking Evidence-to-Action Reliability in Financial Chart Reasoning

arXiv:2609.14302v1 Announce Type: new Abstract: Can financial vision-language models (VLMs) turn chart evidence into reliable action recommendations? Existing hallucination evaluations are mostly clai

Agents

From Process Loss to Assembly Bonus: Human-Grounded Diagnosis of Multi-Agent LLM Collaboration

arXiv:2609.13261v1 Announce Type: cross Abstract: LLM agents are increasingly used for collaborative problem solving and human-group simulation. This makes outcome-only evaluation insufficient: if LLM

agentsarxiv-cs-ai

Uncertainty-Aware Calibrated Clinical Text Classification with Large Language Models

arXiv:2509.19375v2 Announce Type: replace-cross Abstract: Large language models are increasingly used for clinical text classification, where overconfident misclassifications can directly affect patie

ZAPS: Zero-Cost Active Proxy Search for Neural Architecture Search

arXiv:2609.14184v1 Announce Type: new Abstract: Neural Architecture Search (NAS) automates network design, but evaluating a single candidate requires training it to convergence, making exhaustive sear

researcharxiv-cs-lg

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration

arXiv:2609.12002v1 Announce Type: new Abstract: LLMs are increasingly used as automated judges for model training and evaluation, yet individual judges exhibit systematic biases that undermine reliabi

safetyarxiv-cs-lg

Certifying Concept Unlearning in Text-to-Image Diffusion Models

arXiv:2609.12163v1 Announce Type: new Abstract: Existing evaluations of concept unlearning in text-to-image (T2I) diffusion models primarily rely on attack success rates obtained through automated adv

safetyarxiv-cs-lg

ChitraMiti: Benchmarking Visual Grounding and Modality Reliance in Bengali Geometric Reasoning

arXiv:2609.12509v1 Announce Type: new Abstract: Evaluation of vision-language models (VLMs) for multimodal mathematical reasoning remains limited for low-resource languages and for geometry problems t

Is Bash All You Need? An Empirical Study of Tool Interfaces for Enterprise Digital Worker Agents

arXiv:2609.11999v1 Announce Type: cross Abstract: In this study, we examine whether a general shell can outperform specialized tools on enterprise tasks. Shell-based agents have shown strong results i

RiPPLE: Cross-Space Performance Prediction from Early Training for Neural Architecture Search

arXiv:2609.12418v1 Announce Type: cross Abstract: Neural architecture search (NAS) evaluates candidate networks, but fully training enough architectures to rank an entire space is expensive. Zero-cost

Compute-Bounded Security Assurance - Coverage, Verification, and Response under Resource Constraints

arXiv:2609.09229v1 Announce Type: cross Abstract: Additional inference compute can increase the number of correctly resolved security-assurance tasks, but repeated success, unique coverage, accepted e

CS-Guard: Benchmarking LLM Guardrails for Code Generation Security

arXiv:2609.09798v1 Announce Type: cross Abstract: Large language models (LLMs) have been ex- ploited to generate malware, but the effective- ness of guardrails for code generation secu- rity remains u

SoK: Privacy Attacks on Machine Learning via Explainable AI

model-releasesarxiv-cs-lg

arXiv:2609.10627v1 Announce Type: cross Abstract: Machine learning explanations reveal model behavior beyond predictions, creating attack surfaces for model confidentiality and data privacy. We system

Temporal and Multimodal Deep Learning for Cyberattack Detection in LEO Satellite Systems

arXiv:2609.10746v1 Announce Type: cross Abstract: The growing reliance on Low-Earth Orbit (LEO) satellite communication systems has increased the need for intelligent methods capable of detecting cybe

researcharxiv-cs-lg

The Era by Eon Benchmark: A Generated Enterprise Estate with Exact Ground Truth for Benchmarking LLM Agents

arXiv:2609.09853v1 Announce Type: new Abstract: LLM agents for enterprise systems of record cannot be evaluated on customer production data, and no existing substitute provides ground truth. We presen

ACEA: An Adversarial Co-Evolution Arena for Head-to-Head Red-Team and Blue-Team LLM Testing

arXiv:2609.08256v1 Announce Type: cross Abstract: Automated red-team attacks and blue-team defenses for large language models (LLMs) are advancing quickly. However, attackers and defenders are built a

safetyarxiv-cs-ai

AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era

arXiv:2609.07611v1 Announce Type: new Abstract: Scientific ideation is the capacity to formulate novel and testable hypotheses from scientific evidence, and autonomous AI scientists depend on it. Exis

AtomCite: Verification and Correction of Supplied Page-Level Citations in Multi-Page Documents

arXiv:2609.05802v1 Announce Type: cross Abstract: Large language models answering questions over multi-page documents are expected to cite the supporting pages, yet supplied citations are sometimes in

Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review

arXiv:2609.05947v1 Announce Type: new Abstract: Peer review is central to quality control in science. However, existing evaluations of AI-assisted peer review mainly focus on the overall quality of ge

NormViz: A Benchmark and Framework for Grounding Multimodal Reasoning in Global Cultures

arXiv:2609.06831v1 Announce Type: new Abstract: AI systems are used worldwide, but they struggle to serve the needs of culturally diverse populations. Prior work on cultural understanding evaluates AI

OntologyBench: Can Dense Retrieval Satisfy Structured Biomedical Constraints?

arXiv:2609.08174v1 Announce Type: new Abstract: We introduce OntologyBench, a tiered biomedical retrieval benchmark comprising 471,854 training and 125,744 evaluation query-document relevance pairs ac

SWORD: Wikidata-based Distortions Reveal Hidden Cross-Lingual Inconsistencies in LLM Factual Error Rejection

arXiv:2609.09349v1 Announce Type: new Abstract: Modern LLMs demonstrate impressive multilingual performance, yet standard benchmarks primarily reward selecting correct answers rather than evaluating g

Tutorials

uFlowCSP: Crystal Structure Prediction using Mean flow generative models

arXiv:2609.09799v1 Announce Type: cross Abstract: Crystal structure prediction (CSP) is fundamental to computational materials discovery. Generative models including CDVAE, DiffCSP, FlowMM, and Crysta

tutorialsarxiv-cs-lg

Charts Are Beyond Pixels: Probing for Layer-Wise Chart Understanding and Editing

arXiv:2609.08657v1 Announce Type: new Abstract: Charts are structured visual compositions whose elements have distinct functional roles, semantic correspondences, and visibility relations. This struct

9 Sept 2026

STP-BENCH: A Unified Systematic Benchmark for Virtual Spatial Transcriptomics from Histopathology Images

arXiv:2609.05956v1 Announce Type: new Abstract: Spatial transcriptomics (ST) provides unprecedented insights into tumor heterogeneity by capturing spatially resolved gene expression, yet its high expe

9 Sept 2026

Forecast Skill Is Not Decision Skill: Evidence from Weather-Dependent Decision Tasks

arXiv:2512.14779v2 Announce Type: replace Abstract: Standard weather forecast evaluations focus on the forecaster's perspective and on a statistical assessment comparing forecasts and observations. In

researcharxiv-cs-lg

NavArena: Automated Construction of Goal-Oriented Navigation Benchmarks from 3D Gaussian Splatting Reconstructions

model-releasesarxiv-cs-ro

arXiv:2609.04602v1 Announce Type: new Abstract: Fixed 3D Gaussian Splatting (3DGS) reconstructions provide realistic novel views but lack the traversability constraints, valid goals, and closed-loop p

PRISM-Bench: An Audio-Centric Diagnostic Benchmark for Text-to-Audio-Video Generation

arXiv:2609.04867v1 Announce Type: cross Abstract: Text-to-audio-video (T2AV) generation has advanced rapidly, but its evaluation still underestimates the audio modality. Existing benchmarks either tre

WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data

arXiv:2609.05405v1 Announce Type: new Abstract: Recent advances in wearable sensing enable continuous monitoring of physiological and behavioral signals, yet existing benchmarks rarely evaluate whethe

Tutorials

Who Should Grade My Work? Student Perspectives on Transparent AI-Assisted Writing Assessment in Higher Education

arXiv:2609.05346v1 Announce Type: new Abstract: The integration of GenAI tools into higher education assessment raises important questions about how students understand, interpret, and respond to AI-m

tutorialsarxiv-cs-ai

Benchmarking Machine Translation on Chinese Social Media Texts

arXiv:2601.22931v2 Announce Type: replace Abstract: The prevalence of rapidly evolving slang, neologisms, and highly stylized expressions in informal user-generated text, particularly on Chinese socia

4 Sept 2026

Comparing Retrieval Methods for Academic Advisor Discovery: A Six-Method Study of 768 CS Faculty Profiles Across 9 US Universities