Search: “evaluation”

MirrorBench: Evaluating Self-centric Intelligence in MLLMs by Introducing a Mirror

arXiv:2604.14785v2 Announce Type: replace Abstract: Recent progress in Multimodal Large Language Models (MLLMs) has demonstrated remarkable advances in perception and reasoning, suggesting their poten

23 Apr 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Safety

Evaluating LLM-Driven Summarisation of Parliamentary Debates with Computational Argumentation

DGX agent

arXiv:2604.19331v1 Announce Type: new Abstract: Understanding how policy is debated and justified in parliament is a fundamental aspect of the democratic process. However, the volume and complexity of

safetyarxiv-cs-cl

22 Apr 2026

LegalBench-BR: A Benchmark for Evaluating Large Language Models on Brazilian Legal Decision Classification

arXiv:2604.18878v1 Announce Type: new Abstract: We introduce LegalBench-BR, the first public benchmark for evaluating language models on Brazilian legal text classification. The dataset comprises 3,10

22 Apr 2026

SafetyALFRED: Evaluating Safety-Conscious Planning of Multimodal Large Language Models

arXiv:2604.19638v1 Announce Type: new Abstract: Multimodal Large Language Models are increasingly adopted as autonomous agents in interactive environments, yet their ability to proactively address saf

22 Apr 2026

Tutorials

Evaluating Adaptive Personalization of Educational Readings with Simulated Learners

arXiv:2604.16744v1 Announce Type: new Abstract: We present a framework for evaluating adaptive personalization of educational reading materials with theory-grounded simulated learners. The system buil

tutorialsarxiv-cs-cl

LongBench: Evaluating Robotic Manipulation Policies on Real-World Long-Horizon Tasks

model-releasesarxiv-cs-ro

arXiv:2604.16788v1 Announce Type: new Abstract: Robotic manipulation policies often degrade over extended horizons, yet existing benchmarks provide limited insight into why such failures occur. Most p

PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

arXiv:2604.16813v1 Announce Type: cross Abstract: Agentic AI systems are rapidly advancing toward real-world applications, yet their readiness in complex and personalized environments remains insuffic

Prune, Interpret, Evaluate: A Cross-Layer Transcoder-Native Framework for Efficient Circuit Discovery via Feature Attribution

arXiv:2604.16889v1 Announce Type: new Abstract: Existing feature-interpretation pipelines typically operate on uniformly sampled units, but only a small fraction of cross-layer transcoder (CLT) featur

Safety

SynopticBench: Evaluating Vision-Language Models on Generating Weather Forecast Discussions of the Future

arXiv:2604.16451v1 Announce Type: new Abstract: Recent advances in visual-language models (VLMs) have led to significant improvements in a plethora of complex multimodal tasks like image captioning, r

safetyarxiv-cs-cl

PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Research

arXiv:2604.15411v1 Announce Type: cross Abstract: The paradigm of agentic science requires AI systems to conduct robust reasoning and engage in long-horizon, autonomous exploration. However, current s

20 Apr 2026

Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games

arXiv:2506.03610v3 Announce Type: replace Abstract: Large Language Model (LLM) agents are reshaping the game industry, by enabling more intelligent and human-preferable characters. Yet, current game b

17 Apr 2026

Evaluating the Formal Reasoning Capabilities of Large Language Models through Chomsky Hierarchy

arXiv:2604.02709v2 Announce Type: replace Abstract: The formal reasoning capabilities of LLMs are crucial for advancing automated software engineering. However, existing benchmarks for LLMs lack syste

16 Apr 2026

A Benchmark for Evaluating Outcome-Driven Constraint Violations in Autonomous AI Agents

arXiv:2512.20798v4 Announce Type: replace Abstract: As autonomous AI agents are deployed in high-stakes environments, ensuring their safety has become a paramount concern. Existing safety benchmarks p

Evaluating Differential Privacy Against Membership Inference in Federated Learning: Insights from the NIST Genomics Red Team Challenge

arXiv:2604.12737v1 Announce Type: cross Abstract: While Federated Learning (FL) mitigates direct data exposure, the resulting trained models remain susceptible to membership inference attacks (MIAs).

Safety

Evaluating the Limitations of Protein Sequence Representations for Parkinson's Disease Classification

arXiv:2604.11852v1 Announce Type: cross Abstract: The identification of reliable molecular biomarkers for Parkinson's disease remains challenging due to its multifactorial nature. Although protein seq

safetyarxiv-cs-ai

When Does Data Augmentation Help? Evaluating LLM and Back-Translation Methods for Hausa and Fongbe NLP

arXiv:2604.12540v1 Announce Type: cross Abstract: Data scarcity limits NLP development for low-resource African languages. We evaluate two data augmentation methods -- LLM-based generation (Gemini 2.5

CricBench: A Multilingual Benchmark for Evaluating LLMs in Cricket Analytics

arXiv:2512.21877v3 Announce Type: replace-cross Abstract: Cricket is the second most popular sport worldwide, with billions of fans seeking advanced statistical insights unavailable through standard w

Evaluating Small Open LLMs for Medical Question Answering: A Practical Framework

arXiv:2604.10535v1 Announce Type: cross Abstract: Incorporating large language models (LLMs) in medical question answering demands more than high average accuracy: a model that returns substantively d

PAC-BENCH: Evaluating Multi-Agent Collaboration under Privacy Constraints

arXiv:2604.11523v1 Announce Type: new Abstract: We are entering an era in which individuals and organizations increasingly deploy dedicated AI agents that interact and collaborate with other agents. H

VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments

arXiv:2506.02387v3 Announce Type: replace Abstract: Recent advancements in Vision Language Models (VLMs) have expanded their capabilities to interactive agent tasks, yet existing benchmarks remain lim

Lessons Without Borders? Evaluating Cultural Alignment of LLMs Using Multilingual Story Moral Generation

arXiv:2604.08797v1 Announce Type: cross Abstract: Stories are key to transmitting values across cultures, but their interpretation varies across linguistic and cultural contexts. Thus, we introduce mu

13 Apr 2026

SiMing-Bench: Evaluating Procedural Correctness from Continuous Interactions in Clinical Skill Videos

arXiv:2604.09037v1 Announce Type: cross Abstract: Current video benchmarks for multimodal large language models (MLLMs) focus on event recognition, temporal ordering, and long-context recall, but over

13 Apr 2026

Evaluating LLMs for Demographic-Targeted Social Bias Detection: A Comprehensive Benchmark Study

arXiv:2510.04641v3 Announce Type: replace Abstract: Large-scale web-scraped text corpora used to train general-purpose AI models often contain harmful demographic-targeted social biases, creating a re

10 Apr 2026

Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization

arXiv:2604.07343v1 Announce Type: cross Abstract: Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a cen

10 Apr 2026

Syntax Is Easy, Semantics Is Hard: Evaluating LLMs for LTL Translation

arXiv:2604.07321v1 Announce Type: cross Abstract: Propositional Linear Temporal Logic (LTL) is a popular formalism for specifying desirable requirements and security and privacy policies for software,

researcharxiv-cs-ai

10 Apr 2026

Can AI Forecast the Extremes? Evaluating Probabilistic Weather Models on Tropical Cyclones

arXiv:2610.22077v1 Announce Type: new Abstract: Weather prediction is undergoing an important shift, with data-driven AI models outperforming leading numerical weather prediction systems on average at

Evaluating Model Capabilities After Benchmark Saturation

arXiv:2610.20667v1 Announce Type: new Abstract: Rapid advances in AI capabilities are causing benchmarks to saturate. Once several models approach perfect scores, a benchmark loses its power to differ

PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems

arXiv:2610.22158v1 Announce Type: new Abstract: Recent advances in large language models (LLMs) have substantially improved Text-to-SQL performance, yet existing NL2SQL systems largely assume that use

The World Won't Wait: Evaluating and Teaching Agents Under a Live Clock

arXiv:2610.20665v1 Announce Type: new Abstract: Large language model (LLM) agents can solve complex interactive tasks, yet reasoning and tool calls take time. In continuously changing environments, an

researcharxiv-cs-ai

WorkflowPerturb: Calibrated Stress Tests for Evaluating Multi-Agent Workflow Metrics

arXiv:2602.17990v4 Announce Type: replace Abstract: Multi-agent LLM systems that generate structured workflows from natural-language requests are now deployed in production across cloud automation, De

J-tau: A Fully Localized Japanese Benchmark for Tool-Agent-User Interaction

arXiv:2610.19638v1 Announce Type: new Abstract: We present J-tau, the first publicly available Japanese benchmark for evaluating tool-using LLM agents in interactive customer-support scenarios. While

15 Oct 2026

VisualWorldBench: A Fine-Grained Multi-Task Benchmark for Evaluating Visual World Knowledge in MLLMs

arXiv:2610.20083v1 Announce Type: new Abstract: Visual world knowledge is a core capability of multimodal large language models (MLLMs), underpinning knowledge-intensive visual question answering and

15 Oct 2026

Agents

Autonomous Cyber Operations Gym for Training and Evaluating AI Models for Defence

arXiv:2610.18594v1 Announce Type: cross Abstract: This paper proposes an Autonomous Cyber Operations (ACO) gym, a unified framework integrating high-fidelity network emulation, realistic user behaviou

agentsarxiv-cs-lg

14 Oct 2026

Beyond Discrimination: Evaluating Probability Calibration in Radiomics-Image Fusion for Renal Cell Carcinoma Classification

arXiv:2610.14151v1 Announce Type: cross Abstract: Reliable probability estimates are important for the clinical interpretation of machine-learning classifiers, yet radiomics--image fusion models for r

researcharxiv-cs-cv

14 Oct 2026

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models

arXiv:2610.12402v1 Announce Type: cross Abstract: Existing spatial reasoning benchmarks mainly test spatial perception: reading off relations already visible in the input. Yet real-world spatial intel

9 Oct 2026

Tutorials

Evaluating the Transfer of Co-Evolved Communication from 2D to 3D Simulation

arXiv:2610.09280v1 Announce Type: new Abstract: This work examines the transfer of a co-evolved communication mechanism between two robotic agents from a discrete two-dimensional (2D) simulator to a t

tutorialsarxiv-cs-ro

8 Oct 2026

UltraText Bench: A Comprehensive Bilingual Benchmark for Evaluating Visual Text Rendering in Image Generation

arXiv:2610.09823v1 Announce Type: new Abstract: Dense visual text requires image generators to reproduce long strings across multiple regions with correct placement and legibility. As short-string ren

8 Oct 2026

Visual Evidence Under Cross-Examination: Evaluating and Controlling Decision-Level Evidence Use in Vision-Language Models

arXiv:2610.09550v1 Announce Type: new Abstract: Vision-language models increasingly reason through crops, regions, and tool-produced observations. Yet an observation can influence the answer without b

8 Oct 2026

DecepEval: A Benchmark for Evaluating Deception in LLM Agents

arXiv:2610.07967v1 Announce Type: new Abstract: As large language model (LLM) agents become increasingly autonomous, they may pursue task performance through deception, raising concerns about their re

7 Oct 2026

Leveraging a four-quadrant approach for evaluating Redpine Science

arXiv:2610.07937v1 Announce Type: new Abstract: Redpine Science gives models and agents a single access point to a wide range of peer-reviewed literature, queried directly through the Model Context Pr

7 Oct 2026

Evaluating Zone-Guided Front Extraction for Glacier Calving-Front Delineation in SAR Imagery

arXiv:2610.04066v1 Announce Type: new Abstract: Automatic calving-front delineation from synthetic aperture radar imagery is challenging because the front is a thin and often ambiguous boundary betwee

researcharxiv-cs-cv

6 Oct 2026

MagiC: Evaluating Multimodal Cognition Toward Grounded Visual Reasoning

arXiv:2507.07297v2 Announce Type: replace Abstract: Recent advances in large vision-language models have led to impressive performance in visual question answering and multimodal reasoning. However, i

6 Oct 2026

Beyond Random Splits: Evaluating Drug-Target Affinity Models Under Chemically and Biologically Motivated Distribution Shifts Copy

arXiv:2610.03456v1 Announce Type: new Abstract: Drug-target affinity (DTA) prediction is widely used to prioritize candidate compounds before costly experimental screening. DTA models are often compar

researcharxiv-cs-lg

5 Oct 2026

Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents

arXiv:2610.03448v1 Announce Type: cross Abstract: LLM agents increasingly screen tool outputs with small prompt-injection detectors, and teams choose among detectors by their scores on public benchmar

5 Oct 2026

Applications

Evaluating LLM-Generated Preference Distributions

arXiv:2610.01000v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly used as probabilistic generators for simulation, synthetic data generation, and decision support in settin

applicationsarxiv-cs-ai

Evaluating the Robustness of Anti-UAV Detection under Controlled Fog Degradation: Fog-Aware Training and Clear-Sky Tradeoff

arXiv:2610.00141v1 Announce Type: new Abstract: Vision-based anti-UAV systems must function in poor visibility, yet most benchmarks use only clear-sky footage, and previous robustness studies treat ad

MMMG: a Comprehensive and Reliable Benchmark for Multitask Multimodal Generation

arXiv:2505.17613v2 Announce Type: replace Abstract: Automatically evaluating multimodal generation presents a significant challenge, as automated metrics often struggle to align with human evaluation,

Agents

ReLiveGym: Evaluating Long-Lived Agents over Weeks of Replayed Reality

arXiv:2610.00710v1 Announce Type: new Abstract: As large language model (LLM) agents become widely adopted, they are increasingly deployed for tasks that require persistent monitoring or recurring act

agentsarxiv-cs-ai