Search: “evaluation”

EcoAgent-Bench: Evaluating Economic Decision-Making in Budget-Constrained LLM Agents

arXiv:2608.05519v1 Announce Type: new Abstract: Agent benchmarks usually measure task completion and treat resource use as an auxiliary statistic. In deployment, however, the choice among a local look

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Evaluating Investment Logic in Large Language Models: A Real-World Benchmark Towards Personalzied Financial Agents

DGX agent

arXiv:2608.06108v1 Announce Type: new Abstract: Investment competence is inherently personalized: the same market evidence can justify different actions for investors with different goals, horizons, p

model-releasesarxiv-cs-ai

7 Aug 2026

Evaluating Machine Learning Models for Post-Wildfire Debris-Flow Prediction

arXiv:2608.05265v1 Announce Type: cross Abstract: Prediction of post-wildfire debris flows is critical for mitigating hazards to communities, infrastructure, and resources during intense rainfall in r

researcharxiv-cs-cv

FormBharo: Designing and Evaluating a Voice Agent for Conversational Form Filling in Rural India

arXiv:2608.06027v1 Announce Type: cross Abstract: In India, almost every social benefit starts with a form, yet the people who need these benefits most are often unable to read or write. Reaching them

Innocent Panels, Hateful Stories: Evaluating and Detecting Hateful Intent in Multi-Turn Visual Story Generation

arXiv:2608.05210v1 Announce Type: cross Abstract: Picture books and comics have long been used to disseminate hateful narratives because they are easily understood even by children, as exemplified by

Safety

Evaluating the Diagnostic Robustness of Vision-Language Models Under Visual and Textual Perturbations

arXiv:2608.04885v1 Announce Type: cross Abstract: Standard accuracy metrics for VLMs often mask significant reliability failures in sensitive domains. In this work, we utilize a histopathology-validat

safetyarxiv-cs-cl

Global Attention-Fused Image Cropping with Attention-Guided and Global-Aligned Crop Evaluator

arXiv:2608.04821v1 Announce Type: new Abstract: Image cropping aims to improve image aesthetics by preserving important content within an appropriately composed region. However, most existing methods

researcharxiv-cs-cv

IslamicTurathBench: A Multi-Task, Multi-Discipline Benchmark for Evaluating Large Language Models on the Islamic Scholarly Tradition (turath)

arXiv:2608.04703v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for question answering, education, and research, including in religious and cultural domains where an

The Evaluator Is Part of the Experiment: Measuring Open-Ended LLM Conformity

arXiv:2608.04463v1 Announce Type: new Abstract: Prior work on LLM conformity largely measures discrete answer flips under verifiable labels. Open-ended revisions require a different measurement strate

ChartAnno: Evaluating MLLMs for Chart Annotation Generation

arXiv:2608.03464v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have made significant progress in chart understanding, generation, and editing, but their ability to annotate e

5 Aug 2026

Asleep at the Wheel: JEPA's Limitations in Evaluating Novel Driving Data

arXiv:2608.01336v1 Announce Type: new Abstract: Modern autonomous-driving fleets record far more video than human reviewers can inspect. This motivates the need for an automatic clip triage mechanism,

4 Aug 2026

MetaRoute-Bench: Evaluating Meta-Decision Policies for Agentic Workflow Routing

model-releasesarxiv-cs-lg

arXiv:2608.00107v1 Announce Type: new Abstract: Agentic systems must repeatedly decide whether to answer directly, decompose a task, invoke a tool, execute code, delegate to a specialist, verify an in

4 Aug 2026

Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation

arXiv:2608.01666v1 Announce Type: new Abstract: However, whether these judges truly evaluate the scientific substance of ideas or are influenced by superficial stylistic presentation remains an open q

4 Aug 2026

FairFund-Bench: Evaluating Distributive Bias in LLM Resource Allocation

arXiv:2607.28934v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly involved in the distribution of scarce resources, raising concerns about biased allocations based on cha

3 Aug 2026

JigShape: Evaluating Visual-Geometric Reasoning in VLMs through Jigsaw Puzzles

arXiv:2607.27670v1 Announce Type: new Abstract: Jigsaw puzzle solving requires jointly reasoning about visual content and geometric constraints, yet existing benchmarks use rectangular cuts that creat

31 Jul 2026

Evaluating VLMs for Autonomous Agent-Driven Geometry Clipping Detection in Video Game QA

arXiv:2607.25921v1 Announce Type: cross Abstract: In this work, we study the use of Vision-Language Models (VLMs) for anomaly detection in an agent-driven game Quality Assurance (QA) pipeline focusing

MEDit-Bench: A Dataset for Evaluating Message-Driven Narrative Video Editing

arXiv:2607.25300v1 Announce Type: new Abstract: Video editing is fundamentally message-driven: even from the same source footage, the selected shots change depending on the narrative the editor wishes

MyoCardBench: A Real-World Data Benchmark for Evaluating Large Language Models in Clinically Authentic Cardiovascular Care Scenarios

arXiv:2607.25186v1 Announce Type: new Abstract: Background: Most medical large language model (LLM) benchmarks focus on examination knowledge or isolated tasks and may not reflect the longitudinal, mu

Polistemics: Evaluating LLMs as Information Mediators in Politics & Elections

arXiv:2607.25953v1 Announce Type: new Abstract: As LLMs increasingly mediate the political information citizens rely on, there is still no standardized way to assess whether they do so responsibly. We

Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests

arXiv:2607.22883v1 Announce Type: cross Abstract: While Large Language Models (LLMs) show great promise for automating unit test generation, recent studies suggest that the quality of generated tests

researcharxiv-cs-ai

28 Jul 2026

Evaluating Fuzz Testing for Reinforcement Learning Agents

model-releasesarxiv-cs-lg

arXiv:2607.24577v1 Announce Type: new Abstract: Reinforcement Learning (RL) agents are increasingly deployed in safety-critical domains such as robotics, autonomous driving, and drone control, where u

28 Jul 2026

Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks

arXiv:2607.21482v1 Announce Type: new Abstract: Large language models (LLMs) and agents are now widely used tools in code development, with data typically sent to third-party cloud-based models. Their

ConfidenceBench: Evaluating Confidence Calibration in Large Language Models

arXiv:2607.20526v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in settings where fluent but incorrect answers can be costly. In these settings, accuracy alone i

Safety

Evaluating Risks in Weak-to-Strong Alignment: A Bias-Variance Perspective

arXiv:2604.25077v2 Announce Type: replace Abstract: Weak-to-strong alignment offers a promising route to scalable supervision, but it can fail when a strong model becomes confidently wrong on examples

safetyarxiv-cs-ai

SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration

arXiv:2607.20926v1 Announce Type: new Abstract: Scientific research involves complex information-seeking and reasoning workflows across heterogeneous sources. However, existing benchmarks primarily em

SkillCorpus: Consolidating and Evaluating the Open Skill Ecosystem for Real-World LLM Agents

arXiv:2607.15557v4 Announce Type: replace Abstract: Agent skills, SKILL files that package reusable procedural knowledge for an LLM agent, are a popular mechanism for extending agent capabilities. Pub

It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief

arXiv:2607.18232v2 Announce Type: replace Abstract: Users frequently express their beliefs to large language models (LLMs). In some situations, the LLM should accept these contextual beliefs as true.

23 Jul 2026

MeetingToM: Evaluating Multimodal LLMs on Theory-of-Mind Reasoning in Multi-Party Meetings

arXiv:2607.19235v1 Announce Type: cross Abstract: Theory of Mind (ToM), the ability to infer other's beliefs, intentions, and states of knowledge, is central to social interaction, yet remains challen

23 Jul 2026

Applications

Evaluating Reliability in Machine Learning Models for Early Chronic Kidney Disease Prediction: A Systematic Review of Data Leakage and Predictor Stability

arXiv:2607.11963v1 Announce Type: cross Abstract: The early detection of Chronic Kidney Disease using machine learning has attracted significant interest in healthcare-related computer science. Despit

applicationsarxiv-cs-ai

15 Jul 2026

LLMs Can See the Smoke but not the Fire: Evaluating Abductive Reasoning with Elenchos

arXiv:2607.12733v1 Announce Type: new Abstract: Large language models (LLMs) excel at pattern recognition and text generation, but their capacity for abductive inference - inferring latent hypotheses

researcharxiv-cs-ai

15 Jul 2026

Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models

arXiv:2607.08317v1 Announce Type: new Abstract: Modern AI models achieve strong performance on many established benchmarks, yet they still fail on tasks that humans find almost trivial, such as manipu

10 Jul 2026

Safety

MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounters

arXiv:2607.08257v1 Announce Type: new Abstract: Large language models (LLMs) have shown strong performance on isolated psychiatric tasks, including dialogue, diagnosis, and treatment planning, yet exi

safetyarxiv-cs-ai

10 Jul 2026

Evaluating SageMath-Augmented LLM Agents for Computational and Experimental Mathematics

arXiv:2607.06820v1 Announce Type: new Abstract: Recent advances in AI for Mathematics have focused largely on autoformalization and theorem proving, leaving the role of Computer Algebra Systems (CAS)

9 Jul 2026

Applications

Beyond Accuracy: How Humans Evaluate Legally Correct but Socially Controversial Legal Advice from Machines

arXiv:2607.05680v1 Announce Type: cross Abstract: AI systems are increasingly used to provide legal advice, raising questions about whether laypeople accept guidance from algorithms--especially when t

applicationsarxiv-cs-ai

8 Jul 2026

Agents

Faithful or Findable? Evaluating LLM-Generated Metadata for RDF Dataset Search

arXiv:2607.05970v1 Announce Type: cross Abstract: Dataset search depends heavily on metadata, making LLM-generated metadata a consequential form of synthetic content in retrieval systems. We study six

agentsarxiv-cs-ai

8 Jul 2026

ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalization

arXiv:2607.05185v1 Announce Type: new Abstract: Compositional generalization, the ability to understand and produce novel combinations of known components, remains a fundamental challenge for modern a

Criterion-Conditional In-Context Learning: Evaluating Criterion-Shift Adaptation in Vision-Language Models

arXiv:2607.02575v1 Announce Type: cross Abstract: Vision-language models can perform new tasks without parameter updates through in-context learning (ICL), whose core mechanism is utilizing the suppor

Dashboard2Code: Evaluating Multimodal Models on Reconstructing Interactive Dashboards

arXiv:2607.04727v1 Announce Type: cross Abstract: Automatic data visualization generation has advanced rapidly with multi-modal large language models, yet existing efforts largely focus on static char

Evaluating and Understanding Model Editing for Medical Vision Language Models

arXiv:2607.05310v1 Announce Type: new Abstract: Model editing promises a fast, targeted way to correct post-deployment mistakes in medical vision-language models (VLMs) without costly retraining. Howe

Evaluating Intellectual Property Guardrails of Generative Image Models: A Technical Report

arXiv:2607.02582v1 Announce Type: new Abstract: Generative image models are capable of producing images that bear a strong resemblance to, or replicate, recognizable intellectual property (IP). In thi

Evaluating Large Language Models for Antisemitic Incident Classification

arXiv:2607.04890v1 Announce Type: new Abstract: Addressing hate and violence in society requires timely detection of hateful events from public reporting, but automated identification of hateful event

Evaluating LLM Uncertainty in Long-Form Generation Using Deterministic Ground Truth

arXiv:2607.03870v1 Announce Type: new Abstract: As LLMs generate increasingly long outputs, effective uncertainty estimation must identify errors at fine-grained levels rather than discard entire resp

SovereignNegotiation-Bench: Evaluating User-Owned Personal Agents In Delegated Bargaining Under Privacy, Consent, Evidence, And Institutional Pressure

arXiv:2607.02814v1 Announce Type: cross Abstract: Personal agents will increasingly negotiate on behalf of users: splitting costs with other personal agents, appealing platform decisions, escalating s

VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs

arXiv:2511.20272v2 Announce Type: replace Abstract: While Multimodal Large Language Models (MLLMs) have become adept at recognizing objects, they often lack the intuitive, human-like understanding of

EduArt: An educational-level benchmark for evaluating art history knowledge in large language models

arXiv:2607.02007v1 Announce Type: new Abstract: Large language models now score near ceiling on general benchmarks, but these aggregate measures reveal little about how models behave within single dis

3 Jul 2026

Evaluating Chunking Strategies for Retrieval-Augmented Generation on Academic Texts

arXiv:2607.01852v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) systems use the question-answering capabilities of Large Language Models (LLMs) to access information outside the

researcharxiv-cs-ai

3 Jul 2026

Applications

Probabilistic Low-Voltage Peak Load Forecasting with Time Series Foundation Models Evaluated on Application-Oriented Metrics

arXiv:2607.01966v1 Announce Type: new Abstract: Low-voltage load forecasting is an important component in current and future energy systems with a high degree of electrification and decentralized gene

applicationsarxiv-cs-lg

3 Jul 2026

EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards

arXiv:2607.00218v1 Announce Type: cross Abstract: Vision-language models (VLMs) are now proposed as runtime safety guards for embodied agents in homes and factories. A deployable guard must catch genu