Search: “evaluation”

Evaluating Generative Time-Series Models on Data with Point Masses

arXiv:2608.09692v1 Announce Type: cross Abstract: Many of the series that generative time-series models are benchmarked on place a large probability mass on a single value --- it does not rain, no rid

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

How People Evaluate AI-, Expert-, and Peer-Style Financial Advice

DGX agent

arXiv:2608.09019v1 Announce Type: cross Abstract: As generative AI increasingly becomes a common source of daily decision-making, including financial choices, it is critical to understand how people e

researcharxiv-cs-ai

11 Aug 2026

OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agents

arXiv:2608.08775v1 Announce Type: new Abstract: Agentic benchmarks aim to measure how well AI agents plan, search, execute, and recover within realistic multi-tool environments, but they are almost ex

TREAT: Evaluating Access to Formal Knowledge across Equivalent Mathematical Representations

arXiv:2608.07540v1 Announce Type: new Abstract: AI systems increasingly operate between flexible input representations and formal objects used by downstream tools. A key challenge is recognizing when

WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web Tasks

arXiv:2506.01952v2 Announce Type: replace-cross Abstract: Powered by large language models (LLMs), web browsing agents operate graphical user interfaces in a human-like manner, offering a transparent

Confirming Our Biases? Evaluating the Capabilities, Risks, and Societal Impact of Large Language Models

arXiv:2608.06977v1 Announce Type: new Abstract: It is well established that large language models (LLMs) are sensitive to prompt framing, reflecting patterns in their training data or prior prompts. I

researcharxiv-cs-cl

10 Aug 2026

HarnessSafe: Evaluating Safety Across Persistent Carriers in Agent Harnesses

arXiv:2608.06984v1 Announce Type: cross Abstract: Modern agent harnesses persist state across tasks and sessions through persistent carriers like memory, skills, tools, and shared artifacts. However,

10 Aug 2026

CREBench: Evaluating Large Language Models in Cryptographic Binary Reverse Engineering

arXiv:2604.03750v2 Announce Type: replace-cross Abstract: Reverse engineering (RE) is central to software security, particularly for cryptographic programs that handle sensitive data and are highly pr

EcoAgent-Bench: Evaluating Economic Decision-Making in Budget-Constrained LLM Agents

arXiv:2608.05519v1 Announce Type: new Abstract: Agent benchmarks usually measure task completion and treat resource use as an auxiliary statistic. In deployment, however, the choice among a local look

Evaluating Investment Logic in Large Language Models: A Real-World Benchmark Towards Personalzied Financial Agents

arXiv:2608.06108v1 Announce Type: new Abstract: Investment competence is inherently personalized: the same market evidence can justify different actions for investors with different goals, horizons, p

Evaluating Machine Learning Models for Post-Wildfire Debris-Flow Prediction

arXiv:2608.05265v1 Announce Type: cross Abstract: Prediction of post-wildfire debris flows is critical for mitigating hazards to communities, infrastructure, and resources during intense rainfall in r

researcharxiv-cs-cv

FormBharo: Designing and Evaluating a Voice Agent for Conversational Form Filling in Rural India

arXiv:2608.06027v1 Announce Type: cross Abstract: In India, almost every social benefit starts with a form, yet the people who need these benefits most are often unable to read or write. Reaching them

Innocent Panels, Hateful Stories: Evaluating and Detecting Hateful Intent in Multi-Turn Visual Story Generation

arXiv:2608.05210v1 Announce Type: cross Abstract: Picture books and comics have long been used to disseminate hateful narratives because they are easily understood even by children, as exemplified by

Safety

Evaluating the Diagnostic Robustness of Vision-Language Models Under Visual and Textual Perturbations

arXiv:2608.04885v1 Announce Type: cross Abstract: Standard accuracy metrics for VLMs often mask significant reliability failures in sensitive domains. In this work, we utilize a histopathology-validat

safetyarxiv-cs-cl

Global Attention-Fused Image Cropping with Attention-Guided and Global-Aligned Crop Evaluator

arXiv:2608.04821v1 Announce Type: new Abstract: Image cropping aims to improve image aesthetics by preserving important content within an appropriately composed region. However, most existing methods

researcharxiv-cs-cv

IslamicTurathBench: A Multi-Task, Multi-Discipline Benchmark for Evaluating Large Language Models on the Islamic Scholarly Tradition (turath)

arXiv:2608.04703v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for question answering, education, and research, including in religious and cultural domains where an

The Evaluator Is Part of the Experiment: Measuring Open-Ended LLM Conformity

arXiv:2608.04463v1 Announce Type: new Abstract: Prior work on LLM conformity largely measures discrete answer flips under verifiable labels. Open-ended revisions require a different measurement strate

ChartAnno: Evaluating MLLMs for Chart Annotation Generation

arXiv:2608.03464v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have made significant progress in chart understanding, generation, and editing, but their ability to annotate e

5 Aug 2026

Asleep at the Wheel: JEPA's Limitations in Evaluating Novel Driving Data

arXiv:2608.01336v1 Announce Type: new Abstract: Modern autonomous-driving fleets record far more video than human reviewers can inspect. This motivates the need for an automatic clip triage mechanism,

4 Aug 2026

MetaRoute-Bench: Evaluating Meta-Decision Policies for Agentic Workflow Routing

model-releasesarxiv-cs-lg

arXiv:2608.00107v1 Announce Type: new Abstract: Agentic systems must repeatedly decide whether to answer directly, decompose a task, invoke a tool, execute code, delegate to a specialist, verify an in

4 Aug 2026

Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation

arXiv:2608.01666v1 Announce Type: new Abstract: However, whether these judges truly evaluate the scientific substance of ideas or are influenced by superficial stylistic presentation remains an open q

4 Aug 2026

FairFund-Bench: Evaluating Distributive Bias in LLM Resource Allocation

arXiv:2607.28934v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly involved in the distribution of scarce resources, raising concerns about biased allocations based on cha

3 Aug 2026

JigShape: Evaluating Visual-Geometric Reasoning in VLMs through Jigsaw Puzzles

arXiv:2607.27670v1 Announce Type: new Abstract: Jigsaw puzzle solving requires jointly reasoning about visual content and geometric constraints, yet existing benchmarks use rectangular cuts that creat

31 Jul 2026

Evaluating VLMs for Autonomous Agent-Driven Geometry Clipping Detection in Video Game QA

arXiv:2607.25921v1 Announce Type: cross Abstract: In this work, we study the use of Vision-Language Models (VLMs) for anomaly detection in an agent-driven game Quality Assurance (QA) pipeline focusing

MEDit-Bench: A Dataset for Evaluating Message-Driven Narrative Video Editing

arXiv:2607.25300v1 Announce Type: new Abstract: Video editing is fundamentally message-driven: even from the same source footage, the selected shots change depending on the narrative the editor wishes

MyoCardBench: A Real-World Data Benchmark for Evaluating Large Language Models in Clinically Authentic Cardiovascular Care Scenarios

arXiv:2607.25186v1 Announce Type: new Abstract: Background: Most medical large language model (LLM) benchmarks focus on examination knowledge or isolated tasks and may not reflect the longitudinal, mu

Polistemics: Evaluating LLMs as Information Mediators in Politics & Elections

arXiv:2607.25953v1 Announce Type: new Abstract: As LLMs increasingly mediate the political information citizens rely on, there is still no standardized way to assess whether they do so responsibly. We

Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests

arXiv:2607.22883v1 Announce Type: cross Abstract: While Large Language Models (LLMs) show great promise for automating unit test generation, recent studies suggest that the quality of generated tests

researcharxiv-cs-ai

28 Jul 2026

Evaluating Fuzz Testing for Reinforcement Learning Agents

model-releasesarxiv-cs-lg

arXiv:2607.24577v1 Announce Type: new Abstract: Reinforcement Learning (RL) agents are increasingly deployed in safety-critical domains such as robotics, autonomous driving, and drone control, where u

28 Jul 2026

Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks

arXiv:2607.21482v1 Announce Type: new Abstract: Large language models (LLMs) and agents are now widely used tools in code development, with data typically sent to third-party cloud-based models. Their

ConfidenceBench: Evaluating Confidence Calibration in Large Language Models

arXiv:2607.20526v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in settings where fluent but incorrect answers can be costly. In these settings, accuracy alone i

Safety

Evaluating Risks in Weak-to-Strong Alignment: A Bias-Variance Perspective

arXiv:2604.25077v2 Announce Type: replace Abstract: Weak-to-strong alignment offers a promising route to scalable supervision, but it can fail when a strong model becomes confidently wrong on examples

safetyarxiv-cs-ai

SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration

arXiv:2607.20926v1 Announce Type: new Abstract: Scientific research involves complex information-seeking and reasoning workflows across heterogeneous sources. However, existing benchmarks primarily em

SkillCorpus: Consolidating and Evaluating the Open Skill Ecosystem for Real-World LLM Agents

arXiv:2607.15557v4 Announce Type: replace Abstract: Agent skills, SKILL files that package reusable procedural knowledge for an LLM agent, are a popular mechanism for extending agent capabilities. Pub

It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief

arXiv:2607.18232v2 Announce Type: replace Abstract: Users frequently express their beliefs to large language models (LLMs). In some situations, the LLM should accept these contextual beliefs as true.

23 Jul 2026

MeetingToM: Evaluating Multimodal LLMs on Theory-of-Mind Reasoning in Multi-Party Meetings

arXiv:2607.19235v1 Announce Type: cross Abstract: Theory of Mind (ToM), the ability to infer other's beliefs, intentions, and states of knowledge, is central to social interaction, yet remains challen

23 Jul 2026

Applications

Evaluating Reliability in Machine Learning Models for Early Chronic Kidney Disease Prediction: A Systematic Review of Data Leakage and Predictor Stability

arXiv:2607.11963v1 Announce Type: cross Abstract: The early detection of Chronic Kidney Disease using machine learning has attracted significant interest in healthcare-related computer science. Despit

applicationsarxiv-cs-ai

15 Jul 2026

LLMs Can See the Smoke but not the Fire: Evaluating Abductive Reasoning with Elenchos

arXiv:2607.12733v1 Announce Type: new Abstract: Large language models (LLMs) excel at pattern recognition and text generation, but their capacity for abductive inference - inferring latent hypotheses

researcharxiv-cs-ai

15 Jul 2026

Local Ai

3 production patterns for AI agents and how to evaluate each one

A local coding agent, an in-app customer assistant, and an AI SRE triaging production logs may all use the same model class—but not the same harness, eval plan, or rollout risk. Mastra CEO Sam Bhagwat

local-aiarize-ai

10 Jul 2026

Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models

arXiv:2607.08317v1 Announce Type: new Abstract: Modern AI models achieve strong performance on many established benchmarks, yet they still fail on tasks that humans find almost trivial, such as manipu

10 Jul 2026

Safety

MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounters

arXiv:2607.08257v1 Announce Type: new Abstract: Large language models (LLMs) have shown strong performance on isolated psychiatric tasks, including dialogue, diagnosis, and treatment planning, yet exi

safetyarxiv-cs-ai

10 Jul 2026

Evaluating SageMath-Augmented LLM Agents for Computational and Experimental Mathematics

arXiv:2607.06820v1 Announce Type: new Abstract: Recent advances in AI for Mathematics have focused largely on autoformalization and theorem proving, leaving the role of Computer Algebra Systems (CAS)

9 Jul 2026

Applications

Beyond Accuracy: How Humans Evaluate Legally Correct but Socially Controversial Legal Advice from Machines

arXiv:2607.05680v1 Announce Type: cross Abstract: AI systems are increasingly used to provide legal advice, raising questions about whether laypeople accept guidance from algorithms--especially when t

applicationsarxiv-cs-ai

8 Jul 2026

Agents

Faithful or Findable? Evaluating LLM-Generated Metadata for RDF Dataset Search

arXiv:2607.05970v1 Announce Type: cross Abstract: Dataset search depends heavily on metadata, making LLM-generated metadata a consequential form of synthetic content in retrieval systems. We study six

agentsarxiv-cs-ai

8 Jul 2026

ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalization

arXiv:2607.05185v1 Announce Type: new Abstract: Compositional generalization, the ability to understand and produce novel combinations of known components, remains a fundamental challenge for modern a

Criterion-Conditional In-Context Learning: Evaluating Criterion-Shift Adaptation in Vision-Language Models

arXiv:2607.02575v1 Announce Type: cross Abstract: Vision-language models can perform new tasks without parameter updates through in-context learning (ICL), whose core mechanism is utilizing the suppor

Dashboard2Code: Evaluating Multimodal Models on Reconstructing Interactive Dashboards

arXiv:2607.04727v1 Announce Type: cross Abstract: Automatic data visualization generation has advanced rapidly with multi-modal large language models, yet existing efforts largely focus on static char

Evaluating and Understanding Model Editing for Medical Vision Language Models

arXiv:2607.05310v1 Announce Type: new Abstract: Model editing promises a fast, targeted way to correct post-deployment mistakes in medical vision-language models (VLMs) without costly retraining. Howe