Search: “evaluation”

Templated or fully Synthetic? Prompt construction as a confound in measuring LLM political stance beyond writing assistance

arXiv:2608.11008v1 Announce Type: new Abstract: Political stance detection in LLMs has long been dominated by closed-ended, multiple-choice political survey questions---originally designed for humans,

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Safety

The CASE Framework: A Multi-Disciplinary Control Architecture for Governing Enterprise Agentic AI

DGX agent

arXiv:2608.10153v1 Announce Type: new Abstract: Enterprises are deploying autonomous AI agents faster than they can govern them, and prevailing approaches stretch a single discipline, typically DevSec

safetyarxiv-cs-ai

12 Aug 2026

Toward a Theory of Value in AI Alignment

arXiv:2608.10327v1 Announce Type: new Abstract: Can AI systems be aligned to human values? The popularization of large language models (LLMs) and multi-modal foundation models has seen a rise in harms

VIDS-Seg: Towards Reliable Uncertainty Quantification in Pediatric Cardiac Ultrasound Segmentation

arXiv:2608.10903v1 Announce Type: new Abstract: Reliable clinical deployment of machine learning requires models that know when they are likely to fail, particularly for subgroups underrepresented in

safetyarxiv-cs-cv

You chose the best model. Why is your agent still failing?

Public benchmarks can show how a model performs in general. Production reliability depends on the context and harness around it, which only your team can evaluate against its own data, workflows, and

agentsarize-ai

Applications

Accurate but Natural? Diagnosing Grammatical and Idiomatic Gaps in Japanese EFL Writing

arXiv:2608.09289v1 Announce Type: new Abstract: Second language writing research distinguishes grammatical accuracy from native-like idiomaticity, yet automated writing evaluation often conflates thes

applicationsarxiv-cs-cl

Action- and Language-Conditioned Video Assessment for Embodied Control

arXiv:2608.08273v1 Announce Type: cross Abstract: Vision-based embodied agents executing multi-step natural language instructions require feedback mechanisms that assess task progress over complete tr

safetyarxiv-cs-cv

Beyond Aggregate Calibration: Decomposing Income-Conditional Recall Disparities in Automated Credit Default Prediction

arXiv:2608.08202v1 Announce Type: new Abstract: Data-centric curation pipelines frequently rely on model confidence scores to flag and filter noisy or mislabeled training instances. Evaluating this fi

safetyarxiv-cs-lg

Beyond Pixels: Benchmarking and Reward-Based Assessing Framework for Visual Spatial Aesthetics

arXiv:2512.05098v2 Announce Type: replace-cross Abstract: In recent years, Image Quality Assessment (IQA) for AI-generated images (AIGI) has advanced rapidly; however, existing methods primarily targe

Tutorials

Beyond the Node: Clade-level Selection for Efficient MCTS in Automatic Heuristic Design

arXiv:2602.00549v2 Announce Type: replace Abstract: While Monte Carlo Tree Search (MCTS) shows promise in Large Language Model (LLM) based Automatic Heuristic Design (AHD), it suffers from a critical

tutorialsarxiv-cs-lg

Business Arena: Benchmarking LLM Agents in a Realistic Marketplace

arXiv:2608.08621v1 Announce Type: new Abstract: Running a business is a challenging form of intelligent work. Operators must infer opportunities from partial signals, commit capital under uncertainty,

Compiling and Benchmarking Task-State Horizons for Embodied Agents

model-releasesarxiv-cs-ro

arXiv:2608.08036v1 Announce Type: new Abstract: Frontier agentic models are increasingly deployed as high-level planners for long-horizon embodied tasks. Existing robotic benchmarks have advanced long

CORDA: A Benchmark for Hierarchical Harm-Centric Moral Reasoning in Large Language Models

arXiv:2608.08061v1 Announce Type: new Abstract: The key question in moral judgement is not simply whether someone chooses the 'right' answer, but how they decide what matters most when moral principle

CresOWLve: Benchmarking Creative Problem-Solving Over Real-World Knowledge

arXiv:2604.03374v2 Announce Type: replace-cross Abstract: Creative problem-solving requires combining multiple cognitive abilities, including logical reasoning, lateral thinking, analogy-making, and c

CyberAGENTS: Structured Autonomy for Agentic Gamified Learning in Cybersecurity

arXiv:2608.07965v1 Announce Type: new Abstract: Gamification is especially effective in learning domains requiring active problem-solving and iterative skill-building, such as cybersecurity education.

ELBench: A Multi-Dimensional Benchmark for Education-Facing Large Language Models

arXiv:2608.09548v1 Announce Type: cross Abstract: Large language models are increasingly deployed in education as tutors, teaching assistants, and content generators. These roles place demands that or

Estimating Uncertainty in Galaxy Morphology Classification

arXiv:2608.08398v1 Announce Type: new Abstract: Astronomers classify galaxy morphology to investigate cosmic evolution. While deep foundation models are increasingly utilized in Galaxy Morphology Clas

Flow-by-Flow:Content-Judgment Bypass for Governing AI Output in High-Loss Domains

arXiv:2608.07474v1 Announce Type: new Abstract: Prior work showed that human-in-the-loop oversight becomes structurally untenable in high-loss domains when AI output velocity V exceeds human cognitive

From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineering

arXiv:2604.01476v2 Announce Type: replace-cross Abstract: Reinforcement learning for LLMs is vulnerable to reward hacking, where models exploit shortcuts to maximize reward without solving the intende

Geometry Beats Estimated Depth: RGB-Only Multi-Camera 3D Tracking under Sim2Real

arXiv:2608.07579v1 Announce Type: cross Abstract: The AI City Challenge 2026 Track 1 evaluates multi-camera 3D perception in large indoor warehouses under a synthetic-to-real (Sim2Real) setting; depth

IDRAAK: From Multi-Agent NLP to Few-Shot Prompting for Semantic Drift Detection in Technical Requirements

arXiv:2608.08801v1 Announce Type: new Abstract: Translating technical requirements across languages can introduce semantic drift, altering numerical constraints, polarities, modalities, or other speci

agentsarxiv-cs-cl

Introducing ExtractBench, the most comprehensive benchmark for information extraction from complex enterprise documents. The latest models a…

model-releasesjerry-liu--x

Introducing ExtractBench, the most comprehensive benchmark for information extraction from complex enterprise documents. The latest models are pushing the frontier of coding and knowledge work, but su

Large Multimodal Agents for Intelligent Transportation Systems: Architectures, Evidence, and Deployment Challenges

arXiv:2608.08184v1 Announce Type: new Abstract: Large multimodal agents (LMAs) are increasingly proposed for intelligent transportation systems (ITS), but existing studies often conflate multimodality

LazyHMC: Hamiltonian Monte Carlo Simulation for Lazy, Infinite Dimensional Probabilistic Programs

model-releasesarxiv-cs-lg

arXiv:2608.08588v1 Announce Type: cross Abstract: Hamiltonian Monte Carlo (HMC) is a successful generic inference method in probabilistic programming, but in its ordinary formulation it needs gradient

LoRA-based Adaptation Alone Is Not Enough: Understanding the Limits of Foundation Models for Face Presentation Attack Detection

arXiv:2608.09633v1 Announce Type: new Abstract: Face presentation attack detection (PAD) aims to reliably detect a wide range of presentation attacks. While PAD methods achieve strong performance with

researcharxiv-cs-cv

MedCalc-R1: Knowledge-Guided Reward Framework for Medical Mathematical Reasoning

arXiv:2608.08623v1 Announce Type: new Abstract: In Reinforcement Learning with Verifiable Rewards (RLVR) frameworks for mathematical reasoning tasks, floating-point results are typically evaluated usi

Multi-agent discovery of practical quantum LDPC codes

arXiv:2608.08996v1 Announce Type: cross Abstract: Quantum low-density parity-check (qLDPC) codes can encode multiple logical qubits using sparse parity checks, yet searching for useful finite-length i

Penalizing Length: Uncovering Systematic Bias in Quality Estimation Metrics

arXiv:2510.22028v4 Announce Type: replace Abstract: Quality Estimation (QE) metrics are vital in machine translation for reference-free evaluation and increasingly serve as selection criteria in data

Planning/RL for a stochastic single-player merge puzzle: afterstates, previewed chance events, and long-horizon throughput [D]

I am working on an AI for a small single-player merge puzzle and would appreciate pointers to related algorithms, papers, or existing implementations. It resembles 2048 in its action -> afterstate ->

safetyr-machinelearning

Privacy-Preserving Data Drift Detection and Recovery for Large-Scale LLM Applications via Proxy Representations

arXiv:2608.08245v1 Announce Type: cross Abstract: LLM applications deployed at scale face a fundamental challenge: privacy constraints prevent direct inspection of user interactions, making it difficu

ReMIND: Orchestrating Modular Large Language Models for Controllable Serendipity A REM-Inspired System Design for Emergent Creative Ideation

arXiv:2601.07121v3 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used not only for problem solving but also for creative ideation; however, generating ideas that

RISE-RL: Rubric-Informed Selective Exploration for Open-Ended Reinforcement Learning

arXiv:2608.09123v1 Announce Type: new Abstract: Aligning Large Language Models (LLMs) for open-ended tasks is challenging because responses must satisfy multidimensional criteria without following a s

RobustDefect-LLM: Explainable and Robustness-Aware Industrial Surface Defect Classification with Decision Support and AI-Assisted Reporting

arXiv:2608.08589v1 Announce Type: new Abstract: This paper presents RobustDefect-LLM, an industrial surface-defect inspection framework integrating deep-learning classification, operator-facing visual

safetyarxiv-cs-cv

Satellite Trajectory Optimization via Proximal Policy Optimization for Space Debris Avoidance

arXiv:2608.09628v1 Announce Type: new Abstract: Collision avoidance systems are commonly used to avoid fragmentation events occurring in Low-Earth Orbit (LEO) and Geosynchronous Equatorial Orbit (GEO)

safetyarxiv-cs-lg

The Collaboration Gap: Exploration and Benchmarking of Open-World Agentic Cooperation

arXiv:2511.02687v2 Announce Type: replace Abstract: The trajectory of AI development suggests that we will increasingly rely on agent-based systems powered by language models, composed of independentl

Twin Rollouts: Noise-Coupled Counterfactual Branching in Interactive Video World Models

arXiv:2608.08982v1 Announce Type: new Abstract: Interactive video world models generate rollouts autoregressively under an action stream, yet they are trained and evaluated almost exclusively on factu

researcharxiv-cs-lg

ADIAS: Automated Design of Interactive Agentic Systems

arXiv:2608.06410v1 Announce Type: new Abstract: Automated agent design improves agent harnesses through iterative revision, evaluation, and feedback summarization. Existing methods are largely candida

Agentic Planning for Symbolic Execution

arXiv:2608.06397v1 Announce Type: cross Abstract: Symbolic execution seeks to explore feasible program paths, yet a practical run may exhaust its resources while much program behaviour remains unreach

CASA: Classification Augmented with Safety Attention for Robust Multimodal Alignment

arXiv:2604.00310v2 Announce Type: replace-cross Abstract: Multimodal large-language models (MLLMs) often experience degraded safety alignment when harmful queries exploit cross-modal interactions. Mod

CoBa: Cost-Effective Test-Time Scaling via Compute-Balanced Routing

arXiv:2608.07424v1 Announce Type: new Abstract: Test-time scaling is often implemented by spending more compute along one axis: sampling more solutions, extending a chain of thought, or applying a str

Confidence Estimation for Financial Vision-Language Models in Chart and Document Understanding

arXiv:2608.06532v1 Announce Type: new Abstract: LVLMs are increasingly used to read financial charts, tables, and documents, where a single misread figure can move a decision and the most authoritativ

From Test-Time Scaling to Reusable Memory: Measuring Crystallization in Text-to-SQL

arXiv:2608.07213v1 Announce Type: new Abstract: Test-time scaling can correct difficult text-to-SQL queries, but the extra computation is normally discarded after each answer. Systems increasingly ret

researcharxiv-cs-cl

Generative Embedding Benchmark: How Much Information Survives in a Dense Embedding?

model-releasesarxiv-cs-cv

arXiv:2608.06972v1 Announce Type: new Abstract: Embeddings have emerged as a standard representational interface linking foundation models with downstream systems. Most embedding benchmarks assess rep

How Molecular Generative Models Organize Molecular Identity

arXiv:2608.06956v1 Announce Type: new Abstract: Generative models for matter are often evaluated as samplers over output representations, and their latent spaces are commonly used as proxies for navig

researcharxiv-cs-lg

Joint Optimization of Reasoning and Dual-Memory for Self-Learning Diagnostic Agent

arXiv:2604.07269v2 Announce Type: replace Abstract: Clinical expertise improves not only by acquiring medical knowledge, but by accumulating experience that yields reusable diagnostic patterns. Recent

agentsarxiv-cs-cl

Long-Horizon Agent Trajectory Attribution: A Unified Benchmark and Fine-Grained Annotation Framework

arXiv:2608.06909v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly operate through long-horizon trajectories involving user instructions, tool use, external observations, a

Provable Training Data Identification for Large Language Models

arXiv:2510.09717v3 Announce Type: replace-cross Abstract: Identifying training data of large-scale models is critical for copyright litigation, privacy auditing, and ensuring fair evaluation. However,

RegionDet: A Benchmark for Region Detection Beyond Object Instances

model-releasesarxiv-cs-cv

arXiv:2608.06850v1 Announce Type: new Abstract: Object detection is a fundamental task in computer vision and has achieved remarkable progress on standard benchmarks by localizing discrete and well-bo