Search: “evaluation”

Revisiting the Travel Planning Capabilities of Large Language Models

arXiv:2605.03308v1 Announce Type: new Abstract: Travel planning serves as a critical task for long-horizon reasoning, exposing significant deficits in LLMs. However, existing benchmarks and evaluation

7 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

A Three-Stage Offline SDRE-Based Control Framework for Human Motion Reproduction on a Suspended Bipedal Robot

DGX agent

arXiv:2506.04680v2 Announce Type: replace Abstract: During the development of wearable exoskeletons, evaluations involving human subjects pose inherent safety risks. Therefore, systematic testing is o

safetyarxiv-cs-ro

6 May 2026

Agentic AI Systems Should Be Designed as Marginal Token Allocators

arXiv:2605.01214v1 Announce Type: new Abstract: This position paper argues that agentic AI systems should be designed and evaluated as marginal token allocation economies rather than as text generator

Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs

model-releasesarxiv-cs-cv

arXiv:2512.09874v2 Announce Type: replace Abstract: Correctly parsing mathematical formulas from PDFs is critical for training large language models and building scientific knowledge bases from academ

Conditional Diffusion Sampling

arXiv:2605.04013v1 Announce Type: cross Abstract: Sampling from unnormalized multimodal distributions with limited density evaluations remains a fundamental challenge in machine learning and natural s

researcharxiv-cs-lg

Large Language Models are Universal Reasoners for Visual Generation

arXiv:2605.04040v1 Announce Type: new Abstract: Text-to-image generation has advanced rapidly with diffusion models, progressing from CLIP and T5 conditioning to unified systems where a single LLM bac

safetyarxiv-cs-cv

Taking the GP Out of the Loop

arXiv:2506.12818v3 Announce Type: replace Abstract: Bayesian optimization (BO) has traditionally solved black-box problems where function evaluation is expensive and, therefore, observations are few.

researcharxiv-cs-lg

Trojan Hippo: Weaponizing Agent Memory for Data Exfiltration

arXiv:2605.01970v2 Announce Type: cross Abstract: Memory systems enable otherwise-stateless LLM agents to persist user information across sessions, but also introduce a new attack surface. We characte

very fun to collab with @harvey on their Long Horizon Legal Agent Benchmark. We need more industry specific benchmarks, and Harvey is paving…

model-releasesharrison-chase--x

Harrison Chase expresses enthusiasm about collaborating with Harvey on their Long Horizon Legal Agent Benchmark, highlighting the value of developing industry-specific benchmarks for AI evaluation. Th

Voice Mapping of Text-to-Speech Systems: A Metric-Based Approach for Voice Quality Assessment

arXiv:2605.00861v1 Announce Type: cross Abstract: This study investigates voice mapping as an evaluation framework for text-to-speech (TTS) synthesis quality. The study analyzes six TTS models, includ

researcharxiv-cs-ai

ALIGNS: Unlocking nomological networks in psychological measurement through a large language model

arXiv:2509.09723v3 Announce Type: replace Abstract: Psychological measurement is critical to many disciplines. Despite advances in measurement, building nomological networks, theoretical maps of how c

ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review

arXiv:2605.02651v1 Announce Type: cross Abstract: Scientific peer review increasingly struggles to assess reproducibility at the scale and complexity of modern research output. Evaluating reproducibil

Black-box optimization of noisy functions with unknown smoothness

arXiv:2605.02462v1 Announce Type: cross Abstract: We study the problem of black-box optimization of a function f of any dimension, given function evaluations perturbed by noise. The function is assume

researcharxiv-cs-lg

CombinationTS: A Modular Framework for Understanding Time-Series Forecasting Models

arXiv:2605.01231v1 Announce Type: new Abstract: Recent progress in time-series forecasting has led to rapidly increasing architectural complexity, yet many reported State-of-the-Art gains are statisti

Compared to What? Baselines and Metrics for Counterfactual Prompting

arXiv:2605.01048v1 Announce Type: new Abstract: Counterfactual prompting (i.e., perturbing a single factor and measuring output change) is widely used to evaluate things like LLM bias and CoT faithful

Continuous quantification of viral plaque dynamics using ultra-large-area label-free imaging enables rapid antiviral susceptibility testing

arXiv:2605.01738v1 Announce Type: cross Abstract: The plaque reduction assay (PRA) remains the gold standard for antiviral susceptibility testing, evaluating drug potency by measuring reductions in pl

researcharxiv-cs-cv

Applications

ECG-biometrics-bench: A Unified Framework for Reproducible Benchmarking of ECG Biometrics

arXiv:2605.01548v1 Announce Type: cross Abstract: Electrocardiogram (ECG) biometrics have emerged as a promising modality for continuous, liveness-aware authentication in wearable systems. However, ma

applicationsarxiv-cs-cv

Foundation Models to Unlock Real-World Evidence from Nationwide Medical Claims

arXiv:2605.02740v1 Announce Type: cross Abstract: Evidence derived from large-scale real-world data (RWD) is increasingly informing regulatory evaluation and healthcare decision-making. Administrative

Hardware

Maistros: A Greek Large Language Model Adapted Through Knowledge Distillation From Large Reasoning Models

arXiv:2605.01870v1 Announce Type: new Abstract: Large Language Models (LLMs) have substantially advanced the field of Natural Language Processing (NLP), achieving state-of-the-art performance across a

hardwarearxiv-cs-cl

Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Models

arXiv:2511.21086v2 Announce Type: replace Abstract: Large language models must satisfy hard orthographic constraints during controlled text generation, yet systematic cross-family evaluation remains l

Tutorials

Rethinking Electro-Optical Vision Foundation Models for Remote Sensing Retrieval: A Controlled Comparison with Generalist VFM

arXiv:2605.02283v1 Announce Type: new Abstract: Vision foundation models have attracted significant attention for their ability to leverage large-scale unlabeled visual data. This advantage is particu

tutorialsarxiv-cs-cv

SurgCheck: Do Vision-Language Models Really Look at Images in Surgical VQA?

model-releasesarxiv-cs-cv

arXiv:2605.01911v1 Announce Type: new Abstract: Purpose: Vision-language models (VLMs) have shown promising performance in surgical visual question answering (VQA). However, existing surgical VQA data

TRAP: Tail-aware Ranking Attack for World-Model Planning

arXiv:2605.01950v1 Announce Type: new Abstract: World models enable long-horizon planning by internally generating and evaluating imagined trajectories, making them a promising foundation for generali

safetyarxiv-cs-lg

Concolic Testing on Individual Fairness of Neural Network Models

arXiv:2509.06864v2 Announce Type: replace Abstract: This paper introduces PyFair, a formal framework for evaluating and verifying individual fairness of Deep Neural Networks (DNNs). By adapting the co

4 May 2026

InterChart: Benchmarking Visual Reasoning Across Decomposed and Distributed Chart Information

arXiv:2508.07630v2 Announce Type: replace Abstract: We introduce InterChart, a diagnostic benchmark that evaluates how well vision-language models (VLMs) reason across multiple related charts, a task

4 May 2026

Local Ai

TimesNet-Gen: Deep Learning-based Site Specific Strong Motion Generation

arXiv:2512.04694v3 Announce Type: replace Abstract: Effective earthquake risk reduction relies on accurate site-specific evaluations, which require models capable of representing the influence of loca

local-aiarxiv-cs-lg

4 May 2026

Agentic Compilation: Mitigating the LLM Rerun Crisis for Minimized-Inference-Cost Web Automation

arXiv:2604.09718v2 Announce Type: cross Abstract: LLM-driven web agents operating through continuous inference loops -- repeatedly querying a model to evaluate browser state and select actions -- exhi

CBEN -- A Multimodal Machine Learning Dataset for Cloud Robust Remote Sensing Image Understanding

arXiv:2602.12652v2 Announce Type: replace Abstract: Clouds are a common phenomenon that distorts optical satellite imagery, which poses a challenge for remote sensing. However, in the literature cloud

researcharxiv-cs-cv

CoAX: Cognitive-Oriented Attribution eXplanation User Model of Human Understanding of AI Explanations

arXiv:2604.27354v1 Announce Type: new Abstract: Explainable AI (XAI) aims to improve user understanding and decisions when using AI models. However, despite innovations in XAI, recent user evaluations

safetyarxiv-cs-ai

D3-Gym: Constructing Real-World Verifiable Environments for Data-Driven Discovery

arXiv:2604.27977v1 Announce Type: new Abstract: Despite recent progress in language models and agents for scientific data-driven discovery, further advancing their capabilities is held back by the abs

safetyarxiv-cs-ai

From Test-taking to Cognitive Scaffolding: A Pedagogical Diagnostic Benchmark for LLMs on English Standardized Tests

arXiv:2505.17056v2 Announce Type: replace-cross Abstract: As large language models (LLMs) are increasingly integrated into educational tools, current evaluations on standardized tests predominantly fo

Useless but Safe? Benchmarking Utility Recovery with User Intent Clarification in Multi-Turn Conversations

arXiv:2604.27093v1 Announce Type: cross Abstract: Current LLM safety alignment techniques improve model robustness against adversarial attacks, but overlook whether and how LLMs can recover helpfulnes

Applications

When Your LLM Reaches End-of-Life: A Framework for Confident Model Migration in Production Systems

arXiv:2604.27082v1 Announce Type: new Abstract: We present a framework for migrating production Large Language Model (LLM) based systems when the underlying model reaches end-of-life or requires repla

applicationsarxiv-cs-ai

Reasoning Gets Harder for LLMs Inside A Dialogue

arXiv:2603.20133v2 Announce Type: replace Abstract: Large Language Models (LLMs) achieve strong performance on many reasoning benchmarks, yet these evaluations typically focus on isolated tasks that d

30 Apr 2026

Risk Reporting for Developers' Internal AI Model Use

arXiv:2604.24966v1 Announce Type: cross Abstract: Frontier AI companies first deploy their most advanced models internally, for weeks or months of safety testing, evaluation, and iteration, before a p

safetyarxiv-cs-ai

30 Apr 2026

Safety Is Not Universal: The Selective Safety Trap in LLM Alignment

arXiv:2601.04389v2 Announce Type: replace-cross Abstract: Current safety evaluations of large language models (LLMs) create a dangerous illusion of universal protection by aggregating harms under gene

30 Apr 2026

Tutorials

Assessment of the quantitative impact of occlusal positioning splints on temporomandibular joint conditions

arXiv:2604.25322v1 Announce Type: new Abstract: A computational method for quantitative analysis of temporomandibular joint (TMJ) configuration using occlusal positioning splints is proposed and demon

tutorialsarxiv-cs-cv

BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks

arXiv:2604.24955v1 Announce Type: new Abstract: As benchmarks grow in complexity, many apparent agent failures are not failures of the agent at all - they are failures of the benchmark itself: broken

Feasible-First Exploration for Constrained ML Deployment Optimization in Crash-Prone Hierarchical Search Spaces

arXiv:2604.25073v1 Announce Type: new Abstract: Deploying machine learning models under production constraints requires joint optimization over model family, quantization scheme, runtime backend, and

Genie Sim 3.0 : A High-Fidelity Comprehensive Simulation Platform for Humanoid Robot

model-releasesarxiv-cs-ro

arXiv:2601.02078v2 Announce Type: replace Abstract: The development of robust and generalizable robot learning models is critically contingent upon the availability of large-scale, diverse training da

I built a TPU-native medical Q&A fine-tuning pipeline using Gemma 3 with Keras and JAX The project fine-tunes Gemma-3 on medical dialogue da…

model-releasesfrancois-chollet--x

I built a TPU-native medical Q&A fine-tuning pipeline using Gemma 3 with Keras and JAX The project fine-tunes Gemma-3 on medical dialogue data from ChatDoctor and evaluates it on MedMCQA, with a large

Limited Linguistic Diversity in Embodied AI Datasets

arXiv:2601.03136v2 Announce Type: replace Abstract: Language plays a critical role in Vision-Language-Action (VLA) models, yet the linguistic characteristics of the datasets used to train and evaluate

researcharxiv-cs-cl

OMHBench: Benchmarking Balanced and Grounded Omni-Modal Multi-Hop Reasoning

arXiv:2508.16198v3 Announce Type: replace Abstract: Multimodal Large Language Models (MLLMs) have increasingly supported omni-modal processing across text, vision, and speech. However, existing evalua

TEACar: An Open-Source Autonomous Driving Platform

arXiv:2604.24934v1 Announce Type: new Abstract: Intelligent Transportation Systems (ITS) increasingly rely on vision-based perception and learning-based control, necessitating experimental platforms t

agentsarxiv-cs-ro

The Forensic Cost of Watermark Removal

model-releasesarxiv-cs-cv

arXiv:2604.25491v1 Announce Type: new Abstract: Current watermark removal methods are evaluated on two axes: attack success rate and perceptual quality. We show this is insufficient. While state-of-th

Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translation

arXiv:2511.21517v2 Announce Type: replace Abstract: Unlike text, speech conveys information about the speaker, such as gender, through acoustic cues like pitch. This gives rise to modality-specific bi

A Systematic Approach for Large Language Models Debugging

arXiv:2604.23027v1 Announce Type: new Abstract: Large language models (LLMs) have become central to modern AI workflows, powering applications from open-ended text generation to complex agent-based re

28 Apr 2026

GAMMAF: A Common Framework for Graph-Based Anomaly Monitoring Benchmarking in LLM Multi-Agent Systems

arXiv:2604.24477v1 Announce Type: cross Abstract: The rapid integration of Large Language Models (LLMs) into Multi-Agent Systems (MAS) has significantly enhanced their collaborative problem-solving ca