Search: “benchmark”

Generalization Measures under Controlled Covariate Shift: A Regime-Aware Benchmark

model-releasesarxiv-cs-lg

arXiv:2602.01718v2 Announce Type: replace Abstract: Predicting generalization from quantities available before target-test evaluation remains a central challenge in deep learning. The systematic bench

24 Aug 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does

DGX agent

arXiv:2608.19515v1 Announce Type: new Abstract: Prosodic cues can convey task-relevant information that alters the trajectory and outcome of a task-oriented dialogue, even when the words themselves re

researcharxiv-cs-cl

21 Aug 2026

I benchmarked Ox Alpha on SWE-bench Verified Mini (50 tasks): 96% resolved. Now I’m skeptical of myself.

model-releasesr-localllama

TL;DR: The result looks TOO GOOD TO BE TRUE — that's exactly why I'm posting it. I ran Ox alpha on the entire SWE-bench Verified-Mini set with the official mini-swe-agent scaffold — the same agent use

21 Aug 2026

How Quantum Is the Advantage? A Fair, Calibration- and Noise-Aware Benchmark and Attribution Audit of Quantum Machine Learning for Network Intrusion Detection

arXiv:2608.18155v1 Announce Type: cross Abstract: Quantum machine learning (QML) for network intrusion detection (NIDS) is routinely reported to reach near-perfect accuracy, yet the most rigorous stud

20 Aug 2026

An Empirical Study of Reward Specification and Benchmark Reliability in GRPO-based LLM Unlearning

model-releasesarxiv-cs-cl

arXiv:2608.17804v1 Announce Type: cross Abstract: Practical LLM unlearning is usually evaluated through two objectives: suppress target-specific knowledge and preserve non-target utility. In generativ

19 Aug 2026

LIBERO-VIFO: Benchmarking the Capability and Safety of Visual Cue Following in Vision-Language-Action Models

model-releasesarxiv-cs-ro

arXiv:2608.17600v1 Announce Type: new Abstract: Visual cues are increasingly adopted to guide robot learning, but whether Vision-Language-Action (VLA) models can reliably follow authorized cues while

19 Aug 2026

MANIGUARD: A Benchmark and Data Suite for Specification-Grounded Safety Evaluation and Improvement of Robotic Manipulation

model-releasesarxiv-cs-ro

arXiv:2608.17386v1 Announce Type: new Abstract: Foundation-model policies for robotic manipulation are advancing rapidly on task success, but rigorous evaluation of whether they succeed safely is stil

19 Aug 2026

A Human-Centred Approach to Benchmarking LLMs for Parenting Advice

arXiv:2608.14622v1 Announce Type: new Abstract: People are increasingly using large language models (LLMs) to seek advice, including for parenting. Parenting is a critical and socially sensitive domai

researcharxiv-cs-ai

Accuracy and Reliability of Large Language Models in Cosmetic Chemistry and Skin Health: A Benchmarking Study

arXiv:2608.14631v1 Announce Type: new Abstract: As consumers increasingly turn to AI chatbots for skincare advice, the technical accuracy of Large Language Models (LLMs) in cosmetic chemistry remains

researcharxiv-cs-ai

AeroCopilotBench: A Two-Tier Benchmark for Evaluating LLM Agents as Aviation Copilots in an Interactive Virtual Cockpit Environment

arXiv:2608.16349v1 Announce Type: new Abstract: Large language model (LLM) agents may assist flight crews with complex decisions and task execution, but existing aviation evaluations centered on stati

Auxiliary uncertainty signals for LLM-assisted systematic review screening: a benchmark across eight Cohen drug-class reviews

model-releasesarxiv-cs-cl

arXiv:2608.14551v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for title-abstract screening in systematic reviews, but their decisions lack calibrated uncertainty.

Beam-Wise Statistical Background Subtraction for Static Roadside LiDAR: A Cross-Sensor Benchmark Study

arXiv:2608.14868v1 Announce Type: new Abstract: Background subtraction is a key preprocessing step for infrastructure-based LiDAR perception, enabling efficient isolation of dynamic traffic participan

Safety

Beyond Natural-Image Foundation Models: Benchmarking Satellite Pretraining for Ophthalmic Image Analysis

arXiv:2608.15195v1 Announce Type: new Abstract: Vision Foundation Models (VFMs) have emerged as a promising approach in medical imaging, producing broadly applicable systems that can be efficiently ad

safetyarxiv-cs-cv

Decoupling Parcellation from Classification: Systematic Benchmark of Fast Brain Segmentation Methods for Alzheimer's Disease Detection

arXiv:2608.16039v1 Announce Type: cross Abstract: Brain parcellation and classification are typically evaluated in isolation, yet downstream AD detection performance depends on their interaction. We d

Does generative AI supersede supervised XMLC? A Benchmark Study on Automated Subject Indexing with German Scientific Literature

arXiv:2607.14882v2 Announce Type: replace-cross Abstract: With a large controlled vocabulary as the label set, the task of automated subject indexing in a library can be understood as a multi-label cl

HiPHI: A Large-Scale Benchmark for High-Precision Human Motion and Object-Interaction

arXiv:2608.16222v1 Announce Type: cross Abstract: Humanoid intelligence requires learning over an extremely diverse space of whole-body motions and physically grounded interactions. However, existing

INSPIRE: A Benchmark for Instruction-Aware Speech Retrieval

model-releasesarxiv-cs-cl

arXiv:2608.16203v1 Announce Type: cross Abstract: Existing speech retrieval systems rely on fixed similarity matching and cannot adapt to diverse user intents. We introduce INSPIRE, the first benchmar

Qwen3.8-27B on a 24GB M4 Pro Mac mini: benchmarks and the three settings that stop it drowning

When Qwen3.8-27B dropped on Thursday the obvious question came up for us : does a 27B dense model actually fit on the 24GB Mac Mini machines? Ran it properly over the weekend on an M4 Pro (24GB unifie

model-releasesr-ollama

Reconstruction: A Blind Benchmark for Recovering Research Ideas from Pre-Publication Bibliographies

arXiv:2608.16645v1 Announce Type: new Abstract: Can a language model recover the true research idea of a published paper when given only that paper's pre-publication bibliography? We introduce Reconst

The Unwritten Benchmark: A New Challenge for Multimodal Machine Learning in Abstract Perceptual Reasoning

arXiv:2608.14558v1 Announce Type: new Abstract: Current multimodal models have demonstrated remarkable proficiency in recognizing static visual and auditory content. However, their capacity for abstra

When Stories Evolve: Benchmarking LLM Storytelling Across Agent Architectures in Open-Ended World Simulations

arXiv:2608.15654v1 Announce Type: cross Abstract: Large language models can write fluent stories, but open-ended storytelling requires more than local fluency. In evolving world simulations and AI-nat

FlatLab: A Unified Methodology Framework and Simulation-Based Benchmark for Robotic Manipulation of Flat Objects

model-releasesarxiv-cs-ro

arXiv:2608.14049v1 Announce Type: new Abstract: Robotic manipulation of flat objects is challenging due to the ungraspable configurations and strong variations in object geometry and material. Existin

17 Aug 2026

Local agentic coding Benchmark : Qwen 3.8 27B (in many weights quants / cache quants / engine / reasoning effort) vs others.

model-releasesr-localllama

In medium reasoning mode, it both scores higher than the 3.6 version, AND is very much more efficient (almost half requests needed, and a third less tokens generated) - at DeepSeek v4 Flash 3107 MXFP4

17 Aug 2026

Safety

ARAC: Benchmarking Auto-Research's Alignment and Completeness on End-to-End Researchs

arXiv:2608.12788v1 Announce Type: new Abstract: The rapid advancement of Auto-Research has surfaced a fundamental evaluation challenge: how can we measure the alignment, logical coherence, and evoluti

safetyarxiv-cs-ai

14 Aug 2026

Applications

Beyond Simulated Benchmarks: Evaluating Motion Representations for Fall Detection Under Real-World Data Scarcity

arXiv:2608.13197v1 Announce Type: new Abstract: Falls are a major health concern for older adults, and wearable sensors have been widely explored for detecting falls and enabling timely intervention.

applicationsarxiv-cs-lg

14 Aug 2026

PlayWorld: Benchmarking World Models with Agent Players over Long-Horizon Objectives

arXiv:2608.13552v1 Announce Type: new Abstract: Video world models simulate future states conditioned on current observations and user actions. Recent systems have demonstrated impressive video consis

14 Aug 2026

Advancing MLLM-based UAV Image Understanding and Reasoning: A Benchmark and a Training-Free Multi-Agent System

arXiv:2608.11738v1 Announce Type: cross Abstract: Multimodal Large Language Model (MLLM)-based UAV aerial image understanding and reasoning is essential for aerial intelligence yet poses distinct chal

Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence

arXiv:2608.11341v1 Announce Type: new Abstract: Apollo did not reach the Moon merely because its engineers could solve difficult equations. It succeeded by turning a distant ambition into a mission ar

Automated binary classification of hazelnut X-ray images: A deep-learning benchmark for quality assessment

arXiv:2608.11759v1 Announce Type: new Abstract: Non-destructive X-ray imaging can reveal internal hazelnut defects that are difficult to detect by external inspection alone; however, automated interpr

Benchmarking Cyberattack Detection in Electric Vehicle Charging Infrastructure with Benign User Updates

model-releasesarxiv-cs-lg

arXiv:2608.11286v1 Announce Type: cross Abstract: Cyberattack detection in electric vehicle charging infrastructure is complicated by legitimate post-activation revisions to requested energy and depar

Localizing to Debias: A Patch-Level Benchmark and Baseline for Weakly Supervised Spatial Anomaly Detection

arXiv:2608.12045v1 Announce Type: new Abstract: Despite growing interest in weakly supervised video anomaly detection (WSVAD), current methods struggle to bridge the gap between coarse temporal superv

MBA: Multimodal Benchmark and Agents for Real-World Business Ideation

arXiv:2608.11616v1 Announce Type: new Abstract: Agentic systems powered by large language models (LLMs) have opened new opportunities for business ideation. Yet existing approaches remain confined to

On Benchmarking Human-Like Intelligence in Machines

arXiv:2502.20502v2 Announce Type: replace Abstract: Recent advances in Artificial Intelligence (AI) have yielded powerful computational models that, by learning from vast amounts of human-generated da

researcharxiv-cs-ai

Agents

Total Recall at What Cost? Benchmarking the Serving Cost of Agentic Memory Systems

arXiv:2608.11879v1 Announce Type: new Abstract: Long-running conversational agents increasingly rely on a memory system to avoid resending the whole conversation each turn, yet how much that costs to

agentsarxiv-cs-cl

TRACES: A Benchmark for Epistemic Reliability in Scientific Reasoning by LLMs

arXiv:2608.11415v1 Announce Type: cross Abstract: Large language models are being proposed as agents in scientific workflows, in domains where no downstream verifier exists. Such deployment assumes th

Benchmarking LLM-Guided Control-Plane Policies for Backend Fault Isolation in HAProxy

model-releasesarxiv-cs-lg

arXiv:2608.10532v1 Announce Type: cross Abstract: Static load balancers cannot mitigate a backend that is degraded rather than down: round-robin and least-connections keep routing traffic to a server

Cross-View Feature Matching: Survey, Benchmarking, and Foundation-Model Perspectives

arXiv:2608.11093v1 Announce Type: cross Abstract: Cross-view feature matching aims to establish reliable correspondences across images with large viewpoint variations. Over the past decade, the field

researcharxiv-cs-cv

FaithformBench: Benchmarking Faithfulness of Mathematical Chain-of-Thought Autoformalisation

arXiv:2608.10916v1 Announce Type: cross Abstract: Autoformalisation (AF) systems map natural language reasoning steps into formal statements in a proof assistant such as Lean. We consider how to asses

PRMU: A Corpus-Free Benchmark for Person-Centric Knowledge Unlearning in Multimodal Large Language Models

arXiv:2608.11149v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have demonstrated remarkable capabilities in storing and recalling rich person-related knowledge, raising incre

Beyond Pixels: Benchmarking and Reward-Based Assessing Framework for Visual Spatial Aesthetics

arXiv:2512.05098v2 Announce Type: replace-cross Abstract: In recent years, Image Quality Assessment (IQA) for AI-generated images (AIGI) has advanced rapidly; however, existing methods primarily targe

Applications

BRUCE: Benchmarking Robustness Under Corruption Escalation for Scientific Vision-Language Reasoning

arXiv:2608.07742v1 Announce Type: new Abstract: Visual-language models (VLMs) frequently struggle with robustness issues in real-world situations due to low- or varying-quality input images. In this p

applicationsarxiv-cs-cv

Build it, Break it, Repeat: Benchmarking and improving LLM-manipulated disinformation detection in social media posts

arXiv:2608.09510v1 Announce Type: cross Abstract: Detecting machine-generated disinformation on social media is increasingly difficult as large language models (LLMs) make it easier to generate and re

Agents

Business Arena: Benchmarking LLM Agents in a Realistic Marketplace

arXiv:2608.08621v1 Announce Type: new Abstract: Running a business is a challenging form of intelligent work. Operators must infer opportunities from partial signals, commit capital under uncertainty,

agentsarxiv-cs-ai

Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives

arXiv:2608.08160v1 Announce Type: cross Abstract: The rapid advancement of Large Language Models (LLMs) is revolutionizing AI for Games by enabling open-ended and fluid interactive storytelling. Howev

CircuitReason-1k: Benchmarking Long-Horizon Visual-to-Symbolic Reasoning inElectrical Circuits

arXiv:2608.09374v1 Announce Type: new Abstract: Electrical circuit analysis requires more than recognizing components in an image. A solver must ground symbols and labels, recover latent topology, sel

ComplexityWorld: Benchmarking Vision-Language Models on Verifiable Visual Decision Making

arXiv:2608.07584v1 Announce Type: new Abstract: Vision-language models (VLMs) have made rapid progress in visual perception and increasingly support real-world tasks that depend on images. Many such t

CORDA: A Benchmark for Hierarchical Harm-Centric Moral Reasoning in Large Language Models

arXiv:2608.08061v1 Announce Type: new Abstract: The key question in moral judgement is not simply whether someone chooses the 'right' answer, but how they decide what matters most when moral principle

Fair on the Surface? Benchmarking Hidden-Output Fairness Gaps in LLM Recommenders

arXiv:2608.08284v1 Announce Type: new Abstract: Fairness audits for LLM-based recommenders have largely focused on observable outputs, implicitly assuming that stable recommendations reflect stable in