Search: “benchmark”

PolarScale: A Physics-Grounded Benchmark for Radiometrically Consistent RGB-to-Stokes Estimation

arXiv:2610.08346v1 Announce Type: new Abstract: Polarization imaging provides physical cues beyond intensity imaging but typically requires specialized hardware. Recent methods infer polarization from

7 Oct 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation

DGX agent

arXiv:2610.08026v1 Announce Type: new Abstract: In recent years, several methods for detecting when large language models (LLMs) hallucinate have been developed. These methods are often benchmarked wi

model-releasesarxiv-cs-cl

7 Oct 2026

Towards a Unified Misuse Monitoring Benchmark

arXiv:2610.07089v1 Announce Type: cross Abstract: LLM agents increasingly act in multi-actor environments, exposing them to misuse from multiple sources: decomposition attacks, where a harmful request

7 Oct 2026

Detecting Defects that Matter: An Application-Driven Benchmark for Anomaly Detection in Manufacturing and Retail Logistics (VAND 4.0 Challenge)

arXiv:2610.04392v1 Announce Type: new Abstract: Existing Anomaly Detection benchmarks are saturated and often unrealistic. As part of the VAND 4.0 Challenge, we introduce a hidden-test, application-dr

6 Oct 2026

InteractionBench: A Real-Time Interaction Benchmark for Streaming Video Systems

arXiv:2610.05775v1 Announce Type: new Abstract: A video assistant must speak when its instruction warrants a response and stay silent otherwise. We introduce a benchmark that evaluates this decision f

6 Oct 2026

Same Output, Different Gold: Measuring How Reference Choice Moves a Multilingual Benchmark Score

arXiv:2610.03825v1 Announce Type: cross Abstract: A benchmark score compares a system output against a reference, and methodological attention falls almost entirely on the first term. We measure the s

6 Oct 2026

EvoRiskBench: An Evolving Benchmark for Runtime Security Risks in Workspace Agents

arXiv:2610.03153v1 Announce Type: cross Abstract: Workspace agents combine large language models with execution harnesses to perform stateful, multi-step tasks that access or modify external resources

5 Oct 2026

MintEval: Do LLMs Implement the Trading Strategy You Asked For? A Behavioural-Equivalence Benchmark for Natural-Language-to-Strategy Code

arXiv:2610.03080v1 Announce Type: cross Abstract: Large language models are moving from producing trading signals to writing the code that executes them. The failure mode of the second role is silent:

5 Oct 2026

RxnOptBench: Benchmarking LLMs for Reaction-Condition Optimization in Organic Methodology

arXiv:2610.02242v1 Announce Type: cross Abstract: Chemical reaction-condition optimization -- choosing the catalyst, ligand, solvent, reagent, temperature, time, and atmosphere that jointly maximize y

5 Oct 2026

Benchmarking Prompt Optimization of Large Language Models With Chess

arXiv:2610.00416v1 Announce Type: new Abstract: Evaluating large language models becomes increasingly challenging as their capabilities advance: benchmarks can saturate, public test sets risk contamin

CompMat-Bench: Benchmarking AI Agents for Computational Materials Science

arXiv:2610.00636v1 Announce Type: new Abstract: Evaluating AI agents on scientific research tasks is constrained by the time and resources required for the underlying experiments or calculations. In c

Cross-Benchmark Transfer from RL on Agentic Coding Tasks

arXiv:2610.00890v1 Announce Type: cross Abstract: Coding agents often fail in the last mile: they build most of a feature but drop a requirement, test only the cases their implementation already handl

DAYJOB: A Benchmark for Long-Horizon Professional Work

arXiv:2610.01306v1 Announce Type: new Abstract: Professional work often starts with a brief request that leaves the professional to work out what is needed, which documents matter, and whether the req

Detect, Explain, Interpret: An End-to-End Benchmark for Time Series Anomaly Detection, Explainability and Interpretability

arXiv:2610.01168v1 Announce Type: cross Abstract: Time Series Anomaly Detection has received increasing attention, driven by the growing availability of complex time series data. This surge has led to

M^2Weather: A Benchmark for Joint Multi-Station and Multi-Variable Weather Forecasting

arXiv:2610.00370v1 Announce Type: new Abstract: Station weather forecasting is fundamentally shaped by both complex spatial dependencies across stations and strong physical coupling among weather vari

EngramBench: A Capability-Grounded Benchmark for Skill-Evolution Harnesses

arXiv:2609.39284v1 Announce Type: cross Abstract: While large language models have achieved remarkable success in isolated code generation, authentic software engineering requires sustained reasoning,

FFASR: Benchmarking Far-Field Automatic Speech Recognition using High-Fidelity Simulated RIRs

arXiv:2609.38897v1 Announce Type: cross Abstract: Far-field automatic speech recognition(ASR) degrades under reverberation, noise, and talker motion, yet the benchmarks that drive model selection emph

SEABench: Benchmarking Endogenous Misalignment In Self-Evolving Agents

arXiv:2609.35596v2 Announce Type: replace-cross Abstract: Self-evolving LLM agents have gained prominence for their ability to improve after deployment by modifying their harness, including their cont

ViLegalExpert: A Large-Scale Benchmark for Vietnamese Legal Retrieval and Question Answering from Real-World Consultations

arXiv:2609.39189v1 Announce Type: new Abstract: Trustworthy Legal AI requires systems that can answer legal questions while grounding their responses in authoritative sources. However, existing Vietna

WARP: A Unified Benchmark for Invisible Image Watermarking -- Robustness and Protection Against Attacks

arXiv:2609.40031v1 Announce Type: cross Abstract: Digital image watermarking is increasingly critical in media contexts, as emerging regulations and industry practices require marking AI-generated con

Multi-Class, Multi-Tier Network Intrusion Detection: A Comprehensive and Reproducible Benchmark

arXiv:2609.36039v1 Announce Type: cross Abstract: Machine learning (ML) and deep learning (DL) have dominated Intrusion Detection System (IDS) research in recent years. Unfortunately, many existing st

30 Sept 2026

Anytime-Valid LLM Leaderboards via Benchmark-weighted and Block-Factorized e-Processes

arXiv:2609.32248v1 Announce Type: new Abstract: Large language model (LLM) leaderboards compare model capabilities by ranking models according to their mean performance on fixed benchmarks. However, v

29 Sept 2026

SymbolicArena: A Unified Infrastructure for Benchmark Distillation and Dynamic Evaluation in Symbolic Regression

arXiv:2609.35113v1 Announce Type: new Abstract: Symbolic regression (SR) seeks concise and interpretable mathematical expressions from data for scientific equation discovery. Existing SR benchmarks fa

29 Sept 2026

BioEVAL: A global, multi-institutional benchmark of large language and multimodal models for bioengineering

arXiv:2609.30489v1 Announce Type: new Abstract: Large Language Models (LLMs) have demonstrated historic breakthroughs in general reasoning with early successes in biomedical science. However, existing

28 Sept 2026

SatNav: A Scalable Benchmark for Long-Horizon UAV Vision-Language Navigation from Satellite Imagery

arXiv:2609.31507v1 Announce Type: new Abstract: Urban uncrewed aerial vehicle (UAV) vision-language navigation (VLN) requires agents to follow instructions across extended urban spaces, inherently dem

28 Sept 2026

SciHorizon-eLab: An Agentic Protocol-to-Task Compiler for Scalable Benchmarking of Scientific Embodied Agents

arXiv:2609.30971v1 Announce Type: new Abstract: Embodied agents offer a promising route to automating scientific experimentation, yet their progress is constrained by the lack of reliable and systemat

28 Sept 2026

Beyond Spatial Benchmarks: From Spatial Reasoning to Navigation

arXiv:2609.29934v1 Announce Type: new Abstract: Does progress on spatial reasoning benchmarks translate into better navigation? Existing benchmarks test isolated inferences from images or videos, with

25 Sept 2026

Recommended benchmark. I expect voice to become one of the main ways people interact with robots and physical AI. That makes speech recognit…

Recommended benchmark. I expect voice to become one of the main ways people interact with robots and physical AI. That makes speech recognition in real conversations much more important than it seems

model-releasesdair-ai--x

25 Sept 2026

Benchmarking Text-to-SQL under Role-Based Access Control

arXiv:2607.22115v2 Announce Type: replace-cross Abstract: Given a database S and a natural language question Q, text-to-SQL systems aim to generate an SQL query that correctly answers Q when executed

24 Sept 2026

I built a desktop benchmark for local Ollama models — TTFT, tok/s, VRAM, with a HuggingFace GGUF catalogue

Built this because I got tired of guessing whether a model would fit in my 6GB VRAM before downloading it, and because ollama.com's library doesn't let you sort by size. What it does: - Benchmarks Oll

model-releasesr-ollama

24 Sept 2026

PRISM-VLM: A Multi-Axis Discriminative Benchmark for Compact Vision-Language Models

arXiv:2609.27395v1 Announce Type: new Abstract: Compact vision-language models (VLMs) now power a growing share of multimodal applications. The benchmarks used to compare them, however, inherit a fron

24 Sept 2026

Claude Opus 5.5 is now available as the Standard effort level in Perplexity Computer. On our WANDR benchmark, it scored 0.610 at $4.13 per t…

model-releasesperplexity--x

Claude Opus 5.5 is now available as the Standard effort level in Perplexity Computer. On our WANDR benchmark, it scored 0.610 at $4.13 per task, slightly outperforming Claude Fable 5.1 while costing 6

22 Sept 2026

QontoFAQ: A better Information Retrieval Benchmark [R]

model-releasesr-machinelearning

Retrieval benchmarks sometimes feel benchmaxxed by models, so we wanted to find a way to tie it as close as possible to my objective: finding the article that answers a product question right. We work

22 Sept 2026

A Benchmark Suite and Ground-Truth Methodology for Formal Verification of IEC 61131-3 Ladder Diagram Programs

arXiv:2609.18994v1 Announce Type: new Abstract: We present the first benchmark suite for formal verification of Programmable Logic Controller (PLC) programs that combines controlled ground truth with

Behavior2Value: Benchmarking and Empowering LLMs for Consumer Value Measurement from E-commerce Behaviors

arXiv:2609.18203v1 Announce Type: new Abstract: Human values are deep motivational orientations that shape human behaviors. In e-commerce, they reveal the stable drivers behind users' purchase decisio

Benchmarking Large Language Models for Biomedical Relation Extraction

arXiv:2609.19071v1 Announce Type: new Abstract: Extracting SNP-phenotype associations from biomedical literature is vital but challenging. We benchmarked diverse NLP models, including MLMs, hybrid arc

BOOM: Benchmarking Out-Of-distribution Molecular Property Predictions of Machine Learning Models

arXiv:2505.01912v3 Announce Type: replace-cross Abstract: Data-driven molecular discovery leverages artificial intelligence/machine learning (AI/ML) and generative modeling to filter and design novel

CapMem: A Benchmark for Caption-Based Episodic Memory in Egocentric Video

arXiv:2609.17688v1 Announce Type: new Abstract: Wearable assistants require episodic memory over egocentric video, yet current vision-language models face bounded frame budgets, growing visual-token c

Tutorials

Fallacy Benchmarks Measure Scheme Recognition, Not Fallacy Detection

arXiv:2609.18644v1 Announce Type: new Abstract: Fallacy-detection benchmarks pair fallacy classes with a single 'valid' or 'none' class that takes everything data collection did not label as a fallacy

tutorialsarxiv-cs-cl

Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarks

arXiv:2609.17817v1 Announce Type: cross Abstract: Thompson's 'Reflections on Trusting Trust' showed that a compiler can be poisoned to reinsert its own backdoor, so that even recompiling clean source

RoboVAD: A Large Cross-Domain Evaluation Benchmark for Anomaly Detection in Robotic Arm Manipulation Videos

arXiv:2609.17843v1 Announce Type: cross Abstract: Video anomaly detection (VAD) is an actively studied task, having wide applications in typical scenarios such as public surveillance and road traffic

Safety-Flag: A Unified Benchmark for the Reliability and Calibration of LLM Content Moderators

arXiv:2609.19072v1 Announce Type: new Abstract: Large language models are increasingly used for content moderation, but most evaluations still report aggregate accuracy on individual benchmarks. We in

Semantic-ITC: A Frame-wise Indoor Mobile Laser Scanning Dataset and Benchmark for Semantic Segmentation

arXiv:2609.18493v1 Announce Type: new Abstract: Semantic labels for indoor mobile laser scanning (MLS) frames remain largely absent from current point cloud semantic segmentation benchmarks, which mai

VoiceTrace: A Benchmark and Retrieval Framework for Who-Said-What Speech Retrieval

arXiv:2609.18521v1 Announce Type: cross Abstract: Speech retrieval has become increasingly important as spoken content continues to grow across meetings, lectures, podcasts, and videos. Existing bench

BLINDSPOT: A Benchmark for Safety and Refusal Calibration in Long-Horizon Tool-Using Agents

arXiv:2609.16305v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly operate over long-horizon interactions involving tool use, persistent state, evolving authorization, and

FirmCORe: A Benchmark for Structured Reasoning about Inter-Firm Collaboration Opportunities

arXiv:2609.17128v1 Announce Type: new Abstract: Comprehensive structured data on inter-firm relationships is often scarce or inaccessible because many relationships are privately negotiated, selective

Not Another Text Benchmark: Putting the 'Visual' Back in Visual Question Answering for Large Video Models

arXiv:2609.17112v1 Announce Type: new Abstract: Large video models have exhibited impressive performance on a wide range of visual question answering tasks, owing to the rise of powerful, pretrained t

RoleBreak: Benchmarking Long-Horizon Role-Playing Robustness in Spoken Dialogue

arXiv:2609.16614v1 Announce Type: cross Abstract: Speech-to-speech dialogue models increasingly support persona control, yet existing spoken role-playing benchmarks remain largely character-centric an