Search: “benchmark”

arXiv:2605.18059v1 Announce Type: new Abstract: Robustness is a critical requirement for deploying autonomous driving systems in the real world. Existing robustness benchmarks for autonomous driving h

arXiv:2506.03837v2 Announce Type: replace-cross Abstract: The discovery of high-temperature superconducting materials holds great significance for human industry and daily life. In recent years, resea

I've been thinking a lot about the two different groups of evals you need in general agents/agents which handle broad tasks: 1. Benchmark evals - this is a suite of up to 100 eval cases which test the

arXiv:2605.17256v1 Announce Type: cross Abstract: This work introduces a latency-aware benchmarking framework for evaluating deep learning models in power system anomaly detection using high-fidelity,

arXiv:2605.16675v1 Announce Type: new Abstract: We introduce LinAlg-Bench, a diagnostic benchmark evaluating 10 frontier large language models on structured linear algebra computation across a strict

arXiv:2605.16881v1 Announce Type: new Abstract: Digital humanities projects increasingly rely on machine translation and large language models to widen access to classical, religious, and otherwise un

arXiv:2605.18430v1 Announce Type: new Abstract: Text-to-CAD generation aims to create parametric CAD models from natural language, enabling rapid prototyping and intuitive design workflows. However, e

arXiv:2605.16909v1 Announce Type: new Abstract: Tool-using agents are increasingly expected to operate across realistic professional workflows, where they must interpret multimodal inputs, coordinate

arXiv:2504.18361v2 Announce Type: replace-cross Abstract: Recent advances in image manipulation have enabled highly photorealistic content generation, but also lowered the barrier to arbitrary editing

arXiv:2605.16257v1 Announce Type: new Abstract: Achieving human-level manipulation requires dexterous robotic hands capable of complex object interactions. Advancing such capabilities further demands

arXiv:2605.15482v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly being applied to financial analysis, reporting, investment decision support, risk management, compliance,

arXiv:2605.15574v1 Announce Type: new Abstract: Longitudinal chest X-ray (CXR) interpretation requires reasoning over disease evolution across multiple patient visits, yet most existing medical VQA be

arXiv:2605.16116v1 Announce Type: new Abstract: Developing and evaluating e-commerce web agents requires environments that preserve meaningful task structure while enabling controllable, reproducible,

Are your benchmarks actually measuring the capability you think they measure? New paper says they probably not. Coined the 'The Evaluation Trap', it provides a vocabulary for auditing whether your eva

arXiv:2605.13950v1 Announce Type: cross Abstract: Autonomous language-model agents are increasingly evaluated on long-horizon tool-use tasks, but existing benchmarks rarely capture the complexity and

arXiv:2605.14068v1 Announce Type: new Abstract: We introduce CurveBench, a benchmark for hierarchical topological reasoning from visual input. CurveBench consists of extbf{756 images} of pairwise non-

arXiv:2511.00839v2 Announce Type: replace-cross Abstract: Current benchmarks for coding evaluate language models (LMs) on concrete, well-specified tasks such as fixing specific bugs or writing targete

arXiv:2605.13171v1 Announce Type: new Abstract: As automated reasoning systems advance rapidly, there is a growing need for research-level formal mathematical problems to accurately evaluate their cap

arXiv:2605.13172v1 Announce Type: cross Abstract: Recent advances in agent and multi-agent systems have shown strong performance on tool use, reasoning, and collaborative tasks. However, existing benc

arXiv:2605.11224v1 Announce Type: new Abstract: Existing medical-agent benchmarks deliver imaging as pre-selected samples, never as an environment the agent must navigate. We introduce ABRA, a radiolo

arXiv:2505.13770v3 Announce Type: replace-cross Abstract: Reliable causal inference is essential for making decisions in high-stakes areas like medicine, economics, and public policy. However, it rema

arXiv:2605.11680v1 Announce Type: new Abstract: We introduce ShapeCodeBench, a synthetic benchmark for perception-to-program reconstruction: given a rendered raster image, a model must emit an executa

arXiv:2605.11696v1 Announce Type: new Abstract: Recent single-image relighting methods, powered by advanced generative models, have achieved impressive photorealism on synthetic benchmarks. However, t

arXiv:2603.26680v2 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) evolve into lifelong AI assistants, LLM personalization has become a critical frontier. However, progress is c

arXiv:2605.09698v1 Announce Type: new Abstract: As data-science agents shift from co-pilots to auto-pilots, silent misframing becomes a critical failure mode. Agents quietly commit to plausible but un

arXiv:2605.10873v1 Announce Type: cross Abstract: Recovering editable CAD programs from images or 3D observations is central to AI-assisted design, but progress is difficult to measure because existin

arXiv:2605.09708v1 Announce Type: cross Abstract: We present Metal-Sci, a 10-task benchmark of scientific Apple Silicon Metal compute kernels spanning six optimization regimes (stencils, all-pairs in

arXiv:2605.10616v1 Announce Type: cross Abstract: Tabular Foundation Models have recently established the state of the art in supervised tabular learning, by leveraging pretraining to learn generaliza

arXiv:2605.10550v1 Announce Type: new Abstract: Document classification forms the backbone of modern enterprise content management, yet existing benchmarks remain trapped in oversimplified paradigms -

arXiv:2605.08163v1 Announce Type: cross Abstract: Text-in-image editing has become a key capability for visual content creation, yet existing benchmarks remain overwhelmingly English-centric and often

arXiv:2605.08762v1 Announce Type: cross Abstract: Current omni-modal benchmarks mainly evaluate models under settings where multiple modalities are provided simultaneously, while the ability to start

arXiv:2605.09636v1 Announce Type: new Abstract: PDE-to-solver code generation aims to automatically synthesize executable numerical solvers from partial differential equation (PDE) specifications. Thi

arXiv:2605.10032v1 Announce Type: new Abstract: Cell-type-specific marker genes are fundamental to plant biology, yet existing resources primarily rely on curated databases or high-throughput studies

arXiv:2605.10376v1 Announce Type: new Abstract: Vision-Language Models (VLMs) have advanced rapidly in multimodal perception and language understanding, yet it remains unclear whether they can reliabl

arXiv:2605.09610v1 Announce Type: cross Abstract: We introduce SmartEval, a benchmark for systematically evaluating the quality of Solidity smart contracts generated by large language models (LLMs) fr

arXiv:2505.18511v2 Announce Type: replace Abstract: Stochastic Partial Differential Equations (SPDEs) driven by random noise play a central role in modeling physical processes with rough spatio-tempor

arXiv:2605.08366v1 Announce Type: new Abstract: We introduce SWE Atlas, a benchmark suite for coding agents spanning three professional software engineering workflows: Codebase Q&A (124 tasks), Test W

arXiv:2605.07415v1 Announce Type: cross Abstract: Referring expression grounding is a core problem in visual grounding and is widely used as a diagnostic of spatial grounding and reasoning in vision a

arXiv:2605.07133v1 Announce Type: cross Abstract: Graph Anomaly Detection (GAD) is a critical task in graph machine learning with vital applications in financial fraud detection and social platform go

arXiv:2605.07338v1 Announce Type: new Abstract: The decline of global shellfish biodiversity poses a severe threat to coastal ecosystems. Although artificial intelligence (AI) technologies show potent

HiDream-O1-Image debuted at #8 in the Artificial Analysis Text to Image Arena , and the Reddit post likely discusses whether this benchmark inclusion and ranking adequately reflect HiDream-01's perfor

As much as the state of benchmarks in AI is flawed, it is so much easier to track AI progress than robotics. Not sure what you can make of all the videos of robots running races or doing laundry - are

arXiv:2605.05079v1 Announce Type: new Abstract: Video sequence capturing through refractive dynamic media, such as a turbulent air or water surface, often suffer from severe geometric distortions and

arXiv:2605.04503v1 Announce Type: new Abstract: Image Difference Captioning (IDC) generates natural language descriptions that precisely identify differences between two images, serving as a key bench

arXiv:2605.03590v1 Announce Type: new Abstract: Recent large language models (LLMs) show strong speech recognition and translation capabilities for high-resource languages. However, African languages

arXiv:2605.03903v1 Announce Type: new Abstract: Large Multimodal Models (LMMs) have recently shown strong performance on Optical Character Recognition (OCR) tasks, demonstrating their promising capabi