Search: “benchmark”

arXiv:2604.19298v1 Announce Type: cross Abstract: We introduce IndiaFinBench, to our knowledge the first publicly available evaluation benchmark for assessing large language model (LLM) performance on

Let's talk parsing charts 📊📈. Last week we released ParseBench, the first document OCR benchmark for AI agents. New in ParseBench: ChartDataPointMatch. Most document look at a chart and OCR the captio

arXiv:2604.13583v1 Announce Type: new Abstract: Evaluating large language models (LLMs) for legal reasoning requires workflows that span task design, expert annotation, model execution, and metric-bas

arXiv:2604.08538v1 Announce Type: new Abstract: AI agents are changing the requirements for document parsing. What matters is semantic correctness: parsed output must preserve the structure and

arXiv:2604.07054v2 Announce Type: replace Abstract: Sales dialogues require multi-turn, goal-directed persuasion under asymmetric incentives, which makes them a challenging setting for large language

arXiv:2603.18048v3 Announce Type: replace Abstract: Recent Audio Multimodal Large Language Models (Audio MLLMs) demonstrate impressive performance on speech benchmarks, yet it remains unclear whether

arXiv:2610.12954v1 Announce Type: new Abstract: This paper introduces MathVis, a benchmark designed to evaluate LLMs' ability to visualize frontier mathematical problems and perform diagram-assisted r

arXiv:2610.13644v1 Announce Type: cross Abstract: Selecting a subset of features (columns) from a tabular dataset is a ubiquitous challenge in real-world applications. Industry experts seek to minimiz

arXiv:2610.12975v1 Announce Type: new Abstract: We present SmartSalesBench, the first multi-turn conversational benchmark explicitly designed to evaluate large language model (LLM) sales agents agains

Here's a benchmark that uses Lilypond https://x.com/aug5thmusic/status/2096030719156089029 GPT-6 Astra has the best result yet on the Bach Benchmark. Its chorale contains no voice-leading errors, and

arXiv:2610.06413v1 Announce Type: new Abstract: Thinking-enabled vision-language models (VLMs) report ever-higher accuracy on spatial benchmarks, yet final-answer scores cannot reveal whether a correc

arXiv:2610.03276v1 Announce Type: new Abstract: Video saliency prediction is inherently harder to model than static image saliency due to the additional temporal dimension. Video saliency benchmarks r

Hey local AI community, I've been working on this for a while and finally feel ok sharing it. It's a cyber benchmark where the model gets a shell in an isolated docker box and has to find the exact fl

We benchmarked prompt caching across DeepSeek, GLM, GPT, and Claude using Harbor evals and Phoenix traces, so we could compare cache reuse, estimated cost, and latency on the same multi-turn shopping

arXiv:2609.39027v1 Announce Type: new Abstract: AI reviewers can assign different judgments to manuscripts that report the same science in different wording, potentially rewarding rhetorical optimizat

arXiv:2609.38604v1 Announce Type: cross Abstract: Modern LLM agents increasingly tackle complex tasks through interactive, long-horizon exchanges with users, while existing benchmarks generally assume

arXiv:2609.40284v1 Announce Type: cross Abstract: Computer use agents (CUAs), which use graphical user interfaces (GUIs) to complete tasks on a computer, have recently surpassed human performance on m

What is a way to test human-level generality in artificial intelligence? 'The meta-benchmark of being able to pass ARC-AGI-(n+1) immediately upon release' @Jabaluck @DAcemogluMIT The meta-benchmark of

arXiv:2607.18063v2 Announce Type: replace-cross Abstract: LLM-based agents process external content, exposing them to prompt injection and multi-turn manipulation. We present a 21-scenario benchmark f

arXiv:2609.15478v1 Announce Type: new Abstract: Multimodal agents can create complex videos in software such as Blender by coding without relying on diffusion models. Yet video understanding benchmark

arXiv:2603.02790v2 Announce Type: replace Abstract: Foundation models are changing the way we develop medical artificial intelligence. By learning broadly generalizable features across diverse data mo

https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-3 AA shipped a new benchmark last week as part of the Intelligence Index v4.3 update — a brand-new private eval that re

arXiv:2609.09672v1 Announce Type: new Abstract: The rapid advancement of audio and multimodal large language models has unlocked transformative speech understanding capabilities, yet evaluation framew

arXiv:2609.08155v1 Announce Type: new Abstract: Scientific figures are the interface through which research claims are inspected and reused, but final published panels rarely expose the data or plotti

Jensen Huang says that AGI has been achieved. I strongly disagree. GPT-6 performs really well on benchmarks, including ARC-AGI-3. But benchmark performance is not the same as general intelligence. GPT

arXiv:2609.03394v1 Announce Type: new Abstract: Emotion recognition benchmarks often predict one emotion per text, missing many real-world scenarios where two people arrive at opposing emotions from a

arXiv:2609.04173v1 Announce Type: new Abstract: For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases.

arXiv:2609.02277v1 Announce Type: cross Abstract: Perceptual illusions have long served as crucial probes into human cognition, revealing biases and limitations of perception. In the auditory domain,

arXiv:2609.02379v1 Announce Type: cross Abstract: While existing work on LLM authorship attribution (AA) has made progress, available benchmarks remain limited, often focusing on English, controlled s

arXiv:2609.02067v1 Announce Type: new Abstract: Scientific benchmarks are commonly built by domain experts who write tasks and cross-check one another's work, or who adapt existing material from textb

arXiv:2609.01344v1 Announce Type: new Abstract: Multimodal coding and editing systems must map a visible or semantic referent to the exact executable object that can be edited. A wrong reference may s

arXiv:2608.30451v1 Announce Type: new Abstract: Image-based 3D visual grounding is critical for embodied agents, yet existing benchmarks suffer from loose text-observation alignment and neglect tempor

arXiv:2608.30508v1 Announce Type: new Abstract: Long-term memory is increasingly important for conversational agents, yet existing benchmarks primarily measure memory through pointwise factual recall:

arXiv:2608.27169v1 Announce Type: new Abstract: Ancient Chinese artifact text recognition is fundamental to heritage digitization, and benchmarks for ancient texts are essential for evaluating current

Benchmarking Retrieval-Augmented Generation End to End — from Building the Vector Database to Serving an Iterative, Multi-Hop Question-Answering Inference Pipeline The post Introducing the MLPerf End-

arXiv:2510.13394v4 Announce Type: replace Abstract: Spatial reasoning ability is crucial for Vision Language Models (VLMs) to support real-world applications in diverse domains including robotics, aug

arXiv:2608.20153v1 Announce Type: new Abstract: Large language models (LLMs) have shown growing potential for automated theoretical computer science (TCS) research, yet existing benchmarks remain far

arXiv:2608.18618v1 Announce Type: new Abstract: Autonomous laboratories hold great promise for accelerating scientific discovery. To achieve this vision, robots are supposed to dexterously manipulate

qwen3.8 27B has seriously impressive benchmarks on its model card, but that's for the unquantised version. Almost everyone here will run one quant or another. Are there good benchmarks for how those q

arXiv:2608.12343v1 Announce Type: new Abstract: AI chatbots are widely used by students as knowledge sources, yet LLM benchmarks rarely assess interpretative historical reasoning. We evaluate eight le

arXiv:2608.10268v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly mediate legal determinations over what human rights are realized, and how. Yet, no evaluation benchmark exis

ExtractBench is one of the most comprehensive benchmarks for real-world document extraction. ✅ It covers 4869 pages, across 67 document types, spanning 8 real-world domains: finance, energy, gov, auto

arXiv:2608.09802v1 Announce Type: new Abstract: As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluati

arXiv:2608.06549v1 Announce Type: cross Abstract: LLMs are increasingly being applied to tasks involving institutional and political texts, but existing benchmarks evaluate them on isolated documents

arXiv:2608.07341v1 Announce Type: cross Abstract: Test data from public benchmarks inevitably leaks into pretraining corpora, inflating evaluation scores once memorized. extbf{Contamination mitigation

arXiv:2608.06265v1 Announce Type: new Abstract: Synthetic clinical benchmarks for enterprise AI agents can pass existing utility checks and still remain structurally unrealistic, especially in privacy