Search: “benchmark”

BAGEL: Benchmarking Animal Knowledge Expertise in Language Models

arXiv:2604.16241v1 Announce Type: cross Abstract: Large language models have shown strong performance on broad-domain knowledge and reasoning benchmarks, but it remains unclear how well language model

20 Apr 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

PortraitCraft: A Benchmark for Portrait Composition Understanding and Generation

DGX agent

arXiv:2604.03611v2 Announce Type: replace Abstract: Portrait composition plays a central role in portrait aesthetics and visual communication, yet existing datasets and benchmarks mainly focus on coar

model-releasesarxiv-cs-cv

17 Apr 2026

LiveClawBench: Benchmarking LLM Agents on Complex, Real-World Assistant Tasks

arXiv:2604.13072v1 Announce Type: new Abstract: LLM-based agents are increasingly expected to handle real-world assistant tasks, yet existing benchmarks typically evaluate them under isolated sources

16 Apr 2026

Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models

arXiv:2503.21380v3 Announce Type: replace Abstract: The rapid advancement of large reasoning models has saturated existing math benchmarks, underscoring the urgent need for more challenging evaluation

14 Apr 2026

RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies

arXiv:2604.09860v1 Announce Type: cross Abstract: The pursuit of general-purpose robotics has yielded impressive foundation models, yet simulation-based benchmarking remains a bottleneck due to rapid

14 Apr 2026

WearBCI Dataset: Understanding and Benchmarking Real-World Wearable Brain-Computer Interfaces Signals

arXiv:2604.09649v1 Announce Type: cross Abstract: Brain-computer interfaces (BCIs) have opened new platforms for human-computer interaction, medical diagnostics, and neurorehabilitation. Wearable BCI

14 Apr 2026

Many Ways to Be Fake: Benchmarking Fake News Detection Under Strategy-Driven AI Generation

arXiv:2604.09514v1 Announce Type: new Abstract: Recent advances in large language models (LLMs) have enabled the large-scale generation of highly fluent and deceptive news-like content. While prior wo

13 Apr 2026

See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models

arXiv:2512.02231v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) are expected to jointly interpret vision, audio, and language, yet existing video benchmarks rarely a

13 Apr 2026

SessionIntentBench: A Multi-task Inter-session Intention-shift Modeling Benchmark for E-commerce Customer Behavior Understanding

arXiv:2507.20185v2 Announce Type: replace Abstract: Session history is a common way of recording user interacting behaviors throughout a browsing activity with multiple products. For example, if an us

13 Apr 2026

arXiv2Table: Toward Realistic Benchmarking and Evaluation for LLM-Based Literature-Review Table Generation

arXiv:2504.10284v5 Announce Type: replace Abstract: Literature review tables are essential for summarizing and comparing collections of scientific papers. In this paper, we study the automatic generat

Benchmarking LLM Tool-Use in the Wild

arXiv:2604.06185v1 Announce Type: cross Abstract: Fulfilling user needs through Large Language Model multi-turn, multi-step tool-use is rarely a straightforward process. Real user interactions are inh

EVGeoQA: Benchmarking LLMs on Dynamic, Multi-Objective Geo-Spatial Exploration

arXiv:2604.07070v1 Announce Type: new Abstract: While Large Language Models (LLMs) demonstrate remarkable reasoning capabilities, their potential for purpose-driven exploration in dynamic geo-spatial

MedConclusion: A Benchmark for Biomedical Conclusion Generation from Structured Abstracts

arXiv:2604.06505v1 Announce Type: cross Abstract: Large language models (LLMs) are widely explored for reasoning-intensive research tasks, yet resources for testing whether they can infer scientific c

multimodalbenchmarkreasoning

Research

Evidence-Grounded Trustworthy Multimodal Reasoning and Evaluation Benchmark in Complex Urban Scenes

Human

AD2-Bench diagnoses multimodal reasoning via a Chain of Evidence, showing failures from spatial ambiguity and semantic uncertainty in adverse urban scenes.

11 Jun 2025

AI Consilium: a Benchmark of LLM Clinical Multidisciplinary Second Opinions

arXiv:2610.22298v1 Announce Type: cross Abstract: Large language models (LLMs) can generate differential diagnoses for complex cases, but a useful second opinion must also be factually grounded, clini

Auditing Escalation Reliability in Healthcare LLM Agents: A Perturbation-Controlled Benchmark of Workflow-State Failures

arXiv:2610.21270v1 Announce Type: new Abstract: Trustworthy biomedical AI requires reliable action selection when evidence is missing, stale, conflicting, or late, not only accurate answers on static

Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration

arXiv:2512.20159v2 Announce Type: replace-cross Abstract: Code from LLM-assisted development may contain defects like API hallucinations, motivating the development of code evaluation metrics to asses

Can LLMs Tell Stories Out of Order? A Corpus and Benchmark for Nonlinear Narrative Generation

arXiv:2610.21468v1 Announce Type: cross Abstract: Every story has two orders: the order in which its events happen and the order in which they are told. A story is nonlinear when the two differ. We in

From Perception to Prediction and Intervention: Benchmarking Proactive Video Assistants in Real-Life Scenarios

arXiv:2610.21568v1 Announce Type: new Abstract: Recent multimodal large language models have advanced video understanding, but existing benchmarks mainly evaluate whether models can interpret observed

INFUSE: A Unified Surface Benchmark for Indirect Prompt Injection in Agentic Systems

arXiv:2610.20776v1 Announce Type: cross Abstract: LLM-based agents increasingly execute long-horizon, end-to-end workflows involving shell commands, code modification, dependency installation, and int

KBS: A Korean Benchmark Suite for Foundation Model Evaluation

arXiv:2610.21227v1 Announce Type: new Abstract: We present a Korean Benchmark Suite (KBS) comprising Ko-ARC, Ko-GSM8K, Ko-EQ-Bench, Ko-WinoGrande, Ko-LAMBADA, and Ko-IFEval, covering six capabilities

KULTURE Bench: A Benchmark for Assessing Language Model in Korean Cultural Context

arXiv:2412.07251v1 Announce Type: cross Abstract: Large language models have exhibited significant enhancements in performance across various tasks. However, the complexity of their evaluation increas

Beyond Benchmark Accuracy: A Diagnostic Test for LLM Chemical Prediction

arXiv:2610.19906v1 Announce Type: new Abstract: LLMs have achieved strong performance on chemical prediction benchmarks. However, existing evaluation methods typically provide target entity identity i

DecisionRobust: Benchmarking Robustness for JEV Style Decision Models Across Text and Vision

arXiv:2610.19118v1 Announce Type: new Abstract: Emerging decision models such as JEV provide typed interfaces that map task instructions and observations directly to choices, judgments, and scores. As

J-tau: A Fully Localized Japanese Benchmark for Tool-Agent-User Interaction

arXiv:2610.19638v1 Announce Type: new Abstract: We present J-tau, the first publicly available Japanese benchmark for evaluating tool-using LLM agents in interactive customer-support scenarios. While

Last Secure Code Benchmark: Is the Code You Vibe Secure?

arXiv:2610.14766v1 Announce Type: cross Abstract: AI coding agents now write much of the code that ships, yet whether that code is secure remains unclear. We introduce the Last Secure Code Benchmark,

Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving

arXiv:2503.21821v2 Announce Type: replace Abstract: We introduce Physics, a benchmark for evaluating foundation models on advanced physics problem solving. It contains 1034 annotated, open-ended probl

Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving

arXiv:2610.16133v1 Announce Type: cross Abstract: Puzzles offer a natural testbed for Large Language Models (LLMs) reasoning because they pose verifiable goals under partial information and require mu

ThreadBench: Benchmarking Realism in Agentic Simulation of Real-world Discussions

arXiv:2606.14715v2 Announce Type: replace-cross Abstract: LLM agents are increasingly used to simulate real-world interactions, but it remains unclear whether simulated behaviors preserve the content

Machine Learning-Based Prediction of Dengue Infection Using Clinical and Hematological Data from Bangladesh: A Comparative Analysis of Benchmark and Extended Prediction Models

arXiv:2610.14813v1 Announce Type: new Abstract: Dengue is a viral infection caused by the dengue virus (DENV) and a major public health concern in Bangladesh, where symptom overlap with other febrile

MARL-BattleGrounds: A Novel JAX-native Benchmark for Heterogeneous and Competitive Multi-agent RL

arXiv:2610.14883v1 Announce Type: cross Abstract: We present MARL-BattleGrounds (MARL-BGs), an easy-to-use JAX-native benchmark for heterogeneous and competitive multi-agent reinforcement learning (MA

MMTA: Benchmarking Multimodal Temporal Analysis with Time Series, Text, and Vision

arXiv:2610.17856v1 Announce Type: new Abstract: Time-series analysis is increasingly studied in multimodal settings, yet most benchmarks pair temporal signals with text or with visualizations derived

WireCraft: A Simulation Benchmark for Industrial DLO Manipulation

model-releasesarxiv-cs-ro

arXiv:2606.18097v2 Announce Type: replace Abstract: Deformable Linear Objects (DLOs), such as wires and cables, are central to industrial assembly. Unlike rigid objects, whose state is captured by a 6

WorldOracle: Benchmarking Action-Conditioned World Models against Real-World Executions

model-releasesarxiv-cs-cv

arXiv:2610.18826v1 Announce Type: cross Abstract: We introduce WorldOracle, a benchmark specifically designed to evaluate action-conditioned world models (ACWMs) against paired real-world executions.

DRYLAB-Bench: Benchmarking the Biosecurity Risk of Agentic LLM Mutation Design

arXiv:2610.13116v1 Announce Type: new Abstract: As large language models (LLMs) acquire advanced biological design capabilities, they offer substantial research benefits but simultaneously introduce s

GeopoliticalQA: Benchmarking Retrieval and Generation over Intelligence Reports

arXiv:2610.12617v1 Announce Type: new Abstract: Geopolitical and strategic intelligence reports constitute a rich source of expert knowledge and have increasing value in the current political climate,

Mind the Spatial Gap: A Benchmark for Geospatial Reasoning and Constrained Spatial Editing

arXiv:2610.12881v1 Announce Type: new Abstract: Understanding and manipulating spatial relations is fundamental to AI systems that reason about physical environments and construct grounded world model

When Charts Deceive Web Agents: A Paired Benchmark for Misleading Visualization-Grounded Tasks

arXiv:2610.12823v1 Announce Type: new Abstract: Web agents increasingly browse, interpret, and act on web content on behalf of users. In many realistic workflows, however, the visual evidence they con

Research

An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems

arXiv:2508.08833v4 Announce Type: replace-cross Abstract: Frontier large language models (LLMs) now reach near-ceiling accuracy on standard mathematical-reasoning benchmarks and gold-medal-level perfo

researcharxiv-cs-ai

9 Oct 2026

Benchmark Designers Should 'Train on the Test Set' to Expose Exploitable Non-Visual Shortcuts

model-releasesarxiv-cs-cv

arXiv:2511.04655v2 Announce Type: replace Abstract: Multimodal LLMs can answer many questions in vision-centric benchmarks without looking at the image, using linguistic priors and skewed answer distr

9 Oct 2026

Alice: A Large-Scale German Benchmark for Rubric-Based Multi-Dimensional Automatic Short Answer Scoring

arXiv:2610.09661v1 Announce Type: new Abstract: Automatic Short Answer Scoring (ASAS) is central to NLP for Education. However, openly available benchmarks remain scarce, and existing datasets largely

8 Oct 2026

BanglaRhet: Benchmarking Classical and Transformer Models for Rhetorical and Persuasion Detection in Bangla Political Speech

arXiv:2610.09464v1 Announce Type: new Abstract: Political discourse often uses rhetorical and persuasive language to frame narratives, influence public opinion, and mobilize audiences. While Bangla na

8 Oct 2026

Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing

arXiv:2610.09778v1 Announce Type: cross Abstract: Reproducible benchmarking of Large Language Model (LLM) inference is challenging because repeated measurements can vary with execution and system stat

8 Oct 2026

Benchmarking Label-Revealed Online Updates for EEG BCI Decoding

arXiv:2610.07420v1 Announce Type: new Abstract: Electroencephalography (EEG) signals drift over time, which can cause static brain-computer interface (BCI) models to degrade in practice. We present a

BiGym 2.0: Benchmarking Learned and Agent-Developed Policies for Humanoid Household Manipulation

arXiv:2610.07594v1 Announce Type: cross Abstract: Humanoid household manipulation requires the arms to act while the body balances, steps and changes posture. We present BiGym 2.0, an adaptation of Bi

How Well Do LLMs Reason with Noisy Evidence? An Active Visual Reasoning Benchmark

arXiv:2610.07751v1 Announce Type: new Abstract: Real-world reasoning rarely reduces to static question answering: agents must actively gather information from tools and sensors that are often noisy an

OTel: Open Telco AI Datasets, Benchmarks, and Models

arXiv:2610.07766v1 Announce Type: new Abstract: We present Open Telco (OTel), an open telecom AI resource that releases derived telecom datasets for retrieval, reranking, instruction tuning, and safet

ANT: A Multi-Granularity Network Traffic Dataset and Benchmark for Agents Behavior Auditing

arXiv:2610.06514v1 Announce Type: cross Abstract: The growing adoption of large language model (LLM) agents creates a need for network administrators and security teams to audit agent behavior within