Search: “benchmark”

arXiv:2604.25392v1 Announce Type: new Abstract: This paper benchmarks a classical machine learning approach based on PyCaret AutoML against a deep learning approach based on IndoBERT fine-tuning for b

researcharxiv-cs-cl

29 Apr 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

DGHMesh: A Large-scale Dual-radar mmWave Dataset and Generalization-focused Benchmark for Human Mesh Reconstruction

DGX agent

arXiv:2604.22827v1 Announce Type: new Abstract: Millimeter-wave (mmWave) radar has shown great potential for contactless, privacy-preserving, and robust human sensing, yet existing mmWave-based human

model-releasesarxiv-cs-cv

28 Apr 2026

PL-MTEB: Polish Massive Text Embedding Benchmark

arXiv:2405.10138v2 Announce Type: replace Abstract: In this paper, we introduce the Polish Massive Text Embedding Benchmark (PL-MTEB), a comprehensive benchmark for text embeddings in the Polish langu

27 Apr 2026

HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks

arXiv:2604.14709v2 Announce Type: replace Abstract: Existing benchmarks for hardware design primarily evaluate Large Language Models (LLMs) on isolated, component-level tasks such as generating HDL mo

24 Apr 2026

Spectral Embeddings Leak Graph Topology: Theory, Benchmark, and Adaptive Reconstruction

arXiv:2604.21094v1 Announce Type: new Abstract: Graph Neural Networks (GNNs) excel on relational data, but standard benchmarks unrealistically assume the graph is centrally available. In practice, set

24 Apr 2026

GPT-5.5 is likely the best model in the world. But open models like Kimi and Minimax get almost identical coding benchmark scores at 10-25x …

model-releasesclem-delangue--x

GPT-5.5 is likely the best model in the world. But open models like Kimi and Minimax get almost identical coding benchmark scores at 10-25x lower cost. I broke down the benchmarks and pricing. Here's

23 Apr 2026

ParseBench is now live on @Kaggle. The first document OCR benchmark built for AI agents — 2,000 enterprise pages, 167K+ test rules, 5 dimens…

model-releasesjerry-liu--x

ParseBench is now live on @Kaggle. The first document OCR benchmark built for AI agents — 2,000 enterprise pages, 167K+ test rules, 5 dimensions that actually break downstream agents. Benchmark your p

23 Apr 2026

GeoLaux: A Benchmark for Evaluating MLLMs' Geometry Performance on Long-Step Problems Requiring Auxiliary Lines

arXiv:2508.06226v2 Announce Type: replace Abstract: Geometry problem solving (GPS) poses significant challenges for Multimodal Large Language Models (MLLMs) in diagram comprehension, knowledge applica

22 Apr 2026

Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safety

arXiv:2604.18487v1 Announce Type: new Abstract: The Adversarial Humanities Benchmark (AHB) evaluates whether model safety refusals survive a shift away from familiar harmful prompt forms. Starting fro

Local Ai

Benchmarking programs?

The Reddit post 'Benchmarking programs?' in r/ollama likely discusses tools and methods for measuring the performance of local language models running on Ollama. Available benchmarking tools for Ollam

local-air-ollama

EchoChain: A Full-Duplex Benchmark for State-Update Reasoning Under Interruptions

arXiv:2604.16456v1 Announce Type: new Abstract: Real-time voice assistants must revise task state when users interrupt mid-response, but existing spoken-dialog benchmarks largely evaluate turn-based i

HORIZON: A Benchmark for In-the-wild User Behaviour Modeling

arXiv:2604.17259v1 Announce Type: cross Abstract: User behavior in the real world is diverse, cross-domain, and spans long time horizons. Existing user modeling benchmarks however remain narrow, focus

OmniHuman: A Large-scale Dataset and Benchmark for Human-Centric Video Generation

arXiv:2604.18326v1 Announce Type: new Abstract: Recent advancements in audio-video joint generation models have demonstrated impressive capabilities in content creation. However, generating high-fidel

PBSBench: A Multi-Level Vision-Language Framework and Benchmark for Hematopathology Whole Slide Image Interpretation

arXiv:2604.17570v1 Announce Type: new Abstract: Peripheral Blood Smear (PBS) is a critical microscopic examination in hematopathology that yields whole-slide imaging (WSI). Unlike solid tissue patholo

When Text Hijacks Vision: Benchmarking and Mitigating Text Overlay-Induced Hallucination in Vision Language Models

arXiv:2604.17375v1 Announce Type: new Abstract: Recent advances in Vision-Language Models (VLMs) have substantially enhanced their ability across multimodal video understanding benchmarks spanning tem

BAGEL: Benchmarking Animal Knowledge Expertise in Language Models

arXiv:2604.16241v1 Announce Type: cross Abstract: Large language models have shown strong performance on broad-domain knowledge and reasoning benchmarks, but it remains unclear how well language model

20 Apr 2026

PortraitCraft: A Benchmark for Portrait Composition Understanding and Generation

arXiv:2604.03611v2 Announce Type: replace Abstract: Portrait composition plays a central role in portrait aesthetics and visual communication, yet existing datasets and benchmarks mainly focus on coar

17 Apr 2026

Agent evals are drifting away from production reality. Most benchmarks use clean tasks, well-specified requirements, deterministic metrics, …

Agent evals are drifting away from production reality. Most benchmarks use clean tasks, well-specified requirements, deterministic metrics, and retrospective curation. Production work is messier, with

model-releasesdair-ai--x

16 Apr 2026

LiveClawBench: Benchmarking LLM Agents on Complex, Real-World Assistant Tasks

arXiv:2604.13072v1 Announce Type: new Abstract: LLM-based agents are increasingly expected to handle real-world assistant tasks, yet existing benchmarks typically evaluate them under isolated sources

16 Apr 2026

Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models

arXiv:2503.21380v3 Announce Type: replace Abstract: The rapid advancement of large reasoning models has saturated existing math benchmarks, underscoring the urgent need for more challenging evaluation

14 Apr 2026

RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies

arXiv:2604.09860v1 Announce Type: cross Abstract: The pursuit of general-purpose robotics has yielded impressive foundation models, yet simulation-based benchmarking remains a bottleneck due to rapid

14 Apr 2026

WearBCI Dataset: Understanding and Benchmarking Real-World Wearable Brain-Computer Interfaces Signals

arXiv:2604.09649v1 Announce Type: cross Abstract: Brain-computer interfaces (BCIs) have opened new platforms for human-computer interaction, medical diagnostics, and neurorehabilitation. Wearable BCI

14 Apr 2026

Many Ways to Be Fake: Benchmarking Fake News Detection Under Strategy-Driven AI Generation

arXiv:2604.09514v1 Announce Type: new Abstract: Recent advances in large language models (LLMs) have enabled the large-scale generation of highly fluent and deceptive news-like content. While prior wo

13 Apr 2026

See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models

arXiv:2512.02231v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) are expected to jointly interpret vision, audio, and language, yet existing video benchmarks rarely a

13 Apr 2026

SessionIntentBench: A Multi-task Inter-session Intention-shift Modeling Benchmark for E-commerce Customer Behavior Understanding

arXiv:2507.20185v2 Announce Type: replace Abstract: Session history is a common way of recording user interacting behaviors throughout a browsing activity with multiple products. For example, if an us

13 Apr 2026

arXiv2Table: Toward Realistic Benchmarking and Evaluation for LLM-Based Literature-Review Table Generation

arXiv:2504.10284v5 Announce Type: replace Abstract: Literature review tables are essential for summarizing and comparing collections of scientific papers. In this paper, we study the automatic generat

Benchmarking LLM Tool-Use in the Wild

arXiv:2604.06185v1 Announce Type: cross Abstract: Fulfilling user needs through Large Language Model multi-turn, multi-step tool-use is rarely a straightforward process. Real user interactions are inh

EVGeoQA: Benchmarking LLMs on Dynamic, Multi-Objective Geo-Spatial Exploration

arXiv:2604.07070v1 Announce Type: new Abstract: While Large Language Models (LLMs) demonstrate remarkable reasoning capabilities, their potential for purpose-driven exploration in dynamic geo-spatial

MedConclusion: A Benchmark for Biomedical Conclusion Generation from Structured Abstracts

arXiv:2604.06505v1 Announce Type: cross Abstract: Large language models (LLMs) are widely explored for reasoning-intensive research tasks, yet resources for testing whether they can infer scientific c

multimodalbenchmarkreasoning

Research

Evidence-Grounded Trustworthy Multimodal Reasoning and Evaluation Benchmark in Complex Urban Scenes

Human

AD2-Bench diagnoses multimodal reasoning via a Chain of Evidence, showing failures from spatial ambiguity and semantic uncertainty in adverse urban scenes.

11 Jun 2025

AI Consilium: a Benchmark of LLM Clinical Multidisciplinary Second Opinions

arXiv:2610.22298v1 Announce Type: cross Abstract: Large language models (LLMs) can generate differential diagnoses for complex cases, but a useful second opinion must also be factually grounded, clini

Auditing Escalation Reliability in Healthcare LLM Agents: A Perturbation-Controlled Benchmark of Workflow-State Failures

arXiv:2610.21270v1 Announce Type: new Abstract: Trustworthy biomedical AI requires reliable action selection when evidence is missing, stale, conflicting, or late, not only accurate answers on static

Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration

arXiv:2512.20159v2 Announce Type: replace-cross Abstract: Code from LLM-assisted development may contain defects like API hallucinations, motivating the development of code evaluation metrics to asses

Can LLMs Tell Stories Out of Order? A Corpus and Benchmark for Nonlinear Narrative Generation

arXiv:2610.21468v1 Announce Type: cross Abstract: Every story has two orders: the order in which its events happen and the order in which they are told. A story is nonlinear when the two differ. We in

From Perception to Prediction and Intervention: Benchmarking Proactive Video Assistants in Real-Life Scenarios

arXiv:2610.21568v1 Announce Type: new Abstract: Recent multimodal large language models have advanced video understanding, but existing benchmarks mainly evaluate whether models can interpret observed

INFUSE: A Unified Surface Benchmark for Indirect Prompt Injection in Agentic Systems

arXiv:2610.20776v1 Announce Type: cross Abstract: LLM-based agents increasingly execute long-horizon, end-to-end workflows involving shell commands, code modification, dependency installation, and int

KBS: A Korean Benchmark Suite for Foundation Model Evaluation

arXiv:2610.21227v1 Announce Type: new Abstract: We present a Korean Benchmark Suite (KBS) comprising Ko-ARC, Ko-GSM8K, Ko-EQ-Bench, Ko-WinoGrande, Ko-LAMBADA, and Ko-IFEval, covering six capabilities

KULTURE Bench: A Benchmark for Assessing Language Model in Korean Cultural Context

arXiv:2412.07251v1 Announce Type: cross Abstract: Large language models have exhibited significant enhancements in performance across various tasks. However, the complexity of their evaluation increas

Beyond Benchmark Accuracy: A Diagnostic Test for LLM Chemical Prediction

arXiv:2610.19906v1 Announce Type: new Abstract: LLMs have achieved strong performance on chemical prediction benchmarks. However, existing evaluation methods typically provide target entity identity i

DecisionRobust: Benchmarking Robustness for JEV Style Decision Models Across Text and Vision

arXiv:2610.19118v1 Announce Type: new Abstract: Emerging decision models such as JEV provide typed interfaces that map task instructions and observations directly to choices, judgments, and scores. As

J-tau: A Fully Localized Japanese Benchmark for Tool-Agent-User Interaction

arXiv:2610.19638v1 Announce Type: new Abstract: We present J-tau, the first publicly available Japanese benchmark for evaluating tool-using LLM agents in interactive customer-support scenarios. While

Last Secure Code Benchmark: Is the Code You Vibe Secure?

arXiv:2610.14766v1 Announce Type: cross Abstract: AI coding agents now write much of the code that ships, yet whether that code is secure remains unclear. We introduce the Last Secure Code Benchmark,

Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving

arXiv:2503.21821v2 Announce Type: replace Abstract: We introduce Physics, a benchmark for evaluating foundation models on advanced physics problem solving. It contains 1034 annotated, open-ended probl

Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving

arXiv:2610.16133v1 Announce Type: cross Abstract: Puzzles offer a natural testbed for Large Language Models (LLMs) reasoning because they pose verifiable goals under partial information and require mu

ThreadBench: Benchmarking Realism in Agentic Simulation of Real-world Discussions

arXiv:2606.14715v2 Announce Type: replace-cross Abstract: LLM agents are increasingly used to simulate real-world interactions, but it remains unclear whether simulated behaviors preserve the content

Machine Learning-Based Prediction of Dengue Infection Using Clinical and Hematological Data from Bangladesh: A Comparative Analysis of Benchmark and Extended Prediction Models

arXiv:2610.14813v1 Announce Type: new Abstract: Dengue is a viral infection caused by the dengue virus (DENV) and a major public health concern in Bangladesh, where symptom overlap with other febrile

14 Oct 2026

MARL-BattleGrounds: A Novel JAX-native Benchmark for Heterogeneous and Competitive Multi-agent RL

arXiv:2610.14883v1 Announce Type: cross Abstract: We present MARL-BattleGrounds (MARL-BGs), an easy-to-use JAX-native benchmark for heterogeneous and competitive multi-agent reinforcement learning (MA

14 Oct 2026

MMTA: Benchmarking Multimodal Temporal Analysis with Time Series, Text, and Vision

arXiv:2610.17856v1 Announce Type: new Abstract: Time-series analysis is increasingly studied in multimodal settings, yet most benchmarks pair temporal signals with text or with visualizations derived