Search: “benchmark”

HM-Bench: A Comprehensive Benchmark for Multimodal Large Language Models in Hyperspectral Remote Sensing

arXiv:2604.08884v1 Announce Type: cross Abstract: While multimodal large language models (MLLMs) have made significant strides in natural image understanding, their ability to perceive and reason over

13 Apr 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

PinpointQA: A Dataset and Benchmark for Small Object-Centric Spatial Understanding in Indoor Videos

DGX agent

arXiv:2604.08991v1 Announce Type: cross Abstract: Small object-centric spatial understanding in indoor videos remains a significant challenge for multimodal large language models (MLLMs), despite its

model-releasesarxiv-cs-ai

13 Apr 2026

Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures

arXiv:2604.09048v1 Announce Type: cross Abstract: While the large energy consumption of Large Language Models (LLMs) is recognized by the community, system operators lack guidance for energy-efficient

13 Apr 2026

Beyond Social Pressure: Benchmarking Epistemic Attack in Large Language Models

arXiv:2604.07749v1 Announce Type: new Abstract: Large language models (LLMs) can shift their answers under pressure in ways that reflect accommodation rather than reasoning. Prior work on sycophancy h

Fail2Drive: Benchmarking Closed-Loop Driving Generalization

arXiv:2604.08535v1 Announce Type: cross Abstract: Generalization under distribution shift remains a central bottleneck for closed-loop autonomous driving. Although simulators like CARLA enable safe an

PhyAVBench: A Challenging Audio Physics-Sensitivity Benchmark for Physically Grounded Text-to-Audio-Video Generation

arXiv:2512.23994v3 Announce Type: replace-cross Abstract: Text-to-audio-video (T2AV) generation is central to applications such as filmmaking and world modeling. However, current models often fail to

PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models

arXiv:2604.08340v1 Announce Type: new Abstract: While Vision-Language Models (VLMs) have achieved remarkable progress in static visual understanding, their deployment in complex 3D embodied environmen

SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glasses

arXiv:2602.22683v2 Announce Type: replace Abstract: The rapid advancement of AI-powered smart glasses-one of the hottest wearable devices-has unlocked new frontiers for multimodal interaction, with Vi

ToxReason: A Benchmark for Mechanistic Chemical Toxicity Reasoning via Adverse Outcome Pathway

arXiv:2604.06264v1 Announce Type: cross Abstract: Recent advances in large language models (LLMs) have enabled molecular reasoning for property prediction. However, toxicity arises from complex biolog

Digital Typhoon Dataset V3: Benchmarking for Multi-spectral Extension and Analyzing Systematic Bias in Intensity Regression

arXiv:2610.21654v1 Announce Type: new Abstract: This paper presents the Digital Typhoon Dataset V3, a new version of the longest typhoon satellite image dataset spanning nearly 50 years, designed to b

16 Oct 2026

JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

arXiv:2610.21412v1 Announce Type: new Abstract: Large language model (LLM) agents now fix software, operate computers and serve customers, but their harnesses, the programs that run them, call a large

16 Oct 2026

When Does 'Post-Call' Begin? An Availability Contract for Earnings-Call NLP Benchmarks

arXiv:2610.21659v1 Announce Type: cross Abstract: Corporate financial disclosures often unfold over time, with announcements, prepared remarks, and discussion spanning multiple high-frequency return i

16 Oct 2026

Local Ai

DSABench: Benchmarking Source Discovery, Coordination, and Execution for Multi-Database Agents

arXiv:2610.14700v1 Announce Type: new Abstract: Most text-to-SQL benchmarks provide the target database. In a large data catalog, however, choosing the sources is part of the problem: an agent must fi

local-aiarxiv-cs-ai

Frame Evidence:A Benchmark for Reference Frames and Visual Evidence in Multi-Image Spatial Reasoning

arXiv:2610.19564v1 Announce Type: new Abstract: A spatial answer depends on the reference frame specified in language and its supporting visual evidence. Item accuracy alone obscures whether correctne

Into the Unknown: Benchmarking AI agents on open scientific research problems

arXiv:2610.19159v1 Announce Type: cross Abstract: Can AI agents carry out research on open scientific problems? We introduce Into the Unknown, a benchmark that provides an agent with a science paper a

QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models

arXiv:2601.08689v3 Announce Type: replace Abstract: Large Language Models (LLMs) have shown strong capabilities in finance-oriented question answering, yet their evaluation on professional quantitativ

UA-Bench: Benchmarking Large Audio-Language Models on Underwater Acoustic Recognition

arXiv:2610.16942v1 Announce Type: cross Abstract: Large audio-language models (LALMs) are fast becoming a general interface to sound, yet how far their broad pretraining carries into specialized domai

UDHRBench: Benchmarking Human Rights in Large Language Models

arXiv:2610.15179v1 Announce Type: cross Abstract: Large language models are increasingly deployed in contexts requiring adherence to ethical and legal norms, yet their alignment with internationally r

CineRecon: Benchmarking Frontier Models for Agentic Cinematic Reconstruction

arXiv:2610.16577v1 Announce Type: new Abstract: Reconstructing reference videos as editable dynamic 3D scenes is an important path toward controllable video generation for filmmaking: these scenes pro

Research

CodeWalk: Generating Coding Benchmarks by Walking a Problem Graph

arXiv:2610.15345v1 Announce Type: new Abstract: High-quality coding problems are expensive to collect. Benchmarks such as SWE-bench rely on substantial human curation, while synthetic alternatives typ

researcharxiv-cs-lg

GIFT-Eval-CTX: A Benchmark For Contextual Forecasting Model Evaluation

arXiv:2610.17794v1 Announce Type: new Abstract: Contextual forecasting seeks to improve time series forecasting by incorporating forward-looking information, such as planned events, external condition

LIBERO-RIG: A Hardware-Setup Robustness Benchmark for Vision-Language-Action Models

arXiv:2610.18843v1 Announce Type: new Abstract: Robot policies are expected to generalize across deployment environments, including variations in the robot itself: cameras are bumped and remounted, ca

VRIQ: Benchmarking and Diagnosing the Visual-Reasoning IQ of Vision-Language Models

arXiv:2602.05382v2 Announce Type: replace Abstract: Vision-language model (VLM) errors on visual reasoning may come from missing visual structure, from mistakes after that structure is extracted, or f

GeoFitBench-LLM: A Benchmark for Assessing Large Language Models' Capabilities in Geospatial Data Requirements Elicitation for Geo-Analytic Tasks

arXiv:2610.12669v1 Announce Type: cross Abstract: Large language models are becoming central to GeoAI systems, yet their evaluation rarely considers the elicitation of geospatial data requirements. Th

12 Oct 2026

What Should Embodied Agents Remember? Benchmarking and Evaluating Memory Modality for Embodied Agents

arXiv:2610.12561v1 Announce Type: new Abstract: Memory allows large language model (LLM) agents to use past experience, often stored as textual records, when responding to requests. For embodied agent

12 Oct 2026

CityDeploy-Bench: Benchmarking Physics-Grounded Spatial Set Planning for Multi-Transmitter Network Deployment

arXiv:2610.11065v1 Announce Type: new Abstract: Automating city-scale wireless deployment remains challenging under complex urban propagation and network-wide interference. We introduce extbf{CityDepl

Elucidating the Space of Enzymatic Reaction: A Unified Benchmark and Pretrained Model

arXiv:2610.11694v1 Announce Type: cross Abstract: Existing reaction models primarily learn molecular transformations, whereas enzy- matic reactions depend jointly on molecular structure and catalytic

Humanity's Sixth Sense: Benchmarking Intuitive Visual Reasoning in Multimodal Models

arXiv:2610.08966v2 Announce Type: replace Abstract: Humans perceive far more in a scene than what is explicitly depicted: a single glance captures past causes and future trajectories; a quick peek det

ManiUnit: A Manipulation Skill Dataset and Benchmark for Long-Horizon Tasks

arXiv:2610.12089v1 Announce Type: new Abstract: Long-horizon mobile manipulation requires a robot to navigate multi-room environments and execute a sequence of manipulation skills under a single natur

MissMAC-Bench: Building Solid Benchmark for Missing Modality Issue in Robust Multimodal Affective Computing

arXiv:2602.00811v2 Announce Type: replace Abstract: Current Multimodal Affective Computing (MAC) systems heavily rely on the completeness of multiple modalities to accurately understand human's affect

Searching for 'Harmful Refusal': A Psychometric Audit of an AI Safety Benchmark

arXiv:2610.12409v1 Announce Type: new Abstract: Safety benchmarks typically report one overall score for a suite of datasets, each of which may target one or more safety-related attributes, so models

AdSpark: A Large-Scale Dataset and Benchmark for Product-Centric Advertisement Video Generation

arXiv:2610.10047v1 Announce Type: new Abstract: Product-centric advertisement video generation aims to create promotional videos that preserve fine-grained product identity while presenting selling po

8 Oct 2026

Research

HomID : Benchmarking Intrinsic Dimension Estimators on Homogenous Manifolds with Anisotropic Embeddings

arXiv:2510.01335v2 Announce Type: replace Abstract: The manifold hypothesis suggests that data lies on manifolds with smaller intrinsic dimension (ID) than their ambient dimension. However there is no

researcharxiv-cs-lg

8 Oct 2026

Lifelong small-object navigation in changing object layouts: a benchmark and method

arXiv:2610.10125v1 Announce Type: cross Abstract: Household robots need to continually navigate to different objects in the same environment, many of which are small and portable, such as tools and to

8 Oct 2026

An Information-Theoretic Evaluation Framework for Benchmark and Model Diagnosis in Knowledge Tracing

arXiv:2610.06988v1 Announce Type: new Abstract: Knowledge tracing (KT) models are predominantly evaluated using aggregate metrics such as area under the curve (AUC) and accuracy. However, these global

Benchmarking Time Series Foundation Models for Load Forecasting Under Covariate Uncertainty

arXiv:2610.07232v1 Announce Type: new Abstract: Accurate short-term load forecasting (STLF) is essential for the reliable and efficient operation of modern power systems. While time series foundation

Nine Trials to Recover: A Reproducible Benchmark for Repertoire-Free Soft-Robot Damage Adaptation

arXiv:2610.07616v1 Announce Type: new Abstract: We present a repertoire-free benchmark for soft-robot damage recovery under nine online trials. The protocol pairs damage masks within each morphology,

Offline AI Modules: Voice-First Offline Architecture, Hardware Reference Stack, Quantization and Benchmarking

arXiv:2610.07026v1 Announce Type: new Abstract: The Offline AI Modules workstream enables practical, low-power, and community-accessible deployment of voice-first AI systems that operate fully offline

RenderBench: Benchmarking Render-to-Real Video Transfer with Reconstructed Digital Twins

arXiv:2610.08684v1 Announce Type: new Abstract: Modern video models can generate realistic videos from real appearance references and proxy renders that specify scene structure, viewpoint changes, and

Symphony for Text Generation: Benchmarking Clinical Note Generation

arXiv:2610.08161v1 Announce Type: cross Abstract: Ambient documentation systems are rapidly gaining adoption, yet their impact on clinical note quality remains poorly characterized. We introduce MedCo

Towards an Extensible Benchmark for Spoken Dialogue with Social Robots

arXiv:2610.08733v1 Announce Type: new Abstract: Language models provide a plug-and-play interface between humans and robots, but important challenges remain when speech, dialogue, fast interaction, an

Towards benchmarking Western Bluebird detection in the wild

arXiv:2610.07802v1 Announce Type: new Abstract: Bird monitoring in natural environments is challenging due to the small size of some species of birds relative to the scene, background clutter, variabi

CHAOSMINING: Benchmarking Post-Hoc Attribution with Sparse Informative Features in High Dimensions

arXiv:2406.12150v2 Announce Type: replace Abstract: Post-hoc attribution is widely used to identify important model inputs, but evaluating whether these attributions identify truly informative feature

ClimateBench v2.0: Probabilistic Climate Model Benchmarking

arXiv:2610.04558v1 Announce Type: cross Abstract: We present ClimateBench v2, a standardized protocol for evaluating climate models on diagnostics expected to be informative for their skill in project

Odyssey: A Closed-Loop Benchmark for Long-Horizon Real-World Driving with Explicit Navigation Routes

arXiv:2610.06469v1 Announce Type: new Abstract: Closed-loop evaluation of end-to-end driving requires continuous rollouts that reveal how earlier decisions affect subsequent driving. However, existing

Radar2Plan: Benchmarking 4D Radar for End-to-End Open-Loop Ego-Trajectory Planning

arXiv:2610.06121v1 Announce Type: new Abstract: Adverse weather and poor illumination remain major challenges for robust ego-trajectory planning in mobile autonomy. 4D radar offers reliable sensing un

Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos

arXiv:2610.04432v1 Announce Type: new Abstract: Building interactive simulators from real-world observations is a promising way to scale embodied data, but current pipelines still rely heavily on manu

Do Large Language Models Know Colombian Law? A Reliability Benchmark for the Colombian Legal System

arXiv:2610.03639v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to support legal practice, education, and research, yet their reliability in national legal systems o