Search: “benchmark”

CESBench: Benchmarking Large Language Models on Cryptographic Engineering Security for IoT Devices

arXiv:2609.21344v1 Announce Type: cross Abstract: For Internet of Things (IoT) devices, a secure algorithm alone is not enough: an attacker with physical access can attack the implementation directly,

21 Sept 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

PolyBridgeBench: Benchmarking Multimodal LLMs for Physics-Grounded Bridge Design

DGX agent

arXiv:2609.21493v1 Announce Type: new Abstract: Multimodal large language models, or MLLMs, perform well at visual understanding and structured generation, yet these capabilities do not establish whet

model-releasesarxiv-cs-ai

21 Sept 2026

RobotEQ-Video: A Video-Centric Benchmark for Social Proactive Intelligence with World-State Taxonomy

arXiv:2609.21371v1 Announce Type: new Abstract: Social Proactive Intelligence (SPI) extends proactive assistance beyond task completeness to consider social appropriateness in diverse embodied scenari

21 Sept 2026

VakQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answering

arXiv:2609.19879v1 Announce Type: new Abstract: Question answering has advanced rapidly with large language models, but predominantly for high-resource languages, in both text and spoken settings. Spo

18 Sept 2026

Map2Route: Benchmarking Compositional Language-Grounded Route Planning over Semantic Maps

arXiv:2609.17910v1 Announce Type: new Abstract: We introduce Map2Route, a human-curated benchmark for compositional language-grounded route planning over pre-built semantic maps. Map2Route contains 1,

17 Sept 2026

TeochewBench: A Human-Reviewed Benchmark for Teochew Hanzi Translation

arXiv:2609.18156v1 Announce Type: new Abstract: Teochew has a substantial speaker community and exhibits distinctive lexical, syntactic, and pragmatic features, yet textual resources for evaluating la

17 Sept 2026

AssemblyGrid v1: A Benchmark for Multi-Robot Production with Temporary Coalitions, Local Information, and Geometric Constraints

arXiv:2609.16075v1 Announce Type: cross Abstract: Flexible robotic production requires joint decisions on process progression, material routing, resource assignment, temporary cooperation, and simulta

16 Sept 2026

E2A-Bench: Benchmarking Evidence-to-Action Reliability in Financial Chart Reasoning

arXiv:2609.14302v1 Announce Type: new Abstract: Can financial vision-language models (VLMs) turn chart evidence into reliable action recommendations? Existing hallucination evaluations are mostly clai

FedLTLib: A Comprehensive Benchmark for Federated Long-Tail Learning

arXiv:2609.15625v1 Announce Type: cross Abstract: Driven by the escalating demand for privacy-preserving computing, Federated Learning (FL) has witnessed remarkable progress, becoming a cornerstone te

Identity Is More Than Recall: A Benchmark for Persistent Identity in Deployed AI Agents

arXiv:2609.13637v1 Announce Type: new Abstract: Persistent agents need evaluations that distinguish identity facts they can recall from those they express and enact. We introduce PAI-Bench, a provider

MCIQA-2K: A Multi-Dimensional Dataset and No-Reference Quality Assessment Benchmark for Colorized Images

arXiv:2609.14495v1 Announce Type: new Abstract: Image colorization is an inherently ill-posed task, since a single grayscale image may correspond to multiple plausible colorized results. Consequently,

UniqueShip: Mitigating Data Leakage in Acoustic Ship Classification Benchmark Datasets

arXiv:2609.13659v1 Announce Type: cross Abstract: Underwater Acoustic Target Recognition (UATR) of ships is well-suited for machine learning, yet its progress is hindered by the lack of large, diverse

ChitraMiti: Benchmarking Visual Grounding and Modality Reliance in Bengali Geometric Reasoning

arXiv:2609.12509v1 Announce Type: new Abstract: Evaluation of vision-language models (VLMs) for multimodal mathematical reasoning remains limited for low-resource languages and for geometry problems t

14 Sept 2026

Can LLMs Normalize Databases? A Benchmark and Multi-Agent Framework for Schema Normalization

arXiv:2609.11141v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly used to generate structured outputs, but their reliability remains unclear when those outputs must satisfy

11 Sept 2026

A Systematic Evaluation of Molecule Generation Models for De Novo Drug Design: From Benchmarks to Practical Insights

arXiv:2609.10099v1 Announce Type: new Abstract: Molecule generation has emerged as a powerful computational tool for de novo drug design, enabling the exploration of chemical space beyond the limits o

Agentic BAIM-LLM Evaluation (ABLE): Benchmarking LLM Use of Protein Design Tools

arXiv:2609.05818v1 Announce Type: new Abstract: We introduce ABLE, a benchmark for evaluating LLM agents' ability to use biological AI models (BAIMs), such as ProteinMPNN and AlphaFold3, in dual-use p

FolDeX: A Physical-World Benchmark for Long-Horizon Robotic Manipulation of Deformable Objects

arXiv:2609.10243v1 Announce Type: new Abstract: Embodied AI, including vision-language-action and world-action models, must operate reliably in the physical world. Yet methods that perform well in sim

IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications

arXiv:2609.10539v1 Announce Type: new Abstract: A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implement

Infra-Bench CLS: A Global, Open-Source Benchmark for Critical Infrastructure Classification with Earth Observation Foundation Models

arXiv:2609.09482v1 Announce Type: new Abstract: Critical infrastructure location data is often incomplete and unevenly distributed globally, especially in developing regions. Earth observation foundat

Hardware

RAGMark: A Comprehensive Framework for Benchmarking Retrieval-Augmented Generation Systems

arXiv:2609.05760v1 Announce Type: cross Abstract: We present RAGMark, a modular benchmarking framework for advanced Retrieval-Augmented Generation (RAG) systems targeting small-scale multi-GPU environ

hardwarearxiv-cs-ai

AhaBench: Do Agents Learn from Prior Experience? A Benchmark for Long-Horizon Continual Learning

arXiv:2609.05435v1 Announce Type: new Abstract: Modern language agents are expected to operate over long horizons: they ask follow-up questions, reuse worked examples, handle tool feedback, and adapt

BarkNet-Lite: A Lightweight Texture and Colour Network with the BarkBD Benchmark for Bark-Based Tree Species Recognition in Bangladesh

arXiv:2609.07600v1 Announce Type: new Abstract: Tree species recognition supports forest inventory and biodiversity monitoring but still depends on scarce taxonomic expertise. Bark is visible year-rou

ChildGaze: A Benchmark Dataset for Collaborative Behavior Understanding in Children

arXiv:2609.06353v1 Announce Type: new Abstract: Understanding collaborative behavior in children is important for analyzing social participation, peer interaction, shared attention, and engagement dur

CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation

arXiv:2609.05518v1 Announce Type: new Abstract: Despite the strong capabilities of multimodal large language models (MLLMs), their parametric knowledge remains incomplete and difficult to update, moti

DriveMotion: A Large-Scale Multi-Source Benchmark for Driver Motion Sequence Modeling and Forecasting

arXiv:2609.08117v1 Announce Type: new Abstract: Driver motion can provide cues to ongoing behavior, attention, and near-term driving intent. However, most existing driver-centric datasets focus on rec

Monkey See, Can Monkey Do? A Benchmark for Evaluating Robot Skill Learning by Observation

arXiv:2609.08209v1 Announce Type: new Abstract: Learning from Observation (LfO) is a fundamental robotic capability that replicates how humans and animals socially learn from each other. Beyond its bi

VEX-Bench: Benchmarking LLM Agents for Assessing Exploitability of Software Supply Chain Vulnerabilities

arXiv:2609.08040v1 Announce Type: cross Abstract: The software supply chain has become an increasingly exposed attack surface because of its reliance on intricate yet fragile dependencies. Existing de

What the Window Does Not Contain: Auditing Provenance in a Document-Grounded Instability Benchmark

arXiv:2609.06147v1 Announce Type: cross Abstract: Ask a language model the same question about the same document twenty times, and it sometimes returns two different answers. We built Probity, a bench

Hardware

A Sim-to-Real Study of Surface-Code Decoder Benchmarking

arXiv:2609.04557v1 Announce Type: cross Abstract: Quantum error-correction decoders are typically benchmarked against synthetic circuit-level noise, under the assumption that a decoder's ranking under

hardwarearxiv-cs-lg

Brain4FMs: A Benchmark of Foundation Models for Electrical Brain Signal

arXiv:2602.11558v2 Announce Type: replace Abstract: Brain foundation models (BFMs) are advancing neurotechnology by learning transferable representations from neural signals, with broad potential in c

MM-IFEval-Pro: A Multilingual and Attack-Resistant Benchmark for Instruction-Following in Vision-Language Models

arXiv:2609.04859v1 Announce Type: new Abstract: As vision-language models (VLMs) rapidly advance in image understanding, cross-modal reasoning, and complex instruction execution, instruction-following

Multi-Step Tool-Calling over Korean Open Public APIs: A Benchmark and a Data-Synthesis Recipe

arXiv:2609.05395v1 Announce Type: new Abstract: Data-sovereignty regulations increasingly require public institutions to deploy open-source, on-premise LLM agents that chain multiple tool-calls across

TRILOGUE: A Trilingual Spoken Dialogue Fact-Checking Benchmark with Evidence and Paired Audio

arXiv:2609.04452v1 Announce Type: new Abstract: Modern misinformation is often heard before it is read, yet fact-checking systems are still evaluated mainly on clean written claims. Spoken dialogue re

Do Video Generators Track the World Across Segments? A Benchmark and Method for World-State Reasoning in Video Continuation

arXiv:2609.03673v1 Announce Type: new Abstract: Video generators build long videos by composing shorter parts, either by generating segments one after another or by autoregressively extending chunks.

FPCO-Dialog: A Multi-Turn False-Premise Benchmark for Correction and Cooperation in Vision-Language Models

arXiv:2609.03331v1 Announce Type: new Abstract: Vision-language models (VLMs) are increasingly deployed in multi-turn settings where users may describe visual content with incorrect assumptions. Yet e

KC-Bench: A Dynamic Interactive Benchmark for Evaluating Knowledge Conflicts in LLM Agents

arXiv:2609.03588v1 Announce Type: new Abstract: As LLMs increasingly act through tools, they must reconcile user instructions, parametric knowledge, and dynamic environmental observations before takin

When Users Don't Ask: Benchmarking Context-Driven Memory Retrieval in Conversational Agents

arXiv:2609.03467v1 Announce Type: cross Abstract: Large language models (LLMs) are increas- ingly deployed as long-horizon conversational agents, motivating growing interest in mem- ory systems. Howev

Blending Concepts: Benchmarking Visual Metaphor Generation in Text-to-Image Models

arXiv:2609.02502v1 Announce Type: cross Abstract: Text-to-image (T2I) models have achieved remarkable success at faithfully rendering specified objects and attributes, yet their ability to produce vis

3 Sept 2026

SoK: Where Do Flow Labels Come From? Auditing Label Provenance in Encrypted Traffic Benchmarks

arXiv:2609.02140v1 Announce Type: cross Abstract: Encrypted traffic classification infers semantics beyond the flow record from transport-layer observables, and supervised training rests on labels tha

3 Sept 2026

Benchmarking Vision-Language Models for Automated Pathology Diagnosis and Report Generation

arXiv:2609.00866v1 Announce Type: cross Abstract: The rapid advancement of vision-language models (VLMs) has accelerated progress in computational pathology; however, whole-slide image (WSI)-based pat

CoMMET: A Psychologically Grounded Benchmark for Evaluating Theory of Mind in Multimodal LLMs

arXiv:2603.11915v2 Announce Type: replace Abstract: Theory of Mind (ToM)-the ability to reason about the mental states of oneself and others-is a cornerstone of human social intelligence. As Multimoda

CopyShield: A Cross-Level Benchmark of Copyright Defenses in LLMs

arXiv:2609.01161v1 Announce Type: new Abstract: Large language models can reproduce memorized text verbatim, yet copyright defenses are usually evaluated under incompatible protocols. We introduce Cop

Do Satellites See Commuters? A Critical Benchmark of Vision Foundation Models

arXiv:2609.00661v1 Announce Type: new Abstract: Satellite foundation models offer a globally available alternative to census data for commuting origin-destination (OD) generation, yet no study has sys

MeshSplatBench: A Unified Benchmark for Triangle-Based Neural Rendering

arXiv:2609.01306v1 Announce Type: cross Abstract: Triangle-based neural rendering bridges neural scene representations and conventional graphics pipelines by optimizing explicit geometric primitives c

MultiGait: A Multi-Sensor Multi-Perspective Multi-Session Biometric Inference Benchmark and its Dataset

arXiv:2609.01036v1 Announce Type: cross Abstract: A lack of suitable datasets has limited the research into the privacy risks of novel smart city sensors, such as thermal cameras, depth cameras, and l

RPCBench: A Benchmark for Proactive Premise Critique in LLM-based Recommendation

arXiv:2609.00918v1 Announce Type: new Abstract: Large language models are increasingly used as interactive recommender assistants. Their evaluation should therefore go beyond plausible item recommenda

Behavior-Skill: A Fine-Grained Benchmark for Evaluating Vision-Language-Action Policies in Long-Horizon Tasks

arXiv:2608.30536v1 Announce Type: new Abstract: Reliable execution of long-horizon mobile manipulation tasks remains challenging because overall task success depends on the successful completion of mu

1 Sept 2026

Benchmarking External Generalization of SPD Matrix Learning for Resting-State fMRI Connectome Prediction

arXiv:2608.30418v1 Announce Type: cross Abstract: Resting-state functional magnetic resonance imaging (rs-fMRI) functional connectivity (FC) matrices are widely used for individual-level prediction, b