Search: “benchmark”

CUDABeaver: Benchmarking LLM-Based Automated CUDA Debugging

arXiv:2605.08455v1 Announce Type: new Abstract: Debugging CUDA programs has long been challenging because failures often arise from subtle interactions among hardware behavior, compiler decisions, mem

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

CUDAHercules: Benchmarking Hardware-Aware Expert-level CUDA Optimization for LLMs

DGX agent

arXiv:2605.08467v1 Announce Type: new Abstract: Large language models show promise for automated CUDA programming, however even the strongest coding models (e.g., Claude-Opus-4.6) may still fall short

model-releasesarxiv-cs-lg

12 May 2026

Deepfake Detection that Generalizes Across Benchmarks

arXiv:2508.06248v4 Announce Type: replace Abstract: The generalization of deepfake detectors to unseen manipulation techniques remains a challenge for practical deployment. Although many approaches ad

DSGBench: A Diverse Strategic Game Benchmark for Evaluating LLM-based Agents in Complex Decision-Making Environments

arXiv:2503.06047v2 Announce Type: replace Abstract: Large language model (LLM)-based agents are increasingly applied to complex strategic environments that demand long-horizon reasoning, multi-agent i

FinTSB: A Comprehensive and Practical Benchmark for Financial Time Series Forecasting

arXiv:2502.18834v2 Announce Type: replace-cross Abstract: Financial time series (FinTS) record the behavior of human-brain-augmented decision-making, capturing valuable historical information that can

FORTIS: Benchmarking Over-Privilege in Agent Skills

arXiv:2605.09163v1 Announce Type: new Abstract: Large language model agents increasingly operate through an intermediate skill layer that mediates between user intent and concrete task execution. This

KARMA-MV: A Benchmark for Causal Question Answering on Music Videos

arXiv:2605.08175v1 Announce Type: cross Abstract: While significant progress has been made in Video Question Answering and cross-modal understanding, causal reasoning about how visual dynamics drive m

MathlibLemma: Folklore Lemma Generation and Benchmark for Formal Mathematics

arXiv:2602.02561v2 Announce Type: replace-cross Abstract: While the ecosystem of Lean and Mathlib has enjoyed celebrated success in formal mathematical reasoning with the help of large language models

PrimeKG-CL: A Continual Graph Learning Benchmark on Evolving Biomedical Knowledge Graphs

arXiv:2605.10529v1 Announce Type: new Abstract: Biomedical knowledge graphs underwrite drug repurposing and clinical decision support, yet the upstream ontologies they depend on update on independent

Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark

arXiv:2509.26574v4 Announce Type: replace Abstract: While large language models (LLMs) with reasoning capabilities are progressing rapidly on high-school math competitions and coding, can they reason

Safety

SalesSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators

arXiv:2605.08334v1 Announce Type: new Abstract: We present SalesSim, a framework and testbed for evaluating the ability of Multimodal Large Language Models (MLLMs) to simulate realistic, persona-drive

safetyarxiv-cs-cl

SciIntegrity-Bench: A Benchmark for Evaluating Academic Integrity in AI Scientist Systems

arXiv:2605.10246v1 Announce Type: new Abstract: AI scientist systems are increasingly deployed for autonomous research, yet their academic integrity has never been systematically evaluated. We introdu

SDiaReward: Modeling and Benchmarking Spoken Dialogue Rewards with Modality and Colloquialness

arXiv:2603.14889v2 Announce Type: replace-cross Abstract: The rapid evolution of end-to-end spoken dialogue systems demands transcending mere textual semantics to incorporate paralinguistic nuances an

Sens-VisualNews: A Benchmark Dataset for Sensational Image Detection

arXiv:2605.10394v1 Announce Type: new Abstract: The detection of sensational content in media items can be a critical filtering mechanism for identifying check-worthy content and flagging potential di

SenseBench: A Benchmark for Remote Sensing Low-Level Visual Perception and Description in Large Vision-Language Models

arXiv:2605.10576v1 Announce Type: cross Abstract: Low-level visual perception underpins reliable remote sensing (RS) image analysis, yet current image quality assessment (IQA) methods output uninterpr

Single-Thread JPEG Decoder Benchmarks Mis-Evaluate ML Data Loaders

arXiv:2605.08731v1 Announce Type: cross Abstract: JPEG decode is routine ML infrastructure, but Python decoder choices are often justified by single-process, single-thread microbenchmarks. We audit th

Benchmarking Foundation Models for Renal Lesion Stratification in CT

arXiv:2605.07749v1 Announce Type: new Abstract: The rapid proliferation of open-source medical foundation models (FMs) raises a practical question: how well do their pre-trained representations transf

Benchmarking World-Model Learning with Environment-Level Queries

arXiv:2510.19788v4 Announce Type: replace Abstract: World models are central to building AI agents capable of flexible reasoning and planning. Yet current evaluations (i) test only properties measurab

CyBiasBench: Benchmarking Bias in LLM Agents for Cyber-Attack Scenarios

arXiv:2605.07830v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed as autonomous agents in offensive cybersecurity. In this paper, we reveal an interesting phenom

EnvSimBench: A Benchmark for Evaluating and Improving LLM-Based Environment Simulation

arXiv:2605.07247v1 Announce Type: new Abstract: Scalable AI agents training relies on interactive environments that faithfully simulate the consequences of agent actions. Manually crafted environments

Have Graph -- Will Lift? The Case for Higher-Order Benchmarks

arXiv:2605.07397v1 Announce Type: new Abstract: After a somewhat rocky start, geometry and topology have established a foothold in machine learning. Message passing, either on graphs or higher-order c

MathlibPR: Pull Request Merge-Readiness Benchmark for Formal Mathematical Libraries

arXiv:2605.07147v1 Announce Type: cross Abstract: The ecosystem of Lean and Mathlib has become the de facto standard for large language model (LLM) assisted formal reasoning with remarkable successes

Real-IAD MVN: A Multi-View Normal Vector Dataset and Benchmark for High-Fidelity Industrial Anomaly Detection

arXiv:2605.07149v1 Announce Type: new Abstract: Industrial Anomaly Detection (IAD) is critical for quality control, but existing methods struggle with subtle, geometric defects. Standard 2D (RGB) imag

Applications

Script Sensitivity: Benchmarking Language Models on Unicode, Romanized and Mixed-Script Sinhala

arXiv:2601.14958v3 Announce Type: replace-cross Abstract: The performance of Language Models (LMs) on low-resource, morphologically rich languages like Sinhala remains largely unexplored, particularly

applicationsarxiv-cs-ai

Setting-Matched and Semantics-Scaled Benchmarking of One-Step Generative Models Against Multistep Diffusion and Flow Models

arXiv:2603.14186v4 Announce Type: replace Abstract: State-of-the-art text-to-image models produce high-quality images, but inference remains expensive as generation requires several sequential ODE or

TAVIS: A Benchmark for Egocentric Active Vision and Anticipatory Gaze in Imitation Learning

arXiv:2605.07943v1 Announce Type: cross Abstract: Active vision -- where a policy controls its own gaze during manipulation -- has emerged as a key capability for imitation learning, with multiple ind

Safety

Beyond Fixed Thresholds and Domain-Specific Benchmarks for Explainable Multi-Task Classification in Autonomous Vehicles

arXiv:2605.04299v1 Announce Type: new Abstract: Scene understanding is a vital part of autonomous driving systems, which requires the use of deep learning models. Deep learning methods are intrinsical

safetyarxiv-cs-cv

7 May 2026

KernelBench-X: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels

arXiv:2605.04956v1 Announce Type: new Abstract: LLM-based Triton kernel generation has attracted significant interest, yet a fundamental empirical question remains unanswered: where does this capabili

7 May 2026

TabEmbed: Benchmarking and Learning Generalist Embeddings for Tabular Understanding

arXiv:2605.04962v1 Announce Type: new Abstract: Foundation models have established unified representations for natural language processing, yet this paradigm remains largely unexplored for tabular dat

7 May 2026

Applications

A Unified Framework for Tabular Generative Modeling: Loss Functions, Benchmarks, and Improved Multi-objective Bayesian Optimization Approaches

arXiv:2405.16971v2 Announce Type: replace Abstract: Deep learning (DL) models require extensive data to achieve strong performance and generalization. Deep generative models (DGMs) offer a solution by

applicationsarxiv-cs-lg

Research

Benchmarking Logistic Regression, SVM, Naive Bayes, and IndoBERT Fine-Tuning for Sentiment Analysis on Indonesian Product Reviews

arXiv:2605.03439v1 Announce Type: new Abstract: The exponential growth of e-commerce platforms in Indonesia has generated a massive volume of user-generated product reviews. Analyzing the sentiment of

researcharxiv-cs-cl

Benchmarking Parameter-Efficient Fine-Tuning of Large Language Models for Low-Resource Tajik Text Generation with the Tajik Web Corpus

arXiv:2605.03742v1 Announce Type: new Abstract: This paper is devoted to the adaptation of generative large language models for the Tajik language, a low-resource language with Cyrillic script. To ove

Donor-Aware scRNA-seq Benchmarks for IBD Classification

arXiv:2605.03281v1 Announce Type: cross Abstract: Donor-level disease classification from single-cell RNA sequencing (scRNA-seq) requires strict donor-aware cross-validation: naive pipelines that spli

Erase Persona, Forget Lore: Benchmarking Multimodal Copyright Unlearning in Large Vision Language Models

arXiv:2605.03547v1 Announce Type: new Abstract: Large Vision-Language Models (LVLMs), trained on web-scale data, risk memorizing and regenerating copyrighted visual content such as characters and logo

LitVISTA: A Benchmark for Narrative Orchestration in Literary Text

arXiv:2601.06445v2 Announce Type: replace Abstract: Computational narrative analysis aims to capture rhythm, tension, and emotional dynamics in literary texts. Existing large language models can gener

MCJudgeBench: A Benchmark for Constraint-Level Judge Evaluation in Multi-Constraint Instruction Following

arXiv:2605.03858v1 Announce Type: new Abstract: Multi-constraint instruction following requires verifying whether a response satisfies multiple individual requirements, yet LLM judges are often assess

PHBench: A Benchmark for Predicting Startup Series A Funding from Product Hunt Launch Signals

arXiv:2605.02974v1 Announce Type: cross Abstract: Structured launch signals on Product Hunt contain statistically significant predictive information for Series A funding outcomes. We construct PHBench

Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use

arXiv:2605.02964v1 Announce Type: new Abstract: Reinforcement learning (RL) trained language model agents with tool access are increasingly deployed in coding assistants, research tools, and autonomou

VEBench:Benchmarking Large Multimodal Models for Real-World Video Editing

arXiv:2605.03276v1 Announce Type: new Abstract: Real-world video editing demands not only expert knowledge of cinematic techniques but also multimodal reasoning to select, align, and combine footage i

WorldJen: An End-to-End Multi-Dimensional Benchmark for Generative Video Models

arXiv:2605.03475v1 Announce Type: new Abstract: Evaluating generative video models remains an open problem. Reference-based metrics such as Structural Similarity Index Measure (SSIM) and Peak Signal t

ATR-Bench: A Federated Learning Benchmark for Adaptation, Trust, and Reasoning

arXiv:2505.16850v2 Announce Type: replace-cross Abstract: Federated Learning (FL) has emerged as a promising paradigm for collaborative model training while preserving data privacy across decentralize

Benchmarking local Hebbian learning rules for memory storage and prototype extraction

arXiv:2605.01074v1 Announce Type: cross Abstract: Associative memory or content-addressable memory is an important component function in computer science and information processing, and at the same ti

Benchmarking Retrieval Strategies for Biomedical Retrieval-Augmented Generation: A Controlled Empirical Study

arXiv:2605.02520v1 Announce Type: new Abstract: Retrieval-Augmented Generation (RAG) offers a well-established path to grounding large language model (LLM) outputs in external knowledge, yet the quest

Benchmarking Wireless Representations: High-Dimensional vs. Compressed Embeddings for Efficiency and Robustness

arXiv:2605.02009v1 Announce Type: cross Abstract: Building on recent advances in representation learning for wireless channels, this work investigates the cost-benefit trade-offs of high-dimensional c

Beyond Perplexity: Character Distribution Signatures and the MDTA Benchmark for AI Text Detection

arXiv:2605.01647v1 Announce Type: new Abstract: Training-free AI text detection methods primarily rely on model log-probabilities, achieving strong performance through approaches like Binoculars and D

Deep Time Series Models: A Comprehensive Survey and Benchmark

arXiv:2407.13278v3 Announce Type: replace Abstract: Time series, characterized by a sequence of data points organized in a discrete-time order, are ubiquitous in real-world scenarios. Unlike other dat

Applications

ECG-biometrics-bench: A Unified Framework for Reproducible Benchmarking of ECG Biometrics

arXiv:2605.01548v1 Announce Type: cross Abstract: Electrocardiogram (ECG) biometrics have emerged as a promising modality for continuous, liveness-aware authentication in wearable systems. However, ma

applicationsarxiv-cs-cv

EmoMM: Benchmarking and Steering MLLM for Multimodal Emotion Recognition under Conflict and Missingness

arXiv:2605.01024v1 Announce Type: new Abstract: Multimodal Emotion Recognition (MER) is critical for interpreting real-world interactions. While Multimodal Large Language Models (MLLM) have shown prom