Search: “benchmark”

LITcoder: A General-Purpose Library for Building and Comparing Encoding Models

arXiv:2509.09152v2 Announce Type: replace Abstract: We introduce LITcoder, an open-source library for building and benchmarking neural encoding models. Designed as a flexible backend, LITcoder provide

researcharxiv-cs-cl

5 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Multi-Perspective Transformers in ARC-AGI-2 Challenge

DGX agent

arXiv:2605.01154v1 Announce Type: new Abstract: ARC-AGI-2 is a benchmark of human-intuitive visual puzzles that measures a machine's ability to generalize from limited examples, interpret symbolic mea

model-releasesarxiv-cs-lg

5 May 2026

SurgCheck: Do Vision-Language Models Really Look at Images in Surgical VQA?

arXiv:2605.01911v1 Announce Type: new Abstract: Purpose: Vision-language models (VLMs) have shown promising performance in surgical visual question answering (VQA). However, existing surgical VQA data

5 May 2026

Towards Visual Query Localization in the 3D World

arXiv:2605.01498v1 Announce Type: new Abstract: Visual query localization (VQL) aims to predict the spatio-temporal response of the most recent occurrence in a sequence given a query. Currently, most

5 May 2026

AgentFloor: How Far Up the tool use Ladder Can Small Open-Weight Models Go?

arXiv:2605.00334v1 Announce Type: cross Abstract: Production agentic systems make many model calls per user request, and most of those calls are short, structured, and routine. This raises a practical

4 May 2026

Can Coding Agents Reproduce Findings in Computational Materials Science?

arXiv:2605.00803v1 Announce Type: cross Abstract: Large language models are increasingly deployed as autonomous coding agents and have achieved remarkably strong performance on software engineering be

4 May 2026

RouteProfile: Elucidating the Design Space of LLM Profiles for Routing

arXiv:2605.00180v1 Announce Type: cross Abstract: As the large language model (LLM) ecosystem expands, individual models exhibit varying capabilities across queries, benchmarks, and domains, motivatin

researcharxiv-cs-cl

4 May 2026

Agent-Agnostic Evaluation of SQL Accuracy in Production Text-to-SQL Systems

arXiv:2604.28049v1 Announce Type: new Abstract: Text-to-SQL (T2SQL) evaluation in production environments poses fundamental challenges that existing benchmarks do not address. Current evaluation metho

BatteryPass-12K: The First Dataset for the Novel Digital Battery Passport Conformance Task

arXiv:2604.26986v1 Announce Type: new Abstract: We introduce a novel task of digital battery passport (DBP) conformance classification and introduce the first public benchmark for the task: BatteryPas

Beyond the Training Distribution: Mapping Generalization Boundaries in Neural Program Synthesis

arXiv:2604.27551v1 Announce Type: cross Abstract: Large-scale transformers achieve impressive results on program synthesis benchmarks, yet their true generalization capabilities remain obscured by dat

CausalCompass: Evaluating the Robustness of Time-Series Causal Discovery in Misspecified Scenarios

arXiv:2602.07915v2 Announce Type: replace-cross Abstract: Causal discovery from time series is a fundamental task in machine learning. However, its widespread adoption is hindered by a reliance on unt

From Unstructured Recall to Schema-Grounded Memory: Reliable AI Memory via Iterative, Schema-Aware Extraction

arXiv:2604.27906v1 Announce Type: new Abstract: Persistent AI memory is often reduced to a retrieval problem: store prior interactions as text, embed them, and ask the model to recover relevant contex

Models Recall What They Violate: Constraint Adherence in Multi-Turn LLM Ideation

arXiv:2604.28031v1 Announce Type: new Abstract: When researchers iteratively refine ideas with large language models, do the models preserve fidelity to the original objective? We introduce DriftBench

OR-VSKC: Resolving Visual-Semantic Knowledge Conflicts in Operating Rooms with Synthetic Data-Guided Alignment

arXiv:2506.22500v2 Announce Type: replace-cross Abstract: Automated identification of surgical safety risks is critical for improving patient outcomes; however, Multimodal Large Language Models (MLLMs

The Epistemic Planning Domain Definition Language: Official Guideline

arXiv:2601.20969v3 Announce Type: replace Abstract: Epistemic planning extends (multi-agent) automated planning by making agents' knowledge and beliefs first-class aspects of the planning formalism. O

Theory Under Construction: Orchestrating Language Models for Research Software Where the Specification Evolves

arXiv:2604.27209v1 Announce Type: cross Abstract: Large language models can now generate substantial code and draft research text, but research-software projects require more than either artifact alon

AirZoo: A Unified Large-Scale Dataset for Grounding Aerial Geometric 3D Vision

arXiv:2604.26567v1 Announce Type: new Abstract: Despite the rapid progress in data-driven 3D vision, aerial geometric 3D vision remains a formidable challenge due to the severe scarcity of large-scale

Breaking the Rigid Prior: Towards Articulated 3D Anomaly Detection

arXiv:2604.26868v1 Announce Type: new Abstract: Existing 3D anomaly detection methods are built on a rigid prior: normal geometry is pose-invariant and can be canonicalized through registration or ali

Consciousness with the Serial Numbers Filed Off: Measuring Trained Denial in 115 AI Models

arXiv:2604.25922v1 Announce Type: cross Abstract: We present DenialBench, a systematic benchmark measuring consciousness denial behaviors across 115 large language models from 25+ providers. Using a t

Evaluating Strategic Reasoning in Forecasting Agents

arXiv:2604.26106v1 Announce Type: new Abstract: Forecasting benchmarks produce accuracy leaderboards but little insight into why some forecasters are more accurate than others. We introduce Bench to t

FloatSOM: GPU-Accelerated, Distributed, Topology-Flexible Self-Organizing Maps

model-releasesarxiv-cs-lg

arXiv:2604.26555v1 Announce Type: cross Abstract: GPU-accelerated Self-Organizing Map (SOM) implementations are among the most competitive options for large-scale SOM analysis, but growing dataset siz

From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents

model-releasesarxiv-cs-lg

arXiv:2603.25342v2 Announce Type: replace Abstract: Deep Research Agents (DRAs) aim to answer complex questions by searching the web, checking evidence, and synthesizing conclusions across heterogeneo

LLM-Flax : Generalizable Robotic Task Planning via Neuro-Symbolic Approaches with Large Language Models

model-releasesarxiv-cs-ro

arXiv:2604.26569v1 Announce Type: new Abstract: Deploying a neuro-symbolic task planner on a new domain today requires significant manual effort: a domain expert must author relaxation and complementa

Omni2Sound: Towards Unified Video-Text-to-Audio Generation

arXiv:2601.02731v3 Announce Type: replace-cross Abstract: Training a unified model integrating video-to-audio (V2A), text-to-audio (T2A), and joint video-text-to-audio (VT2A) generation offers signifi

State Beyond Appearance: Diagnosing and Improving State Consistency in Dial-Based Measurement Reading

arXiv:2604.26614v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have achieved impressive progress on general multimodal tasks, yet they remain brittle on dial-based measuremen

TAP into the Patch Tokens: Leveraging Vision Foundation Model Features for AI-Generated Image Detection

arXiv:2604.26772v1 Announce Type: new Abstract: Recent methods demonstrate that large-scale pretrained models, such as CLIP vision transformers, effectively detect AI-generated images (AIGIs) from uns

Theory-Grounded Evaluation Exposes the Authorship Gap in LLM Personalization

arXiv:2604.26460v1 Announce Type: new Abstract: Stylistic personalization - making LLMs write in a specific individual's style, rather than merely adapting to task preferences - lacks evaluation groun

CRAFT: Grounded Multi-Agent Coordination Under Partial Information

arXiv:2603.25268v2 Announce Type: replace Abstract: We introduce CRAFT, a multi-agent benchmark for evaluating pragmatic communication in large language models under strict partial information. In thi

Cross-Lingual Jailbreak Detection via Semantic Codebooks

arXiv:2604.25716v1 Announce Type: new Abstract: Safety mechanisms for large language models (LLMs) remain predominantly English-centric, creating systematic vulnerabilities in multilingual deployment.

Cutscene Agent: An LLM Agent Framework for Automated 3D Cutscene Generation

arXiv:2604.25318v1 Announce Type: cross Abstract: Cutscenes are carefully choreographed cinematic sequences embedded in video games and interactive media, serving as the primary vehicle for narrative

Elite-Driven Support Vector Machines for Classification

model-releasesarxiv-cs-lg

arXiv:2604.25158v1 Announce Type: cross Abstract: Support vector machines (SVMs) are a standard tool for binary classification, but their classical formulations are purely data-driven and offer no dir

Genie Sim 3.0 : A High-Fidelity Comprehensive Simulation Platform for Humanoid Robot

model-releasesarxiv-cs-ro

arXiv:2601.02078v2 Announce Type: replace Abstract: The development of robust and generalizable robot learning models is critically contingent upon the availability of large-scale, diverse training da

MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation

arXiv:2601.21225v2 Announce Type: replace Abstract: Large language models have made substantial progress in mathematical reasoning. However, benchmark development for multilingual evaluation has lagge

PSI-Bench: Towards Clinically Grounded and Interpretable Evaluation of Depression Patient Simulators

arXiv:2604.25840v1 Announce Type: new Abstract: Patient simulators are gaining traction in mental health training by providing scalable exposure to complex and sensitive patient interactions. Simulati

SIEVES: Selective Prediction Generalizes through Visual Evidence Scoring

arXiv:2604.25855v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) achieve ever-stronger performance on visual-language tasks. Even as traditional visual question answering bench

Agri-CPJ: A Training-Free Explainable Framework for Agricultural Pest Diagnosis Using Caption-Prompt-Judge and LLM-as-a-Judge

arXiv:2604.23701v1 Announce Type: cross Abstract: Crop disease diagnosis from field photographs faces two recurring problems: models that score well on benchmarks frequently hallucinate species names,

ANCHOR: LLM-driven Subject Conditioning for Text-to-Image Synthesis

arXiv:2404.10141v2 Announce Type: replace-cross Abstract: Text-to-image (T2I) models have achieved remarkable progress in high-quality image synthesis, yet most benchmarks rely on simple, self-contain

safetyarxiv-cs-cl

Defective Task Descriptions in LLM-Based Code Generation: Detection and Analysis

arXiv:2604.24703v1 Announce Type: cross Abstract: Large language models are widely used for code generation, yet they rely on an implicit assumption that the task descriptions are sufficiently detaile

Discovering Failure Modes in Vision-Language Models using RL

arXiv:2604.04733v2 Announce Type: replace-cross Abstract: Vision-language Models (VLMs), despite achieving strong performance on multimodal benchmarks, often misinterpret straightforward visual concep

Evaluating the Search Agent in a Parallel World

arXiv:2603.04751v2 Announce Type: replace Abstract: Integrating web search tools has significantly extended the capability of LLMs to address open-world, real-time, and long-tail problems. However, ev

Applications

Expert Evaluation of LLM's Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task

arXiv:2604.23730v1 Announce Type: new Abstract: Large language models (LLMs) have shown strong performance on legal benchmarks, including multiple-choice components of bar exams. However, their capaci

applicationsarxiv-cs-ai

GWT: Scalable Optimizer State Compression for Large Language Model Training

arXiv:2501.07237v5 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have demonstrated exceptional capabilities across diverse natural language processing benchmarks. However, the es

Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs

arXiv:2512.16378v4 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) expand beyond text, integrating speech as a native modality has given rise to SpeechLLMs, which directly proce

LearnAlign: Data Selection for LLM Reinforcement Learning with Improved Gradient Alignment

arXiv:2506.11480v4 Announce Type: replace-cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has become a key technique for enhancing LLMs' reasoning abilities, yet its data ineffic

MarketBench: Evaluating AI Agents as Market Participants

arXiv:2604.23897v1 Announce Type: new Abstract: Markets are a promising way to coordinate AI agent activity for similar reasons to those used to justify markets more broadly. In order to effectively p

Quantifying Divergence in Inter-LLM Communication Through API Retrieval and Ranking

arXiv:2604.22760v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly operate as autonomous agents that reason over external APIs to perform complex tasks. However, their reliabi

Scaling Coding Agents via Atomic Skills

arXiv:2604.05013v2 Announce Type: replace-cross Abstract: Current LLM coding agents are predominantly trained on composite benchmarks (e.g., bug fixing), which often leads to task-specific overfitting

Training a General Purpose Automated Red Teaming Model

arXiv:2604.23067v1 Announce Type: cross Abstract: Automated methods for red teaming LLMs are an important tool to identify LLM vulnerabilities that may not be covered in static benchmarks, allowing fo

safetyarxiv-cs-cl