Search: “benchmark”

Beyond Pass@k: Measuring Reliability and Security of Agentic Code Generation

arXiv:2608.14711v1 Announce Type: new Abstract: AI coding agent benchmarks rank agents with the Chen et al. (2021) pass@k estimator, but current implementations misapply it: they set n to the number o

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

Do Uncertainty Signals Help? A Systematic Study of Uncertainty-Aware Decoding with Rollback Mechanisms

DGX agent

arXiv:2608.14653v1 Announce Type: cross Abstract: Prediction uncertainty is a widely adopted metric for quantifying model confidence, with downstream applications spanning model explanation, data sele

researcharxiv-cs-ai

18 Aug 2026

Graph Machine Learning: An Opportunity for Power Systems

arXiv:2608.16494v1 Announce Type: cross Abstract: Modern power systems face growing operational complexity driven by the integration of renewable energy sources, decentralization, and the need for rea

HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam

arXiv:2602.13964v4 Announce Type: replace Abstract: Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models on challenging, multi-domain questions.

Listen, Reason, and Segment: Aligning LALMs with Editorial Judgment for Media Chapterization

arXiv:2608.16539v1 Announce Type: cross Abstract: Large Audio Language Models (LALMs) have made rapid progress on standardized benchmarks, yet their deployment in practical media workflows, curation,

Safety

LLM Safety Alignment in Low-Resource Languages: A Systematic Literature Review

arXiv:2608.14626v1 Announce Type: cross Abstract: Large Language Models (LLMs) have achieved substantial progress in safety alignment, yet their safety guarantees remain significantly weaker in low-re

safetyarxiv-cs-ai

What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models

arXiv:2608.16852v1 Announce Type: new Abstract: Regulatory compliance monitoring in deployed language models is increasingly implemented as a legal and audit control, checking model outputs against wr

Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards

arXiv:2608.15980v1 Announce Type: new Abstract: Preference benchmarks are built by hiring annotators, and the identity of those annotators is treated as an implementation detail. We measure what that

A Pathway to General-Purpose Scientific AI: Multimodal Comprehension of Scientific Images

arXiv:2608.14075v1 Announce Type: new Abstract: Scientific figures and tables encode essential experimental evidence, yet remain difficult for digital libraries and multimodal AI systems to retrieve a

From Passive Delegates to Strategic Negotiators: Reinforcing Social Reasoning in Small Language Models with SocialRL

arXiv:2608.13787v1 Announce Type: new Abstract: AI agents increasingly act on their users' behalf, handling tasks such as scheduling meetings, comparing offers, and haggling over prices. These princip

Implementing Computational Law in Wolfram Language for the Governance of Artificial Intelligence

arXiv:2608.13958v1 Announce Type: new Abstract: How do we govern AI systems whose reasoning we cannot fully inspect? Governance does not require understanding a system's reasoning. It requires stating

Ordinal-Aware Calibration for Ordinal Classification

arXiv:2410.15658v4 Announce Type: replace-cross Abstract: Deep neural networks frequently produce overconfident, miscalibrated predictions. In ordinal classification, predictions must also adhere to a

PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessment

arXiv:2608.14284v1 Announce Type: cross Abstract: Fine-grained robotic evaluation matters for understanding embodied models, going beyond binary success rates and rule-based process scores. We present

Reinforcement Learning-Based Production Scheduling in an Industry-Based Coating Scenario Using the Digital Model Playground

arXiv:2608.14122v1 Announce Type: new Abstract: Production scheduling in complex manufacturing environments is challenging when sequence-dependent setup times, stochastic disturbances, and due-date co

Beyond Visual Evidence: Revealing and Mitigating Relational Privacy Leakage in Document MLLMs

arXiv:2608.12911v1 Announce Type: new Abstract: While the privacy risks of multimodal large language models (MLLMs) have drawn significant attention, the unique vulnerabilities of domain-specific MLLM

Don't Want Your LLM to Recommend Nuclear Strike? Try Asking It in Japanese

arXiv:2608.12373v1 Announce Type: new Abstract: Large language models are increasingly used in strategic and advisory contexts, yet their safety alignment is typically evaluated in English only. We te

HC-RAG: Evidence-Centric Retrieval-Augmented Generation over Heterogeneous Financial Filings

arXiv:2608.12335v1 Announce Type: new Abstract: Financial question answering over annual reports requires more than retrieving semantically similar passages. It often involves identifying relevant com

How Do VLMs Behave When Blind or Misled? Behavioral Evaluation of VLMs on Scientific Figures

arXiv:2608.13267v1 Announce Type: cross Abstract: Existing vision-language model (VLM) benchmarks emphasize perception and reasoning accuracy (how well VLMs describe and reason about what they see in

Large-scale Testing Global Optimization Methods with Black-box Adversarial Attacks

arXiv:2608.13296v1 Announce Type: cross Abstract: Existing global optimization benchmark suites are of a moderate size and are based on a small number of analytical functions that date back even to th

Research

Numeracy in Large Language Models: Fundamental Limitations and Paths to Improvement

arXiv:2608.13129v1 Announce Type: new Abstract: Large language models (LLMs) achieve strong results on mathematical reasoning benchmarks yet remain unreliable on elementary numerical tasks, including

researcharxiv-cs-ai

Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence

arXiv:2608.12928v1 Announce Type: new Abstract: We introduce a Polish-language medical visual question answering (VQA) benchmark, built from Polish Board Certification Examination questions for licens

Privacy-Preserving RAG by Concealing Sensitive Information from External LLMs

arXiv:2608.12675v1 Announce Type: new Abstract: Retrieval-Augmented Generation (RAG) is widely used to improve the performance of Large Language Models (LLMs) in answering user queries. Existing priva

Revolutionizing Finance with LLMs: An Overview of Applications and Insights

arXiv:2401.11641v5 Announce Type: replace Abstract: In recent years, Large Language Models (LLMs) like ChatGPT have seen considerable advancements and have been applied in diverse fields. Built on the

Scaling Representation Diversity: Modulated Attention and Reconstructive Regularization for Visual Grounding

arXiv:2608.12748v1 Announce Type: new Abstract: Referring Expression Comprehension (REC) is commonly studied under dataset-specific fine-tuning, resulting in specialist models with limited cross-datas

Spatial Memory Agent: Experience-Grounded Procedure Memory for Spatial Intelligence

arXiv:2608.12743v1 Announce Type: new Abstract: Spatial intelligence is becoming a foundation for embodied agents, robotic planning, and multimodal assistants. To improve the spatial reasoning ability

StreamTTT: Reconciling Real-Time Perception and Long-Term Memory in Streaming VLMs

arXiv:2608.13416v1 Announce Type: new Abstract: Humans effortlessly perceive the present while remembering the past, yet streaming VLMs often trade off real-time perception against long-term memory. P

Training AI Scientists to Replicate Research

arXiv:2608.13331v1 Announce Type: cross Abstract: The replicability of papers is a cornerstone of scientific knowledge, ensuring the reliability of existing results and providing a base for further ex

TsuGO: Probing Search Efficiency in LLM Reasoning via Go Life-and-Death Problems

arXiv:2608.13221v1 Announce Type: new Abstract: The evaluation of LLM reasoning is moving from final-answer accuracy to process-level assessment, yet existing methods still fail to capture how models

Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluations

arXiv:2608.11323v1 Announce Type: new Abstract: Enterprise practitioners read agent leaderboards as if they ranked agent capability. We show, across three open agent-trace benchmarks (TheAgentCompany,

From Synthesis to Removal: Physics-Grounded Reflection Simulation and Diffusion-Based Video Dereflection

arXiv:2608.11562v1 Announce Type: cross Abstract: Videos captured through glass often contain reflections that degrade visual quality and interfere with downstream vision tasks. Although single-image

Agents

Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents

arXiv:2608.11727v1 Announce Type: new Abstract: When a coding agent obeys a rule, it may simply have been going to do that anyway. Existing instruction-following benchmarks cannot tell the difference:

agentsarxiv-cs-ai

Localizing Safety Alignment: MLP Layers and Mid-Network Blocks Encode Refusal Behavior in Large Language Models

arXiv:2608.11583v1 Announce Type: new Abstract: Safety alignment in large language models is often treated as a distributed property of the entire network, yet its practical brittleness suggests that

VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies

arXiv:2608.12282v1 Announce Type: new Abstract: Agents deployed in enterprise settings must reason across structured APIs and document collections, yet existing benchmarks evaluate these capabilities

Closed-Loop LLM Co-Pilots for Digital Agriculture

arXiv:2608.09949v1 Announce Type: new Abstract: This study evaluates the application of Large Language Models (LLMs) in complex biological systems, evolving from data analysis to autonomous, AI-guided

From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models

arXiv:2608.10444v1 Announce Type: cross Abstract: Large language models (LLMs) have made substantial progress on reasoning tasks that require increasingly long and complex inferential chains. This pro

Is There Really a Camouflaged Object? Towards Realistic Camouflaged Object Detection

arXiv:2608.11135v1 Announce Type: new Abstract: Camouflaged object detection (COD) aims to segment objects that are visually concealed in their surroundings and has attracted increasing attention in r

Rethinking LLM Verification: Evidence Structure, Uncertainty, and Selective Refinement

arXiv:2608.10725v1 Announce Type: new Abstract: Large language models (LLMs) often rely on shortcuts rather than systematic reasoning, raising safety concerns in medical applications. Allowing models

RLMOpt: Adaptive Prompt Optimization via Recursive Language Models

arXiv:2608.10471v1 Announce Type: new Abstract: Prompt optimizers automate the search for prompts that improve language-model performance, but existing methods rely on a predefined optimization proced

Stream Forcing: Constructing Unified Training Trajectory for Robust Streaming Video Generation

arXiv:2608.10439v1 Announce Type: new Abstract: Streaming video generation holds strong potential for world modeling, where future frames must be inferred online sequentially to form a continuous vide

Surfacing the Unsaid: CUE-Bench for Affective Stance in Chinese Discourse

arXiv:2608.10810v1 Announce Type: cross Abstract: Emotion understanding in discourse requires reasoning beyond surface sentiment because speakers often convey affect through indirect, implicit, polite

VisEditBench: Can Vision-Language Models Edit Visualization Code from Multimodal Feedback?

arXiv:2608.10408v1 Announce Type: new Abstract: Vision-language models (VLMs) have shown strong capabilities in generating visualization code from textual or visual specifications. However, real-world

VoxSumm: A Multilingual Corpus of Long-Form Spoken News for Joint Summarization and Translation

arXiv:2608.10359v1 Announce Type: cross Abstract: As information increasingly traverses linguistic boundaries, users require concise cross-lingual representations of long-form content. Nevertheless, l

When Chain-of-Thought Helps and When It Hurts: An Empirical Investigation of the Serial-Depth Bottleneck in LLM Reasoning

arXiv:2608.09942v1 Announce Type: cross Abstract: It is widely assumed that chain-of-thought (CoT) prompting universally improves LLM reasoning. We investigate this through the conceptual framework of

ActiveFly-Bench: Aligning Embodied Question Answering with Vision-Language-Action for Aerial Embodied Perception

arXiv:2607.10180v2 Announce Type: replace-cross Abstract: We introduce ActiveFly-Bench, the first benchmark to bridge cyberspace reasoning and physical-world interaction for UAV embodied perception. T

DevIntent: How Much Does LLM-Generated Code Violate Developer Intent?

arXiv:2608.07614v1 Announce Type: cross Abstract: Code generated by LLMs can violate a developer's implicit intentions when given an ambiguous prompt, yet standard benchmarks measure only whether code

Eikonal Regularisation in Physics-Informed Neural Networks for Three-Dimensional Level-Set Advection: Transferability of Two-Dimensional Design Principles

model-releasesarxiv-cs-lg

arXiv:2608.08322v1 Announce Type: cross Abstract: Physics-informed neural networks applied to the level-set formulation of interface advection commonly augment the residual and initial-condition losse

FoMoH: A clinically meaningful foundation model evaluation for structured electronic health records

arXiv:2505.16941v4 Announce Type: replace-cross Abstract: Foundation models (FMs) promise to address core limitations of traditional supervised machine learning: (i) reliance on large amounts of label

Research

LookAgain: Closed-Loop GUI Grounding with Visually Grounded Reflection

arXiv:2608.09723v1 Announce Type: new Abstract: Recent graphical user interface (GUI) grounders have significantly advanced single-shot accuracy on standard benchmarks, yet their performance degrades

researcharxiv-cs-cv