Knowledge catalogue
arXiv:2606.01400v1 Announce Type: cross Abstract: Evaluating large language models (LLMs) across comprehensive benchmarks is expensive and time-consuming. We propose a graph-based prompt selection fra
arXiv:2605.26144v1 Announce Type: cross Abstract: We present VISTA (VIsual Spec-To-App Benchmark), a benchmark for evaluating the end-to-end web-app generation capabilities of LLM-based agents. Unlike
arXiv:2605.26079v1 Announce Type: new Abstract: Modern AI benchmarks operate at a complexity that outpaces traditional verification methods. Tasks authored by domain experts often contain implicit ass
arXiv:2507.19219v2 Announce Type: replace Abstract: Overestimation in evaluating large language models (LLMs) has become an increasing concern. Due to the contamination of public benchmarks or imbalan
arXiv:2605.21404v1 Announce Type: new Abstract: We read twelve well-known LLM agent benchmark papers and recorded, dimension by dimension, what each paper actually says about how its evaluation was ru
arXiv:2605.15537v1 Announce Type: new Abstract: This paper introduces RTL-BenchMT, an agentic framework for dynamically maintaining RTL generation benchmarks. Large Language Models (LLMs) assisted aut
There are a lot of coding and reasoning benchmarks for AI agents, but not a lot for document understanding - which is a prerequisite for all downstream knowledge work. We released ParseBench ~a month
arXiv:2605.15118v1 Announce Type: cross Abstract: We introduce a reusable framework for auditing whether LLM attack benchmarks collectively cover the threat surface: a 4imes6 Target imes Technique mat
arXiv:2512.12634v3 Announce Type: replace Abstract: Mobile GUI Agents, AI agents capable of interacting with mobile applications on behalf of users, have the potential to transform human computer inte
arXiv:2605.08838v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) is widely used to augment large language models (LLMs) with external knowledge. However, many benchmark datasets,
arXiv:2508.04325v2 Announce Type: replace-cross Abstract: Large language models (LLMs) show significant potential in healthcare, prompting numerous benchmarks to evaluate their capabilities. However,
arXiv:2604.20087v1 Announce Type: new Abstract: Skills have become the de facto way to enable LLM agents to perform complex real-world tasks with customized instructions, workflows, and tools, but how
arXiv:2604.19533v1 Announce Type: cross Abstract: We introduce the Cyber Defense Benchmark, a benchmark for measuring how well large language model (LLM) agents perform the core SOC analyst task of th
arXiv:2510.07143v3 Announce Type: replace Abstract: Recent efforts to accelerate inference in Multimodal Large Language Models (MLLMs) have largely focused on visual token compression. The effectivene
arXiv:2601.17230v2 Announce Type: replace Abstract: Automated Fact-Checking has largely focused on verifying general knowledge against static corpora, overlooking high-stakes domains like law where tr
I find that open weights models over-perform on benchmarks compared to actual real-world usage, and Kimi feels like no exception. For example, a small amount of use will show that Kimi is not as good
arXiv:2604.18584v1 Announce Type: cross Abstract: Mathematical problem solving remains a challenging test of reasoning for large language and multimodal models, yet existing benchmarks are limited in
arXiv:2604.17944v1 Announce Type: new Abstract: Developing agents capable of navigating fragmented, multi-source information remains challenging, primarily due to the scarcity of benchmarks reflecting
arXiv:2604.17771v1 Announce Type: new Abstract: Large language models (LLMs) have achieved strong performance on natural language to SQL (NL2SQL) benchmarks, yet their reported accuracy may be inflate
arXiv:2509.06477v2 Announce Type: replace Abstract: Shortcuts such as APIs and deep-links have emerged as efficient complements to flexible GUI operations, fostering a promising hybrid paradigm for ML
arXiv:2604.05510v2 Announce Type: replace Abstract: Augmented reality (AR) has rapidly expanded over the past decade. As AR becomes increasingly integrated into daily life, its security and reliabilit
arXiv:2604.08571v1 Announce Type: cross Abstract: While Large Language Models (LLMs) achieve high performance on standard mathematical benchmarks, their underlying reasoning processes remain highly ov
arXiv:2604.02022v2 Announce Type: replace Abstract: Evaluating the safety of LLM-based agents is increasingly important because risks in realistic deployments often emerge over multi-step interactions
arXiv:2406.12009v5 Announce Type: replace Abstract: Accurate and transparent financial information disclosure is essential for market efficiency, investor decision-making, and corporate governance. Ch
arXiv:2604.06846v1 Announce Type: cross Abstract: Interactive medical dialogue benchmarks have shown that LLM diagnostic accuracy degrades significantly when interacting with non-cooperative patients,
arXiv:2505.22976v2 Announce Type: replace-cross Abstract: The ability to simulate the world in a spatially consistent manner is a crucial requirements for effective world models. Such a model enables
arXiv:2610.16426v1 Announce Type: new Abstract: Automated approval systems let coding agents execute high-risk actions with minimal human oversight by delegating the authorization decision to a classi
arXiv:2610.15634v1 Announce Type: new Abstract: Sycophancy in large language models risks amplifying unsupported claims or beliefs across the knowledge ecosystem as these systems increasingly contribu
arXiv:2610.20012v1 Announce Type: new Abstract: Chemistry evaluation has moved past multiple-choice probes toward open-ended expert problems, yet the benchmarks that define the current frontier still
arXiv:2610.19165v1 Announce Type: cross Abstract: Many real-world questions demand not a single fact but an organized collection of them, yet existing factual knowledge benchmarks almost exclusively t
arXiv:2608.28281v2 Announce Type: replace Abstract: Loop Engineering is emerging as a practice for organizing development work around coding agents. Instead of writing each prompt by hand, practitione
arXiv:2610.16538v1 Announce Type: cross Abstract: We present PatTransBench, a 12,000-item Chinese-English/Japanese/Korean patent machine translation (PatMT) benchmark with a four-dimensional evaluatio
arXiv:2610.16412v1 Announce Type: new Abstract: As IT systems grow in complexity, automated agent-driven incident response becomes increasingly critical. A key capability for these agents is summarizi
arXiv:2610.16127v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly explored in the Architecture, Engineering, and Construction (AEC) domain, yet their reliability in regul
arXiv:2610.16067v1 Announce Type: cross Abstract: Agent evaluation harnesses are software systems that run test suites, generate inputs, and decide outcomes with test oracles. This paper audits such s
arXiv:2610.17694v1 Announce Type: new Abstract: We introduce HelpMeOut, an embodied AI benchmark for error-aware monitoring of household task execution. Existing embodied household benchmarks typicall
arXiv:2610.14707v1 Announce Type: cross Abstract: Expert-aligned 'taste' benchmarks should reward craft judgment, not the ability to game the test. We introduce TasteBench, a pairwise good-vs-bad benc
arXiv:2610.15032v1 Announce Type: new Abstract: Industrial anomaly detection benchmarks have advanced rapidly, yet they remain largely static and therefore underrepresent deployment settings in which
arXiv:2610.16398v1 Announce Type: new Abstract: Robot learning has undergone significant advancements in the domain of compliant control and robotic assembly. The development of novel methodologies ha
arXiv:2610.18829v1 Announce Type: new Abstract: With the rapid development of multimodal large language models (MLLMs), evaluating their video understanding capabilities has become increasingly import
arXiv:2610.14635v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) have achieved remarkable progress across a wide range of multimodal tasks. However, object hallucination remains
arXiv:2601.04824v3 Announce Type: replace Abstract: Automatic identification of events and recurrent behavior analysis are critical for video surveillance. However, most existing content-based video r
arXiv:2610.09633v1 Announce Type: cross Abstract: The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engin
arXiv:2610.07580v1 Announce Type: new Abstract: Aerospace electrical-design revisions can contain multiple genuine changes, although an engineering request may authorize only a subset. Propagating eve
arXiv:2610.08662v1 Announce Type: new Abstract: As coding agents increasingly undertake real-world work autonomously, judging whether their risk treatments are warranted has become important. Existing
arXiv:2610.08346v1 Announce Type: new Abstract: Polarization imaging provides physical cues beyond intensity imaging but typically requires specialized hardware. Recent methods infer polarization from
arXiv:2610.08026v1 Announce Type: new Abstract: In recent years, several methods for detecting when large language models (LLMs) hallucinate have been developed. These methods are often benchmarked wi
arXiv:2610.07089v1 Announce Type: cross Abstract: LLM agents increasingly act in multi-actor environments, exposing them to misuse from multiple sources: decomposition attacks, where a harmful request
arXiv:2610.04392v1 Announce Type: new Abstract: Existing Anomaly Detection benchmarks are saturated and often unrealistic. As part of the VAND 4.0 Challenge, we introduce a hidden-test, application-dr
arXiv:2610.05775v1 Announce Type: new Abstract: A video assistant must speak when its instruction warrants a response and stay silent otherwise. We introduce a benchmark that evaluates this decision f
arXiv:2610.03825v1 Announce Type: cross Abstract: A benchmark score compares a system output against a reference, and methodological attention falls almost entirely on the first term. We measure the s
arXiv:2610.03153v1 Announce Type: cross Abstract: Workspace agents combine large language models with execution harnesses to perform stateful, multi-step tasks that access or modify external resources
arXiv:2610.03080v1 Announce Type: cross Abstract: Large language models are moving from producing trading signals to writing the code that executes them. The failure mode of the second role is silent:
arXiv:2610.02242v1 Announce Type: cross Abstract: Chemical reaction-condition optimization -- choosing the catalyst, ligand, solvent, reagent, temperature, time, and atmosphere that jointly maximize y
arXiv:2610.00416v1 Announce Type: new Abstract: Evaluating large language models becomes increasingly challenging as their capabilities advance: benchmarks can saturate, public test sets risk contamin
arXiv:2610.00636v1 Announce Type: new Abstract: Evaluating AI agents on scientific research tasks is constrained by the time and resources required for the underlying experiments or calculations. In c
arXiv:2610.00890v1 Announce Type: cross Abstract: Coding agents often fail in the last mile: they build most of a feature but drop a requirement, test only the cases their implementation already handl
arXiv:2610.01306v1 Announce Type: new Abstract: Professional work often starts with a brief request that leaves the professional to work out what is needed, which documents matter, and whether the req
arXiv:2610.01168v1 Announce Type: cross Abstract: Time Series Anomaly Detection has received increasing attention, driven by the growing availability of complex time series data. This surge has led to
arXiv:2610.00370v1 Announce Type: new Abstract: Station weather forecasting is fundamentally shaped by both complex spatial dependencies across stations and strong physical coupling among weather vari