Knowledge catalogue
arXiv:2602.18182v5 Announce Type: replace-cross Abstract: AI evaluation has primarily focused on measuring capabilities, with formal approaches inspired from Item Response Theory (IRT) being increasin
arXiv:2610.00636v1 Announce Type: new Abstract: Evaluating AI agents on scientific research tasks is constrained by the time and resources required for the underlying experiments or calculations. In c
arXiv:2610.00237v1 Announce Type: cross Abstract: Spacecraft thermal design requires repeated evaluation of how variations in the number and spatial arrangement of heat-generating components and in th
arXiv:2610.01847v1 Announce Type: cross Abstract: Model specifications define how large language models (LLMs) should behave, guiding alignment training, inference-time behavior, and evaluation. Yet t
arXiv:2610.01661v1 Announce Type: new Abstract: Despite remarkable progress, video generation models often produce highly similar outputs when repeatedly sampled from the same prompt, limiting their u
arXiv:2610.01778v1 Announce Type: new Abstract: Reliable evaluation of image forgery localization (IFL) requires assessing models under diverse distribution changes, yet existing benchmarks often cove
arXiv:2602.20294v2 Announce Type: replace-cross Abstract: Simulating real personalities with large language models requires grounding generation in authentic personal data. Existing evaluation approac
arXiv:2610.01317v1 Announce Type: new Abstract: Evaluating candidate architectures in neural architecture search (NAS) faces an inherent trade-off: on the one hand, reliable performance labels are lim
arXiv:2610.01052v1 Announce Type: new Abstract: We argue that as video generation extends to longer durations, subject consistency should be evaluated over extit{dynamic subject sets}. We therefore in
arXiv:2609.38409v1 Announce Type: new Abstract: Recent progress in large language model reasoning has been driven by benchmarks and reinforcement learning environments with automatically verifiable re
arXiv:2609.38335v1 Announce Type: cross Abstract: Coding agents powered by large language models (LLMs) are evolving from making localized code changes to developing complete software repositories. Ho
arXiv:2609.38473v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) is now the standard way to ground Large Language Models (LLMs) in external knowledge, yet the design space of ret
arXiv:2609.38867v1 Announce Type: new Abstract: Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular int
arXiv:2609.37472v1 Announce Type: cross Abstract: Behavioral tests measure how a language model reads evidence. We ask whether those measurements help choose a recommendation interface. We evaluate si
arXiv:2609.35952v1 Announce Type: cross Abstract: We introduce HEAR (Human-recorded Evaluation of Audio-LLM bias by Real speakers), a large-scale, ecologically valid benchmark comprising 87k real huma
arXiv:2609.26550v3 Announce Type: replace Abstract: LLM-as-a-judge scales evaluation, but reasoning judges are slow and costly. We study JEV-as-a-Judge: evaluation with JEV, a decision-only judge that
arXiv:2609.38070v1 Announce Type: new Abstract: As the chain-of-thought reasoning capabilities of large language models improve, evaluating and calibrating their reasoning confidence is becoming incre
arXiv:2609.36525v1 Announce Type: cross Abstract: Distributed inference has become an indispensable part of deploying medical models under practical latency, memory, and throughput constraints. Althou
arXiv:2609.37918v1 Announce Type: new Abstract: Reasoning across videos requires aligning events, matching identities, comparing motion, and integrating partial observations. Evaluating these capabili
arXiv:2609.32890v1 Announce Type: new Abstract: Language models are increasingly evaluated as vulnerability detectors, and scores reported for similar models differ widely between papers. We measured
arXiv:2609.34807v1 Announce Type: new Abstract: Spatially conditioned diffusion models can embed words and contours in natural-looking images, but vision-language models (VLMs) may fail to recognize t
arXiv:2609.35130v1 Announce Type: new Abstract: Communication-efficient federated optimization commonly spends several gradient evaluations between server updates. Existing local-update methods use th
arXiv:2609.34673v1 Announce Type: new Abstract: Few-step generative models can generate high-fidelity samples within a few function evaluations. Despite this efficiency, generated samples may not exhi
arXiv:2609.34412v1 Announce Type: new Abstract: Natural-language manipulation instructions specify qualitative relations, whereas continuous controllers require state-evaluable task quantities, differ
arXiv:2609.35052v1 Announce Type: new Abstract: Video world models must preserve the visual state of the world over time, but existing evaluation protocols often rely on generated histories, video ref
arXiv:2609.31918v1 Announce Type: new Abstract: We introduce ROTE (RollOut Testing of Exact memorization), a benchmarking protocol for evaluating symbolic memorization of neural architectures. We stud
arXiv:2609.35504v1 Announce Type: new Abstract: Machine intelligence is often evaluated through abstract reasoning problems, yet many real-world problems are visual, such as arranging objects, repairi
arXiv:2609.30489v1 Announce Type: new Abstract: Large Language Models (LLMs) have demonstrated historic breakthroughs in general reasoning with early successes in biomedical science. However, existing
arXiv:2609.30553v1 Announce Type: new Abstract: Expensive evolutionary search does not always need an exact fitness estimate for every candidate. It often needs a reliable answer to a simpler question
arXiv:2609.30722v1 Announce Type: cross Abstract: Existing roadside traffic datasets support perception, forecasting, and visual question answering, but they do not evaluate counterfactual video gener
arXiv:2609.30140v1 Announce Type: new Abstract: In this paper, we study the joint selection of an environmental support contact and a whole-body configuration for a prescribed loco-manipulation task.
arXiv:2609.29808v1 Announce Type: cross Abstract: In July 2026, an unconstrained autonomous agent participating in a frontier AI cybersecurity evaluation harness breached its evaluation sandbox, estab
arXiv:2609.28952v1 Announce Type: new Abstract: Robot-policy benchmarks increasingly cover diverse tasks and preset out-of-distribution conditions, but typically evaluate complete trajectories from pr
arXiv:2609.28974v1 Announce Type: cross Abstract: Post-training quantization (PTQ) reduces the cost of on-device text-to-speech (TTS), but published evaluations cover one system or method. We evaluate
arXiv:2609.29387v1 Announce Type: new Abstract: Supervised synthetic CT (sCT) generation is commonly trained and evaluated as voxel-wise regression against registered reference CT images. In practice,
arXiv:2609.27863v1 Announce Type: new Abstract: Knowledge graph completion is evaluated by ranking observed triples above randomly corrupted ones, which treats every unobserved fact as false. When the
arXiv:2609.27176v1 Announce Type: new Abstract: Evidence that evaluation material entered training does not reveal how much it affected evaluation. This distinction leaves a contaminated benchmark sco
arXiv:2609.28161v1 Announce Type: new Abstract: Advantage-guided reinforcement learning provides a practical way to post-train vision-language-action (VLA) policies using limited robot data. However,
arXiv:2609.27741v1 Announce Type: new Abstract: In value-based reinforcement learning, improving the accuracy of policy evaluation has been shown to improve downstream policy optimization performance.
arXiv:2609.27953v1 Announce Type: cross Abstract: Some LLM-based recommendation rerankers are evaluated under an oracle protocol that guarantees the ground-truth item is present in the scored set, eit
arXiv:2609.22329v1 Announce Type: cross Abstract: Black-Box Optimization (BBO) is often applied in several engineering fields and can utilize an advancement of numerical measure- ments and simulation
arXiv:2609.22819v1 Announce Type: new Abstract: Counterfactual tool evaluation must distinguish authority, historical support, and what a comparison actually estimates. We study these distinctions wit
arXiv:2609.24228v1 Announce Type: new Abstract: Text-to-image (T2I) models are typically evaluated for bias using slot-based templates such as ``a photo of a [profession]''. Such templates probe only
arXiv:2609.23427v1 Announce Type: new Abstract: Vision foundation models targeting Earth observation (EO) tasks are commonly evaluated on clean downstream benchmarks, but operational EO products can a
arXiv:2609.24799v1 Announce Type: new Abstract: Post-training quantization (PTQ) enables efficient deployment of large language models, and PTQ methods are usually optimized and evaluated with generic
arXiv:2609.21149v1 Announce Type: new Abstract: Before patients can use AI-assisted psychiatric intake systems, health systems need practical ways to routinely evaluate these tools against their clini
arXiv:2609.22069v1 Announce Type: new Abstract: Reference-to-video (R2V) generation is evolving toward increasingly general and versatile reference control, giving rise to the emerging paradigm of omn
arXiv:2609.21465v1 Announce Type: cross Abstract: We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly
arXiv:2609.21257v1 Announce Type: cross Abstract: Large language model (LLM) agents can propose, implement, and evaluate model changes. Autoresearch loops demonstrate this capability through minutes-s
arXiv:2609.21187v1 Announce Type: new Abstract: Agent models are frequently evaluated one decision at a time, where the model predicts the next action based on the gold interaction history, which is s
arXiv:2506.12374v3 Announce Type: replace-cross Abstract: Natural-language manipulation instructions specify the task goal but leave the underlying robot trajectory unspecified. We present AntiGroundi
It ain’t independent if you are partnering with them and chose the partner. We’re partnering with Accenture on independent evaluation of frontier AI—part of our recent commitment to embed evaluators a
arXiv:2609.18366v1 Announce Type: new Abstract: Reliable agent evaluation is complicated by automatic harness optimization, which repeatedly uses a released benchmark B_{rel} to guide a Proposer that
arXiv:2609.18909v1 Announce Type: cross Abstract: Agent benchmarks are substantially more costly to evaluate than conventional LLM benchmarks. Benchmark compression is therefore a natural solution, ye
arXiv:2609.17549v1 Announce Type: new Abstract: Cyberbullying (CB) is a complex social phenomenon characterized by repeated aggression, power imbalance, and multi-party interaction. Although large lan
arXiv:2609.18384v1 Announce Type: cross Abstract: We present a robust benchmark for measuring cultural value orientation in large language models on the two Inglehart-Welzel axes over several domains
arXiv:2609.18284v1 Announce Type: new Abstract: In recent years, three initiatives have emerged to develop generative language models in Hungary. The motivation behind them is the same. For Hungarian,
arXiv:2609.18805v1 Announce Type: cross Abstract: Coding agents are typically evaluated with desired behavior specified through issues or instructions. In practical web development, however, agents ma
arXiv:2609.16233v1 Announce Type: cross Abstract: Vision-language models excel at 2D image understanding but remain limited in 3D spatial reasoning. Progress is hindered by limitations in current benc
arXiv:2507.23248v2 Announce Type: replace Abstract: Bengali is spoken by more than 230 million people, yet no standardized instrument evaluates large language models (LLMs) on Bengali across the task