Search: “evaluation”

arXiv:2609.17435v1 Announce Type: new Abstract: We submit MeTRON-FR, a 125M GPT-2 pretrained on 92.47M words of French, to the BabyLM 2026 Strict track. It scores 85.97 +/- 0.17% on QFrBLiMP (a native

arXiv:2609.15315v1 Announce Type: new Abstract: Safe reinforcement learning (RL) is commonly formalized as a Constrained Markov Decision Process (CMDP), in which an agent maximizes expected reward whi

arXiv:2609.13250v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) cannot process every frame of a long video because of limitations in visual-token and computational budgets.

arXiv:2609.13725v1 Announce Type: new Abstract: An external record may contain a procedure to apply or text to read, depending on the user's request. IBBench-Light tests both uses against the same rec

arXiv:2609.13611v1 Announce Type: new Abstract: The WMT26 General MT task evaluates systems on 10 language pairs that have no human references (neither translated from scratch nor post-edited from MT

arXiv:2609.13918v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used in specialized university courses, but control-systems questions require coordinated terminology, not

arXiv:2503.06868v2 Announce Type: replace-cross Abstract: Existing long-text generation methods produce lengthy outputs from short inputs, leaving long-input-to-long-output generation underexplored. A

arXiv:2609.14976v1 Announce Type: new Abstract: Long-horizon LLM agents accumulate memory across sessions, creating sparse but high-impact risks: stale facts, conflicting updates, cross-user leakage,

arXiv:2609.12349v1 Announce Type: new Abstract: Socially assistive robots (SARs) can support structured health and well-being interventions, but hardware and cost constraints limit interaction complex

arXiv:2609.10952v1 Announce Type: cross Abstract: Data poisoning corrupts training data to degrade a model or to plant attacker-controlled behavior. This study evaluates two representative training-ti

arXiv:2609.11131v1 Announce Type: new Abstract: Human simultaneous interpreting (SI) is commonly assessed with analytic rubrics separating meaning transfer, delivery quality, and temporal synchrony, y

arXiv:2609.05910v1 Announce Type: cross Abstract: Reinforcement learning (RL) excels on tasks with verifiable rewards, but in open-ended tasks, the reliability of reward models remains a key challenge

arXiv:2609.05783v1 Announce Type: cross Abstract: In autonomous driving, Bird's-Eye View (BEV) representations provide a structured, top-down abstraction of the vehicle's surroundings and have become

arXiv:2609.04409v1 Announce Type: cross Abstract: Multilingual language models often produce inconsistent answers to semantically equivalent questions across languages, motivating methods to improve c

I wanted a quick calories counter for myself, using LLMs to evaluate the calories from pictures of meals + descriptions. I needed to pick a model so I made a quick benchmark. The setup was: - Nutritio

arXiv:2609.04391v1 Announce Type: new Abstract: In this work, a novel evaluation scheme built on a generalized variant of the Rand Index measure, namely, the Hullermeier-Rifqi Index, is proposed in or

arXiv:2609.04286v1 Announce Type: new Abstract: Artificial intelligence in recruitment has shifted the object being automated from profile pairs and ranked lists to multi-stage workflows that retrieve

Really strong benchmark paper on coding agents. Claude Opus 5 running under Claude Code passes 23.9% of the evaluations. An expert human reference scores 82.2%. Here is what the task actually is. The

Emily Forlini / Fortune: OpenAI quietly updates its evaluation metrics for GPT-6 Astra, making changes that appear to favor Astra and continuing to revise other metrics after launch — OpenAI has chang

arXiv:2609.03727v1 Announce Type: new Abstract: Large language model agents can plan, invoke tools, and modify external states, yet most systems still take an explicit user instruction as a fixed star

arXiv:2609.02990v1 Announce Type: cross Abstract: To scale up collective decision-making, participatory democracy platforms such as Polis and Remesh enable online deliberation among thousands of parti

arXiv:2609.03153v1 Announce Type: new Abstract: Visual fluency in generated video does not imply physical reliability, and a scalar quality score alone is incapable of indicating the obligation a clip

arXiv:2609.01617v1 Announce Type: cross Abstract: Getting accurate, grounded answers out of large enterprise document repositories is a difficult problem. Dense vector retrieval alone frequently perfo

arXiv:2609.02000v1 Announce Type: new Abstract: Vision-language models (VLMs) are increasingly evaluated on complex image and video understanding tasks, yet conventional metrics primarily assess final

arXiv:2507.12674v3 Announce Type: replace-cross Abstract: Evaluating Artificial Intelligence (AI) tutor feedback before deployment requires anticipating student engagement, typically assessed through

arXiv:2608.30114v1 Announce Type: new Abstract: Brazilian Portuguese remains under-served by open language models, and the few that exist are difficult to reproduce and are often compared without meas

arXiv:2601.16753v2 Announce Type: replace-cross Abstract: Longitudinal information in radiology reports refers to the sequential tracking of findings across multiple examinations over time, which is c

arXiv:2608.28708v1 Announce Type: cross Abstract: Prospective silent trials provide an important bridge between retrospective validation of artificial intelligence (AI) models and their use in clinica

Anthropic: Anthropic details security efforts following Claude cyber evaluation incidents, including a weeks-long pause on higher-risk RL and work to curb reward hacking — On July 30, we reported thre

arXiv:2608.27831v1 Announce Type: cross Abstract: Coding agents are now commonly evaluated on the SWE-bench family of benchmarks, whose tasks are built from curated GitHub issues--long, structured, an

arXiv:2606.16113v2 Announce Type: replace-cross Abstract: Algorithmic recourse methods provide counterfactual explanations that inform individuals of the actions required to overturn an unfavorable mo

arXiv:2606.17092v2 Announce Type: replace-cross Abstract: Agentic systems are increasingly integrated with geographic information systems (GIS), where multi-agent coordination enables complex conversa

arXiv:2608.26389v1 Announce Type: new Abstract: SVD-based low-rank compression has become a fast-growing direction for reducing the memory and computational cost of large language models (LLMs). Howev

arXiv:2608.25869v1 Announce Type: new Abstract: Large language models (LLMs) increasingly assess generated content, giving rise to the LLM-as-a-Judge paradigm. These systems now score outputs, filter

arXiv:2608.23568v1 Announce Type: new Abstract: Memory and RAG evaluations often treat the answering model's input as an implementation detail, even though systems may render the same history as a mem

arXiv:2608.24881v1 Announce Type: cross Abstract: Generative models are commonly ranked by Frechet Inception Distance (FID) and Kernel Inception Distance (KID), yet FID's first-two-moment summary can

arXiv:2608.21558v1 Announce Type: cross Abstract: Recent advances in LLMs and the adoption of RAG systems in industry have created a need for domain-specific question-answer datasets that can assess R

arXiv:2509.05346v3 Announce Type: replace Abstract: While large language models (LLMs) are increasingly being adopted to support personalized learning, there remains limited understanding of how their

arXiv:2608.21854v1 Announce Type: new Abstract: Weakly supervised video anomaly detectors are trained with video-level labels but are commonly evaluated as temporal localizers using Micro-AUROC or AP

arXiv:2608.22124v1 Announce Type: new Abstract: LLM-assisted writing is often treated as a detection problem, as it raises questions about clarity, integrity, equity, and evaluation. An analysis of 69

arXiv:2508.07173v3 Announce Type: replace Abstract: Omni-modal Large Language Models (OLLMs) that integrate visual, auditory, and textual processing face severe safety risks. They exhibit fragile defe

arXiv:2608.22595v1 Announce Type: cross Abstract: We develop a new framework for flexible, nonlinear, and interpretable off-policy evaluation for infinite-horizon reinforcement learning. To handle lar

arXiv:2608.20432v1 Announce Type: cross Abstract: Formal proofs in Lean 4 that pass the kernel's type checker can nonetheless vary widely in quality. We introduce ProofJudge, an agentic LLM-as-judge s

arXiv:2608.20597v1 Announce Type: cross Abstract: Agentic AI systems are being procured for military command and control (C2) under public commitments to rigorous testing and human oversight. Whether

arXiv:2608.20106v1 Announce Type: new Abstract: We introduce OenoBench, a wine-domain knowledge benchmark of 3,266 multiple-choice questions across six pillars (regions, grape varieties, viticulture,

arXiv:2608.19564v1 Announce Type: new Abstract: Persistent memory can personalize an LLM agent, but an incorrect durable update can silently distort future behavior. We study the memory-clarification