Search: “evaluation”

If you maintain an AGENTS.md or a CLAUDE.md, this is worth a read. (bookmark it) 288 gold-test evaluated runs across Claude Code and Codex, 17 real tasks from 3 repositories, with context-injection st

arXiv:2607.28119v1 Announce Type: new Abstract: In this paper, we draw a comparison between linguists in training, a trained linguist, and annotations generated by large language models (LLMs) to find

arXiv:2607.19321v2 Announce Type: replace-cross Abstract: As AI agents begin to automate AI R&D, we need ways to assess whether their outputs are safe to deploy, even when the agents themselves may be

arXiv:2607.25485v1 Announce Type: new Abstract: Health AI is evolving from answering questions to agentic systems that converse with patients, reason about health records, and act on their behalf. Pri

arXiv:2607.22740v1 Announce Type: new Abstract: Weakly supervised pipelines for medical imaging have become increasingly popular over the years. These systems often include multiple stages and compone

arXiv:2607.20712v1 Announce Type: cross Abstract: Security protocol verification relies on formal tools such as ProVerif and OFMC. This study evaluates whether Large Language Models (LLMs) can perform

arXiv:2607.22609v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly used for decision-making and reasoning tasks, yet their potential as controllers for physical systems rema

arXiv:2607.22554v1 Announce Type: new Abstract: Large language models (LLMs) often achieve strong accuracy on benchmarks, yet it remains unclear how reliably they apply this knowledge when the same qu

arXiv:2510.24668v2 Announce Type: replace Abstract: Language agents have demonstrated remarkable potential in web search and information retrieval. However, many search-agent benchmarks assume that us

One of the least explored questions in AI is how to build rich open-ended environments for learning. Benchmarks evaluate intelligence, environments shape it. iLands is interesting because it doesn't j

Really interesting concept -- instead of optimizing against static benchmarks or internally generated reward models, agents are evaluated through real economic interactions. Using an external market a

We are releasing PerceptionBench, a benchmark that isolates visual perception and evaluates it as a set of atomic capabilities - discovered from how today's models fail, rather than defined in advance

arXiv:2607.20462v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly integrated into clinical workflows, stressing the need for reliable traceability of model-generated output

arXiv:2607.21072v1 Announce Type: new Abstract: Spatial intelligence is essential for agents to move from static semantic understanding toward interacting with the physical world. Many spatial tasks a

Together, Motorway and AWS built an end-to-end evaluation pipeline that reduced incorrect results from 1 in 8 queries to 1 in 50 and cut issue detection time from few hours to few minutes. The pipelin

arXiv:2607.13411v1 Announce Type: cross Abstract: Clinical AI models can expose patients to harm when adversarial vulnerabilities go undetected, yet formal security auditing requires statistical exper

arXiv:2607.13041v1 Announce Type: cross Abstract: Large Language Model (LLM) based AI educational content generation systems are increasingly being developed, yet no standardised benchmark exists to s

arXiv:2603.04710v2 Announce Type: replace-cross Abstract: Recent advances in automatic speech recognition (ASR) and speech enhancement have strengthened the common belief that cleaner audio should lea

arXiv:2602.17659v2 Announce Type: replace Abstract: Vision-Language-Action models (VLAs) promise to ground language instructions in robot control, yet in practice often fail to faithfully follow langu

Use Arize Skills with Kiro CLI to trace coding-agent changes, build datasets from failures, run experiments, and validate prompts before shipping. The post Kiro CLI observability: trace and evaluate a

Local AI is the future. Learning how to run Opensource models (Inference), how to evaluate them systematically (Evals), and how to customize them (Fine-tuning / RL / Post-training) are invaluable skil

Proactive agents that anticipate user needs and autonomously execute tasks hold great promise as digital assistants, yet the lack of realistic user simulation frameworks hinders their development. Exi

congrats on the launch! As the task horizons on agents gets longer, we need more work evaluating and training models to be better in long-running, open-ended, evolving real-world environments. Today w

Grok 4.5 is now available as an orchestrator model in Computer for Consumer Pro and Max subscribers. We evaluated it against five other orchestrator configurations on WANDR. It scored higher than ever

arXiv:2607.08423v1 Announce Type: new Abstract: The rapid integration of Large Vision-Language Models (VLMs) into critical infrastructure promises to revolutionize personalized healthcare and dietary

A comprehensive evaluation of over 30 frontier embodied AI models reveals that current generalist robot policies lack the robustness needed for reliable real-world manipulation tasks. The analysis dem

Vercel announced a significant performance improvement to its Flags SDK, enabling flag evaluations to run 10 times faster than the previous version. This enhancement improves the speed of feature flag

This guide provides criteria and best practices for assessing enterprise analytics platforms, likely covering key evaluation factors such as scalability, performance, integration capabilities, ease of

LeRobot v0.6.0 is a release from Hugging Face that introduces new capabilities for robotic learning, likely including features for imagining/generating robot behaviors, evaluating model performance, a

arXiv:2607.01502v1 Announce Type: new Abstract: Recent advances in automatic speech recognition (ASR) have explored different sequence models, including Conformer-based models and newer state space mo

arXiv:2607.02513v1 Announce Type: cross Abstract: LLMs memorize sensitive training data, including personally identifiable information (PII), creating a pressing need for reliable post hoc removal met

arXiv:2601.01095v3 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) have achieved impressive progress in vision-language reasoning, yet their ability to understand tempo

arXiv:2606.31169v1 Announce Type: new Abstract: Existing AI-assisted oracle bone inscription (OBI) visual recognition and understanding studies mainly focus on character-level, ignoring the long-form

arXiv:2606.32034v1 Announce Type: cross Abstract: LLM agents increasingly act over long horizons, where a single trajectory can contain hundreds or thousands of actions. In these settings, outcome-onl

arXiv:2606.29689v1 Announce Type: new Abstract: Open-ended aesthetic critique is a challenge for multimodal large language models (MLLMs): unlike multiple-choice aesthetic benchmarks, it has no single

This post likely discusses how to evaluate AI agents built with LangChain using Harbor, a tool for testing and monitoring. It covers practical approaches for assessing agent performance, reliability,

arXiv:2504.14969v2 Announce Type: replace Abstract: This paper proposes a framework for evaluating large language models (LLMs) on Chinese topic constructions, focusing on their sensitivity to island

arXiv:2504.02918v3 Announce Type: replace Abstract: Recent advances in image and video generation raise hopes that these models possess world modeling capabilities-the ability to generate realistic, p

arXiv:2606.15032v2 Announce Type: replace Abstract: World models have become a central abstraction in modern AI. The term now refers to several different objects: action-conditioned environment models

arXiv:2606.27061v1 Announce Type: new Abstract: External indexes can be used for cluster evaluation when ground truth is available. We review the most common external validity indexes focusing on set-

arXiv:2601.13300v2 Announce Type: replace Abstract: Benchmarking large language models (LLMs) is critical for understanding their capabilities, limitations, and robustness. In addition to interface ar

arXiv:2606.26350v1 Announce Type: new Abstract: Although large language model agents are increasingly applied to quantitative-finance workflows, their evaluation remains fragmented across isolated tas

arXiv:2606.26041v1 Announce Type: cross Abstract: Vision-language models (VLMs) have achieved strong performance on OCR-based benchmarks and increasingly focused on text-rich understanding, but their

arXiv:2606.19157v2 Announce Type: replace-cross Abstract: AudioLLMs enable speech recognition conditioned on textual prompts such as domain descriptions or entity lists. However, it remains unclear wh

Marina Temkin / TechCrunch: Patronus AI, which builds simulated digital environments for evaluating AI agents, raised a 50M Series B led by Greenfield, bringing its total funding to 70M — AI agents ar

arXiv:2508.05762v3 Announce Type: replace-cross Abstract: Universal machine learning force fields (UMLFFs) promise to revolutionize materials science by enabling rapid atomistic simulations across the