Search: “evaluation”

arXiv:2609.40198v1 Announce Type: cross Abstract: Speech-to-speech systems must solve tasks whose requirements emerge across conversational turns. We introduce SpeechConversationBench (SCB), a focused

arXiv:2609.37501v1 Announce Type: cross Abstract: We propose RegLLM, a diagnostic harness for bounded autonomy in regulated agentic workflows. It instruments six trustworthiness signals: citation vali

arXiv:2602.18998v2 Announce Type: replace Abstract: LLM agents are increasingly expected to operate as general-purpose systems that resolve real-world user requests, yet their dynamic scaling behavior

arXiv:2609.37143v1 Announce Type: cross Abstract: Modern coding agents can deliver increasingly large repository-level changes, and recent benchmarks reflect this by emphasizing long-horizon tasks wit

arXiv:2609.36253v1 Announce Type: cross Abstract: Large language models (LLMs) show considerable potential in simulating human attitudes and preferences. Prior work finds that LLM-generated responses

arXiv:2609.35900v1 Announce Type: cross Abstract: The integration of large language models (LLMs) into scientific workflows is accelerating, yet their ability to reconstruct the reasoning underlying p

arXiv:2609.31675v1 Announce Type: new Abstract: We introduce the Active Causal Discovery Benchmark (ACDB), an SCM-grounded environment for evaluating whether LLM agents recover causal graph structure

arXiv:2609.32157v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models for autonomous driving produce natural-language reasoning alongside predicted trajectories, but whether this reasoni

arXiv:2609.33807v1 Announce Type: new Abstract: How well can general-purpose multimodal models turn visual understanding and reasoning into embodied manipulation via executable code? We introduce Code

arXiv:2609.32696v1 Announce Type: cross Abstract: Data shape determines how features are structured, how patterns are separated, and how distributions cover the underlying domain. Poor data shape can

arXiv:2609.34215v1 Announce Type: cross Abstract: Evaluating how LLM agents recover from mid-task failures is central to deploying reliable agentic systems. Existing checkpoint-based benchmarks measur

Vultr is evaluating NVIDIA’s Agent Safety Platform as enterprise agents move from experimentation to production and receive real autonomy over internal systems. The platform addresses trust by securin

arXiv:2609.30981v1 Announce Type: new Abstract: Reliable online video question answering requires tracking state transitions while selectively abstaining when visual evidence is insufficient. Existing

We comprehensively evaluated 16 recent frontier VLMs - including Opus 5.5 and GPT-6 Sol/Luna - on whether higher effort led to better document parsing performance. Higher effort typically leads to imp

In September 2026, Sebastian Thrun announced **PhilosophyBench**, the first independent, large‑scale benchmark created by Stanford’s AI Lab and HCI group to evaluate artificial intelligence’s philosop

arXiv:2512.04125v2 Announce Type: replace Abstract: Large language models (LLMs) have demonstrated several emergent behaviors with scale, including reasoning and fluency in long-form text generation.

arXiv:2609.27607v1 Announce Type: cross Abstract: An AI-generated radiology report can resemble a physician's report while omitting an abnormality, adding an unsupported finding, or reversing its pres

arXiv:2609.27678v1 Announce Type: new Abstract: Contract inference requires multiple judgments about a shared document, but aggregate accuracy can conceal changes in the individual decisions. Repeated

arXiv:2609.23806v1 Announce Type: new Abstract: Many knowledge-work benchmarks are constructed around individual tasks, with the context needed for each task selected together with or after the task h

arXiv:2609.23980v1 Announce Type: cross Abstract: AI agents now report vulnerabilities faster than maintainers can review them. Reports often depend on security properties specific to the application,

arXiv:2609.23784v1 Announce Type: new Abstract: Robotic bin packing requires long-horizon sequential decision-making, as each object placement affects the available space for subsequent packing. Exist

arXiv:2609.22281v1 Announce Type: new Abstract: Foundation models have recently demonstrated strong capabilities across a wide range of medical imaging tasks. However, their performance in structured

arXiv:2609.21386v1 Announce Type: cross Abstract: Comprehensive video understanding is crucial for advancing artificial intelligence toward the intricate dynamics of the physical world. While recent a

A new release of our Agent Skills and Docs eval library, Cultivar is available! Now, use @typesafeai's Jev model to evaluate how well your agents use docs and Skill against specified tasks in Modal sa

arXiv:2609.20152v1 Announce Type: new Abstract: Generally, most voice agents are cascaded systems, i.e., an ASR model transcribes the caller's audio, a language model reads the transcript and decides

The simple truth: not even Anthropic has a plan for how to make AI safe in advance. There is no theory. I've asked! Outside evaluators don't change the situation. No one at these companies has workabl

🔥 take: @CompleteSkeptic//TypeSafe killed Quantum Computing Jev feels a bit like bringing the quantum-computing intuition to AI: evaluate a distribution of possibilities in parallel instead of generat

arXiv:2609.16093v1 Announce Type: new Abstract: A shopping conversation has many routes to the same cart, and a task-success rate reduces all of them to one score. We build a deterministic and reprodu

Cristina Criddle / Financial Times: Sources: OpenAI and Anthropic staff felt blindsided by Dario Amodei's and Sam Altman's calls to slow the frontier; some fear evaluators may compromise security — St

arXiv:2609.12839v2 Announce Type: replace-cross Abstract: The proliferation of highly capable open-weight Small Language Models (SLMs) democratizes access to advanced cybersecurity capabilities, posin

arXiv:2609.13192v1 Announce Type: new Abstract: This study compares traditional machine learning models and Large Language Model (LLM)-generated rule-based systems for heart disease prediction using t

arXiv:2512.24565v4 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly serving as autonomous agents, and their utilization of external tools via the Model Context Protocol (

arXiv:2609.15683v1 Announce Type: new Abstract: While In-Context Learning (ICL) enables models to adapt from exemplars without parameter updates, multimodal ICL remains largely underexplored, particul

Joseph Cox / 404 Media: Internal OpenAI docs detail contractors evaluating anonymized prompts and chats to improve the models; model training is turned on by default for consumer plans — Humans are re

One point that people aren't discussing re @DarioAmodei's essay. In 2016, DeepMind brought in third-party evaluators to monitor its AI health work. The result was messy. It's a cautionary tale. Gift l

Bloomberg: Amodei says pacing does not mean halting training or progress, but giving companies time to align and safeguard models and third-party evaluators time to verify — Leaders of the world's big

arXiv:2609.05437v1 Announce Type: new Abstract: Previous AI alignment efforts have focused primarily on first-order social norms -- teaching models what is socially acceptable or unacceptable (e.g., `

arXiv:2609.09788v1 Announce Type: new Abstract: Choosing when to retrain a deployed classifier requires assessing subgroup error rates across the sequence of models used, including periods between upd

arXiv:2609.05788v1 Announce Type: new Abstract: Peer-review feedback often arrives too late for authors to make meaningful revisions. We study an author-facing LLM system that moves part of this stres

arXiv:2609.07664v1 Announce Type: new Abstract: Deployment of Large Language Models (LLMs) on memory-constrained edge devices relies heavily on aggressive post-training quantization. However, evaluati

I am pleased to see that OpenAI’s new model is roughly on par with Gemini Flash and Opus 4.8 on prompt injection risk. Nice work! Evaluating and naming other labs turns out to be a great way to encour

// Evaluating and Improving LLM Self-Modeling // Really interesting paper. Can a model answer questions about its own behavior? The questions are deliberately verifiable, such as whether a particular

arXiv:2510.17460v2 Announce Type: replace Abstract: Idioms remain a persistent challenge in natural language processing due to their figurative and culturally grounded meanings, which distinguish them

arXiv:2609.01788v1 Announce Type: cross Abstract: Real-world communication often requires pragmatic reasoning: interpreting meanings implied through context and cultural convention rather than stated

arXiv:2606.12708v2 Announce Type: replace-cross Abstract: Despite their linguistic diversity and global significance, African languages remain underrepresented in research and resources to support NLP

arXiv:2609.00106v1 Announce Type: new Abstract: This paper presents FAIRY, a full-stack smart-agriculture agent system developed for and deployed to an operating soybean research farm at Harbin Instit