Search: “evaluation”

M3-Score: Fidelity, Memorization and Coverage as Separate Axes for Evaluating Generative Radiology Image Models

arXiv:2609.33769v1 Announce Type: new Abstract: Quantitative evaluation of generative models for radiology remains challenging. Clinically relevant structures are often small and infrequent, feature s

researcharxiv-cs-cv

29 Sept 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

SynDORBench: Evaluating LVLM Perceptual Robustness Under Physically Constrained Visibility Conditions

DGX agent

arXiv:2609.31823v1 Announce Type: new Abstract: Large vision-language models (LVLMs) have demonstrated remarkable performance on multimodal reasoning benchmarks, yet their perceptual reliability under

model-releasesarxiv-cs-cv

29 Sept 2026

Timeline-Bench: Evaluating Agents on Realistic Video-Editing Tasks, from Raw Footage to Final Cut

model-releasesarxiv-cs-cv

arXiv:2609.35143v1 Announce Type: new Abstract: AI agents increasingly carry out long-horizon professional work, but their evaluations rarely require a finished creative deliverable. To this end, we i

29 Sept 2026

Artificial Analysis launches the Cyber Index Alliance with partners Collinear, IBM, Nvidia, and Vercel to evaluate how AI agents find and fix vulnerabilities (Artificial Analysis)

Artificial Analysis: Artificial Analysis launches the Cyber Index Alliance with partners Collinear, IBM, Nvidia, and Vercel to evaluate how AI agents find and fix vulnerabilities — The Artificial Anal

model-releasestechmeme

Evaluating Cultural Awareness of LLMs for Haitian Creole

arXiv:2609.31506v1 Announce Type: cross Abstract: Large language models (LLMs) exhibit substantial performance disparities between high- and low-resource languages. Beyond lower task performance, they

Evaluating Sycophancy in Chinese Large Language Models on Factual Questions Derived from Online Search Queries

arXiv:2609.30986v1 Announce Type: new Abstract: As large language models increasingly mediate information access, factually accurate and independent answers are critical. However, these models can exh

Evaluating the accuracy of KV cache reuse techniques

arXiv:2609.31415v1 Announce Type: new Abstract: Position-independent KV cache reuse aims to reduce latency in retrieval-augmented generation by reusing chunk-level KV caches across prompts. We show th

researcharxiv-cs-lg

From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMs

arXiv:2505.14368v3 Announce Type: replace-cross Abstract: Recent studies demonstrate that Large Language Models (LLMs) are vulnerable to attacks that generate harmful or sensitive outputs. As open-sou

ORCA: Evaluating LLMs on Data Science Code Translation

arXiv:2609.30749v1 Announce Type: new Abstract: Data Science Code Translation (DSCT) is the process of converting code between data science libraries while preserving functional equivalence and enabli

Evaluating Explanation-Driven Vision-Language Reasoning via Generation Order Interventions

arXiv:2609.29496v1 Announce Type: new Abstract: Natural language explanation generation serves as a key mechanism for exposing and evaluating vision-language reasoning. Prior work on explanation-drive

researcharxiv-cs-cl

25 Sept 2026

Robots That Take Initiative: A Framework for Building and Evaluating Proactive Robots

arXiv:2609.28910v1 Announce Type: cross Abstract: Effective robot assistance beyond narrow roles and repetitive tasks requires robots to be proactive - to decide what needs to be done rather than wait

researcharxiv-cs-ai

25 Sept 2026

OpenAI releases MentalHealthBench, an open benchmark to evaluate AI responses in realistic mental health conversations, developed with 80+ licensed experts (OpenAI)

OpenAI: OpenAI releases MentalHealthBench, an open benchmark to evaluate AI responses in realistic mental health conversations, developed with 80+ licensed experts — An open benchmark developed with m

model-releasestechmeme

23 Sept 2026

Evaluating Fine-Tuned and Base Language Models in Maternal and Vaccination Healthcare for African Settings

arXiv:2609.22110v1 Announce Type: new Abstract: Background: Large language models (LLMs) can improve healthcare information delivery in low-resource settings but may produce inaccurate or culturally i

Evaluating Personal Information Output from Conversational Interactions in Generative AI Systems

arXiv:2609.22204v1 Announce Type: new Abstract: This exploratory pilot study evaluates the scope and perceived accuracy of personal information output from ongoing conversational interactions in gener

H2RBench: A Real-to-Sim Benchmark for Evaluating Human-to-Robot Transfer

model-releasesarxiv-cs-ro

arXiv:2609.24778v1 Announce Type: new Abstract: Learning robot manipulation policies from human video demonstrations constitutes a promising avenue for scalable robot learning. However, comparing diff

VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks

arXiv:2609.23570v1 Announce Type: cross Abstract: Coding agents operate on real repository coding tasks, and persistent memory systems promise to reuse experience across tasks. Yet existing evaluation

QuranicMMLU: A Cognitively-Aware Benchmark for Evaluating Generative AI Solutions on Quranic Linguistic Knowledge

arXiv:2609.22038v1 Announce Type: new Abstract: We introduce QuranicMMLU, a benchmark for evaluating generative AI on Quranic Arabic across multiple dimensions of linguistic complexity. Existing Quran

21 Sept 2026

Why decontamination reports can't fix benchmark contamination, and what an evaluator has to do instead [D]

model-releasesr-machinelearning

In February OpenAI stopped reporting SWE-bench Verified and recommended other labs stop too. Every frontier model they tested could reproduce the human-written reference fix, or verbatim details of th

20 Sept 2026

Industry

Anthropic partners with Accenture to embed evaluators within Anthropic; they expect to invest $2B+ in building capacity in this area over the next five years (Anthropic)

Anthropic: Anthropic partners with Accenture to embed evaluators within Anthropic; they expect to invest $2B+ in building capacity in this area over the next five years — We're partnering with Accentu

industrytechmeme

18 Sept 2026

Evaluating Communicative Success in Machine-Translated Conversation

arXiv:2609.19885v1 Announce Type: new Abstract: Interpreter agents built on machine translation (MT) increasingly mediate live conversation between people who do not share a language, yet we still eva

18 Sept 2026

Legal LLM Hallucination Should Be Evaluated as Failure of Legal Warrant

arXiv:2609.17546v1 Announce Type: new Abstract: In this position paper, we argue that legal LLMs' hallucinations should be evaluated as a failure of legal warrant rather than as factual inaccuracy or

17 Sept 2026

Rethinking How We Evaluate Methodological Progress in Health AI

arXiv:2609.18134v1 Announce Type: cross Abstract: Methodological progress in artificial intelligence (AI) for electronic health records (EHRs) depends on our ability to determine which algorithms work

researcharxiv-cs-ai

17 Sept 2026

AquiLLM: Evaluating Faithfulness in Open-Weight RAG-LLM Systems for Scientific Research

arXiv:2609.16519v1 Announce Type: new Abstract: Scientific research increasingly relies on large, heterogeneous data sources, motivating interest in retrieval-augmented generation (RAG) systems that p

16 Sept 2026

Crypto Accounting Bench: Evaluating Frontier and Open-Weight Models on Crypto-Asset Accounting Tasks

arXiv:2609.14811v1 Announce Type: new Abstract: We introduce Crypto Accounting Bench (CAB), a benchmark for assessing whether frontier and open-weight language models can reconstruct the complete jour

Does Reasoning Improve Psychological Depth in Large Language Models? It Depends on Who's Judging

arXiv:2609.13773v1 Announce Type: cross Abstract: LLM-as-a-Judge evaluators are increasingly used to score open-ended generation, yet a judge's correlation with human ratings on its development set ma

From Advertised Improvements to Measured Capabilities: Evaluating ChatGPT Images 2.5 on Forgery Tasks

arXiv:2609.13617v1 Announce Type: new Abstract: We evaluate whether the improvements advertised for ChatGPT Images 2.5 translate into better performance on forgery tasks with predetermined answers. We

researcharxiv-cs-cv

K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations

arXiv:2609.15855v1 Announce Type: cross Abstract: % !TEX root = ../main.tex People increasingly use large language models (LLMs) for mental health support, yet their safety in evolving, high-risk conv

Counterfactual Marginalisation: Framework for Evaluating Robustness to Nuisance Variables

arXiv:2609.10778v1 Announce Type: new Abstract: Machine learning models can achieve strong test performance while relying on demographic or acquisition-related shortcuts. We propose counterfactual (CF

researcharxiv-cs-lg

11 Sept 2026

Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering

researchapple-ml-research

Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generated text against ground-truth references. This para

11 Sept 2026

Can Foundation Models Moderate Online Content? Evaluating Instruction- vs. Example-Driven Policy Operationalization

arXiv:2609.10410v1 Announce Type: new Abstract: The growing complexity of content moderation policies presents a critical challenge for their consistent operationalization. While foundation models pos

10 Sept 2026

Safety

Evaluating Steering Techniques using Human Similarity Judgments

arXiv:2505.19333v2 Announce Type: replace Abstract: Current evaluations of Large Language Model (LLM) steering techniques focus on task-specific performance, overlooking how well steered representatio

safetyarxiv-cs-ai

10 Sept 2026

Safety

California Gov. Gavin Newsom signs into law two bills, backed by Anthropic and OpenAI, regulating how outside groups evaluate AI for safety (Christine Mui/Politico)

Christine Mui / Politico: California Gov. Gavin Newsom signs into law two bills, backed by Anthropic and OpenAI, regulating how outside groups evaluate AI for safety — SAN FRANCISCO — California Gov.

safetytechmeme

9 Sept 2026

Applications

Machine learning doesn't end when the model is trained. You still need to evaluate it, deploy it, and build the infrastructure to keep it wo…

applicationsfrancois-chollet--x

Machine learning doesn't end when the model is trained. You still need to evaluate it, deploy it, and build the infrastructure to keep it working in production. Today's Deal of the Day takes you acros

9 Sept 2026

Monkey See, Can Monkey Do? A Benchmark for Evaluating Robot Skill Learning by Observation

model-releasesarxiv-cs-ro

arXiv:2609.08209v1 Announce Type: new Abstract: Learning from Observation (LfO) is a fundamental robotic capability that replicates how humans and animals socially learn from each other. Beyond its bi

9 Sept 2026

How KDDI built Buffmee, a faster, reliable consumer RAG app

model-releasesgoogle-cloud-ai

When building consumer-facing generative AI applications, balancing high generation quality with fast response times across diverse media types, can be challenging. KDDI, a major telecommunications ca

8 Sept 2026

Building and Evaluating Fixed-Voice Thai TTS from Synthetic Speech

arXiv:2609.03502v1 Announce Type: cross Abstract: In low-resource settings, deploying TTS typically requires choosing between a large voice-cloning model with costly inference or a compact fixed-voice

Agents

Does paying 5x more per page actually get you better document extraction? We ran the data to find out. We evaluated 14 frontier systems acro…

Does paying 5x more per page actually get you better document extraction? We ran the data to find out. We evaluated 14 frontier systems across 370 enterprise documents, then plotted their accuracy aga

agentsjerry-liu--x

KC-Bench: A Dynamic Interactive Benchmark for Evaluating Knowledge Conflicts in LLM Agents

arXiv:2609.03588v1 Announce Type: new Abstract: As LLMs increasingly act through tools, they must reconcile user instructions, parametric knowledge, and dynamic environmental observations before takin

VoxPrivacy: A Benchmark for Evaluating Interactional Privacy of Speech Language Models

arXiv:2601.19956v2 Announce Type: replace-cross Abstract: As Speech Language Models (SLMs) transition from personal devices to shared, multi-user environments such as smart homes, a new challenge emer

Counter-GEO-Bench: Evaluating Defenses Against Information-Distorting Generative Engine Optimization

arXiv:2609.02316v1 Announce Type: cross Abstract: Generative engine optimization (GEO) enables content producers to increase the visibility of their web pages in generative search engines, but the sam

3 Sept 2026

Safety

When Prediction Error Is Not Enough: Evaluating Nuisance-Function Prediction for Causal Estimation

arXiv:2609.00071v1 Announce Type: new Abstract: Prediction error is widely used to evaluate nuisance-function estimators in causal inference, but its relationship with causal estimator performance may

safetyarxiv-cs-ai

2 Sept 2026

Evaluating the Effects of Inter-Observer and Model Variability on Radiological Peritoneal Cancer Index Assessment

arXiv:2608.28716v1 Announce Type: cross Abstract: Deep learning segmentation models are often evaluated using geometric metrics such as Dice, HD95, and ASD, yet it remains unclear to what extent impro

researcharxiv-cs-cv

GeoAgent: Evaluating VLM Geolocalization Through Embodied Navigation

arXiv:2608.29483v1 Announce Type: cross Abstract: Modern Vision-Language Models (VLMs) perform well above the human baseline in image geolocalization, a task critically important in disaster response,

OCR-MetaReasoning Benchmark: Evaluating the Meta-Reasoning Ability of MLLMs in Text-Rich Image Understanding

arXiv:2608.30678v1 Announce Type: new Abstract: Text-rich image understanding requires multimodal large language models (MLLMs) to organize OCR (Optical Character Recognition)-grounded evidence across

OmniACBench: A Benchmark for Evaluating Context-Grounded Acoustic Control in Omni-Modal Models

arXiv:2603.23938v2 Announce Type: replace Abstract: Most testbeds for omni-modal models assess multimodal understanding via textual outputs, leaving it unclear whether these models can properly speak

PrivBench: A Holistic and Modular Benchmarking Platform for Evaluating Text-to-Text Privatization

arXiv:2608.29624v1 Announce Type: new Abstract: Natural Language Processing methods have enabled novel solutions and advances in the field of privacy, particularly in the sub-domain of text-to-text pr

Tutorials

Toward Scalable Audio Description Quality Control: A Workflow for Evaluating Human and VLM Raters

arXiv:2602.01390v3 Announce Type: replace-cross Abstract: Digital video is central to communication, education, and entertainment, but without audio description (AD), blind and low-vision users are ex

tutorialsarxiv-cs-ai

Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretation

arXiv:2608.29571v1 Announce Type: new Abstract: Flexible adaptation to context and shared pragmatic intuitions contribute to smooth human conversation. Iterated reference games---in which players repe

researcharxiv-cs-cl