Search: “evaluation”

LOGICAL-COMMONSENSEQA: A Benchmark for Logical Commonsense Reasoning

arXiv:2601.16504v3 Announce Type: replace Abstract: Commonsense reasoning often involves evaluating multiple plausible interpretations rather than selecting a single atomic answer, yet most benchmarks

21 Apr 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Measuring Representation Robustness in Large Language Models for Geometry

DGX agent

arXiv:2604.16421v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly evaluated on mathematical reasoning, yet their robustness to equivalent problem representations remains po

model-releasesarxiv-cs-cl

21 Apr 2026

Applications

Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measure

arXiv:2412.15176v3 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly employed in real-world applications, driving the need to evaluate the trustworthiness of their generat

applicationsarxiv-cs-lg

21 Apr 2026

The impact of postediting on AI generative translation in Yemeni context: Translating literary prose by ChatGPT

arXiv:2604.16704v1 Announce Type: new Abstract: This study examines the role of artificial intelligence in translation, focusing on ChatGPT, specifically ChatGPT-4, and the extent to which human poste

researcharxiv-cs-cl

21 Apr 2026

Import AI 454: Automating alignment research; safety study of a Chinese model; HiFloat4

This newsletter covers three main topics: advances in automating alignment research to improve AI safety processes, a safety evaluation study of a Chinese AI model, and technical details about HiFloat

safetyimport-ai

20 Apr 2026

Agents

LangSmith Insights Agent helps Credit Genie interpret feedback from customers at scale, and continuously improve their product. https://www.…

LangSmith Insights Agent helps Credit Genie interpret feedback from customers at scale, and continuously improve their product. https://www.langchain.com/blog/credit-genie-insights-agent-financial-ass

agentsharrison-chase--x

20 Apr 2026

Agents

some interesting trends in terms of LLM adoption 👀

some interesting trends in terms of LLM adoption 👀 📊 The latest from the LangSmith Signal: We looked at API call volume and developer adoption trends across LLM providers using LangChain open source t

agentsharrison-chase--x

20 Apr 2026

ADAPT: Benchmarking Commonsense Planning under Unspecified Affordance Constraints

arXiv:2604.14902v1 Announce Type: cross Abstract: Intelligent embodied agents should not simply follow instructions, as real-world environments often involve unexpected conditions and exceptions. Howe

CaptionQA: Is Your Caption as Useful as the Image Itself?

arXiv:2511.21025v2 Announce Type: replace Abstract: Image captions serve as efficient surrogates for visual content in multimodal systems such as retrieval, recommendation, and multi-step agentic infe

CoPA: Benchmarking Personalized Question Answering with Data-Informed Cognitive Factors

arXiv:2604.14773v1 Announce Type: new Abstract: While LLMs have demonstrated remarkable potential in Question Answering (QA), evaluating personalization remains a critical bottleneck. Existing paradig

Create Expert Content: Deploying a Multi-Agent System with Terraform and Cloud Run

model-releasesgoogle-cloud-ai

In support of our mission to accelerate the developer journey on Google Cloud, we built Dev Signal: a multi-agent system designed to transform raw community signals into reliable technical guidance by

MEBench: A Novel Benchmark for Understanding Mutual Exclusivity Bias in Vision-Language Models

arXiv:2505.20122v2 Announce Type: replace Abstract: This paper introduces MEBench, a novel benchmark for evaluating mutual exclusivity (ME) bias, a cognitive phenomenon observed in children during wor

ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaboration

arXiv:2509.21823v2 Announce Type: replace Abstract: Reward is critical to the evaluation and training of large language models (LLMs). However, existing rule-based or model-based reward methods strugg

safetyarxiv-cs-ai

ReasonScaffold: A Scaffolded Reasoning-based Annotation Protocol for Human-AI Co-Annotation

arXiv:2603.21094v3 Announce Type: replace Abstract: Human annotation is central to NLP evaluation, yet subjective tasks often exhibit substantial variability across annotators. While large language mo

researcharxiv-cs-cl

RL-STPA: Adapting System-Theoretic Hazard Analysis for Safety-Critical Reinforcement Learning

arXiv:2604.15201v1 Announce Type: new Abstract: As reinforcement learning (RL) deployments expand into safety-critical domains, existing evaluation methods fail to systematically identify hazards aris

safetyarxiv-cs-lg

C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences

arXiv:2604.13618v1 Announce Type: new Abstract: Rubric-augmented verification guides reward models with explicit evaluation criteria, yielding more reliable judgments than single-model verification. H

researcharxiv-cs-cl

Chain of Uncertain Rewards with Large Language Models for Reinforcement Learning

arXiv:2604.13504v1 Announce Type: cross Abstract: Designing effective reward functions is a cornerstone of reinforcement learning (RL), yet it remains a challenging and labor-intensive process due to

Hybrid Retrieval for COVID-19 Literature: Comparing Rank Fusion and Projection Fusion with Diversity Reranking

arXiv:2604.13728v1 Announce Type: cross Abstract: We present a hybrid retrieval system for COVID-19 scientific literature, evaluated on the TREC-COVID benchmark (171,332 papers, 50 expert queries). Th

Synthetic Tabular Generators Fail to Preserve Behavioral Fraud Patterns: A Benchmark on Temporal, Velocity, and Multi-Account Signals

model-releasesarxiv-cs-lg

arXiv:2604.13125v1 Announce Type: new Abstract: We introduce behavioral fidelity -- a third evaluation dimension for synthetic tabular data that measures whether generated data preserves the temporal,

A Sanity Check on Composed Image Retrieval

arXiv:2604.12904v1 Announce Type: new Abstract: Composed Image Retrieval (CIR) aims to retrieve a target image based on a query composed of a reference image, and a relative caption that specifies the

Benchmarking Deflection and Hallucination in Large Vision-Language Models

arXiv:2604.12033v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) increasingly rely on retrieval to answer knowledge-intensive multimodal questions. Existing benchmarks overlook c

Beyond Perception Errors: Semantic Fixation in Large Vision-Language Models

arXiv:2604.12119v1 Announce Type: new Abstract: Large vision-language models (VLMs) often rely on familiar semantic priors, but existing evaluations do not cleanly separate perception failures from ru

Does Visual Token Pruning Improve Calibration? An Empirical Study on Confidence in MLLMs

arXiv:2604.12035v1 Announce Type: new Abstract: Visual token pruning is a widely used strategy for efficient inference in multimodal large language models (MLLMs), but existing work mainly evaluates i

researcharxiv-cs-cv

GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scripts

arXiv:2604.12978v1 Announce Type: new Abstract: Optical character recognition (OCR) has advanced rapidly with the rise of vision-language models, yet evaluation has remained concentrated on a small cl

MAML-KT: Addressing Cold Start Problem in Knowledge Tracing for New Students via Few-Shot Model-Agnostic Meta Learning

arXiv:2603.00137v2 Announce Type: replace-cross Abstract: Knowledge tracing (KT) models are commonly evaluated by training on early interactions from all students and testing on later responses. While

researcharxiv-cs-ai

A Metamorphic Testing Perspective on Knowledge Distillation for Language Models of Code: Does the Student Deeply Mimic the Teacher?

arXiv:2511.05476v4 Announce Type: replace-cross Abstract: Transformer-based language models of code have achieved state-of-the-art performance across a wide range of software analytics tasks, but thei

researcharxiv-cs-lg

Beyond Compliance: A Resistance-Informed Motivation Reasoning Framework for Challenging Psychological Client Simulation

arXiv:2604.10507v1 Announce Type: new Abstract: Psychological client simulators have emerged as a scalable solution for training and evaluating counselor trainees and psychological LLMs. Yet existing

safetyarxiv-cs-ai

BlasBench: An Open Benchmark for Irish Speech Recognition

arXiv:2604.10736v1 Announce Type: new Abstract: No open Irish-specific benchmark compares end-user ASR systems under a shared Irish-aware evaluation protocol. To solve this, we release BlasBench, an o

Both Ends Count! Just How Good are LLM Agents at 'Text-to-Big SQL'?

arXiv:2602.21480v4 Announce Type: replace-cross Abstract: Text-to-SQL and Big Data are both extensively benchmarked fields, yet there is limited research that evaluates them jointly. In the real world

COMPOSITE-Stem

arXiv:2604.09836v1 Announce Type: new Abstract: AI agents hold growing promise for accelerating scientific discovery; yet, a lack of frontier evaluations hinders adoption into real workflows. Expert-w

Edu-MMBias: A Three-Tier Multimodal Benchmark for Auditing Social Bias in Vision-Language Models under Educational Contexts

arXiv:2604.10200v1 Announce Type: new Abstract: As Vision-Language Models (VLMs) become integral to educational decision-making, ensuring their fairness is paramount. However, current text-centric eva

EmbodiedGovBench: A Benchmark for Governance, Recovery, and Upgrade Safety in Embodied Agent Systems

arXiv:2604.11174v1 Announce Type: cross Abstract: Recent progress in embodied AI has produced a growing ecosystem of robot policies, foundation models, and modular runtimes. However, current evaluatio

GLEaN: A Text-to-image Bias Detection Approach for Public Comprehension

arXiv:2604.09923v1 Announce Type: new Abstract: Text-to-image (T2I) models, and their encoded biases, increasingly shape the visual media the public encounters. While researchers have produced a rich

safetyarxiv-cs-ai

Hodoscope: Unsupervised Monitoring for AI Misbehaviors

arXiv:2604.11072v1 Announce Type: new Abstract: Existing approaches to monitoring AI agents rely on supervised evaluation: human-written rules or LLM-based judges that check for known failure modes. H

Local Ai

Limited Perfect Monotonical Surrogates constructed using low-cost recursive linkage discovery with guaranteed output

arXiv:2604.11524v1 Announce Type: new Abstract: Surrogates provide a cheap solution evaluation and offer significant leverage for optimizing computationally expensive problems. Usually, surrogates onl

local-aiarxiv-cs-ai

M2-Verify: A Large-Scale Multidomain Benchmark for Checking Multimodal Claim Consistency

arXiv:2604.01306v2 Announce Type: replace Abstract: Evaluating scientific arguments requires assessing the strict consistency between a claim and its underlying multimodal evidence. However, existing

MERMAID: Memory-Enhanced Retrieval and Reasoning with Multi-Agent Iterative Knowledge Grounding for Veracity Assessment

arXiv:2601.22361v2 Announce Type: replace-cross Abstract: Assessing the veracity of online content has become increasingly critical. Large language models (LLMs) have recently enabled substantial prog

Optimization-Guided Diffusion for Interactive Scene Generation

arXiv:2512.07661v3 Announce Type: replace Abstract: Realistic and diverse multi-agent driving scenes are crucial for evaluating autonomous vehicles, but safety-critical events which are essential for

safetyarxiv-cs-cv

QFS-Composer: Query-focused summarization pipeline for less resourced languages

arXiv:2604.10687v1 Announce Type: new Abstract: Large language models (LLMs) demonstrate strong performance in text summarization, yet their effectiveness drops significantly across languages with res

safetyarxiv-cs-cl

Retrieval-Augmented Large Language Models for Evidence-Informed Guidance on Cannabidiol Use in Older Adults

arXiv:2604.09548v1 Announce Type: cross Abstract: Older adults commonly experience chronic conditions such as pain and sleep disturbances and may consider cannabidiol for symptom management. Safe use

Switch-JustDance: Benchmarking Whole Body Motion Tracking Controllers Using a Commercial Console Game

arXiv:2511.17925v3 Announce Type: replace-cross Abstract: Recent advances in whole-body robot control have enabled humanoid and legged robots to perform increasingly agile and coordinated motions. How

Trust Your Memory: Verifiable Control of Smart Homes through Reinforcement Learning with Multi-dimensional Rewards

arXiv:2604.10110v1 Announce Type: new Abstract: Large Language Models (LLMs) have become a key foundation for enabling personalized smart home experiences. While existing studies have explored how sma

VidAudio-Bench: Benchmarking V2A and VT2A Generation across Four Audio Categories

arXiv:2604.10542v1 Announce Type: cross Abstract: Video-to-Audio (V2A) generation is essential for immersive multimedia experiences, yet its evaluation remains underexplored. Existing benchmarks typic

WebForge: Breaking the Realism-Reproducibility-Scalability Trilemma in Browser Agent Benchmark

arXiv:2604.10988v1 Announce Type: new Abstract: Existing browser agent benchmarks face a fundamental trilemma: real-website benchmarks lack reproducibility due to content drift, controlled environment

AssemLM: Spatial Reasoning Multimodal Large Language Models for Robotic Assembly

model-releasesarxiv-cs-ro

arXiv:2604.08983v1 Announce Type: new Abstract: Spatial reasoning is a fundamental capability for embodied intelligence, especially for fine-grained manipulation tasks such as robotic assembly. While

DRBENCHER: Can Your Agent Identify the Entity, Retrieve Its Properties and Do the Math?

arXiv:2604.09251v1 Announce Type: new Abstract: Deep research agents increasingly interleave web browsing with multi-step computation, yet existing benchmarks evaluate these capabilities in isolation,

Facet-Level Tracing of Evidence Uncertainty and Hallucination in RAG

arXiv:2604.09174v1 Announce Type: new Abstract: Retrieval-Augmented Generation (RAG) aims to reduce hallucination by grounding answers in retrieved evidence, yet hallucinated answers remain common eve

So the concern over Mythos and cybersecurity seems warranted.

model-releasesethan-mollick--x

So the concern over Mythos and cybersecurity seems warranted. We conducted cyber evaluations of Claude Mythos Preview and found that it is the first model to complete an AISI cyber range end-to-end. 🧵