Search: “evaluation”

Amulet: a Python Library for Assessing Interactions Among ML Defenses and Risks

arXiv:2509.12386v3 Announce Type: replace-cross Abstract: Machine learning (ML) models are susceptible to various risks to security, privacy, and fairness. Most defenses are designed to protect agains

safetyarxiv-cs-ai

12 Sept 2026

Local Ai

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Can Edge-Deployable Vision-Language Models Identify Species?

DGX agent

arXiv:2609.11916v1 Announce Type: new Abstract: Camera traps often run in the field on edge hardware with limited or no connectivity, making small, locally-deployable vision-language models (VLMs) --

local-aiarxiv-cs-ai

12 Sept 2026

When Validation Stops Learning: Auditing Update Admission for Continual Embodied Agents

arXiv:2609.10873v1 Announce Type: new Abstract: Independent evaluation can reject harmful policy updates yet also prevent useful continual learning. We argue that update admission must be assessed thr

safetyarxiv-cs-ai

12 Sept 2026

E-CONAN (Entailment, CONtradition And Neutral) Benchmarks: Arabic Textual Entailment and Natural Inference Datasets

arXiv:2609.11334v1 Announce Type: new Abstract: Natural Language Inference processes pairs of sentences to extract their semantic relations. NLI has been a hot research topic, integrated as a main com

LexAgentHallu: A Hierarchical Benchmark for Profiling Hallucinations in Legal Agents

arXiv:2609.09754v1 Announce Type: new Abstract: As large language models are increasingly deployed as tool-augmented legal agents, they introduce agentic hallucinations where tool-call and reasoning e

MedGEN-Bench: A Contextually Entangled Benchmark for Open-ended Multimodal Medical Generation

arXiv:2511.13135v3 Announce Type: replace Abstract: Medical vision-language models (VLMs) are increasingly expected to support clinical workflows through diagnostic text and relevant medical images. H

Nuha-Speech: Building General-Purpose Arabic Speech-LLMs

arXiv:2609.11892v1 Announce Type: new Abstract: As Speech Large Language Models (speech-LLMs) become increasingly multilingual, Arabic remains significantly underrepresented, highlighting the need for

TransClean: A Benchmark for Detecting and Extracting Clean Translations from Large Language Model Outputs

arXiv:2609.11399v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for machine translation, yet their outputs often contain additional text beyond the translation itsel

A Unified Policy Architecture (UPA): The Governance Kernel for Enterprise AI Operating Systems

arXiv:2609.06543v1 Announce Type: new Abstract: Enterprise AI is evolving into an Enterprise Operating System where autonomous AI agents can plan, reason, use memory, invoke tools, execute workflows,

safetyarxiv-cs-ai

Applications

Aegix Pulse: A Traceable Three-Stage Architecture for Personalized Content Generation and Context-Preserving Revision

arXiv:2609.07672v1 Announce Type: new Abstract: Production content-generation systems must integrate a user's immediate task, long-term brand identity, historical evidence, and revision feedback. We p

applicationsarxiv-cs-ai

API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces

arXiv:2609.08861v1 Announce Type: new Abstract: Benchmark scores are a central currency in model releases: they inform purchasing decisions, shape public trust, and influence policy. Yet, a key assump

Artificial Analysis is not 'broken', and they prove it.

model-releasesr-localllama

Like many of you, I have seen many posts and tweets in the last weeks complaining about Artificial Analysis being 'broken', 'meaningless', and 'bought out.' People who say this have done no research a

CIVI: A Framework for Diagnosing Search Agent Failures in Civic Information

arXiv:2609.08094v1 Announce Type: new Abstract: Large Language Models are increasingly deployed in public-sector settings, where incorrect guidance can cause irreversible harm. We introduce CIVI, the

Local Ai

Encrypt What Matters: When Selective Homomorphic Inference Is Efficient

arXiv:2609.09357v1 Announce Type: cross Abstract: Fully homomorphic encryption (FHE) enables inference on private data without revealing it to the server, but evaluating an entire input under FHE is e

local-aiarxiv-cs-lg

From Representation to Enactment: The ABC Framework of the Translating Mind

arXiv:2511.16811v2 Announce Type: replace Abstract: Building on the third-wave Extended Mind (EM) theory and radical enactivism, this article suggests an alternative to representation-based models of

researcharxiv-cs-cl

Agents

GoAnt: Quality-Diversity Multi-Agent Search for Alpha Factor Discovery in Market Microstructure Data

arXiv:2609.08719v1 Announce Type: new Abstract: Automated alpha factor discovery searches symbolic trading signals from price-volume panels and order-book data under a fixed evaluation budget. Existin

agentsarxiv-cs-ai

Mind the Gap: Exposing LLM Translation Blind Spots Using the AlphaMWE Multilingual Parallel Corpus

arXiv:2609.06634v1 Announce Type: cross Abstract: LLMs' performance on machine translation (MT) tasks is often dependent on the data availability in the specific domains and language pairs that they a

researcharxiv-cs-ai

Performance of Clinical AI System and Physicians and Frontier Language Models in primary care diagnostics

arXiv:2609.09070v1 Announce Type: cross Abstract: Clinical AI evaluation should encompass diagnosis and management after adaptive information gathering. We compared Doctorina, eight physicians and fou

RAP: Research Attention Prediction Reveals Target-Conditioned Evidence Acquisition Biases

arXiv:2609.10092v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly act as research agents, yet their ability to track shifts in research attention is difficult to evaluate bec

RAPID: Reliability-Aware Pair Importance Distillation

arXiv:2609.05481v1 Announce Type: new Abstract: Inter example relational distillation transfers a teacher's representation geometry by matching relations among examples within a mini batch. Computing

researcharxiv-cs-ai

SciLitBench: Benchmark and Design Principles for LLM-Powered Systematic Literature Reviews

arXiv:2609.05505v1 Announce Type: new Abstract: Systematic reviews require sustained human judgment across thousands of records, yet existing evaluations of large language models (LLMs) typically exam

Scoring Without the Engine: Validating a Deterministic, Manipulation-Resistant Content Score for Generative Engines, End to End

arXiv:2609.07559v1 Announce Type: new Abstract: How do you validate a cheap, deterministic proxy for an oracle that is expensive, rate-limited, and non-stationary? We present a protocol built on adver

VST: Verifiable Structured Transport for Auditable Agent-to-Agent Alpha Discovery

arXiv:2609.07065v1 Announce Type: new Abstract: Agent-to-agent (A2A) alpha discovery is slowed by repeated feedback cycles between mining and evaluation agents, whose hand-offs, in contemporary LLM mu

What Does MMLU Actually Measure? A Psychometric Audit of Difficulty Structure in Aggregate Benchmark Scores

arXiv:2609.09372v1 Announce Type: cross Abstract: Although MMLU is widely adopted as a benchmark for calibrating general AI capabilities, we psychometrically demonstrate that its aggregate score prima

Tutorials

AlphaRJM: Reward-Jump Memory for Stochastic Return-Guided Alpha Discovery

arXiv:2609.08581v1 Announce Type: new Abstract: Formulaic alpha discovery is a pool-dependent symbolic search problem in which informative feedback is observed primarily when a complete expression is

tutorialsarxiv-cs-lg

AV-SafetyBench: A Safety Benchmark for Text-to-Audio-Video Generation

arXiv:2609.06991v1 Announce Type: new Abstract: Recent text-to-audio-video (T2AV) models jointly generate video, speech, sound effects, and ambience from a single text prompt. This capability poses ne

GeoHeight-Bench: Towards Height-Aware Multimodal Reasoning in Remote Sensing

arXiv:2603.25565v2 Announce Type: replace Abstract: Current Large Multimodal Models (LMMs) in Earth observation are predominantly evaluated on planar optical tasks and often neglect the vertical dimen

Limitations of Automated Simulatability: LLM Simulators Can Bypass Explanations

arXiv:2609.08585v1 Announce Type: cross Abstract: Simulatability is an evaluation protocol for explanations that quantifies their usefulness by how well they help a user predict a task model's outputs

researcharxiv-cs-lg

MEMOBench: A Process Level Memory Benchmark for Robotic Manipulation

model-releasesarxiv-cs-ro

arXiv:2609.07047v1 Announce Type: new Abstract: Robotic manipulation often requires acting on information that is no longer visible, yet Vision-Language-Action policies are usually evaluated when the

PhenoBench: Mapping What a Deeply Phenotyped Human Cohort Can Tell Us

model-releasesarxiv-cs-lg

arXiv:2609.06080v1 Announce Type: new Abstract: Deeply phenotyped cohorts combine clinical, imaging, molecular, and wearable observations across timescales from seconds to years. This breadth can reve

Robust Vision Systems for Connected and Autonomous Vehicles: Security Challenges and Attack Vectors

arXiv:2602.09740v3 Announce Type: replace Abstract: Connected and Autonomous Vehicles (CAVs), which is critical for achieving Level-5 autonomous driving. Safe and reliable CAV navigation depends on ro

When Does a Laugh Begin? Structured Annotator Disagreement in Temporal Laughter Localization

arXiv:2609.06646v1 Announce Type: new Abstract: Annotators routinely disagree on laughter boundaries and subtle chuckles, yet temporal laughter localization typically evaluates against a single refere

Tutorials

Better Understanding, Better Fixes? A Study of Hallucination in LLM-based Automated Program Repair

arXiv:2609.04909v1 Announce Type: cross Abstract: Large language models (LLMs) have significantly advanced automated program repair (APR), yet existing evaluations remain largely result-centric and pr

tutorialsarxiv-cs-ai

7 Sept 2026

Fascinating paper from Anthropic and colleagues. They study whether models can tell when they are being tested. It turns out that capable mo…

Fascinating paper from Anthropic and colleagues. They study whether models can tell when they are being tested. It turns out that capable models can tell when they are being tested rather than deploye

safetydair-ai--x

7 Sept 2026

Risk-Aware Optimal Control with Rulebooks

model-releasesarxiv-cs-ro

arXiv:2609.05199v1 Announce Type: cross Abstract: We consider safety-critical control problems involving multiple requirements with different priorities and uncertainty in their evaluation. We represe

7 Sept 2026

Beyond Majority Vote: Multi-Perspective Adjudication for Medical Hallucination Detection

arXiv:2609.03953v1 Announce Type: new Abstract: Understanding the frequency of factual errors in chatbot-generated text and evaluating systems that detect these errors is critical for determining chat

How Perturbations Propagate: A Multi-Level Analysis of Robustness in Large Language Models

arXiv:2609.03322v1 Announce Type: new Abstract: Language models encounter typos, corrupted text, altered words, and disrupted token order, yet robustness is usually evaluated only through output behav

safetyarxiv-cs-cl

Investigating the Ability of Large Language Models to Analyze Recipes for Diabetes

arXiv:2609.03967v1 Announce Type: cross Abstract: Several studies have evaluated the ability of Large Language Models (LLMs) for meal planning, yielding positive outcomes. These models can process nat

Measuring Harmfulness of Computer-Using Agents

arXiv:2508.00935v3 Announce Type: replace-cross Abstract: Computer-using agents (CUAs), which can autonomously control computers to perform multi-step actions, might pose significant safety risks if m

MetaStructAtlas: A Grounded 3D Vision-Language Dataset and Benchmark for Functional and Structural Reasoning in Whole-Body PET/CT

arXiv:2609.03690v1 Announce Type: new Abstract: The joint interpretation of metabolic function and anatomical structure is essential for clinical diagnosis in whole-body PET/CT. Although recent advanc

Preprocessing Failure and Adversarial Detection in Depthwise-Separable Edge Vision Systems

arXiv:2609.03453v1 Announce Type: new Abstract: Preprocessing-based defenses are the standard first-line response to adversarial attacks on edge vision systems, requiring no retraining, no architectur

researcharxiv-cs-cv

Agents

Speak for Me: Giving LLMs the Situational Awareness to Participate in a Meeting

arXiv:2609.03923v1 Announce Type: new Abstract: In online meeting delegation, LLM agents fail to recognize when to speak. With no structured way to track stances, coverage, and floor, they miss the mo

agentsarxiv-cs-ai

To What Extent Do Large Language Models Understand Bangla Idioms?

arXiv:2609.03410v1 Announce Type: new Abstract: Idiomatic expressions are an integral part of natural language, reflecting cultural nuances and posing unique challenges for computational models, parti

A Physics-Consistent Benchmark for Contact-Rich Human-Robot Interaction in Assistive Care

model-releasesarxiv-cs-ro

arXiv:2609.02402v1 Announce Type: new Abstract: Conventional task-level evaluation asks whether a robot policy completes a specified action, but can miss failures that emerge only during physical huma

Blending Concepts: Benchmarking Visual Metaphor Generation in Text-to-Image Models

arXiv:2609.02502v1 Announce Type: cross Abstract: Text-to-image (T2I) models have achieved remarkable success at faithfully rendering specified objects and attributes, yet their ability to produce vis

CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI

arXiv:2609.02459v1 Announce Type: new Abstract: We present CivBench, an open-source benchmark for evaluating language model agents in long-horizon, tool-mediated environments through the Model Context

Deeply Interleaved Text-Image Contexts for Multimodal LLMs Assessment

arXiv:2609.02573v1 Announce Type: new Abstract: Current evaluations and training of multimodal models predominantly focus on multi-image tasks, largely overlooking interleaved text-image scenarios. In

FairLens: Benchmarking Fairness in Vision-Language Models for High-Stakes Decision-Making

arXiv:2609.01691v1 Announce Type: new Abstract: Vision-language models (VLMs) are increasingly used to make decisions from visual inputs. We introduce FAIRLENS, a benchmark and evaluation framework fo