Search: “evaluation”

HeiCo-FOCUS: A Clinically Grounded Dataset for Long-Context Video Understanding

arXiv:2610.10156v1 Announce Type: new Abstract: Recent advances in Vision-Language Models (VLMs) have led to rapid progress in video understanding across a wide range of benchmark tasks. However, exis

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

How Do LLMs Change Predictions Under Negation?

DGX agent

arXiv:2610.09571v1 Announce Type: new Abstract: Negation is an essential feature of human language, yet large language models (LLMs) remain unreliable in processing it. We evaluate recent open-source

model-releasesarxiv-cs-cl

8 Oct 2026

How Private is Private? A Comparative Study for Face De-Identification

arXiv:2610.10334v1 Announce Type: new Abstract: Face de-identification (FDeID) has emerged as a critical privacy-preserving technology, yet its evaluation remains fundamentally fragmented. Existing pr

Moonworks Lunara: Modeling Artistic Intelligence [R]

model-releasesr-machinelearning

Lunara introduces a novel Diffusion Mixture Transformer architecture with fewer than 10B active parameters for modeling artistic intelligence in image generation. Its CAT training algorithm iterativel

PatchBench: Measuring Collateral Damage in Activation Patching

arXiv:2610.10276v1 Announce Type: cross Abstract: An LLM safety patch can pass a benchmark while still being a poor repair. This risk is especially acute for jailbreak repairs, where the goal is to co

Hardware

PoreML: A Data-Driven Framework for Learning Multiphase Flow in Porous Media

arXiv:2610.10314v1 Announce Type: new Abstract: Multiphase flow in porous microstructures is central to CO_2 storage, fuel-cell operation, and flip-chip packaging. Predicting these flows remains chall

hardwarearxiv-cs-lg

RT-Safe: Benchmarking Agent Safety in Real-Time Embodied Environment

model-releasesarxiv-cs-lg

arXiv:2610.09294v1 Announce Type: cross Abstract: Rapid progress in AI agents has brought growing attention to agent safety, with extensive evaluation focused on digital environments. As agents move i

Shaer: Controlled Arabic Poetry Generation with Meter Subform and Semantic Conditioning

arXiv:2610.09756v1 Announce Type: new Abstract: Classical Arabic poetry generation requires simultaneously satisfying semantic, linguistic, and fine-grained prosodic constraints. Existing systems typi

researcharxiv-cs-cl

World Models Dream of Success: Diagnosing and Repairing Failure Insensitivity in Robot World Models

arXiv:2610.09134v1 Announce Type: new Abstract: Robot world models support policy evaluation, planning, and synthetic data generation, but these applications require predictions that distinguish succe

Auditable Claims about AI Agents

arXiv:2610.07459v1 Announce Type: new Abstract: Organizations make claims about their AI agents: a person approves every external email, every action is logged, an evaluation shows the agent is safe t

safetyarxiv-cs-ai

CroissantMiner: Automated Extraction and Validation of Croissant Metadata for ML Datasets

arXiv:2610.07132v1 Announce Type: cross Abstract: Croissant has emerged as a standard for machine-readable dataset metadata, yet populating its fields remains labor-intensive and requires careful read

EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans

arXiv:2610.05370v2 Announce Type: replace Abstract: Generative reward models (GRMs) are important for LLM optimization. Unlike scalar reward models, GRMs generate natural-language critiques alongside

researcharxiv-cs-ai

EPOCH: Reliable Discovery through Evidence-Governed Search

arXiv:2610.06986v1 Announce Type: new Abstract: AI research agents are increasingly used to search over programs, mathematical constructions, and proofs. However, existing systems typically optimize e

M3SunAgent: Monocular 3D Spatial Understanding Agent for Metric Depth Estimation and 3D Visual Grounding

arXiv:2610.07982v1 Announce Type: new Abstract: Monocular metric depth estimation and 3D visual grounding represent the two complementary cornerstones of monocular 3D spatial understanding (M3Sun), fr

OSGuard: A Benchmark for Safety in Computer-Use Agents

arXiv:2606.15034v2 Announce Type: replace Abstract: Computer-use agents can complete benign user instructions while violating important constraints of the user's environment. We introduce OSGuard, a d

OTel: Open Telco AI Datasets, Benchmarks, and Models

arXiv:2610.07766v1 Announce Type: new Abstract: We present Open Telco (OTel), an open telecom AI resource that releases derived telecom datasets for retrieval, reranking, instruction tuning, and safet

Agents

Partially Observable Zero-shot coordination by Predicting Intention of Partner

arXiv:2610.08142v1 Announce Type: new Abstract: Zero-shot coordination in embodied settings requires acting while the partner is intermittently out of view, leaving existing methods with ambiguous par

agentsarxiv-cs-ai

PERSIST: Who-What-When Memory Across Sessions for Full-Duplex Spoken Dialogue

arXiv:2610.07725v1 Announce Type: new Abstract: Modern voice assistants may be shared by multiple users and should be able to answer questions about earlier conversations such as 'When did I originall

RAG-PIBench: A Leakage-Aware Benchmark for Prompt-Injection Detection in Trustworthy RAG Systems

arXiv:2610.08571v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) systems are vulnerable to prompt-injection attacks embedded in retrieved content. We introduce RAG-PIBench, a ben

RenderBench: Benchmarking Render-to-Real Video Transfer with Reconstructed Digital Twins

arXiv:2610.08684v1 Announce Type: new Abstract: Modern video models can generate realistic videos from real appearance references and proxy renders that specify scene structure, viewpoint changes, and

TasteVal: Measuring the Experimental Research Taste of AI Systems Against Human Experts

arXiv:2610.06824v2 Announce Type: replace Abstract: We introduce TasteVal, a benchmark to evaluate the experimental research taste of frontier models. We define research taste as the ability to pick i

What the Elevation Map Cannot See: Semantic-Aware Locomotion and Execution-Aware Navigation for Humanoid Robot

arXiv:2610.07396v1 Announce Type: new Abstract: Navigation for humanoid robots is critical, yet large-scale evaluation on physical hardware is often impractical due to cost and safety concerns, making

BrainTRACE: Tracing Longitudinal, Multimodal, and Volumetric Evidence in Brain MRI Clinical Reasoning

arXiv:2610.06571v1 Announce Type: new Abstract: Brain MRI interpretation is a longitudinal clinical reasoning problem: radiologists compare serial studies, integrate information across MRI sequences,

EgoExo-Next:Benchmarking Vision-Language Models on Visual-Option Next-State and Cross-View Reasoning

arXiv:2610.04506v1 Announce Type: new Abstract: Vision-language models (VLMs) are increasingly evaluated for egocentric and cross-view video reasoning, yet existing benchmarks largely focus on semanti

Harmful Content Generation in Text-to-Image Models: Capabilities and Moderation Limitations

arXiv:2610.06503v1 Announce Type: new Abstract: Text-to-image generative models can produce highly realistic imagery but also raise concerns about harmful misuse. While safety mechanisms exist, system

safetyarxiv-cs-cv

Hardware

Image-Based Breast Implant Detection for Mammography Dataset Curation and Near-Real-Time Deployment: Comparing Foundation Models and Task-Specific Convolutional Models

arXiv:2610.03817v1 Announce Type: cross Abstract: Purpose: To evaluate the performance-feasibility tradeoffs of foundation models (FMs) and task-specific convolutional neural networks (CNNs) trained f

hardwarearxiv-cs-cv

Odyssey: A Closed-Loop Benchmark for Long-Horizon Real-World Driving with Explicit Navigation Routes

arXiv:2610.06469v1 Announce Type: new Abstract: Closed-loop evaluation of end-to-end driving requires continuous rollouts that reveal how earlier decisions affect subsequent driving. However, existing

SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics

arXiv:2610.06513v1 Announce Type: cross Abstract: Evaluating text generation requires measuring how well the generated distribution matches the data distribution. For autoregressive models, this is do

researcharxiv-cs-lg

Training and Scaling Compute-Optimal Physiological Waveform Foundation Models

arXiv:2610.05649v1 Announce Type: new Abstract: We investigate the scaling laws and compute-optimal training of physiological waveform foundation models (FMs). We train Aether, a family of over one hu

researcharxiv-cs-lg

When Is Enough Enough in Self-Evolving LLM Systems?

model-releasesarxiv-cs-lg

arXiv:2610.04756v1 Announce Type: cross Abstract: Self-evolving large language model (LLM) systems repeatedly propose, evaluate, and incorporate updates to prompts, skills, or other persistent artifac

Benchmarking Literature Retrieval for a Model Organism: A Dictyostelium Case Study

arXiv:2610.03130v1 Announce Type: cross Abstract: Biological literature retrieval systems are often developed and evaluated using broad biomedical corpora and general-purpose search tasks. However, ma

Do Large Language Models Know Colombian Law? A Reliability Benchmark for the Colombian Legal System

arXiv:2610.03639v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to support legal practice, education, and research, yet their reliability in national legal systems o

Language Models that Play Chess and Explain Their Moves

arXiv:2610.03695v1 Announce Type: new Abstract: Modern chess engines are silent experts: they play at a superhuman level, but do not offer explanations for their play. On the other hand, language mode

Large language models exhibit unreliable updating of clinical judgment as patient evidence evolves

arXiv:2610.02684v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly explored for clinical reasoning, but whether they appropriately revise judgments as patient evidence evolv

researcharxiv-cs-ai

NegT2IBench: When Negation Changes the Picture. A Polarity Benchmark for Text-to-Image Models

arXiv:2610.03084v1 Announce Type: cross Abstract: Text-to-image (T2I) models are judged by benchmarks that measure whether requested content appears, but these benchmarks largely overlook the compleme

OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation

arXiv:2610.02781v1 Announce Type: cross Abstract: Many useful language-model tasks cannot be evaluated by exact outcome verification. Rubric-based reinforcement learning (RL) addresses this issue by s

safetyarxiv-cs-ai

OpenGameEval: Benchmarking Agentic Programming and Exploration in a Stateful Game Engine

arXiv:2610.02563v1 Announce Type: cross Abstract: We present OpenGameEval, a benchmark and evaluation framework for agentic game development inside Roblox Studio. It runs language models as agents in

Tutorials

Reliable Self-Evolution with Imperfect Proxy Rewards

arXiv:2610.02975v1 Announce Type: new Abstract: Large language model (LLM)-based self-evolving search is a promising approach to scientific discovery. However, high-fidelity evaluation of every candid

tutorialsarxiv-cs-ai

SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation

arXiv:2610.02304v1 Announce Type: cross Abstract: Existing Simulink benchmarks mainly evaluate whether generated models compile, execute, or resemble a reference model. These criteria do not establish

Understanding Clinical Cognitive Dialogues Using Large Language Models

arXiv:2609.34125v2 Announce Type: replace Abstract: In-person cognitive assessment is both a test and an interaction. Clinicians explain tasks, repair misunderstandings, and adapt to patient responses

From Pixels to Policy: A Multi-Agent System for Intervention and Geo-Spatial Decision Support

arXiv:2610.01870v1 Announce Type: new Abstract: Urban environments are shaped by design choices with long-term implications for health, safety, and quality of life, yet evaluating proposed interventio

safetyarxiv-cs-cv

In Vino Veritas and Vulnerabilities: Examining LLM Safety via Drunk Language Inducement

arXiv:2601.22169v2 Announce Type: replace-cross Abstract: Humans are susceptible to undesirable behaviours and privacy leaks under the influence of alcohol. This paper investigates drunk language, i.e

safetyarxiv-cs-ai

Interpreting Reasoning of Large Language Models via Partial Information Decomposition

arXiv:2610.00571v1 Announce Type: cross Abstract: Large reasoning models (LRMs) have achieved substantial improvements in solving complex mathematical problems, but often produce lengthy, repetitive,

Is Success All You Need? Investigating the Impact of Input Perturbations on VLA Behaviour in Tabletop Manipulation Tasks

arXiv:2610.01351v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models have achieved high task success rates on robot manipulation task benchmarks. More recently, there has been an emphas

KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards

arXiv:2610.02206v1 Announce Type: cross Abstract: LLMs are increasingly applied to cybersecurity workflows, where they are expected to translate analysts' intent into tool invocations. However, existi

Learning to Sell: Reinforcement Learning for Strategic Large Language Model Agents in Multi-Product Markets

arXiv:2609.33289v2 Announce Type: replace Abstract: Autonomous large language model (LLM) agents operating in multi-product markets must make sequential decisions under information asymmetry and resou

Hardware

LESS: Lightweight Evolutionary Supernet Search in Minutes

arXiv:2610.01468v1 Announce Type: cross Abstract: Low-cost NAS must both explore high-performing architectures and identify them reliably, yet reducing evaluation cost often weakens the fidelity of ca

hardwarearxiv-cs-lg

M^2Weather: A Benchmark for Joint Multi-Station and Multi-Variable Weather Forecasting

model-releasesarxiv-cs-lg

arXiv:2610.00370v1 Announce Type: new Abstract: Station weather forecasting is fundamentally shaped by both complex spatial dependencies across stations and strong physical coupling among weather vari