Search: “evaluation”

Fi-ImageNet-1k: An OOD Benchmark From the Inside of the ImageNet-1k Validation Set

arXiv:2609.01027v1 Announce Type: new Abstract: Out-of-distribution (OOD) detection predicts whether a test image belongs to none of the predefined classes. To evaluate this task, benchmarks need imag

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

GenScale: A Benchmark for Relative Object Scale in Image Generation and Editing

DGX agent

arXiv:2609.00525v1 Announce Type: new Abstract: Modern image generation and editing systems can produce photorealistic, prompt-aligned images, but still often render familiar objects at implausible re

model-releasesarxiv-cs-cv

2 Sept 2026

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

arXiv:2609.01437v1 Announce Type: cross Abstract: As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly

HiveTraceGuard-Pro: A Compact Generative Guardrail for Prompt Injection, Jailbreaks, and Adversarial Obfuscation

arXiv:2609.01046v1 Announce Type: cross Abstract: Production LLMs must handle inputs that attempt to override system instructions, bypass safety policies or elicit harmful responses. A common mitigati

Integrating Traffic Noise Emission Modelling into Variable Speed Limit Control

arXiv:2609.01339v1 Announce Type: new Abstract: Road traffic noise remains a major environmental challenge, yet most speed management strategies are static and do not respond to short-term variations

safetyarxiv-cs-ro

Applications

Is Knowledge Distillation Actually Greener? A Case Study in Machine Translation

arXiv:2602.09691v2 Announce Type: replace Abstract: Knowledge distillation (KD) is a technique to compress a larger teacher system into a smaller student. In machine translation, KD is commonly evalua

applicationsarxiv-cs-cl

LifeAgentBench: Benchmarking LLMs for Long-Horizon, Cross-Dimensional Lifestyle Health Reasoning

arXiv:2601.13880v2 Announce Type: replace Abstract: Personalized lifestyle health analysis requires long-horizon, multi-dimensional reasoning over heterogeneous lifestyle signals, and recent advances

MeshSplatBench: A Unified Benchmark for Triangle-Based Neural Rendering

arXiv:2609.01306v1 Announce Type: cross Abstract: Triangle-based neural rendering bridges neural scene representations and conventional graphics pipelines by optimizing explicit geometric primitives c

Scene Graph-based Driving Scenario Extraction for Automotive Egocentric Datasets

model-releasesarxiv-cs-ro

arXiv:2609.00333v1 Announce Type: new Abstract: Extracting scenarios from unlabelled real-world sensor data streams is a critical but challenging task in the development process of automated driving s

StateSwap: Probing Support-Elimination Hidden States in Multiple-Choice Questions

arXiv:2609.01081v1 Announce Type: cross Abstract: Large language models often answer the same multiple-choice question inconsistently when it is posed under support-oriented and elimination-oriented f

Agents

The Rise of Verbal Reinforcement Learning

arXiv:2609.01597v1 Announce Type: cross Abstract: Natural language is emerging as a primary feedback channel for improving language agents, capable of conveying intent, preferences, and causal structu

agentsarxiv-cs-ai

Training-Free Inpainting Across Domains with a Frozen Text-to-Image Diffusion Model

arXiv:2609.00862v1 Announce Type: new Abstract: We show that a frozen generic text-to-image diffusion model can perform conditional inpainting across three evaluated natural-image domains with one fix

researcharxiv-cs-cv

VoiceLongMemEval: Do Assistants Remember How You Sounded?

arXiv:2609.00570v1 Announce Type: new Abstract: With the growing scale of multi-agent architectures and large language models, deployed AI assistants are increasingly tasked with reasoning over long,

Where the Verifier Fails: A Category-Level Audit of Reward Signals in RLVR

arXiv:2609.01354v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) and standard benchmark evaluation both rely on an automatic verifier that turns a free text answer

Are LLMs Ready to Assist Physicians? PhysAssistBench for Interactive Doctor-Patient-EHR Assistance

arXiv:2606.18613v4 Announce Type: replace-cross Abstract: The most plausible near-term role of medical LLMs is to assist rather than replace physicians, yet current evaluations often test isolated cap

Are Online Skill and Memory Modules Always Worth Their Tokens? A Budget-Constrained Study of Web Agents

arXiv:2606.15017v2 Announce Type: replace Abstract: Online web agents often augment a base actor with memory, workflow, or skill modules. These modules can improve performance, but they also consume t

Automated Testing of LLM-Based Post Hoc Explainers Using Model Checking as an Oracle

arXiv:2608.30581v1 Announce Type: new Abstract: Large language models (LLMs) are used as post hoc explainers of sequential decision-making policies, producing natural-language explanations of why an a

safetyarxiv-cs-ai

Beyond Surface Forms: Symbolic Edits as a Test for Logical Reasoning with LLMs

arXiv:2608.30256v1 Announce Type: cross Abstract: Logical reasoning with large language models (LLMs) is a critical capability, as it reflects a system's ability to correctly deduce hypotheses from a

BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditing

arXiv:2608.31105v1 Announce Type: new Abstract: Users of a deployed language model routinely encounter behaviours that testing almost never surfaces, since deployment puts the model through orders of

safetyarxiv-cs-ai

Breaking Darknet CAPTCHAs with general purpose LLM

arXiv:2608.28794v1 Announce Type: cross Abstract: Our work evaluates the effectiveness of automated methods for solving CAPTCHA challenges commonly encountered in darknet environments. These CAPTCHAs

safetyarxiv-cs-cv

Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks

arXiv:2510.01359v3 Announce Type: replace-cross Abstract: Code-capable large language model (LLM) agents are embedded in software engineering workflows where they can read, write, and execute code, ra

CORAL: A Benchmark for Structure-aware and Brain-wide Neuron Reconstruction in Light Microscopy

arXiv:2608.30768v1 Announce Type: new Abstract: Automatic neuron reconstruction from light microscopy images is a central problem in computational neuroanatomy. While recent methods have achieved enco

Local Ai

Cost-Effective Repository Exploration for Agentic Issue Localization

arXiv:2608.29675v1 Announce Type: cross Abstract: Repository exploration is a distinct and costly stage of coding-agent pipelines: before generating a patch, an agent must identify which repository fi

local-aiarxiv-cs-ai

Applications

Foundation Models Meet Agriculture: Challenges Beyond Pretraining

arXiv:2608.30392v1 Announce Type: new Abstract: Global food security and sustainable climate action increasingly rely on robust, scalable agricultural monitoring. Earth observation foundation models h

applicationsarxiv-cs-lg

Generative vs. Encoder Models for Multilingual NER: A Comprehensive Empirical Study on Naamapadam

arXiv:2608.29959v1 Announce Type: new Abstract: Language is humanity's most consequential technology, yet for over a billion speakers across India's twenty-two constitutionally recognised languages, i

GUI-PRA: Process Reward Agent for GUI Tasks

arXiv:2509.23263v3 Announce Type: replace Abstract: Long-horizon GUI automation remains challenging due to error accumulation over extended interaction sequences. Process Reward Models (PRMs) provide

Hydra: A Navigation World Action Model with Discrete Latent Planning and Continuous Flow-Matching Execution

arXiv:2608.28995v1 Announce Type: cross Abstract: World models let robots imagine possible futures, but exploiting this capability for real-time control is bottlenecked by a representation misalignmen

researcharxiv-cs-cv

Introducing TabFM in BigQuery: Predictive analytics reimagined

model-releasesgoogle-cloud-ai

Historically, enterprise predictive analytics tasks such as predicting churn, purchase intent, or fraud scoring have meant building custom models using libraries like XGBoost, Random Forest, or Deep N

Kinship Data Benchmark for Multi-hop Reasoning

arXiv:2601.07794v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly evaluated on their ability to perform multi-hop reasoning, i.e., to combine multiple pieces of i

Agents

Last Step Matters: Early Uncertainty Cannot Predict Failure in Long-Horizon Agents

arXiv:2608.29685v1 Announce Type: new Abstract: Early failure prediction is important for long-horizon agents, as it enables timely intervention and can reduce inference and tool-use costs. Uncertaint

agentsarxiv-cs-lg

Linguistic Distance Segregates Latent Representations in Automatic Speech Recognition Systems

arXiv:2608.30853v1 Announce Type: new Abstract: While automatic speech recognition (ASR) models have achieved remarkable improvements in recent years, performance disparities persist across different

researcharxiv-cs-cl

Paper Pilot: A Human-in-the-Loop Expert System for Evidence-Traceable Scientific Manuscript Generation in Applied Sciences

arXiv:2608.28596v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly embedded in scientific workflows for literature analysis, drafting, and review. Existing systems adva

Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Models

arXiv:2608.29996v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) achieve strong performance across many multimodal tasks; however, they often exploit spurious object-background c

Read the Room, Read the Image: Understanding Indirect Speech Acts in Multimodal Visual Contexts

arXiv:2608.30270v1 Announce Type: new Abstract: Indirect speech acts (ISAs) require pragmatic reasoning over context, as directive intent can- not be inferred from surface form alone. Prior text-based

Reliable Benchmarking of Artifact Detection in Computational Pathology: A Reproducibility and Uncertainty Analysis

arXiv:2608.30835v1 Announce Type: cross Abstract: Background and Objective: Quality control is a prerequisite for whole-slide image analysis, yet the benchmarks on which quality-control methods are co

S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?

arXiv:2608.31100v1 Announce Type: new Abstract: Large language models (LLMs) increasingly interact with external environments and accumulate substantial behavioral experience, yet existing agent bench

SIC-Agents: Benchmarking and Building an Adaptive Simulator for Pediatric Serious Illness Communication Training

arXiv:2608.29481v1 Announce Type: new Abstract: Pediatric serious illness communication (SIC) is critically important, yet scalable communication training for clinicians remains limited. Compared with

Agents

SkillZip Pro: Execution-Aware Dynamic Compression of Progressively Loaded Skills for Self-Evolving Agents

arXiv:2608.30785v1 Announce Type: new Abstract: Production agent skills are directory bundles, not isolated prompts. The root is loaded at activation; references, schemas, scripts, assets, and nested

agentsarxiv-cs-ai

Sleight of Word Benchmark: Can Language Models Notice If Their Own Output Was Tampered With?

arXiv:2608.29921v1 Announce Type: cross Abstract: The output of a Language Model can be tampered with while the model is writing it. A simple test can thus be constructed by evaluating the model's per

SVI2LoD3: Agent-Driven Reconstruction of LoD3 Facade Openings in Semantic 3D City Models from Volunteered Street View Imagery using Large Language and Visual Models

arXiv:2608.29992v1 Announce Type: new Abstract: This paper presents an end-to-end, agent-driven pipeline for the LoD3 reconstruction of facade openings in 3D city models, producing directly usable Cit

TAKE 85: Testing Audiovisual filmmaKer's intEnt across 85 Hours of Film

arXiv:2608.30068v1 Announce Type: new Abstract: Films communicate through deliberate creative choices, including lighting, color, composition, editing, dialogue, music, and sound. Humans naturally int

TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models

arXiv:2608.29999v1 Announce Type: cross Abstract: Large audio-language models (LALMs) describe audio at the clip level but cannot assign timestamps to the events, speakers, or sounds they identify. De

Terminal-Bench-LILT: Multilingual Agentic Coding Benchmark Grounded in Language, Region, and Culture

arXiv:2608.28641v1 Announce Type: cross Abstract: Most evaluations for coding agents are conducted exclusively in English, which does not reflect real-world multilingual deployment. We present Termina

Text-Guided Diffusion-Based Adversarial Attacks on Chest X-Ray Images

arXiv:2608.29456v1 Announce Type: new Abstract: As artificial intelligence is increasingly integrated into chest X-ray (CXR) interpretation, triage, and clinical decision support, understanding its vu

researcharxiv-cs-cv

Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation

arXiv:2508.14273v3 Announce Type: replace Abstract: As researchers increasingly adopt LLMs as writing assistants, generating high-quality research paper introductions remains both challenging and esse

Towards Stream Learning on Embedded Systems: Benchmarking the Memory Consumption of Stream Learning Methods

arXiv:2608.30923v1 Announce Type: cross Abstract: Stream learning is commonly evaluated through predictive performance and adaptation to concept drift. However, sustained operation of a stream learner

UReason: Benchmarking Reasoning-to-Generation Alignment in Unified Multimodal Models

arXiv:2602.08336v3 Announce Type: replace Abstract: Unified multimodal models (UMMs) aim to integrate multimodal understanding and generation within a unified architecture, yet it remains unclear to w

UTILMEM: Benchmarking Evidence Utilization in Long-Term Conversational Memory

arXiv:2608.30508v1 Announce Type: new Abstract: Long-term memory is increasingly important for conversational agents, yet existing benchmarks primarily measure memory through pointwise factual recall: