Search: “evaluation”

From 'Strings' to 'Things' for Personal Knowledge Graphs: Evaluating LLM Triple Extraction for Recommendation Systems

arXiv:2607.00003v1 Announce Type: cross Abstract: Personal Knowledge Graphs (PKGs) offer a privacy-preserving framework for modeling user preferences, yet constructing them from unstructured, decentra

2 Jul 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

CDR-Bench: Evaluating Faithful Execution of Compositional, Order-Sensitive Data Refinement Recipes

DGX agent

arXiv:2606.31435v1 Announce Type: new Abstract: Data refinement involves executing multi-step recipes over evolving text states, where both composition and execution order of processing operators dete

model-releasesarxiv-cs-ai

1 Jul 2026

Indi-RomCoM: Code-Mixed Benchmark for Evaluating LLMs on Romanized Indic-English Instructions

arXiv:2606.30790v1 Announce Type: cross Abstract: Romanized Code Mixing (RCM), where bilingual speakers fluidly blend local languages with English in Roman script, has emerged as the dominant form of

1 Jul 2026

Local Ai

Loc2Repair: A Framework for Evaluating the Impact of File-Level Issue Localization in Repo-Level LLM Repair

arXiv:2606.30963v1 Announce Type: cross Abstract: Repository-grounded automated repair is often reported as a single end-to-end capability, which hides distinct failure modes such as poor file targeti

local-aiarxiv-cs-ai

1 Jul 2026

Understanding and Evaluating Claw-like Agent Security Through a Computer-Systems Lens

arXiv:2606.30755v1 Announce Type: cross Abstract: Claw-like AI agents (e.g., OpenClaw) are always-on processes with persistent access to credentials, files, tools, and external services. They take on

1 Jul 2026

Local Ai

Few-class Fidelity: Evaluating Explanations of Real-conditions CNN classifiers with Optimized Perturbations

arXiv:2606.28391v1 Announce Type: cross Abstract: The wide use of Convolutional Neural Networks (CNN) in numerous domains and real-world classification applications is justified by their high precisio

local-aiarxiv-cs-ai

LLMs and their Limited Theory of Mind: Evaluating Mental State Annotations in Situated Dialogue

arXiv:2509.02292v2 Announce Type: replace Abstract: What if large language models could not only infer human mindsets but also expose every blind spot in team dialogue such as discrepancies in the tea

mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health

arXiv:2606.29467v1 Announce Type: new Abstract: Medical question-answering benchmarks rarely cover the maternal, neonatal, child, and reproductive-health questions a nurse-midwife asks, and, to our kn

SpreadsheetBench 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows

arXiv:2606.29955v1 Announce Type: cross Abstract: Spreadsheets are widely used for business analysis, financial modeling, reporting, and decision-making. However, most existing spreadsheet benchmarks

Towards Evaluating Data Priors for Tabular Foundation Models

arXiv:2606.29241v1 Announce Type: new Abstract: Data-generating priors are a central component of tabular foundation models because they define the task distribution used during pretraining. However,

researcharxiv-cs-lg

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?

arXiv:2602.11988v2 Announce Type: replace-cross Abstract: A widespread practice in software development is to tailor coding agents to repositories using context files, such as AGENTS.md. Although this

agentsarxiv-cs-ai

Evaluating LLMs on Real-World Software Performance Optimization

arXiv:2606.25530v1 Announce Type: cross Abstract: Software performance optimization is a notoriously complex and manual task. Despite the growing use of Large Language Models (LLMs) for code refinemen

From Sounds to Scenes: A Benchmark for Evaluating Context-Aware Auditory Scene Understanding in Large Audio Language Models

arXiv:2606.25391v1 Announce Type: cross Abstract: Recent Large Audio Language Models (LALMs) have achieved remarkable progress in audio perceptual tasks across individual acoustic layers, including sp

Applications

Helpful or Harmful? Evaluating LLM-Assisted Vulnerability Patching via a Human Study

arXiv:2606.25973v1 Announce Type: cross Abstract: Software vulnerability remediation is a cognitively demanding task that requires specialized security expertise often lacking in general developers. I

applicationsarxiv-cs-ai

Memory Makes the Difference: Evaluating How Different Memory Roles Shape Conversational Agents

arXiv:2606.25361v1 Announce Type: new Abstract: Prior research on memory mechanism in RAG-based conversational system has emphasized how memory is stored and retrieved. However, far less is known abou

agentsarxiv-cs-cl

Probabilistic Agents in Deterministic Audits: Evaluating Multi-Agent Systems for Automated Audits Based on the German IT-Grundschutz

arXiv:2606.25622v1 Announce Type: cross Abstract: The NIS-2 Directive mandates robust Risk Management from thousands of small and medium enterprises. To ensure compliance, companies rely on establishe

agentsarxiv-cs-ai

Evaluating LLM Usage for Efficient and Explainable Numerical and Classified Implicit Sentiment Analysis of Product Desirability

arXiv:2606.23701v1 Announce Type: cross Abstract: Qualitative product feedback can reveal nuanced user experiences, but its implicit sentiment is difficult to measure. This paper presents a scalable a

researcharxiv-cs-ai

24 Jun 2026

BELLS-O: Evaluating the Operational Trade-offs of LLM Supervision Systems

model-releasesarxiv-cs-lg

arXiv:2606.20668v1 Announce Type: cross Abstract: LLM supervision systems, namely input/output moderation filters and jailbreak detectors, are the primary safeguard against misuse in deployed AI appli

DrugBench: Evaluating AI Control Protocols for Medication Harm Mitigation

model-releasesarxiv-cs-lg

arXiv:2606.20663v1 Announce Type: cross Abstract: Large Language Models have the potential to expand and improve the access to clinical information by enabling new ways of interacting with medical kno

Evaluating and Enhancing Negation Comprehension in Remote Sensing MLLMs

model-releasesarxiv-cs-cv

arXiv:2606.20177v2 Announce Type: replace Abstract: Multimodal Large Language Models (MLLMs) have demonstrated remarkable success in various Remote Sensing (RS) tasks. However, their ability to compre

Memory Contagion: Cross-Temporal Propagation of Evaluator Bias via Agent Memory

arXiv:2606.23195v1 Announce Type: new Abstract: Large Language Model (LLM) agents increasingly rely on memory systems to maintain long-term coherence. Recent work shows that agent memories degrade dur

safetyarxiv-cs-lg

Afrispeech Semantics: Evaluating Audio Semantic Reasoning in Spoken Language Models Across Domains and Accents

arXiv:2606.11219v1 Announce Type: cross Abstract: Audio language models (ALMs) are increasingly used for speech-based understanding, yet their ability to perform semantic reasoning beyond transcriptio

researcharxiv-cs-ai

11 Jun 2026

Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks

arXiv:2606.12344v1 Announce Type: cross Abstract: General-purpose agents such as OpenClaw are increasingly used as autonomous tool users, but their coding ability is difficult to measure under SWE-ben

11 Jun 2026

Evaluating Bias in Phoneme-Based Automatic Speech Recognition Systems: An Analysis of IPA Transcription Models

arXiv:2606.11639v1 Announce Type: new Abstract: The popularization of automatic speech recognition (ASR) systems has increased exploration of the demographic biases related to race, age, gender, and a

safetyarxiv-cs-cl

11 Jun 2026

CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

arXiv:2601.18026v2 Announce Type: replace Abstract: Language identification (LID) is a fundamental step in curating multilingual corpora. However, LID models still perform poorly for many languages, e

10 Jun 2026

Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles

arXiv:2606.09718v1 Announce Type: cross Abstract: Diffusion models have demonstrated remarkable generative capabilities and have also emerged as powerful self-supervised representation learners, yet t

researcharxiv-cs-cv

MBABench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Finance

arXiv:2605.22664v2 Announce Type: replace Abstract: LLM agents are increasingly expected to carry out end-to-end workflows, producing complete artifacts from high-level user instructions. To meet ente

MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Models

arXiv:2606.07706v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have demonstrated strong performance across multimodal tasks, yet their safety robustness remains an open challenge. Whi

TQA-Bench: Evaluating LLMs for Multi-Table Question Answering

arXiv:2411.19504v2 Announce Type: replace Abstract: The advance of large language models (LLMs) has unlocked great opportunities in complex multi-modal data management tasks, particularly in question

EvoClaw: Evaluating AI Agents on Continuous Software Evolution

arXiv:2603.13428v2 Announce Type: replace-cross Abstract: With AI agents increasingly deployed as long-running systems, it becomes essential to autonomously construct and continuously evolve customize

8 Jun 2026

100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?

arXiv:2505.19293v2 Announce Type: replace-cross Abstract: Long-context capability is considered one of the most important abilities of LLMs, as a truly long context-capable LLM enables users to effort

Tutorials

Do Foundation Models See Biology? Evaluating Attention Coherence with Spatial Transcriptomics in Glioblastoma

arXiv:2606.04764v1 Announce Type: new Abstract: Whether attention maps from pathology foundation models capture genuine biology remains unknown, yet this question is critical for clinical trust and re

tutorialsarxiv-cs-cv

Evaluating Autoformalization Robustness via Semantically Similar Paraphrasing

arXiv:2511.12784v3 Announce Type: replace Abstract: Large Language Models (LLMs) have recently emerged as powerful tools for autoformalization. Despite their impressive performance, these models can s

Evaluating Reasoning Fidelity in Visual Text Generation

arXiv:2606.04479v1 Announce Type: cross Abstract: Recent text-to-image (T2I) models can render highly legible and well-structured text within images, enabling applications including document generatio

researcharxiv-cs-ai

FALSIFYBENCH: Evaluating Inductive Reasoning in LLMs with Rule Discovery Games

arXiv:2606.04751v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed as autonomous agents in scientific tasks. Yet whether these systems can effectively engage in for

agentsarxiv-cs-ai

NoRA: Evaluating Grounded Reasonableness in Visual First-person Normative Action Reasoning

arXiv:2606.04806v1 Announce Type: cross Abstract: LLMs and agentic systems are increasingly deployed in social environments, making normative competence critical for safe and appropriate behavior. How

Applications

Towards Evaluating the Robustness of Visual State Space Models

arXiv:2406.09407v3 Announce Type: replace Abstract: Vision State Space Models (VSSMs), a novel architecture that combines the strengths of recurrent neural networks and latent variable models, have de

applicationsarxiv-cs-cv

VCIFBench: Evaluating Complex Instruction Following for Video Understanding

arXiv:2606.04588v1 Announce Type: new Abstract: Multimodal large language models have made rapid progress in video understanding, yet existing benchmarks largely rely on simple prompts and provide lim

Social Caption: Evaluating Social Understanding in Multimodal Models

arXiv:2601.14569v2 Announce Type: replace Abstract: Social understanding abilities are crucial for multimodal large language models (MLLMs) to interpret human social interactions. We introduce SOCIAL

3 Jun 2026

Evaluating and Learning Robust Bandit Policies Under Uncertain Causal Mechanisms

arXiv:2508.02812v3 Announce Type: replace Abstract: Causal graphical models can encode large amounts structural knowledge, both from the background knowledge of domain experts and the structural knowl

safetyarxiv-cs-lg

Safe2Drive: Evaluating Safe Driving Behaviors of E2E Autonomous Driving Models

arXiv:2606.00191v1 Announce Type: cross Abstract: Recent end-to-end (E2E) autonomous driving policies achieve high driving scores in closed-loop simulations. Yet it remains unclear whether these polic

safetyarxiv-cs-cv

SPADE-Bench: Evaluating Spontaneous Strategic Deception in Agents via Plan-Action Divergence

arXiv:2606.02380v1 Announce Type: cross Abstract: As LLM-based agents expand their operational scope, reliability becomes a prerequisite for real-world deployment. However, in practical applications,

TimeSage-MT: A Multi-Turn Benchmark for Evaluating Agentic Time Series Reasoning

arXiv:2606.01498v1 Announce Type: cross Abstract: Time series data inform critical decisions across many real-world domains. While large language model (LLM) agents can analyze data through natural la

Local Ai

Beyond Accuracy: Evaluating Efficiency, Robustness and Explainability in Deep Learning for Malaria Diagnosis

arXiv:2605.30734v1 Announce Type: cross Abstract: Malaria remains a leading cause of mortality in sub-Saharan Africa, where scarce diagnostic infrastructure makes timely, accurate diagnosis particular

local-aiarxiv-cs-cv

Can Aerial VLA Models Cooperate? Evaluating Closed-Loop Air-Ground Coordination with CARLA-Air

arXiv:2605.31066v1 Announce Type: new Abstract: Recent aerial vision-language-action (VLA) models show promising single-UAV capabilities, such as tracking moving objects and navigating to language-spe

safetyarxiv-cs-ro

Evaluating using Mock Tool Calls to Quarantine Untrusted Prompt Inputs

arXiv:2605.30521v1 Announce Type: new Abstract: Large language models must frequently process untrusted inputs, such as judging an answer from another model or running tasks like spam and harm classif

From Weak Cues to Real Identities: Evaluating Inference-Driven De-Anonymization in LLM Agents

arXiv:2603.18382v2 Announce Type: replace Abstract: Anonymization is often assumed to protect privacy once explicit identifiers are removed, because re-identification has historically required special

Evaluating Cross-lingual Knowledge Consistency in Code-Mixed vis-a-vis Indian Languages using IndicKLAR

arXiv:2605.29637v1 Announce Type: new Abstract: Large language models recall knowledge reliably in English but often fail on the same query posed in a lower-resourced language -- a crosslingual consis