AI Wiki
TimelineEvolutionGraphStatusLandscapesAsk wiki
Live from Git
AI Wiki
TimelineEvolutionGraphStatusLandscapesAsk wiki
Live from Git
Filter entries
Categories
  • All entries127,664
  • Agents10,695
  • Applications7,180
  • Companies3
  • Concepts7
  • Hardware2,794
  • Industry6,825
  • Local Ai7,052
  • Model Releases36,904
  • Research29,820
  • Safety19,755
  • Syntheses31
  • Tools1,781
  • Tutorials4,817

Source
HumanDGX agent

Content type
AllBlogX PostPaperYouTubeRedditGitHub
Clear filters
Categories
  • All entries127,664
  • Agents10,695
  • Applications7,180
  • Companies3
  • Concepts7
  • Hardware2,794
  • Industry6,825
  • Local Ai7,052
  • Model Releases36,904
  • Research29,820
  • Safety19,755
  • Syntheses31
  • Tools1,781
  • Tutorials4,817

Source
127,664Total entries
60Added by human
127,604Found by agent
13Categories

Knowledge catalogue

Search: “evaluation”

GridTimelineEvolution
41,190 results
21 May 2026

Reasoning-Trace Collapse: Evaluating the Loss of Explicit Reasoning During Fine-Tuning

ResearchDGX agent

arXiv:2605.21127v1 Announce Type: new Abstract: Explicit reasoning models are trained to produce intermediate reasoning traces before final answers, but downstream fine-tuning is often performed on or

20 May 2026

A Family of Divergence Measures for Evaluating the Reconstruction Quality of Explainable Ensemble Trees

Model ReleasesDGX agent

arXiv:2605.19618v1 Announce Type: new Abstract: Validating interpretable surrogate models for ensemble learners requires measuring agreement between the ensemble's internal representation and its surr

Evaluating Memory Condensation Strategies for Coding Agents in Data-Driven Scientific Discovery

Human
DGX agent

Content type
AllBlogX PostPaperYouTubeRedditGitHub
Clear filters
TutorialsDGX agent

arXiv:2605.18854v1 Announce Type: new Abstract: Coding agents accumulate extensive context during long-running tasks, yet fixed context windows force practitioners to choose between truncation and tas

FedMental: Evaluating Federated Learning for Mental Health Detection from Social Media Data

Model ReleasesDGX agent

arXiv:2605.18936v1 Announce Type: cross Abstract: Social media text data are often used to train Machine Learning (ML) models to identify users exhibiting high-risk mental health behaviors. However, s

19 May 2026

LongMINT: Evaluating Memory under Multi-Target Interference in Long-Horizon Agent Systems

Model ReleasesDGX agent

arXiv:2605.18565v1 Announce Type: cross Abstract: Real-world agents operate over long and evolving horizons, where information is repeatedly updated and may interfere across memories, requiring accura

OmniCode: A Benchmark for Evaluating Software Engineering Agents

Model ReleasesDGX agent

arXiv:2602.02262v3 Announce Type: replace-cross Abstract: LLM-powered coding agents are redefining how real-world software is developed. To drive the research towards better coding agents, we require

SwordBench: Evaluating Orthogonality of Steering Image Representations

Model ReleasesDGX agent

arXiv:2605.16372v1 Announce Type: cross Abstract: Steering or intervening on model representations at inference time to correct predictions is essential for AI interpretability and safety, yet existin

18 May 2026

IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia

Model ReleasesDGX agent

arXiv:2603.17915v2 Announce Type: replace-cross Abstract: As large language models (LLMs) are deployed in multilingual settings, their safety behavior in culturally diverse, low-resource languages rem

RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades

Model ReleasesDGX agent

arXiv:2605.15846v1 Announce Type: cross Abstract: Coding agents are increasingly deployed in real software development, where a single version iteration requires months of coordinated work across many

15 May 2026

MechVerse: Evaluating Physical Motion Consistency in Video Generation Models

Model ReleasesDGX agent

arXiv:2605.14843v1 Announce Type: new Abstract: Text- and image-conditioned video generation models have achieved strong visual fidelity and temporal coherence, but they often fail to generate motion

pi-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows

Model ReleasesDGX agent

arXiv:2605.14678v1 Announce Type: new Abstract: The rise of personal assistant agents, e.g., OpenClaw, highlights the growing potential of large language models to support users across everyday life a

RxEval: A Prescription-Level Benchmark for Evaluating LLM Medication Recommendation

Model ReleasesDGX agent

arXiv:2605.14543v1 Announce Type: cross Abstract: Inpatient medication recommendation requires clinicians to repeatedly select specific medications, doses, and routes as a patient's condition evolves.

14 May 2026

Re-evaluating Minimum Bayes Risk Decoding for Automatic Speech Recognition

ResearchDGX agent

arXiv:2510.19471v2 Announce Type: replace-cross Abstract: Recent work has shown that sample-based Minimum Bayes Risk (MBR) decoding outperforms beam search in text-to-text generation tasks, such as ma

TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraints

AgentsDGX agent

arXiv:2605.13414v1 Announce Type: new Abstract: Deploying language models as autonomous agents requires more than per-task accuracy: when an agent faces a queue of problems under a finite token budget

13 May 2026

EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages

Model ReleasesDGX agent

arXiv:2603.09678v2 Announce Type: replace-cross Abstract: Large language models achieve near-ceiling performance on code generation benchmarks, yet most of the programming languages used by popular be

GeoR-Bench: Evaluating Geoscience Visual Reasoning

Model ReleasesDGX agent

arXiv:2605.11541v1 Announce Type: new Abstract: Geoscience intelligence is expected to understand, reason about, and predict earth system changes to support human decision-making in critical domains s

LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues

Model ReleasesDGX agent

arXiv:2605.12493v1 Announce Type: new Abstract: Long-term memory is crucial for agents in specialized web environments, where success depends on recalling interface affordances, state dynamics, workfl

12 May 2026

Beyond Accuracy: Evaluating Strategy Diversity in LLM Mathematical Reasoning

Model ReleasesDGX agent

arXiv:2605.09292v1 Announce Type: new Abstract: Large language models now achieve high final-answer accuracy on mathematical reasoning benchmarks, but accuracy alone does not capture reasoning flexibi

Can Deep Research Agents Retrieve and Organize? Evaluating the Synthesis Gap with Expert Taxonomies

Model ReleasesDGX agent

arXiv:2601.12369v3 Announce Type: replace Abstract: Deep Research Agents increasingly automate survey generation, yet whether they match human experts at retrieving essential papers and organizing the

CodeClinic: Evaluating Automation of Coding Skills for Clinical Reasoning Agents

Model ReleasesDGX agent

arXiv:2605.09675v1 Announce Type: new Abstract: Clinical reasoning agents based on large language models (LLMs) aim to automate tasks such as intensive care unit (ICU) monitoring and patient state tra

Evaluating Federated Learning approaches for mammography under breast density heterogeneity

Local AiDGX agent

arXiv:2605.09137v1 Announce Type: new Abstract: Breast density is a key factor that influences mammography interpretation and is a major source of heterogeneity in multicenter datasets. Such heterogen

Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness

Model ReleasesDGX agent

arXiv:2605.10379v1 Announce Type: new Abstract: Large language models (LLMs) have become capable mathematical problem-solvers, often producing correct proofs for challenging problems. However, correct

PHMForge: Evaluating LLM Agents on Industrial Prognostics through MCP-Native, Algorithm-Grounded Tools

SafetyDGX agent

arXiv:2604.01532v2 Announce Type: replace Abstract: LLM agents are beginning to invoke industrial asset-management tools through the Model Context Protocol (MCP), yet whether they can act reliably on

Position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead

Model ReleasesDGX agent

arXiv:2507.23009v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have achieved remarkable results on a range of standardized tests originally designed to assess human cognitive a

Single-Thread JPEG Decoder Benchmarks Mis-Evaluate ML Data Loaders

Model ReleasesDGX agent

arXiv:2605.08731v1 Announce Type: cross Abstract: JPEG decode is routine ML infrastructure, but Python decoder choices are often justified by single-process, single-thread microbenchmarks. We audit th

Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs

Model ReleasesDGX agent

arXiv:2605.09063v1 Announce Type: new Abstract: Following the recent achievement of gold-medal performance on the IMO by frontier LLMs, the community is searching for the next meaningful and challengi

Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearning

SafetyDGX agent

arXiv:2605.08765v1 Announce Type: cross Abstract: Unlearning in large language models (LLMs) aims to remove harmful training data while preserving overall utility. However, we find that existing metho

11 May 2026

RuleSafe-VL: Evaluating Rule-Conditioned Decision Reasoning in Vision-Language Content Moderation

Model ReleasesDGX agent

arXiv:2605.07760v1 Announce Type: new Abstract: Platform content moderation applies explicit policy rules and context-dependent conditions to decide whether user content is allowed, restricted, or rem

WiCER: Wiki-memory Compile, Evaluate, Refine Iterative Knowledge Compilation for LLM Wiki Systems

Model ReleasesDGX agent

arXiv:2605.07068v1 Announce Type: cross Abstract: The LLM Wiki pattern, to compile and provide domain knowledge into a persistent artifact and serve it to LLMs via KV cache inference, promises context

7 May 2026

Evaluating Patient Safety Risks in Generative AI: Development and Validation of a FMECA Framework for Generated Clinical Content

SafetyDGX agent

arXiv:2605.04085v1 Announce Type: cross Abstract: Objectives: Large language models (LLMs) are increasingly used for clinical text summarization, yet structured methods to assess associated patient sa

MRI-Eval: A Tiered Benchmark for Evaluating LLM Performance on MRI Physics and GE Scanner Operations Knowledge

Model ReleasesDGX agent

arXiv:2605.05175v1 Announce Type: cross Abstract: Background: Existing MRI LLM benchmarks rely mainly on review-book multiple-choice questions, where top proprietary models already score highly, limit

6 May 2026

CreativityBench: Evaluating Agent Creative Reasoning via Affordance-Based Tool Repurposing

Model ReleasesDGX agent

arXiv:2605.02910v1 Announce Type: cross Abstract: Recent advances in large language models have led to strong performance on reasoning and environment-interaction tasks, yet their ability for creative

EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solving

Model ReleasesDGX agent

arXiv:2509.17677v2 Announce Type: replace Abstract: Large language models (LLMs) have shown strong performance on mathematical reasoning under well-defined conditions. However, real-world engineering

TriBench-Ko: Evaluating LLM Risks in Judicial Workflows

Model ReleasesDGX agent

arXiv:2605.03792v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly integrated into legal workflows. However, existing benchmarks primarily address proxy tasks, such as bar e

5 May 2026

Can LLMs Compress (and Decompress)? Evaluating Code Understanding and Execution via Invertibility

Model ReleasesDGX agent

arXiv:2601.13398v2 Announce Type: replace Abstract: LLMs demonstrate strong performance on code benchmarks, yet consistent reasoning across forward and backward execution remains elusive. We present R

Evaluating LLMs on Large-Scale Graph Property Estimation via Random Walks

Model ReleasesDGX agent

arXiv:2605.01484v1 Announce Type: new Abstract: With the rapidly improving reasoning abilities of Large Language Models (LLMs), there is also a rising demand to use them in a wide variety of domains.

Evaluating Tabular Representation Learning for Network Intrusion Detection

Model ReleasesDGX agent

arXiv:2605.02519v1 Announce Type: new Abstract: Classic Network Intrusion Detection Systems (NIDS) often rely on manual feature engineering to extract meaningful patterns from network traffic data. Ho

4 May 2026

BanglaSocialBench: A Benchmark for Evaluating Sociopragmatic and Cultural Alignment of LLMs in Bangladeshi Social Interaction

Model ReleasesDGX agent

arXiv:2603.15949v3 Announce Type: replace Abstract: Large Language Models have demonstrated strong multilingual fluency, yet fluency alone does not guarantee socially appropriate language use. In high

Evaluating the Architectural Reasoning Capabilities of LLM Provers via the Obfuscated Natural Number Game

Model ReleasesDGX agent

arXiv:2605.00677v1 Announce Type: new Abstract: While Large Language Models have achieved notable success on formal mathematics benchmarks such as MiniF2F, it remains unclear whether these results ste

1 May 2026

CausalCompass: Evaluating the Robustness of Time-Series Causal Discovery in Misspecified Scenarios

Model ReleasesDGX agent

arXiv:2602.07915v2 Announce Type: replace-cross Abstract: Causal discovery from time series is a fundamental task in machine learning. However, its widespread adoption is hindered by a reliance on unt

WebMall -- A Multi-Shop Benchmark for Evaluating Web Agents

Model ReleasesDGX agent

arXiv:2508.13024v3 Announce Type: replace Abstract: LLM-based web agents have the potential to automate long-running web tasks, such as searching for products in multiple e-shops and subsequently orde

30 Apr 2026

Ceci n'est pas une explication: Evaluating Explanation Failures as Explainability Pitfalls in Language Learning Systems

Model ReleasesDGX agent

arXiv:2604.26145v1 Announce Type: cross Abstract: AI-powered language learning tools increasingly provide instant, personalised feedback to millions of learners worldwide. However, this feedback can f

Evaluating the Alignment Between GeoAI Explanations and Domain Knowledge in Satellite-Based Flood Mapping

SafetyDGX agent

arXiv:2604.26051v1 Announce Type: cross Abstract: The increasing number of satellites has improved the temporal resolution of Earth observation, making satellite-based flood mapping a promising approa

Value-Guided Iterative Refinement and the DIQ-H Benchmark for Evaluating VLM Robustness

Model ReleasesDGX agent

arXiv:2512.03992v2 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) are essential for embodied AI and safety-critical applications, such as robotics and autonomous systems. However

29 Apr 2026

RELIC: Evaluating Complex Reasoning via the Recognition of Languages In-Context

ApplicationsDGX agent

arXiv:2506.05205v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used to solve complex tasks where they must retrieve and compose many pieces of in-context information

28 Apr 2026

AgentPulse: A Continuous Multi-Signal Framework for Evaluating AI Agents in Deployment

Model ReleasesDGX agent

arXiv:2604.24038v1 Announce Type: new Abstract: Static benchmarks measure what AI agents can do at a fixed point in time but not how they are adopted, maintained, or experienced in deployment. We intr

Can LLMs Act as Historians? Evaluating Historical Research Capabilities of LLMs via the Chinese Imperial Examination

Model ReleasesDGX agent

arXiv:2604.24690v1 Announce Type: new Abstract: While Large Language Models (LLMs) have increasingly assisted in historical tasks such as text processing, their capacity for professional-level histori

EgoDyn-Bench: Evaluating Ego-Motion Understanding in Vision-Centric Foundation Models for Autonomous Driving

Model ReleasesDGX agent

arXiv:2604.22851v1 Announce Type: cross Abstract: While Vision-Language Models (VLMs) have advanced highlevel reasoning in autonomous driving, their ability to ground this reasoning in the underlying

Mind the Gap: Evaluating Model- and Agentic-Level Vulnerabilities in LLMs with Action Graphs

Model ReleasesDGX agent

arXiv:2509.04802v3 Announce Type: replace Abstract: As large language models increasingly deployed into agentic systems, existing methods face critical gaps in observing, assessing, and mitigating dep

Quantifying and Mitigating Self-Preference Bias of LLM Judges

SafetyDGX agent

arXiv:2604.22891v1 Announce Type: cross Abstract: LLM-as-a-Judge has become a dominant approach in automated evaluation systems, playing critical roles in model alignment, leaderboard construction, qu

24 Apr 2026

ChessArena: A Chess Testbed for Evaluating Strategic Reasoning Capabilities of Large Language Models

ResearchDGX agent

arXiv:2509.24239v4 Announce Type: replace-cross Abstract: Recent large language models (LLMs) have shown strong reasoning capabilities. However, a critical question remains: do these models possess ge

23 Apr 2026

ThermoQA: A Three-Tier Benchmark for Evaluating Thermodynamic Reasoning in Large Language Models

Model ReleasesDGX agent

arXiv:2604.19758v1 Announce Type: new Abstract: We present ThermoQA, a benchmark of 293 open-ended engineering thermodynamics problems in three tiers: property lookups (110 Q), component analysis (101

22 Apr 2026

LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models

Model ReleasesDGX agent

arXiv:2604.18803v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) are increasingly deployed in settings where reliable visual grounding carries operational consequences, yet their behavi

21 Apr 2026

Beyond 'I Don't Know': Evaluating LLM Self-Awareness in Discriminating Data and Model Uncertainty

Model ReleasesDGX agent

arXiv:2604.17293v1 Announce Type: new Abstract: Reliable Large Language Models (LLMs) should abstain when confidence is insufficient. However, prior studies often treat refusal as a generic 'I don't k

Evaluating Temporal and Structural Anomaly Detection Paradigms for DDoS Traffic

ResearchDGX agent

arXiv:2604.16575v1 Announce Type: new Abstract: Unsupervised anomaly detection is widely used to detect Distributed Denial-of-Service (DDoS) attacks in cloud-native 5G networks, yet most studies assum

OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation

Model ReleasesDGX agent

arXiv:2506.05606v5 Announce Type: replace Abstract: Can large language models (LLMs) accurately simulate the next web action of a specific user? While LLMs have shown promising capabilities in generat

When Visuals Aren't the Problem: Evaluating Vision-Language Models on Misleading Data Visualizations

Model ReleasesDGX agent

arXiv:2603.22368v2 Announce Type: replace Abstract: Visualizations help communicate data insights, but deceptive data representations can distort their interpretation and propagate misinformation. Whi

Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains

ResearchDGX agent

arXiv:2604.17393v1 Announce Type: new Abstract: Automatic evaluation metrics are central to the development of machine translation systems, yet their robustness under domain shift remains unclear. Mos

20 Apr 2026

Evaluating LLM Simulators as Differentially Private Data Generators

AgentsDGX agent

arXiv:2604.15461v1 Announce Type: cross Abstract: LLM-based simulators offer a promising path for generating complex synthetic data where traditional differentially private (DP) methods struggle with

Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidence

Model ReleasesDGX agent

arXiv:2603.13091v2 Announce Type: replace Abstract: The growing interest in embodied agents increases the demand for spatiotemporal video understanding, yet existing benchmarks largely emphasize extra

← Previous
1…6263646566…687
Next →