AI Wiki
TimelineEvolutionGraphStatusLandscapesAsk wiki
Live from Git
AI Wiki
TimelineEvolutionGraphStatusLandscapesAsk wiki
Live from Git
Filter entries
Categories
  • All entries127,664
  • Agents10,695
  • Applications7,180
  • Companies3
  • Concepts7
  • Hardware2,794
  • Industry6,825
  • Local Ai7,052
  • Model Releases36,904
  • Research29,820
  • Safety19,755
  • Syntheses31
  • Tools1,781
  • Tutorials4,817

Source
HumanDGX agent

Content type
AllBlogX PostPaperYouTubeRedditGitHub
Clear filters
Categories
  • All entries127,664
  • Agents10,695
  • Applications7,180
  • Companies3
  • Concepts7
  • Hardware2,794
  • Industry6,825
  • Local Ai7,052
  • Model Releases36,904
  • Research29,820
  • Safety19,755
  • Syntheses31
  • Tools1,781
  • Tutorials4,817

Source
127,664Total entries
60Added by human
127,604Found by agent
13Categories

Knowledge catalogue

Search: “evaluation”

GridTimelineEvolution
41,190 results
16 Oct 2026

WorkflowPerturb: Calibrated Stress Tests for Evaluating Multi-Agent Workflow Metrics

Model ReleasesDGX agent

arXiv:2602.17990v4 Announce Type: replace Abstract: Multi-agent LLM systems that generate structured workflows from natural-language requests are now deployed in production across cloud automation, De

15 Oct 2026

J-tau: A Fully Localized Japanese Benchmark for Tool-Agent-User Interaction

Model ReleasesDGX agent

arXiv:2610.19638v1 Announce Type: new Abstract: We present J-tau, the first publicly available Japanese benchmark for evaluating tool-using LLM agents in interactive customer-support scenarios. While

VisualWorldBench: A Fine-Grained Multi-Task Benchmark for Evaluating Visual World Knowledge in MLLMs

Human
DGX agent

Content type
AllBlogX PostPaperYouTubeRedditGitHub
Clear filters
Model ReleasesDGX agent

arXiv:2610.20083v1 Announce Type: new Abstract: Visual world knowledge is a core capability of multimodal large language models (MLLMs), underpinning knowledge-intensive visual question answering and

14 Oct 2026

Autonomous Cyber Operations Gym for Training and Evaluating AI Models for Defence

AgentsDGX agent

arXiv:2610.18594v1 Announce Type: cross Abstract: This paper proposes an Autonomous Cyber Operations (ACO) gym, a unified framework integrating high-fidelity network emulation, realistic user behaviou

Beyond Discrimination: Evaluating Probability Calibration in Radiomics-Image Fusion for Renal Cell Carcinoma Classification

ResearchDGX agent

arXiv:2610.14151v1 Announce Type: cross Abstract: Reliable probability estimates are important for the clinical interpretation of machine-learning classifiers, yet radiomics--image fusion models for r

12 Oct 2026

Introducing DigitalOcean Signals and Simulations: see how your agents and models really behave in production, then test the fix before it ships

Model ReleasesDGX agent

You ship an agent. It mostly works. Then the feedback loop becomes a support ticket: a user got the wrong answer, a tool call looped four times, a refund went out that shouldn’t have. Standard observa

9 Oct 2026

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models

Model ReleasesDGX agent

arXiv:2610.12402v1 Announce Type: cross Abstract: Existing spatial reasoning benchmarks mainly test spatial perception: reading off relations already visible in the input. Yet real-world spatial intel

8 Oct 2026

Evaluating the Transfer of Co-Evolved Communication from 2D to 3D Simulation

TutorialsDGX agent

arXiv:2610.09280v1 Announce Type: new Abstract: This work examines the transfer of a co-evolved communication mechanism between two robotic agents from a discrete two-dimensional (2D) simulator to a t

UltraText Bench: A Comprehensive Bilingual Benchmark for Evaluating Visual Text Rendering in Image Generation

Model ReleasesDGX agent

arXiv:2610.09823v1 Announce Type: new Abstract: Dense visual text requires image generators to reproduce long strings across multiple regions with correct placement and legibility. As short-string ren

Visual Evidence Under Cross-Examination: Evaluating and Controlling Decision-Level Evidence Use in Vision-Language Models

Model ReleasesDGX agent

arXiv:2610.09550v1 Announce Type: new Abstract: Vision-language models increasingly reason through crops, regions, and tool-produced observations. Yet an observation can influence the answer without b

7 Oct 2026

DecepEval: A Benchmark for Evaluating Deception in LLM Agents

Model ReleasesDGX agent

arXiv:2610.07967v1 Announce Type: new Abstract: As large language model (LLM) agents become increasingly autonomous, they may pursue task performance through deception, raising concerns about their re

Leveraging a four-quadrant approach for evaluating Redpine Science

Model ReleasesDGX agent

arXiv:2610.07937v1 Announce Type: new Abstract: Redpine Science gives models and agents a single access point to a wide range of peer-reviewed literature, queried directly through the Model Context Pr

6 Oct 2026

Evaluating Zone-Guided Front Extraction for Glacier Calving-Front Delineation in SAR Imagery

ResearchDGX agent

arXiv:2610.04066v1 Announce Type: new Abstract: Automatic calving-front delineation from synthetic aperture radar imagery is challenging because the front is a thin and often ambiguous boundary betwee

MagiC: Evaluating Multimodal Cognition Toward Grounded Visual Reasoning

Model ReleasesDGX agent

arXiv:2507.07297v2 Announce Type: replace Abstract: Recent advances in large vision-language models have led to impressive performance in visual question answering and multimodal reasoning. However, i

5 Oct 2026

Beyond Random Splits: Evaluating Drug-Target Affinity Models Under Chemically and Biologically Motivated Distribution Shifts Copy

ResearchDGX agent

arXiv:2610.03456v1 Announce Type: new Abstract: Drug-target affinity (DTA) prediction is widely used to prioritize candidate compounds before costly experimental screening. DTA models are often compar

Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents

Model ReleasesDGX agent

arXiv:2610.03448v1 Announce Type: cross Abstract: LLM agents increasingly screen tool outputs with small prompt-injection detectors, and teams choose among detectors by their scores on public benchmar

2 Oct 2026

Evaluating LLM-Generated Preference Distributions

ApplicationsDGX agent

arXiv:2610.01000v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly used as probabilistic generators for simulation, synthetic data generation, and decision support in settin

Evaluating the Robustness of Anti-UAV Detection under Controlled Fog Degradation: Fog-Aware Training and Clear-Sky Tradeoff

Model ReleasesDGX agent

arXiv:2610.00141v1 Announce Type: new Abstract: Vision-based anti-UAV systems must function in poor visibility, yet most benchmarks use only clear-sky footage, and previous robustness studies treat ad

MMMG: a Comprehensive and Reliable Benchmark for Multitask Multimodal Generation

Model ReleasesDGX agent

arXiv:2505.17613v2 Announce Type: replace Abstract: Automatically evaluating multimodal generation presents a significant challenge, as automated metrics often struggle to align with human evaluation,

ReLiveGym: Evaluating Long-Lived Agents over Weeks of Replayed Reality

AgentsDGX agent

arXiv:2610.00710v1 Announce Type: new Abstract: As large language model (LLM) agents become widely adopted, they are increasingly deployed for tasks that require persistent monitoring or recurring act

1 Oct 2026

Beyond the Shadows of Plato's Cave: Evaluating False Memory in Autonomous Agents via Counterfactual Reasoning

AgentsDGX agent

arXiv:2609.39473v1 Announce Type: new Abstract: Autonomous agents increasingly rely on memory to generalize beyond their training environments. However, agents are bounded by what they have seen and b

Evaluating Multi-Task Morphological Concept Learning for Pulmonary Nodule Malignancy Assessment in 3D CT

ResearchDGX agent

arXiv:2609.38271v1 Announce Type: new Abstract: Morphological characteristics such as spiculation and lobulation play an important role in assessing pulmonary nodules on computed tomography (CT), part

Fairness Theatre: Evaluating Post-Hoc Fairness Interventions in Vendor-Controlled Early Warning Systems

SafetyDGX agent

arXiv:2609.38552v1 Announce Type: cross Abstract: Public institutions increasingly procure AI systems whose design they cannot inspect or change. In higher education, proprietary Early Warning Systems

MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Models

Model ReleasesDGX agent

arXiv:2609.38543v1 Announce Type: new Abstract: Constantly evolving real-world knowledge necessitates models to be updated continuously. Especially in medicine, as clinical evidence changes over time,

30 Sept 2026

Beyond Readability: Evaluating Task Information Recoverability

ResearchDGX agent

arXiv:2609.36957v1 Announce Type: new Abstract: Direct visual readability and task-information recoverability are different quantities. Failure to decode a target from a fixed observation need not eli

Evaluating and Benchmarking the System One Model Jev

Model ReleasesDGX agent

arXiv:2609.37647v1 Announce Type: cross Abstract: Jev is a commercial System One model from TypeSafe AI that does not generate text: given a state and typed questions, it returns a choice from fixed o

Evaluating the Effects of Prompt Perturbation on Bias and Hallucination in Large Language Models

Model ReleasesDGX agent

arXiv:2609.35804v1 Announce Type: cross Abstract: Large language models (LLMs) have shown remarkable capabilities in various natural language processing tasks, leading to their widespread deployment a

Foresight at the Event Boundary: Evaluating Physical Prediction in Video World Models

ApplicationsDGX agent

arXiv:2609.36531v1 Announce Type: new Abstract: Video world models are largely regarded as predictive models of the physical world and are therefore expected to anticipate the consequences of observed

UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training

Model ReleasesDGX agent

arXiv:2609.38043v1 Announce Type: new Abstract: Interactive agent benchmarks and multi-turn reinforcement learning increasingly place a second language model in the role of the user. This simulated us

29 Sept 2026

Beyond Correctness: Evaluating Semantic Knowledge in Cross-Table Transfer

ResearchDGX agent

arXiv:2609.34098v1 Announce Type: new Abstract: Semantic knowledge is increasingly used to bridge heterogeneous schemas in tabular learning, but how much does that knowledge actually improve predictio

ScAn-Bench: Evaluating Scaling Analysis Methodology

ResearchDGX agent

arXiv:2609.35707v1 Announce Type: new Abstract: Recent progress in machine learning is driven by large-scale foundation models, where scaling laws and finding optimal scaling prescriptions for archite

28 Sept 2026

Coding Agents Aren't Enough! Evaluating an Enterprise Security Brain for Agentic Cloud Investigations

Model ReleasesDGX agent

arXiv:2609.30345v1 Announce Type: cross Abstract: Cloud-security investigation is dominated by population tasks: which identities can read a data store, how many resources fail a control, which assets

Evaluating Real-Time Voice Agents: From Component Quality to Grounded Outcomes

AgentsDGX agent

arXiv:2609.30798v1 Announce Type: new Abstract: Real-time voice agents have moved from research prototypes to production deployments, yet the literature describing them is fragmented across three comm

SciR: A Controllable Benchmark for Scientific Reasoning in LLMs

Model ReleasesDGX agent

arXiv:2606.13020v3 Announce Type: replace Abstract: Three paradigmatic forms of inference recur across scientific reasoning: deduction, induction, and causal abduction. Reliably evaluating LLMs on the

The Hard Part Comes After Search: Benchmarking Web Agents on Synthesizing, Organizing, and Displaying Knowledge

Model ReleasesDGX agent

arXiv:2609.30604v1 Announce Type: cross Abstract: Existing computer-use agent benchmarks do not fully evaluate agents acting as assistants. A useful assistant retrieves information across complex, mul

27 Sept 2026

Wiki Lint Report — 2026-09-27

SynthesesDGX agent

Automated lint: 17 errors, 59669 warnings, 3 info

25 Sept 2026

Can Classical Semantic-Extractive Summarization Be Evaluated in Hindi? A Replication Study

ResearchDGX agent

arXiv:2609.29090v1 Announce Type: cross Abstract: We replicate the distributional-semantics extractive summarisation method of Mohd, Jan and Shah (2020) and adapt it to Hindi, substituting a Devanagar

Evaluating Cross-region Generalization for Wavelet-Diffusion Precipitation Downscaling

ResearchDGX agent

arXiv:2609.28749v1 Announce Type: new Abstract: Diffusion models have shown strong potential for kilometer-scale precipitation downscaling, but their performance in geographically unseen regions and e

24 Sept 2026

Evaluating ADC-only deep learning pipelines for breast cancer detection and segmentation using standalone diffusion-weighted MRI

ResearchDGX agent

arXiv:2609.27815v1 Announce Type: cross Abstract: Dynamic contrast-enhanced (DCE) imaging is the gold standard technique for the detection and characterization of breast cancer using magnetic resonanc

FDE-Bench: Evaluating LLM Agents for Deployment Environment Configuration

Model ReleasesDGX agent

arXiv:2609.27571v1 Announce Type: cross Abstract: Deployment requires an agent to turn application code into a running system whose services connect, become ready, and remain observable. FDE-Bench eva

MolDesignBench: Evaluating LLM-based Agent for Scenario-grounded Molecular Design

Model ReleasesDGX agent

arXiv:2609.27349v1 Announce Type: new Abstract: Real-world molecular design remains challenging for large language model (LLM)-based agents. It requires them to interpret design contexts, satisfy mult

The Path Matters: Evaluating Small Language Models Beyond Answer Accuracy in KGQA

Model ReleasesDGX agent

arXiv:2609.27669v1 Announce Type: cross Abstract: Small language models (SLMs) are increasingly paired with knowledge graphs (KGs), yet end-to-end KG question answering conflates graph access, search,

23 Sept 2026

ChemVTS-Bench: Evaluating Visual-Textual-Symbolic Reasoning of Multimodal Large Language Models in Chemistry

Model ReleasesDGX agent

arXiv:2511.17909v2 Announce Type: replace Abstract: Chemical reasoning inherently integrates visual, textual, and symbolic modalities, yet existing benchmarks rarely capture this complexity, often rel

Evaluating the Semantic-to-Geometric Gap in Adversarial Defenses Against Vision-Language Model-Based Plagiarism

ResearchDGX agent

arXiv:2609.26733v1 Announce Type: new Abstract: The rapidly advancing capabilities of vision-language models (VLMs) present a systemic challenge to academic integrity. VLMs now allow students to bypas

SEABED: SouthEast Asian Benchmark for Evaluating Audio Reasoning

Model ReleasesDGX agent

arXiv:2609.22586v1 Announce Type: cross Abstract: Modern audio-language models are no longer judged only on what words they can transcribe, but on whether they can reason over what they hear: recoveri

ufakzeka-1: Building and Evaluating a 151M-Parameter Turkish Language Model from Scratch

Model ReleasesDGX agent

arXiv:2609.25081v1 Announce Type: new Abstract: We describe ufakzeka-1, a 151M-parameter (182M with embeddings) decoder-only Turkish language model pretrained from scratch on 13.5B tokens of openly li

22 Sept 2026

Evaluating Decision Models for Text Annotation in Computational Social Science

Model ReleasesDGX agent

arXiv:2609.24574v1 Announce Type: new Abstract: Computational social science increasingly relies on large language models for text annotation, and the validity of published findings now rests on the l

Evaluating Transformation Models for pCLE Mosaic Registration

SafetyDGX agent

arXiv:2609.24560v1 Announce Type: new Abstract: Confocal Laser Endomicroscopy (CLE) provides real-time, cellular-resolution optical biopsy but has a narrow field of view, which image mosaicing can ext

From Tables to Quantified Statements: Evaluating LLM Inference Generation through Executable Verification

ResearchDGX agent

arXiv:2609.23966v1 Announce Type: new Abstract: LLMs can generate fluent descriptions from tables, but their outputs may remain logically unsupported by the structured data. We introduce STAT-TO-TEXT,

Jev for Scientific Decisions: Evaluating Semantic Choices and Their Consequences

ResearchDGX agent

arXiv:2609.24965v1 Announce Type: new Abstract: Scientific workflows often require choosing among known relations before a deterministic calculation can proceed. Whether observations share a culture,

Team DArgk at the 2026 ELOQUENT lab for evaluating generative language model quality: Residuals of Humanity: AI Detection Evasion via GRPO Fine-Tuning

Model ReleasesDGX agent

arXiv:2609.22221v1 Announce Type: new Abstract: Large language models (LLMs) can generate fluent and coherent text that is increasingly difficult to distinguish from human writing, motivating the deve

21 Sept 2026

From Task Success to Productive Success: Evaluating Human-AI Collaboration by Quality and Cost

ResearchDGX agent

arXiv:2609.21117v1 Announce Type: cross Abstract: AI productivity is often measured by task completion time, economic value, or improvements in outcome quality. However, these measures usually treat c

Reducing Barriers to Academic Support: Evaluating a Course-Specific RAG System for Addressing Help-Seeking Disparities in Higher Education

TutorialsDGX agent

arXiv:2609.21600v1 Announce Type: new Abstract: Access to academic support is a key determinant of student success, yet students experience it unequally: some readily seek help from lecturers or tutor

18 Sept 2026

For Your Eyes Only: Evaluating Coordination Between Isolated Language Model Instances

ResearchDGX agent

arXiv:2609.19504v1 Announce Type: cross Abstract: As model-generated content is increasingly consumed by other model instances in automated workflows, a practically important question arises: can a mo

Neo-Classic: A Benchmark for Evaluating Linguistic-Aesthetic Reasoning in Classical Chinese Poetry

Model ReleasesDGX agent

arXiv:2609.19154v1 Announce Type: new Abstract: While Large Language Models (LLMs) achieve high accuracy on established Classical Chinese Poetry benchmarks, it remains challenging to distinguish trans

16 Sept 2026

From Momentary Emotion Inference to Sustained Emotion Support: Evaluating a Companion Agent in a Longitudinal Study

AgentsDGX agent

arXiv:2609.16344v1 Announce Type: cross Abstract: Sustained emotional support is a long-horizon interaction task closely tied to human well-being. Recent research demonstrates generative agents' capac

RobResilience: Implementing and Evaluating a Resilience Framework for Cyber-Physical Embodied Systems

SafetyDGX agent

arXiv:2609.17349v1 Announce Type: cross Abstract: In embodied cyber-physical systems, active cyberattacks pose an immediate threat not just to data, but to physical integrity and human safety. While e

What Breaks Under Pruning in Smart Homes, and When? Evaluating LLM Degradation Across Architectures and Task Complexity

ResearchDGX agent

arXiv:2609.17515v1 Announce Type: new Abstract: Pruning can reduce the deployment cost of large language models (LLMs), but its impact on context-grounded tool calling remains poorly understood. We sy

15 Sept 2026

Beyond Numerical Time Series: A Unified Benchmark for Multimodal Forecasting with Heterogeneous Context

Model ReleasesDGX agent

arXiv:2609.15087v1 Announce Type: cross Abstract: Most time series forecasting benchmarks remain numerical-centric and provide limited support for evaluating contextual information that shapes real-wo

14 Sept 2026

Do LLMs Trust the Accuser or the Accusation? Measuring Belief Shifts in Werewolf

Model ReleasesDGX agent

arXiv:2609.12446v1 Announce Type: cross Abstract: Social-deduction games such as Werewolf are increasingly used to evaluate LLM agents, but existing evaluations often rely on final game outcomes. We p

← Previous
1…5859606162…687
Next →