AI Wiki
TimelineEvolutionGraphStatusLandscapesAsk wiki
Live from Git
AI Wiki
TimelineEvolutionGraphStatusLandscapesAsk wiki
Live from Git
Filter entries
Categories
  • All entries127,664
  • Agents10,695
  • Applications7,180
  • Companies3
  • Concepts7
  • Hardware2,794
  • Industry6,825
  • Local Ai7,052
  • Model Releases36,904
  • Research29,820
  • Safety19,755
  • Syntheses31
  • Tools1,781
  • Tutorials4,817

Source
HumanDGX agent

Content type
AllBlogX PostPaperYouTubeRedditGitHub
Clear filters
Categories
  • All entries127,664
  • Agents10,695
  • Applications7,180
  • Companies3
  • Concepts7
  • Hardware2,794
  • Industry6,825
  • Local Ai7,052
  • Model Releases36,904
  • Research29,820
  • Safety19,755
  • Syntheses31
  • Tools1,781
  • Tutorials4,817

Source
127,664Total entries
60Added by human
127,604Found by agent
13Categories

Knowledge catalogue

Search: “evaluation”

GridTimelineEvolution
41,190 results
20 May 2026

MapAnything: Evaluating Monocular Metric Depth Models for 3D Urban Asset Localization

ResearchDGX agent

arXiv:2509.14839v2 Announce Type: replace Abstract: City administrations increasingly rely on comprehensive databases and urban digital twins of city assets, such as traffic signs and trees, as well a

Multimodal evaluators: MLLM-as-a-judge for image-to-text tasks in Strands Evals

IndustryDGX agent

If you’re building visual shopping, image or document understanding, or chart analysis, you need a way to verify whether your model’s response is actually grounded in the source image. A text-only eva

ViroGym: Realistic Large-Scale Benchmarks for Evaluating Viral Proteins

Model ReleasesDGX agent

arXiv:2603.06740v2 Announce Type: replace-cross Abstract: Protein language models (pLMs) have shown strong potential for zero-shot prediction of missense variant effects, yet systematic benchmarking o

Human
DGX agent

Content type
AllBlogX PostPaperYouTubeRedditGitHub
Clear filters
19 May 2026

Evaluating AI Alignment in LLMs: Output Analysis of Value Priorities Across 75 Models with Human Benchmarking

SafetyDGX agent

arXiv:2506.12617v4 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used in human-AI interaction research and practice, yet existing capability and safety benchmarks reve

Evaluating Cognitive Age Alignment in Interactive AI Agents

Model ReleasesDGX agent

arXiv:2605.17894v1 Announce Type: new Abstract: While agentic AI and its core multimodal large language models (MLLMs) have demonstrated remarkable promise in language and visual reasoning across doma

Evaluating Deep Research Agents on Expert Consulting Work: A Benchmark with Verifiers, Rubrics, and Cognitive Traps

Model ReleasesDGX agent

arXiv:2605.17554v1 Announce Type: new Abstract: Frontier deep research agents (DRAs) plan a research task, synthesize across documents, and return a structured deliverable on demand. They are being de

From Pixels to Places: A Systematic Benchmark for Evaluating Image Geolocalization Ability in Large Language Models

Model ReleasesDGX agent

arXiv:2508.01608v2 Announce Type: replace Abstract: Image geolocalization, the task of identifying the geographic location depicted in an image, is important for applications in crisis response, digit

Hunt Instead of Wait: Evaluating Deep Data Research on Large Language Models

Model ReleasesDGX agent

arXiv:2602.02039v2 Announce Type: replace Abstract: The agency expected of Agentic Large Language Models goes beyond answering correctly, requiring autonomy to set goals and decide what to explore. We

The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort

Model ReleasesDGX agent

arXiv:2605.17062v1 Announce Type: cross Abstract: Spracklen et al. (USENIX Security '25) showed that code-generating large language models hallucinate package names that do not exist on PyPI or npm at

Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedback

Model ReleasesDGX agent

arXiv:2605.17453v1 Announce Type: cross Abstract: Tool-using LLM agents increasingly rely on external tools to make consequential decisions, yet most existing agent-security benchmarks and defenses im

18 May 2026

Evaluating Chinese Ambiguity Understanding in Large Language Models

Model ReleasesDGX agent

arXiv:2605.15635v1 Announce Type: new Abstract: Linguistic ambiguity is critical to the robustness of Large Language Models (LLMs), yet existing research focuses mostly on English, with limited attent

Large Language Models Could Be Rote Learners

Model ReleasesDGX agent

arXiv:2504.08300v5 Announce Type: replace-cross Abstract: Benchmark-based evaluation, e.g., multiple-choice questions (MCQs) and open-ended questions (OEQs), is widely used for evaluating Large Langua

LPDS: Evaluating LLM Robustness Through Logic-Preserving Difficulty Scaling

ResearchDGX agent

arXiv:2605.15393v1 Announce Type: new Abstract: As large language models (LLMs) are increasingly deployed to perform tasks with minimal human oversight, it is crucial that these models operate robustl

15 May 2026

Editor's Choice: Evaluating Abstract Intent in Image Editing through Atomic Entity Analysis

Model ReleasesDGX agent

arXiv:2605.14842v1 Announce Type: new Abstract: Humans naturally communicate through abstract concepts like 'mood'. However, current image editing benchmarks focus primarily on explicit, literal comma

FutureSim: Replaying World Events to Evaluate Adaptive Agents

Model ReleasesDGX agent

arXiv:2605.15188v1 Announce Type: cross Abstract: AI agents are being increasingly deployed in dynamic, open-ended environments that require adapting to new information as it arrives. To efficiently m

14 May 2026

KamonBench: A Grammar-Based Dataset for Evaluating Compositional Factor Recovery in Vision-Language Models

Model ReleasesDGX agent

arXiv:2605.13322v1 Announce Type: new Abstract: Kamon (family crests) are an important part of Japanese culture and a natural test case for compositional visual recognition: each crest combines a smal

LLMs as annotators of credibility assessment in Danish asylum decisions: evaluating classification performance and errors beyond aggregated metrics

Model ReleasesDGX agent

arXiv:2605.13412v1 Announce Type: cross Abstract: Off-the-shelf large language models (LLMs) are increasingly used to automate text annotation, yet their effectiveness remains underexplored for underr

13 May 2026

increasing levels of autonomy: /skill: preset prompts /plan: human-refined inputs /goal: AI-evaluated outputs

ToolsDGX agent

This post outlines a framework for progressively automating AI workflows through three levels of increasing autonomy: using preset prompts for consistency, refining inputs with human judgment to impro

12 May 2026

Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Values

Model ReleasesDGX agent

arXiv:2605.10365v1 Announce Type: new Abstract: Autonomous agents have rapidly matured as task executors and seen widespread deployment via harnesses such as OpenClaw. Safety concerns have rightly dra

Beyond Multiple Choice: Evaluating Steering Vectors for Summarization

SafetyDGX agent

arXiv:2505.24859v3 Announce Type: replace-cross Abstract: Steering vectors are a lightweight method for controlling text properties by adding a learned bias to language model activations at inference

BiAxisAudit: A Novel Framework to Evaluate LLM Bias Across Prompt Sensitivity and Response-Layer Divergence

Model ReleasesDGX agent

arXiv:2605.09041v1 Announce Type: new Abstract: Bias audits of large language models now operate within governance frameworks such as the EU AI Act, making benchmark reliability a security concern in

Can Language Models Analyze Data? Evaluating Large Language Models for Question Answering over Datasets

ResearchDGX agent

arXiv:2605.10419v1 Announce Type: cross Abstract: This paper investigates the effectiveness of large language models (LLMs) in answering questions over datasets. We examine their performance in two sc

Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability

AgentsDGX agent

arXiv:2605.10516v1 Announce Type: new Abstract: This paper establishes a rigorous measurement science for AI agent reliability, providing a foundational framework for quantifying consistency under sem

Evaluating Developmental Cognition Capabilities of LLMs

ResearchDGX agent

arXiv:2605.08549v1 Announce Type: new Abstract: Conversational AI is increasingly personalized around users' preferences, histories, goals, and knowledge, but much less around how users interpret and

Matching Meaning at Scale: Evaluating Semantic Search for 18th-Century Intellectual History through the Case of Locke

ResearchDGX agent

arXiv:2605.09236v1 Announce Type: cross Abstract: While digitized corpora have transformed the study of intellectual transmission, current methods rely heavily on lexical text reuse detection, capturi

Med-StepBench: A Hierarchical Reasoning Framework for Evaluating Hallucinations in Medical Vision-Language Models

Model ReleasesDGX agent

arXiv:2605.10002v1 Announce Type: new Abstract: Large vision-language models (VLMs) demonstrate strong performance in medical image understanding, but frequently generate clinically plausible yet inco

Neuroprobe: Evaluating Intracranial Brain Responses to Naturalistic Stimuli

ResearchDGX agent

arXiv:2509.21671v2 Announce Type: replace Abstract: High-resolution neural datasets enable foundation models for the next generation of brain-computer interfaces and neurological treatments. The commu

OPT-BENCH: Evaluating the Iterative Self-Optimization of LLM Agents in Large-Scale Search Spaces

Model ReleasesDGX agent

arXiv:2605.08904v1 Announce Type: new Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in reasoning and tool use. However, the fundamental cognitive faculties essential

The Cartesian Shortcut: Re-evaluate Vision Reasoning in Polar Coordinate Space

ResearchDGX agent

arXiv:2605.09883v1 Announce Type: cross Abstract: As current Multimodal Large Language Models rapidly saturate canonical visual reasoning benchmarks, a key question emerges: do these strong scores gen

11 May 2026

EnvSimBench: A Benchmark for Evaluating and Improving LLM-Based Environment Simulation

Model ReleasesDGX agent

arXiv:2605.07247v1 Announce Type: new Abstract: Scalable AI agents training relies on interactive environments that faithfully simulate the consequences of agent actions. Manually crafted environments

TeamBench: Evaluating Agent Coordination under Enforced Role Separation

Model ReleasesDGX agent

arXiv:2605.07073v1 Announce Type: new Abstract: Agent systems often decompose a task across multiple roles, but these roles are typically specified by prompts rather than enforced by access controls.

7 May 2026

KernelBench-X: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels

Model ReleasesDGX agent

arXiv:2605.04956v1 Announce Type: new Abstract: LLM-based Triton kernel generation has attracted significant interest, yet a fundamental empirical question remains unanswered: where does this capabili

NoisyCausal: A Benchmark for Evaluating Causal Reasoning Under Structured Noise

Model ReleasesDGX agent

arXiv:2605.04313v1 Announce Type: new Abstract: Causal reasoning in natural language requires identifying relevant variables, understanding their interactions, and reasoning about effects and interven

ReasonAudio: A Benchmark for Evaluating Reasoning Beyond Matching in Text-Audio Retrieval

Model ReleasesDGX agent

arXiv:2605.03361v2 Announce Type: new Abstract: As multimodal content continues to expand at a rapid pace, audio retrieval has emerged as a key enabling technology for media search, content organizati

StoryAlign: Evaluating and Training Reward Models for Story Generation

Model ReleasesDGX agent

arXiv:2605.04831v1 Announce Type: new Abstract: Story generation aims to automatically produce coherent, structured, and engaging narratives. Although large language models (LLMs) have significantly a

6 May 2026

12 Angry AI Agents: Evaluating Multi-Agent LLM Decision-Making Through Cinematic Jury Deliberation

Model ReleasesDGX agent

arXiv:2605.01986v1 Announce Type: new Abstract: What if the twelve jurors of Sidney Lumet's 12 Angry Men (1957) were not men, but large language models? Would the one juror who disagrees still be able

BIT.UA-AAUBS at ArchEHR-QA 2026: Evaluating Open-Source and Proprietary LLMs via Prompting in Low-Resource QA

Local AiDGX agent

arXiv:2605.03618v1 Announce Type: new Abstract: This paper presents the joint participation of the BIT.UA and AAUBS groups in the ArchEHR-QA 2026 shared task, which focuses on clinical question answer

Evaluating Generative Models as Interactive Emergent Representations of Human-Like Collaborative Behavior

ResearchDGX agent

arXiv:2605.03855v1 Announce Type: new Abstract: Human-AI collaboration requires AI agents to understand human behavior for effective coordination. While advances in foundation models show promising ca

Evaluating Reasoning Models for Queries with Presuppositions

ResearchDGX agent

arXiv:2605.03050v1 Announce Type: new Abstract: Millions of users turn to AI models for their information needs. It is conceivable that a large number of user queries contain assumptions that may be f

Repurposing and Evaluating the (In)Feasibility of Dataset Poisoning enabled Watermarking for Contrastive Learning

ResearchDGX agent

arXiv:2605.01834v1 Announce Type: cross Abstract: Contrastive learning (CL) reduces annotation cost via auto-derived supervisory signals. Since large-scale in-house CL datasets are infeasible, relianc

5 May 2026

Creating and Evaluating Figurative Language Dataset for Sindhi

Model ReleasesDGX agent

arXiv:2605.01323v1 Announce Type: new Abstract: In this article, we introduce SiNFluD, a novel benchmark dataset for Sindhi figurative language classification. We first collect raw text from various b

Dependency Parsing Across the Resource Spectrum: Evaluating Architectures on High and Low-Resource Languages

ResearchDGX agent

arXiv:2605.02608v1 Announce Type: new Abstract: Transformer-based models achieve state-of-the-art dependency parsing for high-resource languages, yet their advantage over simpler architectures in low-

From Concept to Capability: Evaluating 3D Gaussian Splatting for Synthetic Scene Editing in Autonomous Driving

SafetyDGX agent

arXiv:2605.01995v1 Announce Type: new Abstract: The perception of an Autonomous Driving System (ADS) critically depends on relevant, comprehensive, and diverse datasets to ensure its safety while oper

GR-Ben: A General Reasoning Benchmark for Evaluating Process Reward Models

Model ReleasesDGX agent

arXiv:2605.01203v1 Announce Type: cross Abstract: Currently, process reward models (PRMs) have exhibited remarkable potential for test-time scaling. Since large language models (LLMs) regularly genera

Pandora's Regret: A Proper Scoring Rule for Evaluating Sequential Search

Model ReleasesDGX agent

arXiv:2605.01936v1 Announce Type: new Abstract: In sequential search, alternatives are tested until the true class is found. Standard proper scoring rules like log loss are local, ignoring the ranking

4 May 2026

FinSafetyBench: Evaluating LLM Safety in Real-World Financial Scenarios

Model ReleasesDGX agent

arXiv:2605.00706v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly applied in financial scenarios. However, they may produce harmful outputs, including facilitating illegal

Image Score: Learning and Evaluating Human Preferences for Mercari Search

ResearchDGX agent

arXiv:2408.11349v2 Announce Type: replace Abstract: Mercari is the largest C2C e-commerce marketplace in Japan, having more than 20 million active monthly users. Search being the fundamental way to di

Linking Behaviour and Perception to Evaluate Meaningful Human Control over Partially Automated Driving

SafetyDGX agent

arXiv:2605.00556v1 Announce Type: cross Abstract: Partial driving automation creates a tension: drivers remain legally responsible for vehicle behaviour, yet their active control is significantly redu

SCAN: Structured Capability Assessment and Navigation for LLMs

ResearchDGX agent

arXiv:2505.06698v4 Announce Type: replace Abstract: Evaluating Large Language Models (LLMs) has become increasingly important, with automatic evaluation benchmarks gaining prominence as alternatives t

1 May 2026

Evaluating Epistemic Guardrails in AI Reading Assistants: A Behavioral Audit of a Minimal Prototype

ResearchDGX agent

arXiv:2604.27275v1 Announce Type: cross Abstract: Large language model (LLM) reading assistants are increasingly used in settings that require interpretation rather than simple retrieval. In these con

Evaluating TabPFN for Mild Cognitive Impairment to Alzheimer's Disease Conversion in Data Limited Settings

ResearchDGX agent

arXiv:2604.27195v1 Announce Type: new Abstract: Accurate prediction of conversion from Mild Cognitive Impairment (MCI) to Alzheimers Diseases (AD) is essential for early intervention, however, develop

30 Apr 2026

Beyond Screenshots: Evaluating VLMs' Understanding of UI Animations

ResearchDGX agent

arXiv:2604.26148v1 Announce Type: cross Abstract: AI agents operating on user interfaces must understand how interfaces communicate state and feedback to act reliably. As a core communicative modality

Evaluating Strategic Reasoning in Forecasting Agents

SafetyDGX agent

arXiv:2604.26106v1 Announce Type: new Abstract: Forecasting benchmarks produce accuracy leaderboards but little insight into why some forecasters are more accurate than others. We introduce Bench to t

Talent or Luck? Evaluating Attribution Bias in Large Language Models

SafetyDGX agent

arXiv:2505.22910v2 Announce Type: replace Abstract: When a student fails an exam, do we tend to blame their effort or the test's difficulty? Attribution, defined as how reasons are assigned to event o

29 Apr 2026

Evaluating Computational Pathology Foundation Models for Prostate Cancer Grading under Distribution Shifts

Model ReleasesDGX agent

arXiv:2410.06723v2 Announce Type: replace-cross Abstract: Pathology foundation models (PFMs) have emerged as powerful pretrained encoders for computational pathology, but their robustness under clinic

Thinking About Thinking: Evaluating Reasoning in Post-Trained Language Models

SafetyDGX agent

arXiv:2510.16340v2 Announce Type: replace Abstract: Recent advances in post-training techniques have endowed Large Language Models (LLMs) with enhanced capabilities for tackling complex, logic-intensi

28 Apr 2026

Evaluating CUDA Tile for AI Workloads on Hopper and Blackwell GPUs

Model ReleasesDGX agent

arXiv:2604.23466v1 Announce Type: cross Abstract: NVIDIA's CUDA Tile (CuTile) introduces a Python-based, tile-centric abstraction for GPU kernel development that aims to simplify programming while ret

Evaluating Jailbreaking Vulnerabilities in LLMs Deployed as Assistants for Smart Grid Operations: A Benchmark Against NERC Standards

Model ReleasesDGX agent

arXiv:2604.23341v1 Announce Type: cross Abstract: The deployment of Large Language Models (LLMs) as assistants in electric grid operations promises to streamline compliance and decision-making but exp

Game-Time: Evaluating Temporal Dynamics in Spoken Language Models

Model ReleasesDGX agent

arXiv:2509.26388v3 Announce Type: replace-cross Abstract: Conversational Spoken Language Models (SLMs) are emerging as a promising paradigm for real-time speech interaction. However, their capacity of

Reheat Nachos for Dinner? Evaluating AI Support for Cross-Cultural Communication of Neologisms

TutorialsDGX agent

arXiv:2604.23842v1 Announce Type: cross Abstract: Neologisms and emerging slang are central to daily conversation, yet challenging for non-native speakers (NNS) to interpret and use appropriately in c

← Previous
1…6768697071…687
Next →