Search: “benchmark”

arXiv:2603.08206v5 Announce Type: replace Abstract: Modern tabular foundation models such as TabPFN and TabICL naturally produce full predictive distributions, while the benchmarks used to evaluate th

researcharxiv-cs-lg

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

EBench: Elemental Diagnosis of Generalist Mobile Manipulation Policies

DGX agent

arXiv:2606.18239v2 Announce Type: replace Abstract: We present EBench, a simulation benchmark that diagnoses generalist mobile manipulation policies beyond a single success-rate scalar. EBench compris

model-releasesarxiv-cs-ro

23 Jun 2026

ELADO: Elliptic PDE Assessment Datasets for Operator Learning

arXiv:2606.20771v1 Announce Type: new Abstract: We introduce ELADO (Elliptic PDE Assessment Datasets for Operator Learning), a systematic benchmark suite constructed to show and quantify failure modes

HalluHard update: We’ve added GLM-5.2, using adaptive thinking with maximum reasoning effort, to our leaderboard. Despite its impressive per…

model-releasesgary-marcus--x

HalluHard update: We’ve added GLM-5.2, using adaptive thinking with maximum reasoning effort, to our leaderboard. Despite its impressive performance on other benchmarks, GLM-5.2 still hallucinates fre

Next-Gen CAPTCHAs: Leveraging the Cognitive Gap for Scalable and Diverse GUI-Agent Defense

arXiv:2602.09012v2 Announce Type: replace Abstract: The rapid evolution of GUI-enabled agents has rendered traditional CAPTCHAs obsolete. While previous benchmarks like OpenCaptchaWorld established a

Hardware

ParallelKernelBench: Frontier LLMs can't write fast multi-GPU kernels (yet)

ParallelKernelBench is a benchmark from Together AI that evaluates frontier large language models' ability to write optimized multi-GPU CUDA kernels, revealing significant limitations in current LLMs'

hardwaretogether-ai-blog

AI Coding Agents Can Reproduce Social Science Findings

arXiv:2606.11447v1 Announce Type: new Abstract: Recent anecdotal evidence suggests that AI coding agents can reproduce published findings when provided with original data and code; yet systematic eval

Every Act Has Its Price: Compressed Moral Composition in Frontier LLMs

arXiv:2606.11232v1 Announce Type: cross Abstract: Existing LLM moral benchmarks usually ask which isolated moral act, value, or foundation a model prefers. This is useful but incomplete. Realistic jud

LLMpedia: A Transparent Framework to Materialize an LLM's Encyclopedic Knowledge at Scale

arXiv:2603.24080v2 Announce Type: replace Abstract: Benchmarks like MMLU suggest flagship language models approach factuality saturation above 90%. LLMpedia shows this picture is incomplete. We materi

Sparse probes and murky physics: a case study of interpretability challenges in a foundation model for continuum dynamics

arXiv:2606.11657v1 Announce Type: cross Abstract: Generative AI emulators are increasingly used in scientific domains where we already have strong theory, benchmarks, and physical intuition. This rais

A Constrained Natural-Language Interface for Variational Multi-Physics Finite Element Simulations in FEniCS

arXiv:2606.10928v1 Announce Type: cross Abstract: Large language models can reduce the manual effort required to set up finite element simulations, but they introduce reliability risks when generated

Flaws in the LLM Automation Narrative

arXiv:2606.11166v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly described as performing at the level of human experts on knowledge economy tasks. These claims are prima

KCSAT-ML: Probing Reasoning Models with Nationwide-Cohort Human Difficulty

arXiv:2606.10403v1 Announce Type: new Abstract: Math reasoning benchmarks have proliferated, yet most lack a per-item difficulty signal grounded in actual human performance. We introduce KCSAT-ML, a d

LIBERO-Occ: Evaluating and Improving Vision-Language-Action Models under Scene-Induced Occlusion via Viewpoint Imagination

arXiv:2606.10862v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models achieve strong performance on standard manipulation benchmarks, but most evaluations assume that task-relevant obj

Research

RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty

arXiv:2602.12424v2 Announce Type: replace-cross Abstract: Benchmarks establish a standardized evaluation framework to systematically assess the performance of large language models (LLMs), facilitatin

researcharxiv-cs-ai

WorldOlympiad: Can Your World Model Survive a Triathlon?

model-releasesarxiv-cs-cv

arXiv:2606.11129v1 Announce Type: new Abstract: We introduce WorldOlympiad, a benchmark for diagnosing video-based world models across physical faithfulness, geometric consistency, and interaction fid

A Dataset for Dynamic Human Preferences for Vision Language Models

arXiv:2606.07653v1 Announce Type: cross Abstract: Given the increased adoption of Vision Language Models (VLMs) in human-interactive settings, it is important that we evaluate how well these models ca

ComplexConstraints and Beyond: Expert Rubrics for RLVR

arXiv:2606.09118v1 Announce Type: new Abstract: As LLM capabilities advance rapidly, the evaluation methods used to assess them increasingly lag behind. Traditional benchmarks relied on programmatic v

Safety

Culturally-Adapted Red-Teaming Across East and Southeast Asian Contexts: A Methodological and Comparative Analysis

arXiv:2606.09178v1 Announce Type: cross Abstract: Multilingual safety evaluation of large language models (LLMs) has predominantly relied on direct translation (DT) of English benchmarks into target l

safetyarxiv-cs-ai

Applications

Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm that Provably Exploits Model Similarity

arXiv:2606.07726v1 Announce Type: new Abstract: Large Language Models are typically benchmarked by evaluating every model on every test query. For practitioners seeking the best model to deploy, this

applicationsarxiv-cs-lg

Finite Certificates for In-Context Determinacy and a Threshold Theory of Emergence in Language Models

arXiv:2606.07623v1 Announce Type: new Abstract: This paper develops a model-theoretic framework for verifying context-conditioned language-model behavior by replacing benchmark labels with finite sema

GEAR-VLA: Learning Geometry-Aware Action Representations for Generalizable Robotic Manipulation

arXiv:2606.08530v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models achieve strong benchmark performance but still struggle in real-world deployment with unseen objects, background s

Report: GKE Inference Gateway delivers up to 92% faster AI responses

model-releasesgoogle-cloud-ai

As generative AI moves from experimental pilots to massive production environments, the efficiency of your infrastructure becomes the ultimate differentiator. One way to get the most out of it and min

Towards Personalized Bangla Book Recommendation: A Large-Scale Heterogeneous Book Graph Dataset

arXiv:2602.12129v2 Announce Type: replace-cross Abstract: Personalized book recommendation in Bangla literature has been constrained by the lack of structured, large-scale, and publicly available data

Local Ai

[3090] Gemma4 QAT + MTP quick TPS numbers [TLDR 1.2-1.8x better]

I'd need to search for this specific Reddit post to provide accurate details about the actual findings and technical specifics of this GPU performance benchmark. This post likely discusses throughput

local-air-localllama

Adaptive Conditional Forest Sampling for Spectral Risk Optimisation under Decision-Dependent Uncertainty

arXiv:2603.12507v2 Announce Type: replace Abstract: Minimising a spectral risk objective, defined as a weighted combination of expected cost and Conditional Value-at-Risk (CVaR), is challenging when t

CrowdMath: A Dataset of Crowdsourced Mathematical Research Discussions

arXiv:2606.06526v1 Announce Type: new Abstract: Large language models have made substantial progress on mathematical reasoning, but existing benchmarks typically evaluate well-specified problems with

Diagnosing Visual Ignorance in Vision-Language Models

model-releasesarxiv-cs-cv

arXiv:2606.06890v1 Announce Type: new Abstract: Vision-Language Models (VLMs) frequently rely on language priors, producing confident answers that are weakly grounded in visual evidence. While this be

mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?

arXiv:2606.07069v1 Announce Type: new Abstract: We introduce mmPISA-bench, a compact high-quality multilingual reasoning benchmark derived from the OECD Programme for International Student Assessment

Tree-of-Experience: A Structured Experience-Management Solution for Self-Evolving Agents under Low-Repetition and Implicit-Reward Environments

arXiv:2606.06960v1 Announce Type: new Abstract: Experience-based self-evolution is crucial for LLM agents, but existing benchmarks often assume explicit goals, stable task patterns, and clear feedback

Stability vs. Manipulability: Evaluating Robustness Under Post-Decision Interaction in LLM Judges

arXiv:2606.05384v1 Announce Type: cross Abstract: LLM-as-judge evaluation is widely used in benchmarking pipelines, where model outputs are compared and ranked using automated evaluators. These pipeli

5 Jun 2026

Research

A Latent Variable Framework for Scaling Laws in Large Language Models

arXiv:2512.06553v2 Announce Type: replace-cross Abstract: We propose a statistical framework built on latent variable modeling for scaling laws of large language models (LLMs). Our work is motivated b

researcharxiv-cs-lg

AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?

arXiv:2606.05080v1 Announce Type: new Abstract: Scientific and engineering progress is fundamentally a long-horizon iterative process: proposing changes, running experiments, measuring outcomes, and c

Tools

EVA-Bench Data 2.0: 3 Domains, 121 Tools, 213 Scenarios

EVA-Bench Data 2.0 is an expanded benchmark dataset containing tools and scenarios across 3 domains, featuring 121 tools and 213 test scenarios for evaluating AI agent performance. This dataset enable

toolshugging-face

Food-R1: A Unified Multi-Task Food Vision-Language Model with Reinforcement Learning

model-releasesarxiv-cs-cv

arXiv:2606.04986v1 Announce Type: new Abstract: Recent studies have explored Vision-Language Models (VLMs) for food analysis. However, most existing methods rely primarily on supervised fine-tuning (S

GENEB: Why Genomic Models Are Hard to Compare

arXiv:2606.04525v1 Announce Type: new Abstract: Progress in genomic foundation models is difficult to assess due to fragmented benchmarks, incompatible evaluation protocols, and task-specific reportin

Knowledge Index of Noah's Ark

arXiv:2606.05104v1 Announce Type: new Abstract: Knowledge benchmarks for LLMs face three issues: scaling-driven designs that do not operationalize disciplinary representativeness; flat-payment annotat

OckBench: Measuring the Efficiency of LLM Reasoning

arXiv:2511.05722v3 Announce Type: replace-cross Abstract: Large language models (LLMs) such as GPT-5 and Gemini 3 have pushed the frontier of automated reasoning and code generation. Yet current bench

Tools

Reality: The Final Eval — Lukas Petersson and Axel Backlund of Andon Labs

Lukas Petersson and Axel Backlund of Andon Labs discuss their work on evaluation methods and benchmarking for AI systems, likely covering approaches to assessing AI capabilities and safety in real-wor

toolslatent-space

TaDA: Calibrated Probe Gating for Task-Domain LoRA Merging

arXiv:2606.05016v1 Announce Type: new Abstract: Combining a task LoRA adapter with a domain LoRA adapter into a single unified model is a practical yet largely unexplored challenge. Existing methods t

Tree-Based Formalization of Multi-Agent Complementarity in Human-AI Interactions

arXiv:2606.04779v1 Announce Type: new Abstract: Complementarity is the case in which a human--AI interaction (HAI) outperforms the best prediction benchmark available among its members. Although this

Can Structural Cues Save LLMs? Evaluating Language Models in Massive Document Streams

arXiv:2603.19250v2 Announce Type: replace Abstract: Evaluating language models in streaming environments is critical, yet underexplored. Existing benchmarks either focus on single complex events or pr

Applications

Does Language Shift Break Medical Vision-Language Models? Indonesian Radiology Visual Question Answering Case Study

arXiv:2606.03693v1 Announce Type: new Abstract: Medical Vision-Language Models (VLMs) are typically evaluated on English radiology visual question answering benchmarks, leaving their robustness under

applicationsarxiv-cs-cl

eMEM: A Hybrid Spatio-Temporal Memory System For Embodied Agents

model-releasesarxiv-cs-ro

arXiv:2606.03374v1 Announce Type: new Abstract: We present eMEM (Embodied Memory), a hybrid graph-based memory system for embodied agents operating in physical environments. Current agent memory archi

RESCAST-100K: A Comprehensive Dataset for Cross-Domain Residential Load and Indoor Temperature Forecasting

arXiv:2606.02852v1 Announce Type: new Abstract: Accurate short-term forecasting of residential energy load and indoor temperature is essential for home energy management systems, grid-level demand res

Safety

Towards a Science of AI Agent Reliability

arXiv:2602.16666v3 Announce Type: replace Abstract: AI agents are increasingly deployed to execute important tasks. While rising accuracy scores on standard benchmarks suggest rapid progress, many age

safetyarxiv-cs-ai

Can we design legal agent verifiers that are up to 1,000x cheaper? Verifiers are LLM judges that check an agent’s work against rubric criter…

model-releasesharrison-chase--x

Can we design legal agent verifiers that are up to 1,000x cheaper? Verifiers are LLM judges that check an agent’s work against rubric criteria: they're used both in agent benchmarking and as reward si

2 Jun 2026

Cross-Environment Neural Reranking for Sample-Efficient Action Selection in Text-Based Agents

arXiv:2606.02204v1 Announce Type: new Abstract: Large language model agents achieve strong performance on text-based benchmarks but incur prohibitive inference costs, motivating the use of compact neu