Search: “benchmark”

RewardBench 2: Advancing Reward Model Evaluation

arXiv:2506.01937v2 Announce Type: replace Abstract: Reward models are used throughout the post-training of language models to capture nuanced signals from preference data and provide a training target

24 Apr 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledge

DGX agent

arXiv:2604.20389v1 Announce Type: cross Abstract: The rapid evolution and use of Large Language Models (LLMs) in professional workflows require an evaluation of their domain-specific knowledge against

model-releasesarxiv-cs-ai

23 Apr 2026

Interesting, OpenAI just released a free healthcare version of ChatGPT-5.4 for clinicians that beat specialty-matched physicians with unlimi…

model-releasesethan-mollick--x

Interesting, OpenAI just released a free healthcare version of ChatGPT-5.4 for clinicians that beat specialty-matched physicians with unlimited time + web access on a benchmark of real & hard clinical

23 Apr 2026

Owner-Harm: A Missing Threat Model for AI Agent Safety

arXiv:2604.18658v1 Announce Type: cross Abstract: Existing AI agent safety benchmarks focus on generic criminal harm (cybercrime, harassment, weapon synthesis), leaving a systematic blind spot for a d

22 Apr 2026

Automatic Slide Updating with User-Defined Dynamic Templates and Natural Language Instructions

arXiv:2604.17894v1 Announce Type: new Abstract: Presentation slides are a primary medium for data-driven reporting, yet keeping complex, analytics-style decks up to date remains labor-intensive. Exist

CaTS-Bench: Can Language Models Describe Time Series?

model-releasesarxiv-cs-cv

arXiv:2509.20823v5 Announce Type: replace-cross Abstract: Time series captioning, the task of describing time series in natural language, requires numeric and temporal reasoning, trend interpretation,

Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact

model-releasesarxiv-cs-lg

arXiv:2603.00883v2 Announce Type: replace Abstract: LLMs increasingly excel on AI benchmarks, but doing so does not guarantee validity for downstream tasks. This study contrasts LLM alignment on bench

Medical thinking with multiple images

arXiv:2604.16506v1 Announce Type: cross Abstract: Large language models perform well on many medical QA benchmarks, but real clinical reasoning often requires integrating evidence across multiple imag

MerLin: A Discovery Engine for Photonic and Hybrid Quantum Machine Learning

model-releasesarxiv-cs-lg

arXiv:2602.11092v2 Announce Type: replace Abstract: Identifying where quantum models may offer practical benefits in near term quantum machine learning (QML) requires moving beyond isolated algorithmi

MetaLint: Easy-to-Hard Generalization for Code Linting

arXiv:2507.11687v4 Announce Type: replace-cross Abstract: Large language models excel at code generation but struggle with code linting, particularly in generalizing to unseen or evolving best practic

Unmasking the Illusion of Embodied Reasoning in Vision-Language-Action Models

model-releasesarxiv-cs-ro

arXiv:2604.18000v1 Announce Type: new Abstract: Recent Vision-Language-Action (VLA) models report impressive success rates on standard robotic benchmarks, fueling optimism about general-purpose physic

Who is the richest club in the championship? Detecting and Rewriting Underspecified Questions Improve QA Performance

arXiv:2602.11938v5 Announce Type: replace Abstract: Large language models (LLMs) perform well on well-posed questions, yet standard question-answering (QA) benchmarks remain far from solved. We argue

Research

Comparison Study: Glacier Calving Front Delineation in Synthetic Aperture Radar Images With Deep Learning

arXiv:2501.05281v2 Announce Type: replace Abstract: Continuous monitoring of glacier calving fronts is essential for sea level rise projections. This study benchmarks Deep Learning systems for front d

researcharxiv-cs-cv

Agents

Discover and Prove: An Open-source Agentic Framework for Hard Mode Automated Theorem Proving in Lean 4

arXiv:2604.15839v1 Announce Type: new Abstract: Most ATP benchmarks embed the final answer within the formal statement -- a convention we call 'Easy Mode' -- a design that simplifies the task relative

agentsarxiv-cs-ai

DiZiNER: Disagreement-guided Instruction Refinement via Pilot Annotation Simulation for Zero-shot Named Entity Recognition

arXiv:2604.15866v1 Announce Type: cross Abstract: Large language models (LLMs) have advanced information extraction (IE) by enabling zero-shot and few-shot named entity recognition (NER), yet their ge

EvoTest: Evolutionary Test-Time Learning for Self-Improving Agentic Systems

arXiv:2510.13220v2 Announce Type: replace Abstract: A fundamental limitation of current AI agents is their inability to learn complex skills on the fly at test time, often behaving like 'clever but cl

'Excuse me, may I say something...' CoLabScience, A Proactive AI Assistant for Biomedical Discovery and LLM-Expert Collaborations

arXiv:2604.15588v1 Announce Type: cross Abstract: The integration of Large Language Models (LLMs) into scientific workflows presents exciting opportunities to accelerate biomedical discovery. However,

MEDLEY-BENCH: Scale Buys Evaluation but Not Control in AI Metacognition

arXiv:2604.16009v1 Announce Type: new Abstract: Metacognition, the ability to monitor and regulate one's own reasoning, remains under-evaluated in AI benchmarking. We introduce MEDLEY-BENCH, a benchma

Research

GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models

arXiv:2604.14262v1 Announce Type: new Abstract: GUI grounding models report over 85% accuracy on standard benchmarks, yet drop 27-56 percentage points when instructions require spatial reasoning rathe

researcharxiv-cs-lg

17 Apr 2026

Safety

Pushing the Boundaries of Multiple Choice Evaluation to One Hundred Options

arXiv:2604.14634v1 Announce Type: new Abstract: Multiple choice evaluation is widely used for benchmarking large language models, yet near ceiling accuracy in low option settings can be sustained by s

safetyarxiv-cs-cl

17 Apr 2026

Rethinking Patient Education as Multi-turn Multi-modal Interaction

arXiv:2604.14656v1 Announce Type: cross Abstract: Most medical multimodal benchmarks focus on static tasks such as image question answering, report generation, and plain-language rewriting. Patient ed

17 Apr 2026

From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs

arXiv:2604.14137v1 Announce Type: new Abstract: Evaluating LLMs is challenging, as benchmark scores often fail to capture models' real-world usefulness. Instead, users often rely on ``vibe-testing'':

LiteParse should be the default document parser you use with any AI agent (Claude Code, Claude Cowork, OpenClaw, Codex, and more) The core i…

model-releasesjerry-liu--x

LiteParse should be the default document parser you use with any AI agent (Claude Code, Claude Cowork, OpenClaw, Codex, and more) The core is extremely fast text and accurate parsing from any document

Qwen3.6-35B-A3B on my laptop drew me a better pelican than Claude Opus 4.7

model-releasessimon-willison

For anyone who has been (inadvisably) taking my pelican riding a bicycle benchmark seriously as a robust way to test models, here are pelicans from this morning's two big model releases - Qwen3.6-35B-

Scaling Test-Time Compute to Achieve IOI Gold Medal with Open-Weight Models

arXiv:2510.14232v2 Announce Type: replace-cross Abstract: Competitive programming has become a rigorous benchmark for evaluating the reasoning and problem-solving capabilities of large language models

WorkRB: A Community-Driven Evaluation Framework for AI in the Work Domain

arXiv:2604.13055v1 Announce Type: new Abstract: Today's evolving labor markets rely increasingly on recommender systems for hiring, talent management, and workforce analytics, with natural language pr

Beyond Scores: Diagnostic LLM Evaluation via Fine-Grained Abilities

arXiv:2604.12191v1 Announce Type: new Abstract: Current evaluations of large language models aggregate performance across diverse tasks into single scores. This obscures fine-grained ability variation

15 Apr 2026

Modality-Agnostic Prompt Learning for Multi-Modal Camouflaged Object Detection

model-releasesarxiv-cs-cv

arXiv:2604.12380v1 Announce Type: new Abstract: Camouflaged Object Detection (COD) aims to segment objects that blend seamlessly into complex backgrounds, with growing interest in exploiting additiona

15 Apr 2026

Research

To score 100% on a game, you just need to beat the median action efficiency of an unfiltered pool of random people. Easy if you're a bit sma…

researchfrancois-chollet--x

Francois Chollet makes the observation that achieving a perfect score on certain AI benchmark games or tasks only requires surpassing the median performance of an unfiltered random population sample,

15 Apr 2026

BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows

arXiv:2604.11304v1 Announce Type: new Abstract: Existing AI benchmarks lack the fidelity to assess economically meaningful progress on professional workflows. To evaluate frontier AI agents in a high-

ChatCLIDS: Simulating Persuasive AI Dialogues to Promote Closed-Loop Insulin Adoption in Type 1 Diabetes Care

arXiv:2509.00891v3 Announce Type: replace Abstract: Real-world adoption of closed-loop insulin delivery systems (CLIDS) in type 1 diabetes remains low, driven not by technical failure, but by diverse

DecepGPT: Schema-Driven Deception Detection with Multicultural Datasets and Robust Multimodal Learning

arXiv:2603.23916v2 Announce Type: replace-cross Abstract: Multimodal deception detection aims to identify deceptive behavior by analyzing audiovisual cues for forensics and security. In these high-sta

EgoFun3D: Modeling Interactive Objects from Egocentric Videos using Function Templates

model-releasesarxiv-cs-cv

arXiv:2604.11038v1 Announce Type: new Abstract: We present EgoFun3D, a coordinated task formulation, dataset, and benchmark for modeling interactive 3D objects from egocentric videos. Interactive obje

HiPRAG: Hierarchical Process Rewards for Efficient Agentic Retrieval Augmented Generation

arXiv:2510.07794v2 Announce Type: replace-cross Abstract: Agentic RAG is a powerful technique for incorporating external information that LLMs lack, enabling better problem solving and question answer

Learning World Models for Interactive Video Generation

arXiv:2505.21996v3 Announce Type: replace-cross Abstract: Foundational world models must be both interactive and preserve spatiotemporal coherence for effective future planning with action choices. Ho

LLMs for Text-Based Exploration and Navigation Under Partial Observability

arXiv:2604.09604v1 Announce Type: new Abstract: Exploration and goal-directed navigation in unknown layouts are central to inspection, logistics, and search-and-rescue. We ask whether large language m

What Users Leave Unsaid: Under-Specified Queries Limit Vision-Language Models

arXiv:2601.06165v2 Announce Type: replace-cross Abstract: Current vision-language benchmarks predominantly feature well-structured questions with clear, explicit prompts. However, real user queries ar

Safety

Do LLMs Follow Their Own Rules? A Reflexive Audit of Self-Stated Safety Policies

arXiv:2604.09189v1 Announce Type: cross Abstract: LLMs internalize safety policies through RLHF, yet these policies are never formally specified and remain difficult to inspect. Existing benchmarks ev

safetyarxiv-cs-ai

Do We Really Need to Approach the Entire Pareto Front in Many-Objective Bayesian Optimisation?

arXiv:2604.09417v1 Announce Type: new Abstract: Many-objective optimisation, a subset of multi-objective optimisation, involves optimisation problems with more than three objectives. As the number of

DRBENCHER: Can Your Agent Identify the Entity, Retrieve Its Properties and Do the Math?

arXiv:2604.09251v1 Announce Type: new Abstract: Deep research agents increasingly interleave web browsing with multi-step computation, yet existing benchmarks evaluate these capabilities in isolation,

From Paper to Program: Accelerating Quantum Many-Body Algorithm Development via a Multi-Stage LLM-Assisted Workflow

arXiv:2604.04089v2 Announce Type: replace-cross Abstract: Large language models (LLMs) can generate code rapidly but remain unreliable for scientific algorithms whose correctness depends on structural

Hidden in Plain Sight: Visual-to-Symbolic Analytical Solution Inference from Field Visualizations

arXiv:2604.08863v1 Announce Type: new Abstract: Recovering analytical solutions of physical fields from visual observations is a fundamental yet underexplored capability for AI-assisted scientific rea

HTNav: A Hybrid Navigation Framework with Tiered Structure for Urban Aerial Vision-and-Language Navigation

arXiv:2604.08883v1 Announce Type: cross Abstract: Inspired by the general Vision-and-Language Navigation (VLN) task, aerial VLN has attracted widespread attention, owing to its significant practical v

Memory-efficient Continual Learning with Prototypical Exemplar Condensation

arXiv:2603.13804v2 Announce Type: replace-cross Abstract: Rehearsal-based continual learning (CL) mitigates catastrophic forgetting by maintaining a subset of samples from previous tasks for replay. E

Temperature-Dependent Performance of Prompting Strategies in Extended Reasoning Large Language Models

arXiv:2604.08563v1 Announce Type: cross Abstract: Extended reasoning models represent a transformative shift in Large Language Model (LLM) capabilities by enabling explicit test-time computation for c

TiAb Review Plugin: A Browser-Based Tool for AI-Assisted Title and Abstract Screening

arXiv:2604.08602v1 Announce Type: cross Abstract: Background: Server-based screening tools impose subscription costs, while open-source alternatives require coding skills. Objectives: We developed a b

A Systematic Study of Retrieval Pipeline Design for Retrieval-Augmented Medical Question Answering

arXiv:2604.07274v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated strong capabilities in medical question answering; however, purely parametric models often suffer from

10 Apr 2026

CAMO: A Class-Aware Minority-Optimized Ensemble for Robust Language Model Evaluation on Imbalanced Data

arXiv:2604.07583v1 Announce Type: new Abstract: Real-world categorization is severely hampered by class imbalance because traditional ensembles favor majority classes, which lowers minority performanc