Search: “claude”

MultiEmo-Bench: Multi-label Visual Emotion Analysis for Multi-modal Large Language Models

arXiv:2605.14635v1 Announce Type: cross Abstract: This paper introduces a multi-label visual emotion analysis benchmark dataset for comprehensively evaluating the ability of multimodal large language

15 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

NodeSynth: Socially Aligned Synthetic Data for AI Evaluation

DGX agent

arXiv:2605.14381v1 Announce Type: cross Abstract: Recent advancements in generative AI facilitate large-scale synthetic data generation for model evaluation. However, without targeted approaches, thes

model-releasesarxiv-cs-cl

15 May 2026

Sycophancy is an Educational Safety Risk: Why LLM Tutors Need Sycophancy Benchmarks

arXiv:2605.14604v1 Announce Type: new Abstract: This position paper argues that effective tutoring requires corrective friction: surfacing misconceptions and challenging them supportively to drive con

15 May 2026

VerbalValue: A Socially Intelligent Virtual Host for Sales-Driven Live Commerce

arXiv:2605.14542v1 Announce Type: new Abstract: A skilled live-commerce host is not merely a narrator, but a sales agent who converts viewer curiosity into purchase intent through expert product knowl

15 May 2026

From Pixels to BFS: High Maze Accuracy Does Not Imply Visual Planning

model-releasesarxiv-cs-cv

arXiv:2603.26839v2 Announce Type: replace-cross Abstract: How do multimodal models solve visual spatial tasks -- through genuine planning, or through brute-force search in token space? We introduce ex

Grid-Orch: An LLM-Powered Orchestrator for Distribution Grid Simulation and Analytics

arXiv:2605.12728v1 Announce Type: cross Abstract: The power distribution engineering workforce faces a projected shortage of up to 1.5 million engineers by 2030, creating urgent demand for more access

GUIGuard-Bench: Toward a General Evaluation for Privacy-Preserving GUI Agents

arXiv:2601.18842v3 Announce Type: replace-cross Abstract: As GUI agents increasingly rely on screenshots to perceive and operate digital environments, they may inadvertently expose sensitive informati

Language Model Goal Selection Differs from Humans' in a Self-Directed Learning Task

arXiv:2603.03295v2 Announce Type: replace-cross Abstract: Whether in agentic workflows, social studies, or chat settings, large language models (LLMs) are increasingly being asked to replace humans in

RTLC -- Research, Teach-to-Learn, Critique: A three-stage prompting paradigm inspired by the Feynman Learning Technique that lifts LLM-as-judge accuracy on JudgeBench with no fine-tuning

arXiv:2605.13695v1 Announce Type: cross Abstract: LLM-as-a-judge is now the default measurement instrument for open-ended generation, but on the public JudgeBench benchmark even strong instruction-tun

AgentDisCo: Towards Disentanglement and Collaboration in Open-ended Deep Research Agents

arXiv:2605.11732v1 Announce Type: cross Abstract: In this paper, we present AgentDisCo, a novel Disentangled and Collaborative agentic architecture that formulates deep research as an adversarial opti

ClinicalBench: Stress-Testing Assertion-Aware Retrieval for Cross-Admission Clinical QA on MIMIC-IV

arXiv:2605.11143v1 Announce Type: new Abstract: Reasoning benchmarks measure clinical performance on clean inputs. We evaluate the step before reasoning: retrieval over real EHR notes, where negation,

Drop the Act: Probe-Filtered RL for Faithful Chain-of-Thought Reasoning

model-releasesarxiv-cs-lg

arXiv:2605.11467v1 Announce Type: new Abstract: Reasoning models post-hoc rationalize answers they have already committed to internally, producing chains of *reasoning theater*: deliberative-looking s

ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?

model-releasesarxiv-cs-lg

arXiv:2605.11086v1 Announce Type: cross Abstract: AI agents are rapidly gaining capabilities that could significantly reshape cybersecurity, making rigorous evaluation urgent. A critical capability is

MEME: Multi-entity & Evolving Memory Evaluation

arXiv:2605.12477v1 Announce Type: cross Abstract: LLM-based agents increasingly operate in persistent environments where they must store, update, and reason over information across many sessions. Whil

ShapeCodeBench: A Renewable Benchmark for Perception-to-Program Reconstruction of Synthetic Shape Scenes

model-releasesarxiv-cs-cv

arXiv:2605.11680v1 Announce Type: new Abstract: We introduce ShapeCodeBench, a synthetic benchmark for perception-to-program reconstruction: given a rendered raster image, a model must emit an executa

Solve the Loop: Attractor Models for Language and Reasoning

arXiv:2605.12466v1 Announce Type: cross Abstract: Looped Transformers offer a promising alternative to purely feed-forward computation by iteratively refining latent representations, improving languag

Three Regimes of Context-Parametric Conflict: A Predictive Framework and Empirical Validation

arXiv:2605.11574v1 Announce Type: new Abstract: The literature on how large language models handle conflict between their training knowledge and a contradicting document presents a persistent empirica

Beyond Accuracy: Evaluating Strategy Diversity in LLM Mathematical Reasoning

arXiv:2605.09292v1 Announce Type: new Abstract: Large language models now achieve high final-answer accuracy on mathematical reasoning benchmarks, but accuracy alone does not capture reasoning flexibi

Byte-Exact Deduplication in Retrieval-Augmented Generation: A Three-Regime Empirical Analysis Across Public Benchmarks

arXiv:2605.09611v1 Announce Type: new Abstract: This preprint presents an empirical analysis of byte-exact chunk-level deduplication in Retrieval-Augmented Generation (RAG) pipelines. We measure conte

CIVeX: Causal Intervention Verification for Language Agents

arXiv:2605.09168v1 Announce Type: new Abstract: A valid tool call is not necessarily a valid intervention. Tool-using language agents are guarded by schema validators, policy filters, provenance check

Combining Mechanical and Agentic Specification Inference for Move

arXiv:2605.10005v1 Announce Type: cross Abstract: In this paper, we describe early work on a specification inference tool for the Move Prover that combines a weakest-precondition (WP) analysis over Mo

ConFit v3: Improving Resume-Job Matching with LLM-based Re-Ranking

arXiv:2605.09760v1 Announce Type: new Abstract: A reliable resume-job matching system helps a company find suitable candidates from a pool of resumes and helps a job seeker find relevant jobs from a l

Context-Augmented Code Generation: How Product Context Improves AI Coding Agent Decision Compliance by 49%

arXiv:2605.08112v1 Announce Type: cross Abstract: AI coding agents powered by large language models can read codebases and produce functional code, but they routinely violate team-specific product dec

CrackMeBench: Binary Reverse Engineering for Agents

arXiv:2605.10597v1 Announce Type: cross Abstract: Benchmarks for coding agents increasingly measure source-level software repair, and cybersecurity benchmarks increasingly measure broad capture-the-fl

DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules

arXiv:2605.08614v1 Announce Type: new Abstract: Monitoring complex industrial assets relies on engineer-authored symbolic rules that trigger based on sensor conditions and prompt technicians to perfor

Explanation Fairness in Large Language Models: An Empirical Analysis of Disparities in How LLMs Justify Decisions Across Demographic Groups

arXiv:2605.08671v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed not only to make decisions but to explain them. While AI decision fairness has been studied ext

FormalRewardBench: A Benchmark for Formal Theorem Proving Reward Models

arXiv:2605.10141v1 Announce Type: new Abstract: Recent neural theorem provers use reinforcement learning with verifiable rewards (RLVR), where proof assistants provide binary correctness signals. Whil

GUARD: Guideline Upholding Test through Adaptive Role-play and Jailbreak Diagnostics for LLMs

arXiv:2508.20325v3 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) become increasingly integral to various domains, their potential to generate harmful responses has prompted si

Instruction Adherence in Coding Agent Configuration Files: A Factorial Study of Four File-Structure Variables

arXiv:2605.10039v1 Announce Type: cross Abstract: Frontier coding agents read configuration files (CLAUDE.md, AGENTS.md, Cursor Rules) at session start and are expected to follow the conventions insid

LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments

arXiv:2605.10779v1 Announce Type: cross Abstract: The rapid proliferation of LLM-based autonomous agents in real operating system environments introduces a new category of safety risk beyond content s

LLMs with in-context learning for Algorithmic Theoretical Physics

arXiv:2605.08212v1 Announce Type: cross Abstract: There is an increasing number of algorithmic computations in theoretical physics. These, while conceptually simple, can nevertheless be time-consuming

Magis-Bench: Evaluating LLMs on Magistrate-Level Legal Tasks

arXiv:2605.08437v1 Announce Type: cross Abstract: Existing benchmarks for legal AI focus primarily on tasks where LLMs must produce legal arguments or documents, yet the capacity to judge such argumen

MDGYM: Benchmarking AI Agents on Molecular Simulations

arXiv:2605.08941v1 Announce Type: new Abstract: The promise of AI-driven scientific discovery hinges on whether AI agents can autonomously design and execute the computational workflows that underpin

Metal-Sci: A Scientific Compute Benchmark for Evolutionary LLM Kernel Search on Apple Silicon

arXiv:2605.09708v1 Announce Type: cross Abstract: We present Metal-Sci, a 10-task benchmark of scientific Apple Silicon Metal compute kernels spanning six optimization regimes (stencils, all-pairs in

Overconfident and Blind to Details: Fixing Prompt Insensitivity with Abductive Preference Learning

arXiv:2510.09887v2 Announce Type: replace Abstract: Vision and language models frequently ignore semantically critical input edits, defaulting to pretraining priors. For example, models will confident

Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmark

arXiv:2410.14702v2 Announce Type: replace Abstract: Multi-modal Large Language Models (MLLMs) exhibit impressive problem-solving abilities in various domains, but their visual comprehension and abstra

Process Matters more than Output for Distinguishing Humans from Machines

arXiv:2605.06524v2 Announce Type: replace Abstract: Reliable human-machine discrimination is becoming increasingly important as large language models and autonomous agents are deployed in online setti

Recursive Language Models

arXiv:2512.24601v3 Announce Type: replace Abstract: We study allowing large language models (LLMs) to process arbitrarily long prompts through the lens of inference-time scaling. We propose Recursive

Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs

arXiv:2605.09063v1 Announce Type: new Abstract: Following the recent achievement of gold-medal performance on the IMO by frontier LLMs, the community is searching for the next meaningful and challengi

What's the plan? Metrics for implicit planning in LLMs and their application to rhyme generation and question answering

arXiv:2601.20164v2 Announce Type: replace-cross Abstract: Prior work suggests that language models, while trained on next token prediction, show implicit planning behavior: they may select the next to

When to Re-Commit: Temporal Abstraction Discovery for Long-Horizon Vision-Language Reasoning

arXiv:2605.09860v1 Announce Type: new Abstract: Long-horizon reasoning requires deciding not only what actions to take, but how deeply to commit before the next observation. We formalize this as commi

IntentGrasp: A Comprehensive Benchmark for Intent Understanding

arXiv:2605.06832v1 Announce Type: cross Abstract: Accurately understanding the intent behind speech, conversation, and writing is crucial to the development of helpful Large Language Model (LLM) assis

MathlibPR: Pull Request Merge-Readiness Benchmark for Formal Mathematical Libraries

arXiv:2605.07147v1 Announce Type: cross Abstract: The ecosystem of Lean and Mathlib has become the de facto standard for large language model (LLM) assisted formal reasoning with remarkable successes

MedVIGIL: Evaluating Trustworthy Medical VLMs Under Broken Visual Evidence

model-releasesarxiv-cs-cv

arXiv:2605.07919v1 Announce Type: new Abstract: Medical vision--language models (VLMs) are usually evaluated on intact image--question pairs, but trustworthy clinical use requires a stronger property:

MobileDev-Bench: A Benchmark for Issue Resolution in Mobile Application Development

model-releasesarxiv-cs-lg

arXiv:2603.24946v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown strong performance on automated software engineering tasks, yet existing benchmarks focus primarily on

MultiSoc-4D: A Benchmark for Diagnosing Instruction-Induced Label Collapse in Closed-Set LLM Annotation of Bengali Social Media

arXiv:2605.06940v1 Announce Type: new Abstract: Annotation automation via Large Language Models (LLMs) is the core approach for scaling NLP datasets; however, LLM behavior with respect to closed-set i

NSMQ Riddles: A Benchmark of Scientific and Mathematical Riddles for Quizzing Large Language Models

arXiv:2605.07051v1 Announce Type: new Abstract: Large Language Models (LLMs) have shown good performance on various science educational benchmarks, demonstrating their potential for use in science and

Quality-Conditioned Agreement in Automated Short Answer Scoring: Mid-Range Degradation and the Impact of Task-Specific Adaptation

arXiv:2605.07647v1 Announce Type: cross Abstract: Automated short answer scoring (ASAS) is shifting from discriminative, fine-tuned models to large language models (LLMs) used in few-shot settings. Th