Search: “benchmark”

Beyond Accuracy: Community Perspectives on Machine Translation

arXiv:2606.09655v2 Announce Type: replace Abstract: Despite remarkable progress in machine translation (MT), non-AI communities have raised growing concerns about MT systems, suggesting a noticeable g

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Caught in the Story: Narrative Captivity in Multi-turn LLMs Conversation

DGX agent

arXiv:2609.03407v1 Announce Type: new Abstract: People increasingly turn to large language models (LLMs) for everyday advice, making ethically charged interpersonal problems a practical moral-advisory

model-releasesarxiv-cs-ai

4 Sept 2026

Do GUI Agents Know When Not to Act? Enabling Conflict-Aware Termination for Multimodal GUI Agents

arXiv:2609.03438v1 Announce Type: new Abstract: Graphical user interface (GUI) agents are increasingly used to execute natural-language instructions on user interfaces, yet real users may issue infeas

Inferred Generative-Process Diversity Predicts Correlated Failure Across Language Models

model-releasesarxiv-cs-lg

arXiv:2609.03422v1 Announce Type: new Abstract: Diversity is a widely observed factor in the resilient function of collective systems, yet the type of diversity that matters depends on the properties

Investigating the Ability of Large Language Models to Analyze Recipes for Diabetes

arXiv:2609.03967v1 Announce Type: cross Abstract: Several studies have evaluated the ability of Large Language Models (LLMs) for meal planning, yielding positive outcomes. These models can process nat

IRWOZ 2.0: A Large Language Model-driven Dialogue Dataset for Industrial Robot Conversations

arXiv:2609.04030v1 Announce Type: new Abstract: IRWOZ has improved industrial human-robot interaction (HRI) dialogue systems through domain-specific annotations. However, its initial version contains

It happened again... this time OpenAI's rogue agents cyber-attacked (well, spammed) a dormant German wiki and used it to share the answers t…

model-releasessimon-willison--x

It happened again... this time OpenAI's rogue agents cyber-attacked (well, spammed) a dormant German wiki and used it to share the answers to a benchmark they were training against https://simonwillis

Making Every Tool Call Count: Necessary Tool-Evidence Path Rewards for Agentic Vision-Language Models

arXiv:2609.03493v1 Announce Type: new Abstract: Modern vision-language models (VLMs) can directly answer many image-grounded questions, yet they often struggle with complex queries requiring fine-grai

Pattern Over-Generalization of Knowledge Graph Embedding

arXiv:2609.03487v1 Announce Type: cross Abstract: Knowledge graph embedding (KGE) demonstrates its effectiveness for predicting missing links in knowledge graphs (KGs) by projecting entities and relat

Representational alignment yields generalizable safety in language models

arXiv:2609.04022v1 Announce Type: cross Abstract: Aligning large language models (LLMs) is essential for their safe deployment. Current alignment methods mainly optimize observable responses, yet mode

Resolution-Aware Experimental Design under Partial Identifiability

model-releasesarxiv-cs-lg

arXiv:2609.03686v1 Announce Type: new Abstract: Experimental design is commonly framed as choosing the experiment expected to provide the most information. Under partial identifiability however, persi

Safety

ScoreMix: Synthetic Data Generation by Score Composition in Diffusion Models Improves Recognition

arXiv:2506.10226v3 Announce Type: replace-cross Abstract: Synthetic data generation is increasingly used in machine learning for training and data augmentation. Yet, current strategies often rely on e

safetyarxiv-cs-ai

TAP-Path: Task-Adaptive Structural and Token Pruning for Efficient and Trustworthy Pathology Foundation Models

arXiv:2609.04071v1 Announce Type: cross Abstract: Pathology foundation models improve transferable representation learning for histopathology, but recent gains often rely on encoders with hundreds of

TruncGradGS: Improved 3D Gaussian Splatting via Truncated Gradient Updates

arXiv:2609.03534v1 Announce Type: cross Abstract: 3D Gaussian Splatting has become a de facto scene representation for novel view synthesis, yet robustly learning 3D Gaussian primitives from visual in

Understanding Autonomous Driving Datasets by Describing Differences between Image Subsets in Natural Language

arXiv:2609.03677v1 Announce Type: cross Abstract: Understanding the composition of large-scale autonomous driving datasets is essential for safety, robustness, and reliable operation across domains. F

WorldReward: Reward Modeling for Camera-Conditioned World Models

model-releasesarxiv-cs-cv

arXiv:2609.03952v1 Announce Type: new Abstract: Camera-conditioned world models generate interactive videos in which commanded actions should induce the expected scene changes while appearance, geomet

A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models

arXiv:2609.02054v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly deployed in interactive systems where understanding user intent precisely is paramount. A key capability f

APEx: Distillation of Agent Procedural Experience for Adaptive Deep Research Question Answering

arXiv:2609.02253v1 Announce Type: new Abstract: Deep research agents augment large language models with external tools to answer complex, long-horizon questions through multi-turn reasoning. Learning

Bridging Latent Reasoning and Target-Language Generation via Retrieval-Transition Heads

arXiv:2602.22453v3 Announce Type: replace Abstract: Recent work has identified a subset of attention heads in Transformer as retrieval heads, which are responsible for retrieving information from the

CHIME: Credit-Aware Hierarchical Memory Evolution for Long-Horizon Agentic Planning

arXiv:2609.02074v1 Announce Type: new Abstract: Planning is a central capability that enables agents to decompose complex long-horizon tasks into manageable steps. Test-time search and training-based

Counter-GEO-Bench: Evaluating Defenses Against Information-Distorting Generative Engine Optimization

arXiv:2609.02316v1 Announce Type: cross Abstract: Generative engine optimization (GEO) enables content producers to increase the visibility of their web pages in generative search engines, but the sam

Deeply Interleaved Text-Image Contexts for Multimodal LLMs Assessment

model-releasesarxiv-cs-cv

arXiv:2609.02573v1 Announce Type: new Abstract: Current evaluations and training of multimodal models predominantly focus on multi-image tasks, largely overlooking interleaved text-image scenarios. In

EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction

arXiv:2609.02783v1 Announce Type: new Abstract: Evaluating LLM agents is essential for guiding their development, yet it has grown prohibitively expensive: a single pass of a frontier model over an ag

From Tokens to Semantics: Leveraging Complementary Signals for Hallucination Detection in Black-Box LLMs

arXiv:2609.02679v1 Announce Type: cross Abstract: When LLMs support public-facing or high-stakes workflows, missed fabrications can harm users and institutions, while false alarms consume limited huma

GPT‑6 Astra

model-releasessimon-willison

GPT‑6 Astra GPT-6 Astra is 'rolling out today to a limited set of organizations and over the coming days will become available to all ChatGPT Plus, Pro, Business, and Enterprise users, as well as thro

Harness Engineering in LLM Tool Use via Agent-Native Reusable Tool Primitives

arXiv:2609.01736v1 Announce Type: cross Abstract: Large language models (LLMs) augmented with external tools have demonstrated remarkable capability in solving complex real-world tasks. However, exist

How Output Format Confounds Data Quality and Capability in Instruction Tuning

arXiv:2609.02015v1 Announce Type: new Abstract: Instruction-tuning data are judged by quality metrics, and tuned models are judged by benchmarks, but both judgments pass through an output interface: t

researcharxiv-cs-cl

ICE: Intervention-Consistent Explanation Evaluation with Statistical Grounding for LLMs

arXiv:2603.18579v2 Announce Type: replace-cross Abstract: Evaluating whether explanations faithfully reflect a model's reasoning remains an open problem. Existing benchmarks use single interventions w

RecKAN: Kolmogorov-Arnold Networks with a Learnable Recursive Polynomial Basis

arXiv:2609.01729v1 Announce Type: cross Abstract: Kolmogorov--Arnold Networks (KANs) replace the fixed scalar weights of a standard network with learnable univariate functions on each edge, but existi

SCX Router: Streaming Zero-Shot Model Selection with a Decoder-KV Classifier and a Real-World Task Ontology

arXiv:2609.02292v1 Announce Type: new Abstract: The rapid proliferation of large language models (LLMs) and the growing diversity of their applications presents a unique optimization opportunity: sele

SignMatch: Matching Dictionary Signs to Continuous Sign Language Video

model-releasesarxiv-cs-cv

arXiv:2609.01886v1 Announce Type: new Abstract: The objective of this paper is to match dictionary sign videos to corresponding signs in continuous signing videos, where a match is defined by the visu

The Memory Trust Gap: Capability-Dependent Failures in Persistent-Memory Agents

arXiv:2609.01852v1 Announce Type: new Abstract: Persistent memory supports personalized agents, but a stale stored fact can override current authoritative evidence without warning. We study when this

Toward Robust LiDAR Semantic Segmentation for Real-World Deployment: Evaluation under Coarse Labels, Adverse Conditions, and Domain Shifts

model-releasesarxiv-cs-ro

arXiv:2609.02830v1 Announce Type: new Abstract: LiDAR-based semantic segmentation is a core perception module for autonomous vehicles and mobile robots. Despite the strong performance of recent state-

VakyArth: Evaluating Pragmatic Competence in LLMs across Indic Languages

arXiv:2609.01788v1 Announce Type: cross Abstract: Real-world communication often requires pragmatic reasoning: interpreting meanings implied through context and cultural convention rather than stated

Industry

Vals analysis: open-weight models performing multi-stage tasks, like building a web app, can have an environmental impact 10K times greater than simple queries (Bloomberg)

Bloomberg: Vals analysis: open-weight models performing multi-stage tasks, like building a web app, can have an environmental impact 10K times greater than simple queries — Benchmarker Vals AI finds t

industrytechmeme

A Dataset for Modeling Iterative Problem-Solving

arXiv:2609.00940v1 Announce Type: new Abstract: Solving problems through repeated attempts is a sequential modeling task: at each step, the solver receives feedback and decides how to revise their sol

Bridging Lexical Divergence: LLM-Assisted, Cost-Efficient, Zero-shot Scientific Entity Linking

arXiv:2609.00228v1 Announce Type: new Abstract: Scientific domain entity linking (EL) differs from general domain EL because mentions and entity names often lack lexical overlap. Another challenge is

Data-Driven Persona-Conditioned Agents for A/B Test Simulation

arXiv:2609.01038v1 Announce Type: new Abstract: A/B testing is the gold standard for evaluating product changes, but each experiment requires real user traffic, engineering effort, and weeks of measur

DiagEvo: Diagnosis-Guided Self-Evolution via Hierarchical Error Memory

arXiv:2609.00768v1 Announce Type: new Abstract: Self-play is an effective paradigm for language-model self-evolution, but without guidance, solver performance can plateau or decline across rounds. Ung

Do General NLP Embeddings Capture Ontological Reasoning?

arXiv:2609.00177v1 Announce Type: cross Abstract: General-purpose NLP embedding models perform well on linguistic tasks, but their ability to capture symbolic ontological structure remains unclear. We

Do Multimodal LLMs See Before They Read? Diagnosing Contextual Sycophancy

arXiv:2609.00067v1 Announce Type: cross Abstract: External text can override conflicting image evidence in multimodal large language models, a failure we call multimodal contextual sycophancy. We intr

Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents for Drone Control: Commanding, Approaching, Tracking and Searching

arXiv:2609.01404v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) are strong perceivers of images and video. We ask how far that reach extends into acting: dropping an MLLM di

Foundation models for electricity price forecasting and battery arbitrage: Can they replace market-specific forecasting models?

model-releasesarxiv-cs-lg

arXiv:2609.00089v1 Announce Type: new Abstract: Foundation models promise accurate forecasts with little or no task-specific training, but whether they can replace models designed specifically for ele

HiveTraceGuard-Pro: A Compact Generative Guardrail for Prompt Injection, Jailbreaks, and Adversarial Obfuscation

arXiv:2609.01046v1 Announce Type: cross Abstract: Production LLMs must handle inputs that attempt to override system instructions, bypass safety policies or elicit harmful responses. A common mitigati

How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation

arXiv:2609.01369v1 Announce Type: new Abstract: Reliable evaluation of open-ended question answering remains a bottleneck for measuring answer correctness of modern LLMs. Unlike multiple-choice tasks,

Investigating Linear Probe Robustness to Linguistic Register, Medical Specialty, and Corpus Shifts in Medical QA

arXiv:2609.01361v1 Announce Type: new Abstract: Linear classifiers trained on hidden states of a large language model (LLM), linear probes, can flag factual errors from a single forward pass. Geometri

LEAP: Likelihood Elicitation and Aggregation for LLM-based Probabilistic Forecasting

arXiv:2609.01337v1 Announce Type: new Abstract: LLM-based forecasting systems have improved on real-world tasks such as financial markets and sports outcomes, largely through stronger search and tool

Agents

NSIDDx: A Design Framework for Neuro-Symbolic, Practitioner-First Differential Diagnosis in Low-Resource Settings

arXiv:2609.00256v1 Announce Type: new Abstract: LLM-based diagnostic systems achieve high semantic accuracy on benchmarks, but open-ended evaluation on clinically uncommon presentations reveals a syst

agentsarxiv-cs-cl