Search: “benchmark”

PDDLCoder: Agentic PDDL Generation for LLM-Assisted Symbolic Planning

arXiv:2608.16637v1 Announce Type: new Abstract: LLMs remain unreliable for long-horizon planning, often generating logically inconsistent or non-applicable plans. Recent hybrid methods instead transla

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Synthetic Image Detection with CLIP: Understanding and Assessing Predictive Cues

DGX agent

arXiv:2602.12381v2 Announce Type: replace Abstract: Recent generative models produce near-photorealistic images, challenging the trustworthiness of photographs. Synthetic image detection (SID) methods

model-releasesarxiv-cs-cv

18 Aug 2026

The Synthetic Web: Adversarially-Curated Mini-Internets for Diagnosing Epistemic Weaknesses of Language Agents

arXiv:2603.00801v2 Announce Type: replace Abstract: Language agents increasingly act as web-enabled systems that search, browse, and synthesize information from diverse sources. However, these sources

Toward Better Assessment of LLMs' Performance in Clinical Error Detection

arXiv:2608.16643v1 Announce Type: cross Abstract: Automated detection of errors in clinical documentation is a promising application of large language models (LLMs), yet decisions to deploy such model

Local Ai

TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation

arXiv:2608.16765v1 Announce Type: cross Abstract: Despite recent advances in unified multimodal models for multi-reference image generation, existing benchmarks remain organized around predefined task

local-aiarxiv-cs-ai

UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations

arXiv:2608.15930v1 Announce Type: new Abstract: Foundation GUI agents can automate complex digital tasks, but deployment is hindered by scarce and biased training data, ambiguous prompts, and unreliab

Walk Before You Run: The Importance of Data Exploration for Data Analysis Agents

arXiv:2608.16045v1 Announce Type: cross Abstract: LLM-based data-analysis tools are increasingly used to help users analyze messy spreadsheets and workbooks, from answering questions over uploaded fil

Safety

When Context Misleads: Intent-Guided Decoding for Robust Retrieval-Augmented Generation

arXiv:2608.16515v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) improves large language models by grounding generation in external evidence, but it also introduces a source trus

safetyarxiv-cs-ai

When Single-Dataset Conclusions Fail: A 45-Task Study of Threshold Tuning and Resampling for Imbalanced Classification

arXiv:2608.16147v1 Announce Type: new Abstract: Class-imbalance handling is routinely evaluated on a single benchmark dataset, and the resulting conclusions are reported as if they were properties of

Applications

A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing

arXiv:2608.13573v1 Announce Type: new Abstract: Large Language Model (LLM) serving has become a critical cloud workload, and realistic traces are essential for motivating and benchmarking serving syst

applicationsarxiv-cs-ai

Agentic Transaction: Towards ACID-Compliant Agent Systems

arXiv:2608.13900v1 Announce Type: cross Abstract: Large language model (LLM) agents are evolving from conversational assistants into autonomous systems that execute long-horizon tasks through reasonin

AgentRewind: Recoverable Execution for Long-Horizon LLM Agents

arXiv:2608.14380v1 Announce Type: new Abstract: Many real-world tasks require LLM agents to interact with their environments over long execution horizons. Errors that occur early in execution may prop

AI Research Preference Models

arXiv:2608.13940v1 Announce Type: new Abstract: AI research agents (AIRA) can now propose, implement, and evaluate their own machine learning experiments, but progress on frontier tasks is throttled b

Bootstrapping Niche Multilingual Code Translation via Reinforcement Learning with Execution-Based Verifiable Supervision

model-releasesarxiv-cs-cl

arXiv:2608.13854v1 Announce Type: new Abstract: Code translation must preserve executable behavior across many programming languages, yet neural code translation has largely focused on a few popular l

ChartProbe: A Diagnostic Study on Visual Reasoning through Perception, Grounding, and Simple Reasoning

model-releasesarxiv-cs-cv

arXiv:2608.13766v1 Announce Type: new Abstract: Vision-language models (VLMs) remain unreliable on chart questions that require reasoning over visual quantities, and this weakness is usually attribute

Content Depth Matters in Short-Video Recommendation: Rethinking the Attention Economy

arXiv:2608.13990v1 Announce Type: new Abstract: Driven by the attention economy, short-video Recommender Systems (RSs) are primarily optimized to maximize user engagement by promoting videos that capt

CVT-Bench: Probing Spatial-State Integrity through Counterfactual Viewpoint Transformations

model-releasesarxiv-cs-cv

arXiv:2603.21114v2 Announce Type: replace Abstract: Multimodal large language models (MLLMs) perform strongly on isolated spatial tasks, but whether their predictions remain persistent and mutually co

Debiasing Text-to-Image Evaluation via Implicit Cultural Alignment Reward Modeling

arXiv:2607.15740v2 Announce Type: replace-cross Abstract: As Text-to-Image (T2I) systems rapidly advance, evaluating the cultural authenticity of synthesized content has become increasingly important

DeepSeek V4 Pro 0813 vs Claude Fable 5 on DeepSWE: Cost, Coding, and Routing

model-releasestogether-ai-blog

DeepSeek V4 Pro 0813 and Claude Fable 5 were benchmarked on DeepSWE. A cascading strategy that starts with Pro 0813 and escalates to Fable only when it fails solves 82.7 % of tasks at an average cost

Do AI chatbots find what experts would? Effects of model, user role, and sample size on study retrieval for medical questions

arXiv:2608.13786v1 Announce Type: cross Abstract: Large language model (LLM) chatbots are increasingly used to answer clinical questions with citations to relevant clinical studies. Prior research has

Research

Does ISO-Grounded NFR Specification Improve LLM Code Generation? A Comparison of Rich and Structured Interventions against a Natural-Language Baseline

arXiv:2608.13742v1 Announce Type: cross Abstract: In LLM-based code generation, Non-Functional Requirements (NFRs) are often specified as terse one-line phrases. We ask whether grounding those specifi

researcharxiv-cs-ai

EchoChange: A Diffusion Language Model with Dual Pass Remasking for Factual Remote Sensing Disaster Change Captioning

arXiv:2608.01856v2 Announce Type: replace Abstract: Bi-temporal remote-sensing disaster change captioning often needs to identify sparse and spatially localized changes across large pre- and post-even

Evaluating Agentic Learning Harness Capabilities Without Labels via the Scaling Hypothesis

arXiv:2608.13608v1 Announce Type: new Abstract: Agentic 'Continual Learning Harnesses', systems that pair an LLM with retrieval or memory to improve from feedback without retraining, have shown growin

Forecast Collapse in Time-Series Foundation Models

arXiv:2608.14106v1 Announce Type: cross Abstract: When forecasting hourly returns for 1,000 US equities, we observe an unexpected phenomenon: predictions become nearly flat and show poor stock ranking

High-dimensional nonparametric changepoint detection via low-rank degree-two density projection

arXiv:2608.13922v1 Announce Type: new Abstract: Detecting distributional changes in high dimension is difficult when neither the pre-change nor post-change density is parametrically specified. We intr

Safety

Language-Specific Gaps in AI Safety Training Datasets

arXiv:2608.13695v1 Announce Type: cross Abstract: Large language model providers routinely cite multilingual safety benchmarks spanning a dozen or more languages as evidence that their models are safe

safetyarxiv-cs-lg

Safety

Measuring Fairness in Large Audio Language Models via Semantic-Aware Bias Estimation

arXiv:2608.13624v1 Announce Type: cross Abstract: Large Audio Language Models (LALMs) have seen increasing use for audio understanding tasks such as speech recognition and audio question answering, ra

safetyarxiv-cs-ai

MedClaw: Heuristic Agent Harness for Long-Horizon Surgical Video Reasoning

arXiv:2608.14015v1 Announce Type: cross Abstract: Understanding tens-of-minutes surgical videos requires long-horizon temporal reasoning, answering what happens before, after, or across stages of a pr

Model-agnostic Retrieval-Augmented Extended Forecasting for time series

arXiv:2608.14054v1 Announce Type: new Abstract: Time series forecasting with pretrained foundation models has demonstrated strong zero-shot capabilities. However, achieving optimal performance on time

NAST: Improving Negation Handling in Medical Vision-Language Models through Negation-Aware Selective Training

model-releasesarxiv-cs-cv

arXiv:2602.12498v2 Announce Type: replace Abstract: Negation is a fundamental linguistic operation in clinical reporting, yet vision-language models (VLMs) frequently fail to distinguish affirmative f

P2Skill: Privacy Preserving Skill Distillation for Cloud-Local LLM Inference Systems

arXiv:2608.14094v1 Announce Type: cross Abstract: Cloud-local LLM inference systems have the potential to use the reasoning capability of large cloud models while protecting sensitive user data on per

Recent Advances in Deep Learning-Based Drug-Target Binding Affinity Prediction

arXiv:2608.13797v1 Announce Type: new Abstract: Computational approaches to drug discovery involve multiple sub-problems, and among them, drug-target binding affinity prediction plays an important rol

Research

Research-Oriented Human-Centric Evaluation for Foundation Models

arXiv:2506.01793v2 Announce Type: replace Abstract: Most current evaluations of foundation models focus on objective benchmarks, such as knowledge coverage and reasoning accuracy, often overlooking us

researcharxiv-cs-cl

Rewrite Once, Validate Anywhere: Producing OWL-Aware SHACL Constraints (Extended Version)

arXiv:2608.14104v1 Announce Type: cross Abstract: The Shapes Constraint Language (SHACL) is a W3C recommendation to express syntactic constraints, called shapes, on RDF graphs. SHACL validators are us

Second Thought: Reasoning in Parallel as LLM Agents Act and Observe

arXiv:2608.13667v1 Announce Type: new Abstract: LLM agents in the ReAct paradigm alternate between reasoning, acting, and observing, but deliberate reasoning is confined to the Thought phase: while th

Applications

Trajectory Dynamics in Self-Supervised Learning Latent Space for Audio Deepfake Detection

arXiv:2608.13817v1 Announce Type: cross Abstract: Human speech production is constrained by physiology, giving rise to characteristic temporal structure on acoustic signals. We hypothesise that these

applicationsarxiv-cs-lg

Twin: Playing an Unknown Game with a Test-Time Digital Twin

arXiv:2608.14490v1 Announce Type: new Abstract: We present a Test-time World-model Inference (Twin) system, in which a frontier coding agent writes an executable world model for completing continual l

Based on an accelerating frontier -> local trajectory, expect a ~30b param 'Mythos at home' by as soon as Jan 2027 (rationalisation below)

model-releasesr-localllama

Including the rationalisation for the data below - this is a more robust version of an earlier post I did similar to this - explaining below: How I chose the comparisons The basic question I’m trying

16 Aug 2026

Applications

Physical AI evals are starting to become a real category. Until recently, most VLA evaluation was basically: LIBERO → success rate → leaderb…

applicationsyann-lecun--x

Physical AI evals are starting to become a real category. Until recently, most VLA evaluation was basically: LIBERO → success rate → leaderboard. Now we're seeing a much broader stack emerge: • Allen

16 Aug 2026

We tuned an AI agent that can do large-scale document extraction from long docs (50+ pages, some with 10k-100k fields) with 94%+ accuracy 📈…

model-releasesjerry-liu--x

We tuned an AI agent that can do large-scale document extraction from long docs (50+ pages, some with 10k-100k fields) with 94%+ accuracy 📈 It uses a harness + model set that is tuned specifically for

16 Aug 2026

A Contract-Grade Verifier for LLM-Generated GPU Kernels, and a Native Blackwell Backward for the Gated-Linear-Recurrence Family

arXiv:2608.12700v1 Announce Type: new Abstract: Systems that generate GPU kernels with language models report high correctness rates. Those rates come from a single loose test: run the kernel on a few

A preliminary Qwen3.8-27B model card is live!

model-releasesr-localllama

If you scroll down from the countdown at https://huggingface.co/Qwen/Qwen3.8-27B, you see a big model card with a bunch of sections: Highlights, Model Overview, Quickstart, Best Practices, Citation, e

arepsilon-MemEvo: Adaptive Cross-Task Memory Transfer for LLM Program Evolution

arXiv:2608.12522v1 Announce Type: new Abstract: LLM-based program evolution systems such as FunSearch and AlphaEvolve have shown strong ability to discover novel algorithms, but typically optimize eac

CAKE: Compiler-Agent Co-Design for Frontier Kernel Evolution

arXiv:2608.12629v1 Announce Type: new Abstract: GPU kernel agents and GPU programming languages have advanced separately, leaving expert kernels difficult to reproduce. Agents usually treat the compil

CAS: A Causal Attribution Score for Local and Global Explainable Artificial Intelligence

arXiv:2608.12555v1 Announce Type: new Abstract: Predictive explanation methods attribute a model output; they do not, by themselves, attribute an intervention effect on the real-world outcome. We intr

Diagnostic Foundation for Evaluating LLMs' Research Integrity as Co-Scientists

arXiv:2608.12345v1 Announce Type: new Abstract: Language models are increasingly deployed as co-scientists, yet their ability to uphold research integrity under institutional pressure remains unmeasur

Dual Spatial-Temporal Attribution: Architecture-Aligned Post-Hoc Explainability for Recurrent Graph Anomaly Detection

arXiv:2608.12441v1 Announce Type: cross Abstract: Deep learning detectors for anomalies in dynamic graphs have reached strong accuracy, yet they remain opaque: when an edge is flagged, the analyst rec

ERSkill: Evolving for Skill-Guided Adaptive Memory Retrieval

arXiv:2608.12720v1 Announce Type: cross Abstract: While Large Language Model (LLM) agents increasingly rely on long-term memory for persistent interactions, the retrieval mechanisms governing this mem