Search: “benchmark”

arXiv:2608.26332v1 Announce Type: new Abstract: Managed LLM services are now part of real production systems, but model selection and service planning still rely heavily on capability benchmarks that

applicationsarxiv-cs-lg

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

BTS-AgentBench: A Deterministic, Replayable Pipeline from Read-Only Telemetry Logs to Agent Benchmarks

DGX agent

arXiv:2608.27334v1 Announce Type: new Abstract: Industrial sites contain large volumes of read-only telemetry, but few benchmarks specify how to compile these records into executable multi-turn agent

model-releasesarxiv-cs-cl

28 Aug 2026

From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench

arXiv:2608.27442v1 Announce Type: cross Abstract: In real-world software development, code review typically involves iterative interactions between developers and reviewers to improve software quality

SynthCharge: An Electric Vehicle Routing Instance Generator with Feasibility Screening to Enable Learning-Based Optimization and Benchmarking

arXiv:2603.03230v2 Announce Type: replace-cross Abstract: The electric vehicle routing problem with time windows (EVRPTW) extends the classical VRPTW by introducing battery capacity constraints and ch

Research

Technical Comparative Benchmarking Study: Advanced AI Hybrid Methods for Renewable Energy Farm Optimization and Forecasting

arXiv:2608.26613v1 Announce Type: new Abstract: This study provides a comprehensive benchmarking of conventional machine learning (ML), ensemble learning, deep neural networks, recurrent architectures

researcharxiv-cs-lg

Asymmetric Cross-Modal Fine-Grained Visual Categorization: ACF-Net and the BirdPro Benchmark

arXiv:2608.25520v1 Announce Type: new Abstract: Audio-visual cross-modal Fine-Grained Visual Categorization (FGVC) aims to identify fine-grained categories by jointly leveraging visual and auditory in

27 Aug 2026

Gemma4 31B vs Qwen3.8 27B - why the huge difference in benchmarks?

model-releasesr-localllama

Hi all, I'm looking for the best model for a hobby project and trying to make sense of the various data I came across. I know benchmarks do not often translate to the real world, especially to your pa

No reward hacking was found in GLM-5.2: solve the task, not the benchmark.

No reward hacking was found in GLM-5.2: solve the task, not the benchmark. Introducing reward hacking score corrections to the Artificial Analysis Coding Agent Index In v1.4 of the Artificial Analysis

model-releasesollama--x

Local Ai

Self-hosting LLMs on budget hardware: general principles, hardware, benchmarks and frontends

Hello, I've been self-hosting LLMs on various budget hardware for a while (6x RTX 3060 12 GB, Intel Arc Pro B60 24 GB, RX 9070 XT, etc). Over the last few months, I wrote about it in 4 articles: Gener

local-air-localllama

UHI-Bench: Benchmarking Dual-Source Urban Heat Island Modeling Across Cities in Diverse Climate Regimes

model-releasesarxiv-cs-lg

arXiv:2608.23857v1 Announce Type: new Abstract: Urban heat islands (UHIs) are intensifying under climate change, exacerbating thermal exposure risks. Their two primary observations, land surface tempe

With only 6B active parameters, Qwen3.8-Flash-Next-Base tops 8 of 14 benchmarks, including MMLU-Pro, SuperGPQA, BBH and GSM8K. And it remain…

With only 6B active parameters, Qwen3.8-Flash-Next-Base tops 8 of 14 benchmarks, including MMLU-Pro, SuperGPQA, BBH and GSM8K. And it remains competitive with Qwen3.7-Plus-Base on the rest. Its 51B N-

model-releasesqwen--x

Agents

Advances in models, chips, or devices will continually improve local-first capabilities. We will open our benchmarks soon. Read the full pos…

Advances in models, chips, or devices will continually improve local-first capabilities. We will open our benchmarks soon. Read the full post: https://www.perplexity.ai/hub/blog/a-local-first-agent-fo

agentsperplexity--x

MRMAD: A Multi-Round Multi-Audio Benchmark for Evaluating Acoustic Degradation Perception in Large Audio-Language Models

model-releasesarxiv-cs-lg

arXiv:2608.22236v1 Announce Type: cross Abstract: Large audio-language models (LALMs) have shown promising progress in understanding speech, music, and general sound events, yet their ability to reaso

NetConfArena: An Executable Benchmark for LLM Agents in Closed-Loop Network Configuration

arXiv:2608.23179v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly attractive for automating network configuration, yet their reliability and failure patterns are poo

OmniCAD: A Large-Scale Benchmark for 3D Spatial Reasoning in Robotics Assemblies

arXiv:2608.22637v1 Announce Type: new Abstract: Recent vision-language models (VLMs) show strong capabilities in robotic perception and spatial reasoning, yet their ability to reason about complex mec

StocBench: A Benchmark for Generative Modeling of Stochastic Dynamics

model-releasesarxiv-cs-lg

arXiv:2608.22309v1 Announce Type: new Abstract: We benchmark transport-based generative models as well as distillation-based few-step methods for the probabilistic forecasting of stochastic fluid flow

Text-ADBench: Text Anomaly Detection Benchmark Based on LLM Embeddings

arXiv:2507.12295v2 Announce Type: replace-cross Abstract: Text anomaly detection is a critical task in natural language processing (NLP), with applications spanning fraud detection, misinformation ide

What Proves You Wrong: Benchmarking Language Models on Falsifiable Research Ideation

arXiv:2608.22948v1 Announce Type: cross Abstract: Large language models are increasingly used to propose research ideas, yet the prevailing ways of judging such ideas supply no shared decision rule: f

Clarify-Then-Search: A Clarification Benchmark for Deep Search with End-to-End Nugget Restoration

arXiv:2608.20357v1 Announce Type: cross Abstract: Deep search is brittle on underspecified user queries: missing constraints such as time, location, scope, or definitions can lead to retrieval drift a

Nvidia says its Groq 3 LPX racks delivered 3,400 tokens per second in an Artificial Analysis benchmark running Gemma 4 31B with a 100,000-token input sequence (The Register)

The Register: Nvidia says its Groq 3 LPX racks delivered 3,400 tokens per second in an Artificial Analysis benchmark running Gemma 4 31B with a 100,000-token input sequence — Nvidia's $20 billion bet

model-releasestechmeme

VIALS: A Benchmark for Visual Interpretation of Artifacts in the Life Sciences

arXiv:2608.21357v1 Announce Type: new Abstract: In professional life sciences workflows, scientists routinely interpret visual artifacts (gel blots, microscopy images, plasmid maps, flow cytometry plo

VisTa3D: A Dataset and Benchmark for Thin Object Reconstruction from Vision, Tactile, and 3D Point Clouds

arXiv:2608.20740v1 Announce Type: new Abstract: State-of-the-art 3D reconstruction models, whether from visual, range, or both, tend to underperform on thin objects. This is partially due to the small

When does fusing hand-crafted knowledge with learned representations pay? A cost-normalized benchmark of stacking, substitution, and interference

arXiv:2608.21098v1 Announce Type: new Abstract: Fusing prior knowledge with data-driven learning is attractive where data is scarce, yet no controlled account says when it helps, is redundant, or harm

Industry

A year ago today we published The Last Economy by @EMostaque. Since then, scores on the hardest AI benchmarks have risen sharply, while the …

A year ago today we published The Last Economy by @EMostaque. Since then, scores on the hardest AI benchmarks have risen sharply, while the cost of useful intelligence has kept falling. The book asks

industryemad-mostaque--x

22 Aug 2026

ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Models

arXiv:2608.20338v1 Announce Type: new Abstract: Large Language Models (LLMs) increasingly require selective removal of harmful or sensitive knowledge, called unlearning, yet existing methods and bench

21 Aug 2026

The real benchmark for an AI agent isn’t what it says. It’s what it gets done. AutoClaw, @Zai_org 's AI Agent for Work, is built to turn int…

model-releaseszhipu-ai--x

The real benchmark for an AI agent isn’t what it says. It’s what it gets done. AutoClaw, @Zai_org 's AI Agent for Work, is built to turn intent into finished output. Give it a complex task. AutoClaw b

20 Aug 2026

#1 open-weight model on Harvey's Legal Agent benchmark! 🙌 Strong enough to handle professional tasks. Small enough to run on your local mac…

#1 open-weight model on Harvey's Legal Agent benchmark! 🙌 Strong enough to handle professional tasks. Small enough to run on your local machine. Qwen3.8-27B is becoming part of your everyday workflows

model-releasesqwen--x

I benchmarked Qwen3.8 27B on browser tasks. It's on par with GPT 5.6 Luna (xhigh)

The benchmark I used is BU bench v1, and the open source harness is Browser Agent. Qwen3.8 27B beat all other affordable or open models I tested. It's insanely good! submitted by /u/pierreb5 [link] [c

model-releasesr-ollama

Agents

It's felt like harness month on Twitter. We're seeing much faster and cheaper gains on a bunch of benchmarks by focusing on harness improvem…

It's felt like harness month on Twitter. We're seeing much faster and cheaper gains on a bunch of benchmarks by focusing on harness improvement rather than model improvement. Allowing the harness to b

agentsswyx--x

MobileWorldSafety: Benchmarking GUI Agent Safety Against Environmental Injection Attacks in Android Apps

arXiv:2608.17659v1 Announce Type: cross Abstract: LLM-powered GUI agents that autonomously operate smartphones are rapidly transitioning from research prototypes to early real-world deployment. Howeve

PTXBench: Benchmark and Adapt LLMs for GPU Kernel Optimization with Architecture-specific PTX

arXiv:2608.17379v1 Announce Type: cross Abstract: We introduce PTXBench, a benchmark for evaluating and adapting large language models (LLMs) to use architecture-specific PTX for GPU kernel optimizati

When Writing Style Drifts: Benchmarking Authorship Verification under Distribution Shifts in Genre, Time and the AI-Era

arXiv:2608.17979v1 Announce Type: new Abstract: Authorship verification (AV) assumes that an author's writing style remains sufficiently stable to distinguish it from that of other writers. In practic

WorldBench: Benchmarking Physical Understanding of World Models by Isolating Physics Concepts

arXiv:2601.21282v2 Announce Type: replace Abstract: Recent advances in generative foundational models, often termed 'world models,' have propelled interest in applying them to critical tasks like robo

Agentic-SQL Revisited: Autonomy-Based Taxonomy and Empirical Benchmark Analysis for LLM Text-to-SQL

arXiv:2608.15389v1 Announce Type: new Abstract: LLM-based Text-to-SQL progress is reported across heterogeneous benchmarks, backbones, and inference protocols, making cross-system comparison fragile.

Brex’s summer benchmark shows how quickly the infrastructure behind the agent economy is growing. 14 of the 25 fastest-growing software vend…

model-releasestogether-ai--x

Brex’s summer benchmark shows how quickly the infrastructure behind the agent economy is growing. 14 of the 25 fastest-growing software vendors serve teams building AI products, and Together AI ranks

Chronocooked: A Benchmark for Implicit Interval Timing in Reinforcement Learning Agents

arXiv:2608.16666v1 Announce Type: new Abstract: This paper presents Chronocooked, a reinforcement learning (RL) benchmark suite for studying implicit interval timing in RL agents. Inspired by Overcook

DeepInsight II: One Trace from Benchmark to Robot

arXiv:2608.16556v1 Announce Type: new Abstract: Across a Physical AI stack, evaluation maturity is inversely aligned with deployment risk: foundation models enjoy mature, standardized harnesses, while

FloodReasonBench: Benchmarking VLM Reasoning Segmentation for Embodied Flood Response at the Edge

arXiv:2608.15410v1 Announce Type: cross Abstract: Reasoning segmentation enables vision-language models (VLMs) to translate mission-relevant language requests into pixel-level visual grounding, offeri

Gated Against One Model, Open to the Next: Option-Only Solvability in Legal Multiple-Choice Benchmarks

arXiv:2608.15428v1 Announce Type: cross Abstract: Multiple-choice benchmarks are graded on whether a model picks the right option, not on whether it needed the question. Measuring that gap takes care:

Agents

I am bullish on simple agent harnesses. Love this idea from Vercel. fx is a new coding agent harness for model benchmarking, sandboxing, eva…

I am bullish on simple agent harnesses. Love this idea from Vercel. fx is a new coding agent harness for model benchmarking, sandboxing, evals, and gyms. Introducing fx, a tiny, open, native coding ag

agentsdair-ai--x

MedMCP-Calc: Benchmarking LLMs for Realistic Medical Calculator Scenarios via MCP Integration

arXiv:2601.23049v2 Announce Type: replace Abstract: Medical calculators are fundamental to quantitative, evidence-based clinical practice. However, their real-world use is an adaptive, multi-stage pro

NumerosityVLM: A Cognitively Inspired Benchmark for Interpreting Numerosity Representations in Vision-Language Models

arXiv:2608.15425v1 Announce Type: cross Abstract: Vision-language models (VLMs) achieve strong performance on high-level multimodal tasks, yet numerosity perception, a cognitive ability that emerges i

PersonaShot: Benchmarking Person-Centric Narrative Continuity in Multi-Shot Video Generation

arXiv:2608.16717v1 Announce Type: new Abstract: Video generation is rapidly evolving from single-shot clips to multi-shot narratives, where the human character serves as the core narrative anchor. How

ReRef-3D: A Benchmark for Spatial Referring Expression-Guided 3D Scene Rearrangement

arXiv:2608.16011v1 Announce Type: new Abstract: We introduce ReRef-3D, a benchmark for language-guided placement in 3D scenes. It contains 33,826 instructions across 998 CLEVR-derived scenes, spanning

Second-Order Policy Effects as State Transitions: A Source-Linked Benchmark for Policy Simulation

arXiv:2608.15101v1 Announce Type: new Abstract: Policy evaluation often estimates direct benefits and costs while treating the institutional environment as fixed. In practice, a policy changes the sys

Task- and Session-Level Model Routing: A Common-Interface Hybrid Evaluation of Four Open-Source Routers Across Four Benchmarks

arXiv:2608.14641v1 Announce Type: new Abstract: Agentic systems increasingly delegate model selection to a router, yet open-source routers are usually evaluated with different tasks, candidate pools,

The Commercial Tax: Rent-vs-Own Blind Spots in Multi-Hop Retrieval Benchmarks

arXiv:2608.16096v1 Announce Type: cross Abstract: Enterprises connect language models to their own data through retrieval. The benchmarks that rank multi-hop retrieval systems leave out two facts a bu

VARM-Bench: Benchmarking Verifiable Structured Reasoning in Chinese Abusive Speech Moderation

arXiv:2608.15600v1 Announce Type: new Abstract: The widespread circulation of abusive online content has increased the need for reliable moderation of Chinese social-media text. Existing Chinese bench