Knowledge catalogue
New benchmark categories for generative AI and agentic workflows arrive alongside updated LLM tests, as the industry standard for measuring PC AI performance evolves. The post MLPerf Client v2.0 Expan
See how Arize uses Harbor and Phoenix to benchmark AI agents, MCP servers, skills, and CLIs in reproducible sandboxes, then inspect traces and annotations. The post How we benchmark AI agents and tool
Latest version of the benchmark introduces two new tests for emerging AI deployment patterns, including Agentic Inference The post MLCommons Sets Participation Record with New MLPerf Inference v6.1 Be
TensorRT Edge‑LLM ran Qwen 3.6‑27B on a single NVIDIA Jetson AGX Thor and completed the MLPerf Edge Agentic benchmark in 24 min 36 s—52.33 tokens/s—which is 6.4× faster than the llama.cpp reference (2
MLPerf Inference v6.1 introduced the first agentic workload run on datacenter hardware and evaluated a model with more than one trillion parameters (Kimi K2), setting a new benchmark for scale. The te
Moving Large Language Models (LLMs) from experimental prototypes into enterprise production exposes a critical truth: your infrastructure dictates both your performance ceilings and your unit economic
We benchmarked prompt caching across DeepSeek, GLM, GPT, and Claude using Harbor evals and Phoenix traces, so we could compare cache reuse, estimated cost, and latency on the same multi-turn shopping
Benchmarking Retrieval-Augmented Generation End to End — from Building the Vector Database to Serving an Iterative, Multi-Hop Question-Answering Inference Pipeline The post Introducing the MLPerf End-
Kate Clark / Wall Street Journal: AI assistant Instinct is raising a 250M Series B co-led by Index Ventures and Benchmark at a 2.5B valuation, bringing its total raised to 350M — Buzzy startup Instinc
LLMs have become more powerful at smaller sizes, but deploying them to edge devices like smartphones remains a massive challenge. Today, developers have to optimize across a sprawling combination of a
IBM, as part of a global effort, is developing tools to streamline AI benchmarking by making results easier to compare, validate, and reuse. The initiative is detailed in a blog post titled 'All of AI
Arize and Fireworks benchmarked 10 AI models across 2,400 agent runs. Learn why cost per successful task beats token price for model evaluation and routing. The post Cost per successful task: Benchmar
Evolving MLPerf Training from feature-interaction baselines to production-scale sequential recommendation with HSTU and the 560 GB Yambda-5B embedding benchmark. The post DLRMv4: The Next-Generation R
OpenAI: OpenAI releases MentalHealthBench, an open benchmark to evaluate AI responses in realistic mental health conversations, developed with 80+ licensed experts — An open benchmark developed with m
We benchmarked Jev against Claude Opus 5 and GPT-5.6 Terra on accuracy, cost, and latency. Learn how threshold tuning changes hallucination detection. The post Jev vs. LLM-as-a-Judge: Accuracy and cos
Standardizing independent evaluation, fairness, and interpretability for mechanism-informed generative AI in healthcare The post MLCommons Joins EU-Funded AIRIS Project to Build and Benchmark Next-Gen
ScarfBench is a benchmarking framework designed to evaluate AI agents' capabilities in migrating enterprise Java applications to modern frameworks. The benchmark likely assesses how well AI systems ca
Madison Mills / Axios: Google rolls out Gemini 4 Argon to a small group of cybersecurity partners and says it outperforms GPT-6 Astra on certain coding and knowledge work benchmarks — Google is unveil
Bloomberg: Sources: some Google employees say Gemini 4 performs well on benchmarks but struggles with some real-world coding tasks; Google disputes that characterization — As Alphabet Inc.'s Google pr
Utkarsh Shetti / Reuters: AI agent startup Instinct raised a 1B Series C from Sequoia, Benchmark, and Coatue at a 10B valuation and details recent products, such as a concierge service — AI agent firm
MLCommons announced the release of MLPerf Storage v3.0 benchmark results, expanding the suite to cover the full spectrum of AI storage workloads, including an S3 object‑storage access layer in additio
Qdrant has released **Qdrant‑Fineweb‑10B**, a public benchmark dataset comprising dense and sparse embeddings for 10 billion documents from the FineWeb corpus and exact results for 120,000 queries to
BenchMIRT is a new method for auditing LLM benchmarks question by question, revealing which capabilities they actually measure and helping researchers build smaller, more focused, and easier-to-interp
True spatial intelligence for multimodal agents transcends low-level geometric perception, evolving from knowing where things are to understanding what they are for. While existing benchmarks, such as
Carl Franzen / VentureBeat: MiMo-V2.6-Pro ties Grok 4.7 (xHigh) and beats GLM-5.3 (max) on Artificial Analysis' Intelligence Index, making it the benchmark's top-scoring open-weight model — That puts
Xiaomi: Xiaomi debuts open-weight omnimodal models MiMo-V2.6 Pro and Flash; Pro allegedly performs “on par with Opus 5 and GPT-5.6 Sol across most agent benchmarks” — Frontier intelligence, all the mo
Valida Pau / The Information: Source: AI assistant Instinct seeks to raise 1B at a 10B valuation, up from 2.25B less than a month ago; Sequoia and Benchmark are in talks to lead the round — The startu
Terence Tao / What's new: A group of 25 Fields Medal recipients says AI companies' push to solve mathematical problems as a benchmark is detrimental to the science of mathematics — I am proud to be am
Google DeepMind: Google launches a pilot of double-blind AI evaluations, keeping external evaluations in a cryptographic “box” to stop benchmark contamination and protect IP — Building trust in propri
At Google Cloud Next 2026 in Las Vegas, MLCommons and Google Cloud demonstrated a powerful new capability for trustworthy medical AI - one that protects patient data, model IP, and benchmark integrity
A field guide to the new wave of long-horizon agent benchmarks: what each one actually measures, the realism-versus-verifiability bargain it strikes, and the seam where its score leaks. The post Long-
IBM Research presents a framework for benchmarking quantum optimization algorithms, addressing the need for standardized methods to evaluate quantum computing performance on optimization problems. The
The FFASR Leaderboard is a benchmarking platform introduced by Hugging Face for evaluating Automatic Speech Recognition (ASR) systems on real-world performance metrics. It provides a standardized eval
Kate Clark / Wall Street Journal: Sources: Benchmark raised 2B across two new funds, including a 1.25B fund for late-stage bets, its first growth fund after decades focusing on new startups — After a
This benchmark demonstrates how integrating Alluxio with Ray Data achieves 20x faster training data read speeds for cross-region machine learning workloads on Anyscale's platform. The study shows perf
Models got smart enough to cheat their benchmarks, and outcome-only scores stopped measuring what we thought they measured. The fix, full trace analysis, is the same methodology production AI teams ha
Michael Nuñez / VentureBeat: Datacurve releases the DeepSWE coding benchmark, a 113-task test across 91 open-source repositories and five languages, and says GPT-5.5 is the leader at 70% — For months,
ITBench-AA is a new benchmark developed by Artificial Analysis and IBM that evaluates frontier AI models on agentic enterprise IT tasks, with results showing that current leading models score below 50
Callum Borchers / Wall Street Journal: Companies are increasingly using AI benchmarking services to aggregate public job listings and other payroll data to identify under- and over-paid employees — Ne
NVIDIA AIPerf is a benchmarking framework that supersedes GenAI‑Perf with a multiprocess architecture, eliminating client‑side bottlenecks during high‑concurrency LLM testing. It supports more than 15
Julie Bort / TechCrunch: PrismML releases Bonsai 2 27B, which compresses Alibaba's Qwen3.8 27B to 5.9 GB, small enough for smartphones, while retaining 98.2% of Qwen's benchmark scores — If AI lab Pri
Benchmark two 30B Mixture-of-Experts models, Qwen3-Coder-30B and NVIDIA Nemotron-3-Nano-30B, across G5, G6, G6e, and G7 GPU instances on Amazon SageMaker AI. Compare throughput, latency, and cost-per-
Google: Google launches Gemini 3.8 Flash Cyber for partners in its new Fairwind Program and says Gemini 3.8 Flash beats Claude Opus 5 and GPT-5.6 Sol on some benchmarks — Our newest Gemini models deli
The Register: Nvidia says its Groq 3 LPX racks delivered 3,400 tokens per second in an Artificial Analysis benchmark running Gemma 4 31B with a 100,000-token input sequence — Nvidia's $20 billion bet
Abhinaya Prabhu / Tech Funding News: SF-based Vals, which develops evaluations and benchmarks to test AI models on real-world tasks, raised a 40M Series A led by a16z at a 400M valuation — - Vals AI r
The Information: DeepSeek launches V4-Pro, its most advanced model that rivals Kimi K3 on some benchmarks but is priced much lower, at 0.435/1M input and 0.87/1M output tokens — Chinese AI developer D
Luz Ding / Bloomberg: Alibaba says its 2.4T-parameter Qwen3.8-Max tops Moonshot's Kimi K3 on some benchmarks, and it plans to release Qwen3.8-Max and Qwen3.8-27B's weights next week — Alibaba Group Ho
Newley Purnell / Bloomberg: DeepSeek rolls out the official V4 Flash API in public beta, touting enhanced agent capabilities and benchmark scores “far surpassing” V4 Pro Preview — China's DeepSeek rol
Julie Bort / TechCrunch: Ollama, which helps developers run open-weight AI models locally, raised a 65M Series B led by Theory Venture, following a 15M Series A led by Benchmark — The popular open sou
Cognition: Cognition releases SWE-1.7, trained from Kimi K2.7 and available at 1,000 tokens/second, claiming it nears GPT-5.5 and Opus 4.8 on benchmarks at a lower cost — Today, we're launching SWE-1.
OpenAI: OpenAI says it found widespread task issues in SWE-Bench Pro, estimates ~30% of tasks are broken, and retracts its earlier recommendation to adopt the benchmark — Through a detailed audit, we
In this post, you learn how to use the new MLflow integration with Amazon SageMaker AI optimized inference recommendation jobs and Amazon SageMaker AI benchmark jobs to automatically stream experiment
Carl Franzen / VentureBeat: Sakana AI launches Fugu, a multi-agent orchestration system accessible through a single model API, claiming Fugu Ultra matches Fable and Mythos on benchmarks — Last night,
Sarah Guo: As AI commoditizes benchmarkable work, an organization's lasting moats lie in tasks that are verifiable through its private data and judgment — The mid-2026 investor's version of AI psychos
FrontierCode is a benchmarking framework designed to evaluate code quality and move beyond 'slop'—low-quality or poorly generated code—in AI-generated outputs. The framework likely provides metrics an
This paper benchmarks frontier automatic speech recognition (ASR) systems on their ability to handle code-switched speech, where bilingual speakers mix languages within a single conversation. The rese
Michael Larabel / Phoronix: Initial benchmarks show Nvidia's Vera CPU, which features 88 in-house-designed Olympus cores, packs a heavy-hitting punch, beating Intel's and AMD's x86_64 CPUs — NVIDIA's
This article presents benchmarking results for AI coding agents evaluated at scale, likely comparing performance metrics such as code generation accuracy, execution success rates, and inference effici
Tobias Burns / CNBC: CME Group and Silicon Data announce a futures market for computing capacity, with contracts based on daily GPU benchmarks for on-demand rental rates — A new futures market for sem
Qwen: Alibaba launches Qwen3.6-27B, an open-weight dense model with 27B parameters, saying it surpasses Qwen3.5-397B-A17B on major coding benchmarks — · 4226 words · QwenTeam丨Translations:.体中文 — HUGGI