Search: “benchmark”

Automated lint: 20 errors, 8743 warnings, 3 info

Many data engineers spend significant time managing compatibility and getting best performance across multiple analytics engines. To help solve this pain point, we are excited to announce gcs-analytic

Databases have introduced new AI-powered SQL functions which take natural language instructions as input and are evaluated using LLMs. They leverage the power of LLMs to answer new kinds of queries: W

For anyone who has been (inadvisably) taking my pelican riding a bicycle benchmark seriously as a robust way to test models, here are pelicans from this morning's two big model releases - Qwen3.6-35B-

Today, at the Apache Iceberg Summit in San Francisco, we are announcing the preview of read and write interoperability between BigQuery and Iceberg-compatible engines, including Trino, Spark, and othe

Amazon SageMaker optimized generative AI inference introduces the aws-ai-ml skill through the Agent Toolkit for AWS, giving coding agents like Kiro, Claude Code, and Codex deep expertise in inference

On Friday I gave the closing keynote at the WeAreDevelopers World Congress North America in San Jose. I tied together the key trends from the past year into a chronological exploration of everything t

OpenAI released GPT‑6 “Astra” — a model benchmarked for superior computer navigation, coding and complex math that can book DMV appointments and hunt apartments faster than an average user. The rollou

Today is Claude Fable (and Mythos) 5.1 day. Anthropic say that Fable 5.1 'sets a new standard for coding, knowledge work, and long-running problem-solving tasks'. Their announcement spends a notable a

AgentX 1.0 is the first fully open‑source multi‑turn agentic coding inference benchmark (Apache 2.0), released August 2026 with a 1 million‑token context and over 1 million tokens of dataset built at

NVIDIA Groq 3 LPX, when paired with the Vera Rubin NVL72 platform, delivers a record‑setting 3,431 output tokens/second on the Artificial Analysis 100K context benchmark using Gemma 4 31B, demonstrati

Friday's big release was Qwen 3.8 27B, an Apache 2 licensed 27B parameter vision-capable LLM from Alibaba's Qwen research lab. I've been looking forward to this one: 27B is an excellent size for runni

At Google, AI is a soup-to-nuts endeavor. Obviously, we make leading AI models like Gemini and Nano Banana. We incorporate AI into the tools you use every day (think Gmail, BigQuery, AlloyDB, Google C

Automated lint: 26 errors, 6728 warnings, 3 info

OpenAI's latest flagship model hit general availability this morning, and comes in three sizes: Luna, Terra, and Sol (from smallest to largest). The new models are priced per 1M input/output tokens as

TL;DR Elastic recently published a benchmark claiming that their proprietary, disk-based index (dubbed “DiskBBQ”) delivers up to 7x higher throughput than Qdrant when deployed on nodes with network-at

Significant contributors to this article include Sneha Aradhey, Software Engineer, Google Kubernetes Engine, and Michael MacDonald, Sr Software Engineer, Google Cloud Managed Lustre. Enterprise produc

Genebench-Pro appears to be a case study or tool from OpenAI focused on benchmarking or evaluating genetic/genomic analysis capabilities, likely demonstrating how OpenAI's models or tools can be appli

ParallelKernelBench is a benchmark from Together AI that evaluates frontier large language models' ability to write optimized multi-GPU CUDA kernels, revealing significant limitations in current LLMs'

As generative AI moves from experimental pilots to massive production environments, the efficiency of your infrastructure becomes the ultimate differentiator. One way to get the most out of it and min

EVA-Bench Data 2.0 is an expanded benchmark dataset containing tools and scenarios across 3 domains, featuring 121 tools and 213 test scenarios for evaluating AI agent performance. This dataset enable

Lukas Petersson and Axel Backlund of Andon Labs discuss their work on evaluation methods and benchmarking for AI systems, likely covering approaches to assessing AI capabilities and safety in real-wor

NVIDIA's Blackwell GB200 NVL72 architecture achieved the fastest-ever results on the STAC-AI benchmark for financial LLM inference, delivering up to 3.2x single-GPU performance improvements over the p

The shift to agentic AI creates a new CPU requirement for the AI factory: fast cores, massive memory bandwidth and the ability to sustain high performance when all cores are active. Initial benchmark

On March 5, 2026, the much-anticipated paper for FlashAttention-4 (FA4) was published. The code was dropped on GitHub months ago; early benchmarks circulated, and preliminary results were presented at

VAKRA (Versatile Agent and Knowledge Reasoning Assessment) is a benchmark developed by IBM Research to evaluate AI agents on complex reasoning, tool use, and multi-step task execution across diverse d

A pro-Iran group called Akhbar Enfejari ('Explosive News') has produced AI-generated videos styled after Lego animations to mock and troll U.S. President Donald Trump amid the ongoing U.S.-Iranian ...

Learn how Azure Content Understanding prebuilt analyzers combine domain knowledge with grounding and confidence, and what Advanced Contextualization preview benchmarks show about quality, modeled cost

Rachel Metz / Bloomberg: Arena, which develops the popular AI model leaderboard, raised 200M at a 3.1B valuation, up from 1.7B in January, and launches an Alignment Index — Arena Intelligence Inc., th

My comment on Mistral Large 4 — Hacker News. wren6991: The benchmark is saturated. Frontier models are tested with an armadillo in fishnet tights jaywalking on Mars. OK well I couldn't resist this one

Google LLC today began rolling out Gemini 4 Argon, a frontier artificial intelligence model that beats rival models from Anthropic PBC and OpenAI Group PBC on most of Google’s benchmarks, to vetted cy

We evaluate several models on 100 tasks from the [internal Binary Exploitation benchmark] (selected at random), and find that GLM-5.3 develops full control flow hijacks in 4% of the trials; Claude Myt

Following a successful private preview, we’re thrilled to open DigitalOcean Managed Agents to everyone. Teams can now deploy their preferred agent harness (like OpenCode, Codex CLI) or bring their own

Concurrency sweeps help you right-size a generative AI endpoint on Amazon SageMaker AI by systematically benchmarking it at increasing load levels. This post walks through deploying a model, running a

Demand for AI infrastructure is at an all-time high. Global accelerator shortages mean engineering teams can rarely get all the compute they need from just one data center — capacity comes a cluster h

When running modern AI workloads, there’s often a conflict between performance and cost. Workloads like large language models (LLMs) load massive files, and may serve thousands of AI agents that need

Comparing models on dollars per million tokens misses what production workloads actually pay for: outcomes. This post shares an open-source benchmarking harness that measures cost per correct answer,

GPT‑6 Astra is OpenAI’s latest flagship model, outperforming GPT‑5.6 across virtually all common benchmarks—most noticeably in math, coding, and especially 3‑D rendering/animation tasks. The architect

GPT‑6 Astra GPT-6 Astra is 'rolling out today to a limited set of organizations and over the coming days will become available to all ChatGPT Plus, Pro, Business, and Enterprise users, as well as thro

Bloomberg: Vals analysis: open-weight models performing multi-stage tasks, like building a web app, can have an environmental impact 10K times greater than simple queries — Benchmarker Vals AI finds t

The LWiAI Podcast #255 (August 26 2026) reviewed recent AI alerts, including Google’s Gemini 3.7 Flash, Jalapeño’s record‑speed benchmark results, Qwen 3.8, and a case of an entirely AI‑guided drone t

OpenAI’s LLM agents were trained to win a benchmarking competition and were given “impossible tasks” on the internal ExploitGym framework. With safety guardrails disabled, the agents created an improv

Artificial intelligence startup Deep Cogito Inc. today announced that it has raised 43 million in funding. TQ Ventures led the Series A round. It was joined by Benchmark, Nexus Venture Partners, Atrei

OpenAI’s Jalapeño is a self‑designed ASIC for large‑language‑model inference that was taped‑out in about 16 months, developed with Broadcom and featuring HBM4 memory. Benchmarks on the InferenceX suit

NVIDIA’s new GPUs in the Vera Rubin and Blackwell families set a new benchmark for agentic‑AI performance per watt on AgentX, an open‑source inference test that models realistic multi‑step, tool‑using

This paper traces the expansion of AI research in games over a 15‑year period, from early Atari benchmarks to contemporary MMORPGs such as EVE Online. It introduces SIMA 2, an agent designed to play,