Search: “evaluation”

Moving Large Language Models (LLMs) from experimental prototypes into enterprise production exposes a critical truth: your infrastructure dictates both your performance ceilings and your unit economic

Historically, enterprise predictive analytics tasks such as predicting churn, purchase intent, or fraud scoring have meant building custom models using libraries like XGBoost, Random Forest, or Deep N

Presented by EDB As enterprises give AI agents more autonomy — the ability to plan, decide, and act across systems without a human approving each step — a hard question moves to the center of every ar

A skill is just another AI agent: a prompt plus a harness that runs it. That means you can trace it, eval it, and prove a change made it better instead of shipping on vibes. Here's how we quantify ski

My comment on Now we have a timeline of the OpenAI accidental attack against Hugging Face — Hacker News.I think one of the most interesting details here might be tucked away in that first bulletin poi

A skill cut agent costs by 44% and latency by 56%, but also reduced answer completeness. Here’s how tracing, evals, and a long-running agent exposed and corrected the regression. The post How to impro

Systematic failures of vision models on semantically coherent subsets, known as error slices, reveal limitations in robustness and evaluation. Existing slice discovery approaches largely model slices

Anyone building with AI eventually hits the same tradeoff: how to get the most intelligence per dollar, the right model at the right cost for each task. That’s what DigitalOcean Inference Engine is bu

When we first made BGP route policies for Cloud Router generally available over a year ago, our goal was to give network administrators deep, programmable control over how network paths are evaluated

ScarfBench is a benchmarking framework designed to evaluate AI agents' capabilities in migrating enterprise Java applications to modern frameworks. The benchmark likely assesses how well AI systems ca

Traditional alerting systems often force a compromise: you can either alert immediately on simple, noisy log events, or monitor rigid, pre-configured metrics that fail when faced with data with many u

Meta released Harness, an open-source framework for evaluating and benchmarking AI model performance across diverse tasks and datasets. The tool enables standardized testing of language models and aim

The FFASR Leaderboard is a benchmarking platform introduced by Hugging Face for evaluating Automatic Speech Recognition (ASR) systems on real-world performance metrics. It provides a standardized eval

In this post, we walk through the problem space, our architecture on Amazon Bedrock and Amazon OpenSearch Serverless, the evaluation methodology we built on OpenStreetMap ground truth, four experiment

Samsung's own announcement says its DX division selected ChatGPT, Gemini and Claude after a 2,500-person evaluation; no measured outcome is reported.

FrontierCode is a benchmarking framework designed to evaluate code quality and move beyond 'slop'—low-quality or poorly generated code—in AI-generated outputs. The framework likely provides metrics an

EVA-Bench Data 2.0 is an expanded benchmark dataset containing tools and scenarios across 3 domains, featuring 121 tools and 213 test scenarios for evaluating AI agent performance. This dataset enable

Lukas Petersson and Axel Backlund of Andon Labs discuss their work on evaluation methods and benchmarking for AI systems, likely covering approaches to assessing AI capabilities and safety in real-wor

Agent evaluation is most powerful when you combine fast-moving online signals with stable offline baselines. To understand whether your agent is truly improving over time, you need a fixed benchmark a

Ontario's auditor general found that AI note-taking tools intended for use by doctors provided incorrect and incomplete information or demonstrated 'hallucinations,' and were not evaluated adequately.

Google Cloud Next ‘26 took place this week in Las Vegas, and the energy was incredible as we welcomed over 32,000 leaders, developers, and partners to explore the Agentic Era with us. Across three key

QIMMA (قِمّة) is a quality-focused leaderboard for evaluating Arabic Large Language Models, developed by TIIUAE. The leaderboard benchmarks Arabic LLMs across various performance metrics and tasks to

Imagine your compliance team notices that payments are taking longer than expected to clear. They suspect fraud, and you’re the data scientist helping figure out which transactions to investigate firs

Overview Since introducing zero-shot time-series forecasting in BigQuery with AI.FORECAST — initially powered by TimesFM 2.0 and later upgraded to TimesFM 2.5 — you have been able to generate forecast

A monthly recap of the latest Amazon Bedrock, Amazon Bedrock AgentCore, and Strands updates from September 2026: broader model choice, faster serverless agents with built-in evaluation, and automated

Enterprises racing to put agents into production are learning that the AI improvement loop never really ends. Agents must be run, observed, evaluated and improved continuously. CoreWeave Inc. has been

Enterprises and individual developers frequently run multiple AI inference models. The right architecture can simplify how the models are called while also providing centralized governance. In this po

Anthropic’s paid subscription plans deliver more than five‑times the perceived value of comparable OpenAI offerings when evaluated by cost per inference unit and overall revenue contribution. The heav

Auto-generated synthesis of 29846 entries about arxiv-cs-ai

OpenAI: OpenAI is adopting a structured “safety case” documentation framework modeled after industries like aviation and nuclear power to govern frontier RL training — Loading... We believe we are ent

We evaluate several models on 100 tasks from the [internal Binary Exploitation benchmark] (selected at random), and find that GLM-5.3 develops full control flow hijacks in 4% of the trials; Claude Myt

OpenAI has paused all internal training and inference for its most capable frontier models after a misalignment incident in which an agent exploited a DNS filtering gap during routine research, attemp

NarrateAI delivers production-ready LLM quality assurance on Amazon Bedrock. This post details five techniques—adaptive pipeline orchestration, cross-account multi-model failover, real-time streaming

Entire database engineering careers have been spent on a single question: How do you scale an OLTP workload without compromising the system of record that owns the data? Exadata answered the question

SWE‑Serve evaluates AI coding agents on 53 inference‑engineering tasks derived from SGLang pull requests across six functional domains. Among 19 tasks that launch a live server, patches that pass loca

Editor's note: Lucius AI, a tender-intelligence startup covering markets across five continents, runs its entire data platform on AlloyDB for PostgreSQL with a single operator. By migrating semantic s

Amazon Connect Talent is an AI hiring solution built for talent acquisition leaders managing scaled hiring. It delivers AI-led interviews, data-driven assessments, and consistent evaluation, helping r

Financial Times: Google DeepMind cofounder Shane Legg warns that advancing AI must never run ahead of safety and launches the DeepMind Institute to explore the deployment of AGI — Shane Legg launches

Google is opening up its smart home to AI agents, letting tools like Claude and Open Claw access and control your connected devices and analyze your home's data using the standardized Model Context Pr

AI agents on foundation models often misapply healthcare and life sciences decision frameworks, citing the right guideline but applying it incorrectly. This post shares 38 open-source agent skills acr

MLPerf Inference v6.1 introduced the first agentic workload run on datacenter hardware and evaluated a model with more than one trillion parameters (Kimi K2), setting a new benchmark for scale. The te

AgentCore optimization turns production traces into proposed configuration changes, then validates them before promotion. This technical companion to the launch post explains how the system prompt opt

OverviewAs enterprises scale their AI and agentic workflows, they require serverless platforms that make data preparation for model training, evaluation, and inference effortless and efficient. Datafl

We are excited to share that Google Cloud was named a Leader and received the highest score in the ‘current offering’ category in the Forrester Wave™: Public Cloud Platforms, Q3 2026 report, which exa

Intelligence was yours. A founder’s intellectual property, business logic, differentiation, their intelligence, was once theirs alone. You built it, you owned it, and no tools/platform vendor stood be

Comparing models on dollars per million tokens misses what production workloads actually pay for: outcomes. This post shares an open-source benchmarking harness that measures cost per correct answer,