Knowledge catalogue
Coinbase CEO Brian Armstrong recently posed the question every company scaling AI is asking: how do you keep spend flat while token usage grows exponentially? This isn’t hypothetical. It’s confronting
Amazon Bedrock now offers OpenAI GPT-5.6 models (Sol, Terra, and Luna) in more than 25 AWS Regions with cross-Region inference. Learn how US geographic and global inference profiles route requests for
Enterprises and individual developers frequently run multiple AI inference models. The right architecture can simplify how the models are called while also providing centralized governance. In this po
Together AI has partnered with IBM and NVIDIA to launch the first dedicated, large‑scale AI inference cluster on IBM Cloud, consisting of NVIDIA B300 GPUs and Spectrum‑X Ethernet networking engineered
Developers now have a much cheaper way to access cutting-edge artificial intelligence inference capabilities with the launch of Antseed today. It’s a new peer-to-peer marketplace that makes it possibl
Demand for AI infrastructure is at an all-time high. Global accelerator shortages mean engineering teams can rarely get all the compute they need from just one data center — capacity comes a cluster h
As generative AI moves from experimental pilots to massive production environments, the efficiency of your infrastructure becomes the ultimate differentiator. One way to get the most out of it and min
Anyone building with AI eventually hits the same tradeoff: how to get the most intelligence per dollar, the right model at the right cost for each task. That’s what DigitalOcean Inference Engine is bu
The race to build AI infrastructure systems has moved beyond chip specifications into a battle over entire rack-scale platforms, as inference and agentic workloads redefine what counts as a computer.
OpenAI has updated its inference platform to give users full control over performance, cost, and quality without building their own stack—models go live in minutes and support multiple deployments beh
NVIDIA TensorRT's multi-device inference feature scales inference across multiple GPUs, enabling deployment of models that exceed single-GPU memory or reducing latency for memory-bound workloads. Each
Ben Thompson / Stratechery: Agentic inference is set to be different than today's inference, and will change compute infrastructure because speed won't matter when humans aren't involved — Subscribe t
Every few weeks, a new model steps up and claims best-in-class at something. The launch chart makes the case, the link makes the rounds, and your team is back to the question nobody has time for: shou
Building and serving models on infrastructure is a strong use case for businesses. In Google Cloud, you have the ability to design your AI infrastructure to suit your workloads. Recently, I experiment
Amazon SageMaker AI shipped 13 inference launches in the first half of 2026 across two deployment paths: fully managed endpoints and Amazon SageMaker HyperPod Inference. This post reviews each launch,
Amazon SageMaker HyperPod Inference Gateway is a Kubernetes-native, GPU-aware routing add-on for Amazon EKS. It uses real-time GPU signals to send each inference request to the best-suited pod, cuttin
DigitalOcean's Dedicated Inference offering provides cloud infrastructure optimized for running machine learning inference workloads with guaranteed resources and performance isolation. This technical
Amazon SageMaker optimized generative AI inference introduces the aws-ai-ml skill through the Agent Toolkit for AWS, giving coding agents like Kiro, Claude Code, and Codex deep expertise in inference
Marina Temkin / TechCrunch: Sources: AI inference-chip startup Etched is in early talks to raise funding at a 40B-50B valuation, up from 21B in August — Although it's only been a couple of months sinc
What happens when your workload fails in one region but you need access to service? This is a common case for availability and uptime. With recent enhancement to the Kubernetes ecosystem and capabilit
DigitalOcean introduced a unified batch inference solution designed to enable scalable and cost-efficient AI model inference workloads. The offering allows users to process multiple inference requests
Enterprises are rapidly moving AI workloads from experimentation to production on Google Kubernetes Engine (GKE), using its scalability to serve powerful inference endpoints. However, as these models
Learn how to run OpenAI GPT-6 Sol, Luna, and Astra on Amazon Bedrock through global cross-Region inference from Singapore, Jakarta, Thailand, Malaysia, and Taipei. This post covers the Converse and Op
Distributed inference splits a large AI model into sub‑components that run concurrently on multiple compute nodes, allowing each node to process only part of the workload. This parallelism boosts thro
Microsoft Corp. today pulled back the curtains on Maia 300, its newest internally developed artificial intelligence accelerator. The chip is optimized for inference. Microsoft estimates that it can pr
The rise of AI-native cloud provider CoreWeave Inc. is part of the greater story emerging around operationalizing AI. As enterprises shift from training to inference, neoclouds such as CoreWeave are p
The future of artificial intelligence inference isn’t premium; it’s ubiquitous and commoditized. That may sound odd coming from someone in the AI semiconductor business. Conventional wisdom suggests c
Latest version of the benchmark introduces two new tests for emerging AI deployment patterns, including Agentic Inference The post MLCommons Sets Participation Record with New MLPerf Inference v6.1 Be
MLPerf Inference v6.1 introduced the first agentic workload run on datacenter hardware and evaluated a model with more than one trillion parameters (Kimi K2), setting a new benchmark for scale. The te
In this analysis, MLPerf Inference Working Group chairs Miro Hodak and Frank Han break down a record field of 30 submitters and 120 systems, the arrival of agentic and end-to-end benchmarks, and what
Amazon SageMaker Inference now offers prefix-aware routing, a routing strategy that sends requests sharing the same prompt prefix to the same instance so the KV cache stays warm. In benchmarks on Llam
Positron AI Inc., a developer of inference appliances that don’t include HBM memory, today announced that it has raised 875 million in funding. NEA, Atreides Management, Valor Equity Partners, Andra C
Mike Wheatley / SiliconANGLE: Nvidia says its inference accelerator Groq 3 LPX has entered full production and Nebius has signed on as the first customer, and SpaceX will deploy Vera CPUs — Chipmaker
The next era of AI inference won’t be defined by a single breakthrough chip, network or system. It’ll be defined by how every layer of the AI factory works together. That’s why NVIDIA is extending Ver
Chipmaker Nvidia Corp. says its dedicated artificial intelligence inference accelerator Groq 3 LPX has now entered full production as it strives to maintain its dominance in the world of AI compute. T
AI inference infrastructure is becoming a system-level challenge as organizations move generative and agentic applications into production. Graphics processing unit performance remains essential, but
A few weeks after closing a 300 million round backed by Nvidia Corp., Etched Inc. today announced that it has raised an additional 700 million in funding. Investment firm Jane Street led the deal. It
As inference becomes the dominant workload in AI infrastructure, multi-tier storage architectures are emerging as a key method for cost control and enhanced performance. These architectures combine fl
Tobias Mann / The Register: AMD acquires Toronto-based Taalas, which integrates model weights directly into silicon to boost inference performance, for an undisclosed sum — Early tech demos show model
The Together AI platform’s dedicated inference architecture consists of three immutable entities: **configs** (engine, GPU type/count, parallelism and optimization profile), **deployments** (a specifi
DigitalOcean describes the architecture and engineering approach behind their Inference Router, a system designed to route and manage machine learning inference requests efficiently across distributed
Today, Amazon SageMaker AI supports optimized generative AI inference recommendations. By delivering validated, optimal deployment configurations with performance metrics, Amazon SageMaker AI keeps yo
DigitalOcean's Inference Cloud Memory Layer is a technical architecture component designed to optimize database performance by implementing an in-memory caching layer for faster data access and reduce
Speculative decoding is a technique used to accelerate the slow, sequential token generation (decode stage) in LLM inference. This method significantly reduces latency and improves hardware utilizatio
Dr. Ian Cutress / More Than Moore: Q&A with OpenAI VP of Hardware Richard Ho on its Jalapeño inference chip co-designed with Broadcom, using internal OpenAI models to design the chip, and more — VP Ha
NVIDIA’s PAIR Virtual Inference Router distributes AI inference requests across multiple local machines—supporting Ollama and LM Studio without any agent or harness changes—to mitigate multi‑agent bot
Inference workloads fall into four use‑case buckets (Chatbots/Copilots, Agents, Content Generation, Translation), each with distinct token patterns that dictate the required GPU memory and compute. Si
NVIDIA TensorRT Model Connect delivers a set of reference implementations for deploying open‑source models from Hugging Face checkpoints to native C++ inference in two steps: a Python CLI builds a dep
Serving automatic speech recognition (ASR) models at scale is costly when each request uses only a fraction of a GPU. Learn how NVIDIA CUDA Multi-Process Service (MPS) with NVIDIA Triton Inference Ser
AI storage infrastructure is becoming a more consequential planning issue as organizations move from model training toward agentic AI. As agents reason, act and reassess, they build longer contexts an
OpenAI reports Jalapeño delivers 1.5–1.9× more AI work per watt and 1.7–3.6× lower latency vs comparison systems on InferenceX.
Counterpoint Research: In Q2 2026, server-led eSSDs reached 48% of NAND flash shipments as AI workloads shifted from training to inference, driving 5x YoY industry revenue growth — - AI inference work
Learn how to build an inference meta-monitoring system for Amazon SageMaker AI endpoints using Amazon Quick. This governance layer sits above production ML inference pipelines to continuously track pr
In this post, we introduce the UI for optimized generative AI inference recommendations in Amazon SageMaker AI Studio, a low-code no-code (LCNC) experience. The API already gives you programmatic acce
Mike Wheatley / SiliconANGLE: Inference cloud startup DeepInfra raised a 107M Series B co-led by 500 Global and Georges Harik; it currently supports more than 190 open models — Dedicated inference clo
This DigitalOcean guide covers strategies and best practices for deploying and optimizing very large language models (600 billion+ parameters) on cloud infrastructure, focusing on inference performanc
Agentic workflows are making speed a condition for whole classes of AI applications to exist, and the fastest tokens will determine which platforms win the next phase of AI. Cerebras Systems Inc., whi
Token economics are becoming a defining measure of artificial intelligence infrastructure efficiency. As AI workloads move beyond training, the economics of producing useful intelligence are starting
NVIDIA Confidential Computing (CC) secures large‑language‑model inference by providing encrypted virtual machines, GPUs, and NVLink, enabling processing of sensitive data in trusted hardware. Running
**Summary for AI knowledge base** Vera Rubin NVL72 is a co‑designed platform combining the Rubin GPU, Vera CPU, NVLink 6 Switch, ConnectX‑9, BlueField‑4, and Spectrum‑6, targeting agentic inference wo