Search: “inference”

DeepSeek-V4 makes million-token context a serving-systems problem. Together AI explores the inference work behind V4 on NVIDIA HGX B200, including compressed KV layouts, prefix caching, kernel maturit

Ryan Whitwam / Ars Technica: Google releases Multi-Token Prediction drafters for its Gemma 4 models, which use a form of speculative decoding to guess future tokens for faster inference — Google launc

This article from Together AI discusses foundational research techniques and methodologies used to enable efficient large-scale language model inference. It likely covers optimization strategies, hard

The Information: Sources: Anthropic is in early talks to buy AI inference chips from UK-based Fractile when they become available in 2027 — As Anthropic's sales explode, straining the servers it uses,

Dina Bass / Bloomberg: Cloud computing provider Nebius agrees to buy Eigen AI, which optimizes the performance of chips running AI inference tasks, for $615M in stock and cash — Cloud computing provid

Matt Day / Bloomberg: Meta and Amazon reach a multibillion-dollar, multiyear deal for Meta to rent hundreds of thousands of Amazon's Graviton chips for its AI inference needs — Amazon.com Inc. and Met

This article examines the fundamental trade-offs in large language model inference operations, specifically the competing priorities of maximizing throughput, minimizing latency, and reducing costs. I

We're building AI Gateway into a unified inference layer for AI, letting developers call models from 14+ providers. New features include Workers AI binding integration and an expanded catalog with mul

Google has begun allowing third‑party customers to acquire or rent its TPU chips—starting with the TPUv7 “Ironwood” generation. In comparative tests against NVIDIA Volta (B200/B300), Ironwood shows up

At its first Horizon customer and partner event this week, Equinix Inc. argued that the architecture of enterprise artificial intelligence is being reshaped by a simple yet hard-to-answer question: Wh

Shadow Engine Recovery in NVIDIA Dynamo keeps a fully-initialized standby LLM engine on the same GPU, using the GPU Memory Service (GMS) to persist model weights and share them across engines without

The transition from chatbots to autonomous agents is changing the shape of demand itself, and tokenomics — the economics of AI token consumption — is emerging as the defining constraint on enterprise

Boundless Network Inc., a distributed computing company that built its graphics processing unit network to generate zero-knowledge proofs for the cryptocurrency industry, is expanding its network towa

The race for fast token generation has moved from benchmark sheets into production data centers, and the hardware blueprint for winning it is no longer a GPU-only story. As agentic AI use cases multip

Significant contributors to this article include Sneha Aradhey, Software Engineer, Google Kubernetes Engine, and Michael MacDonald, Sr Software Engineer, Google Cloud Managed Lustre. Enterprise produc

Seven months after inking a 20 billion chip licensing deal with Nvidia Corp., Groq Inc. today announced that it has raised 650 million in funding. Growth investment firm Disruptive and hedge fund Infi

Developer-centered artificial intelligence cloud provider Runpod Inc. today announced the launch of Flash, a software development kit and platform that removes the infrastructure overhead for deployin

Learn how to deploy one large language model across two GPU nodes on Amazon EKS using the multi-node Ray LLM Deep Learning Containers, with separate prefill and decode nodes behind one OpenAI-compatib

Amazon Bedrock AgentCore payments gives AI agents a managed way to pay for services on demand, with spending limits enforced by the infrastructure. See how Incarna's agents pay BlockRun for model infe

Scribd, Inc. is home to one of the world's largest collections of human-created content. Scribd’s products leverage one of the world's largest collections of human-created content and intelligent tool

Benchmark two 30B Mixture-of-Experts models, Qwen3-Coder-30B and NVIDIA Nemotron-3-Nano-30B, across G5, G6, G6e, and G7 GPU instances on Amazon SageMaker AI. Compare throughput, latency, and cost-per-

Artificial intelligence hardware startup Olix Computing Ltd. today announced that it has raised 312 million in funding. The Series C round included contributions from Arm Holding plc, Netflix Inc. co-

This paper was accepted at the AI4TCI (Workshop on AI for Secure and Trustworthy Critical Infrastructure Systems) Workshop at the International Conference on Availability, Reliability and Security (AR

Chip startup SambaNova Inc. today announced that it has raised 1 billion in funding at a 11 billion valuation. General Atlantic led the Series F round with contributions from more than a dozen others.

OpenAI Group PBC today revealed a custom chip called Jalapeño that it will use to power its large language models. The processor is the fruit of a collaboration with Broadcom Inc., which is no strange

As AI factories evolve into “data centers of the future,” the infrastructure stack must also transform into a mix of CPU and GPU platforms that can deliver a full set of AI computing solutions. This r

This article discusses strategies for optimizing the performance of advanced AI frontier models when running on AMD hardware infrastructure. It likely covers deployment best practices, hardware config

With access to the latest generative AI models and high-performance accelerated compute in high global demand, AWS customers need tools to take advantage of model availability and capacity across mult

This article discusses how prefix-aware routing and prefix caching techniques can reduce the computational overhead and costs associated with running large language models at scale by eliminating redu

This post demonstrates a comprehensive observability solution using Amazon Managed Grafana dashboards that provides a holistic view of both quality and quantity for LLMs served on Amazon SageMaker AI

This article from Databricks discusses strategies and best practices for deploying large language models in production environments at scale while maintaining reliability and performance. It likely co

IBM Corp. subsidiary Red Hat today is unveiling a broad set of product and partnership announcements aimed at helping enterprises put artificial intelligence into operation, modernize infrastructure a

This article discusses AWS infrastructure, tools, and services designed to support the training and deployment of large foundation models, enabling machine learning practitioners to leverage AWS's com

Anthropic PBC today announced that it will use SpaceX Corp.’s Colossus 1 supercomputer to power its Claude chatbot. The system was originally built in 2024 by xAI Holdings Corp., an artificial intelli

Media and entertainment is undergoing a structural shift driven by fragmented audiences, new content formats and the growing role of AI in production workflows. The traditional broadcast model — built

Artificial intelligence infrastructure startup Parasail Inc. today announced that it has raised 32 million in early-stage funding. Touring Capital and Kindred Ventures jointly led the Series A round.

Physical Intelligence (Pi) is building a general-purpose robotic intelligence system whose core Visual-Language-Action (VLA) model takes visual observations, natural-language instructions, and the...

The NVIDIA Vera Rubin platform attains superior performance‑per‑watt by combining factory‑level DSX MaxLPS power shifting, rack‑level capacitors with Intelligent Power Smoothing, and the Groq 3 LPX de

Robotics forces AI models to run on‑device because real‑time control loops cannot tolerate the latency or capital cost of sending data to a datacenter, so manufacturers must embed compute within each

The TileRT InferenceX article (Aug 10 2026) examines whether the TileRT software stack on NVIDIA GPUs can compete with dedicated inference systems such as Cerebras, Groq LPUs and SambaNova for ultra‑h

As organizations move from AI pilots to production AI factories, infrastructure decisions have shifted from peak chip specifications to cost per token: how many useful tokens they can deliver per doll

The TensorRT for RTX plugin provides a runtime for Unreal Engine's Neural Network Engine (NNE), enabling efficient deployment of AI models directly within real-time applications. Developers can see 1.

Today, we are thrilled to announce the availability of G7e instances powered by NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs on Amazon SageMaker AI. You can provision nodes with 1, 2, 4, and 8 RT

AgentX 1.0 is the first fully open‑source multi‑turn agentic coding inference benchmark (Apache 2.0), released August 2026 with a 1 million‑token context and over 1 million tokens of dataset built at

Auto-generated index of all companies mentioned across the wiki.

Auto-generated index of all tools mentioned across the wiki.