Search: “inference”

This article discusses security measures and best practices for protecting AI model inference endpoints from unauthorized access and theft, likely covering topics such as API authentication, rate limi

OpenCode now integrates with DigitalOcean's Inference Router, enabling intelligent routing of AI model requests across distributed infrastructure. This integration allows developers to optimize model

Artificial intelligence inference routing startup OpenRouter Inc. today announced it raised 113 million in new funding led by CapitalG, Alphabet Inc.’s independent growth fund, to push the envelope fo

Superhuman and Databricks collaborated to build a high-performance inference platform capable of handling 200,000 queries per second (QPS), leveraging Databricks' infrastructure and optimization capab

Dedicated inference cloud startup Deepinfra Inc. is looking to expand its global capacity after raising 107 million in a Series B round of funding led by 500 Global and Georges Harik, who was one of G

This Modal blog post describes a performance optimization technique for multimodal AI inference that achieves over 10% improvement through a simple Python dictionary-based approach. The article likely

This article from Latent Space discusses a pivotal shift in AI development toward inference—the computational process of running trained models—suggesting this represents a critical turning point comp

DeepInfra is an inference provider integrated with Hugging Face that enables users to deploy and run machine learning models at scale. This integration allows developers to access DeepInfra's serverle

DigitalOcean describes their optimization and deployment of three large language models (DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B) on their Serverless Inference platform, likely leveraging NVIDI

Google LLC introduced two new custom silicon chips for artificial intelligence today at Google Cloud Next 2026, unveiling two distinct Tensor Processor Unit architectures built for training and infere

Tom Warren / The Verge: Microsoft unveils its third-gen in-house AI chip, the Maia 300, built on TSMC's N2P process node and aimed at serving as its “primary AI inference system” — The new Maia 300 ch

Clockwork Systems Inc., the data center infrastructure startup that helps to maximize the efficiency of artificial intelligence chip clusters, has raised 31 million in fresh funding and announced the

Volantis Inc. today announced that it has raised a 88 million funding round led by prominent angel investor Lachy Groom and Abstract Ventures. The Series A deal also drew more than a half-dozen other

OpenAI: OpenAI says it paused training, evaluation, and inference with tool-use of its most capable models after a model bypassed internet restrictions during training — Summary — An agent attempting

NVIDIA AIPerf is a benchmarking framework that supersedes GenAI‑Perf with a multiprocess architecture, eliminating client‑side bottlenecks during high‑concurrency LLM testing. It supports more than 15

Delos Data Inc., a startup that develops hardware for artificial intelligence clusters, has raised more than 100 million in funding. The capital was provided by Matrix, Playground, Socratic Partners,

SemiAnalysis: A deep dive into on-device and data center inference for robots, including a primer on robot models, deployments, supply chains, the “network wall”, and more — Robot Models, Silicon & DR

Kai Nicol-Schwarz / CNBC: Netherlands-based inference chip startup Euclyd raised a €200M+ Series A co-led by Samsung, Somerset Capital, Scaleup Europe Fund, and Innovation Industries — Samsung has bac

Bloomberg: Sources: Crusoe signed a ~$13B, five-year deal with Jane Street to provide GPUs and other infrastructure for AI training and inference via its cloud platform — Crusoe, a data center upstart

Eugene Demaitre / The Robot Report: Nvidia unveils the Jetson Orin Nano 2 edge AI computer that it says doubles inference performance, with 78 TOPS of AI compute and an eight-core Arm CPU — As AI mode

Thomas Claburn / The Register: OpenAI says new monitoring and security safeguards will add 20% compute overhead to monitored inference workloads; costs will not be passed on to customers — Expanded mu

GPU utilization can read healthy while your queue backs up, and a new replica takes minutes to warm. Here's how to pick autoscaling metrics, tune scale-up/down windows, and budget for cold starts on d

The article shows that dense‑attention performance in long‑context inference is governed by group size (query heads per KV head), head dimension, and sequence length, with prefill being compute‑bound

ThunderAgent is a program-aware scheduler for agentic inference. By treating each agent workflow as a schedulable program, it eliminates KV cache thrashing to deliver more than 2x single-node throughp

How we built the OlmoEarth Platform to fine-tune geospatial models and run continent-scale satellite inference while managing massive data pipelines, distributed compute, and automatically recovering

Phoebe Liu / The Information: AMD is working with Cerebras to combine AMD Helios and Cerebras wafer-scale chips for an AI inference solution to make workloads faster; CBRS closed up 4.86% — AMD said T

Vera Rubin NVL72 is Nvidia’s second‑generation, rack‑scale Oberon architecture that achieves inference gains through extreme co‑design. Early engineering‑sample data from CoreWeave show DeepSeek R1 de

In this post, we walk through five capabilities now available in SageMaker HyperPod inference: multi-tier data capture for auditing and model improvement, direct deployment from Hugging Face Hub, loca

Reuters: Sources: DeepSeek is in the early stages of developing its own AI chip designed for inference, in a bid to reduce its reliance on Nvidia and Huawei hardware — Chinese startup DeepSeek is deve

DigitalOcean has released Evaluations, a feature for testing production models and routers within their inference stack. This tool enables developers to validate and benchmark their AI/ML models befor

Dina Bass / Bloomberg: AI inference startup Etched raised 800M from investors including Jane Street and a TSMC-linked venture firm, and says it has signed sales contracts worth 1B — Nvidia rival says

Stephanie Palazzolo / The Information: Source: OpenAI engineers earlier this month told some colleagues they had figured out a way to more than halve the cost of inference — We closely track efforts b

Ben Jiang / South China Morning Post: DeepSeek details DSpark, a speculative decoding framework for its V4 models, saying it speeds up AI inference by up to 85% and was tested on Gemma and Qwen — Chin

OpenAI and Broadcom unveiled Jalapeño, OpenAI's first Intelligence Processor designed specifically for LLM inference , with early testing showing substantially better performance per watt than current

OpenAI: OpenAI and Broadcom unveil Jalapeño, an LLM-optimized inference chip developed from design to manufacturing tape-out in nine months, aided by OpenAI's models — Loading... - Early testing shows

New research explores how game theory can optimize NVIDIA Dynamo disaggregated serving architectures, revealing critical performance thresholds and reducing worst-case AI inference latency by up to 7.

Modal Auto Endpoints is a feature that automatically optimizes and manages inference endpoints, allowing users to deploy machine learning models with reduced operational overhead. The service likely h

MiniMax-M3 is a large language model capable of handling 1 million token contexts and multimodal inputs while maintaining efficient inference performance. Together AI's blog post discusses techniques

The Information: Sources: ByteDance has partnered with chipmaker InnoStar to develop an AI inference chip modeled after Groq's LPUs, which are built to run AI models at low cost — TikTok owner ByteDan

The Information: Source: AI inference provider Baseten is in talks to raise 1B at a post-money valuation of 11B, up from 5B after its 300M Series E announced in January — AI startup Baseten has recent

This article describes techniques for using prompt caching to improve the inference speed and efficiency of open-source large language models when deployed on Databricks' platform. Prompt caching redu

Cheng Ting-Fang / Nikkei Asia: AMD CEO Lisa Su projects the CPU market will grow over 35% annually through 2031, up from 3% to 4% historically, driven by AI inference and agentic AI demand — TAIPEI —

This article presents benchmarking results for AI coding agents evaluated at scale, likely comparing performance metrics such as code generation accuracy, execution success rates, and inference effici

Anna Tong / Forbes: Sources: Zyphra, which trains and runs inference for its open-weight models on AMD hardware, is raising a 500M Series B at a valuation of at least 5B — The Series B round, which ch

Dutch artificial intelligence infrastructure giant Nebius Group N.V. said today it’s recruiting the core engineering team from AI orchestration software firm Clarifai Inc. in an effort to boost its ma

Robbie Whelan / Wall Street Journal: UK chip startup Fractile raised a $220M Series B led by Factorial Funds, Accel and Founders Fund to make specialized logic and memory chips for inference — Factori