Knowledge catalogue
fyi @FireworksAI_HQ launched inference AND training for K3 on day 0. having 'specialized intelligence' isn't just inference, it's inference + training, together. That's why K3 on Fireworks offers both
Coinbase CEO Brian Armstrong recently posed the question every company scaling AI is asking: how do you keep spend flat while token usage grows exponentially? This isn’t hypothetical. It’s confronting
Web agent economics are set by inference volume and each step is one inference call. A simple task (extract a field, fill a form, navigate a site) is 10 to 20 calls. A multi-site workflow runs into th
arXiv:2604.19760v1 Announce Type: new Abstract: We present a simulation-based evaluation of the Inference Headroom Ratio (IHR), a dimensionless diagnostic quantity for characterizing inference stabili
Sam Altman praises the inference team's work on efficiently serving a model, suggesting that becoming proficient in AI inference is crucial to the field's progress. The post appears to highlight the t
This is a HANDWRITTEN post. I spent way too much time trying to get fast inference on Apple Silicon. This post is for people who want to know what's the latest on running local models on their mac, an
Enterprises and individual developers frequently run multiple AI inference models. The right architecture can simplify how the models are called while also providing centralized governance. In this po
arXiv:2610.04862v1 Announce Type: cross Abstract: Long-horizon problem solving and scientific research require computation to accumulate across successive attempts. Partial solutions, experimental fin
how inference engines actually work - releasing full talk slides! i cover everything in the lifetime of a request e2e: > the inference engine > kv + prefix caching > continuous batching > paged attent
Next YC Paper Club: Inference Inference is no longer just: y = model.forward(x) It has now become as complex and important as training. The difference between naive inference and well tuned could be 1
Demand for AI infrastructure is at an all-time high. Global accelerator shortages mean engineering teams can rarely get all the compute they need from just one data center — capacity comes a cluster h
arXiv:2605.26419v1 Announce Type: new Abstract: Amortized inference promises fast test-time Bayesian inference, but existing methods are inherently tied to fixed models. Extending amortization to unse
arXiv:2605.11511v1 Announce Type: cross Abstract: The validity of statistical inference depends critically on how data are collected. When data gathered through active data collection (ADC) are reused
arXiv:2605.20281v1 Announce Type: cross Abstract: We develop a unified microeconomic and monetary theory of artificial intelligence inference costs and their pass-through to inflation, welfare, and op
Inference is becoming the largest compute market and energy consumer in AI. Pearl turns inference CapEx of hyperscalers into a profit center: every LLM token produced by GPUs can simultaneously genera
arXiv:2609.23130v1 Announce Type: new Abstract: Large language model (LLM) inference is evolving from an engine-local optimization problem into a distributed control problem involving reusable state,
As generative AI moves from experimental pilots to massive production environments, the efficiency of your infrastructure becomes the ultimate differentiator. One way to get the most out of it and min
arXiv:2606.04935v1 Announce Type: new Abstract: Active inference casts decision-making as inference, with the Expected Free Energy (EFE) unifying goal-directed and information-seeking behavior. Recent
Minimax M3 is a remarkable model in many ways, but esp for inference efficiency. Minimax model designers and @togethercompute inference team will discuss it tomorrow live. MiniMax M3 is live and Toget
Many research labs only consider inference efficiency after the fact. Step 3.7 Flash is a 196B MoE model, and built for inference from the start by @StepFun_ai. Multi-Matrix Factorization Attention (M
arXiv:2511.17826v2 Announce Type: replace-cross Abstract: Deterministic inference is increasingly critical for large language model (LLM) applications such as LLM-as-a-judge evaluation, multi-agent sy
Together AI has long been a proud supporter of open-source innovation in inference. We're excited for the new TokenSpeed inference engine, available in preview today with MIT licence, and can't wait t
arXiv:2609.37381v1 Announce Type: new Abstract: Neural simulation-based inference (SBI) has been widely successful in inferring a relatively small number of interpretable parameters from potentially h
2/ New deep dive: Autoscaling endpoints for LLM inference. Dedicated Inference can scale on eight metrics. inflight_requests is the default because it sees queue pressure before latency degrades. We t
Anyone building with AI eventually hits the same tradeoff: how to get the most intelligence per dollar, the right model at the right cost for each task. That’s what DigitalOcean Inference Engine is bu
NVIDIA TensorRT's multi-device inference feature scales inference across multiple GPUs, enabling deployment of models that exceed single-GPU memory or reducing latency for memory-bound workloads. Each
Ben Thompson / Stratechery: Agentic inference is set to be different than today's inference, and will change compute infrastructure because speed won't matter when humans aren't involved — Subscribe t
Every few weeks, a new model steps up and claims best-in-class at something. The launch chart makes the case, the link makes the rounds, and your team is back to the question nobody has time for: shou
arXiv:2512.19905v3 Announce Type: replace-cross Abstract: Recent developments in large language models have shown advantages in reallocating a notable share of computational resource from training tim
arXiv:2607.26504v1 Announce Type: new Abstract: Many discrete reasoning tasks, such as code generation, are inherently non-causal: programmers move between high-level structure and local details, a pr
arXiv:2602.18196v3 Announce Type: replace Abstract: Structured dilated attention has an appealing inference-time efficiency knob: it reduces the FLOPs of attention and the KV cache size by a factor of
arXiv:2605.00174v1 Announce Type: cross Abstract: Video and image streaming on edge devices requires low latency. To address this, Neural Networks (NNs) are widely used, and prior work mainly focuses
arXiv:2609.38239v1 Announce Type: cross Abstract: A Technical Report: Operating a large language model (LLM) as a service requires more than inference infrastructure: the provider must also defend aga
arXiv:2608.13863v1 Announce Type: new Abstract: Deep neural network (DNN) inference on mobile devices often incurs high latency and energy consumption due to limited computing and memory resources. To
arXiv:2603.11799v2 Announce Type: replace Abstract: Membership inference attacks (MIAs) are becoming standard tools for auditing the privacy of machine learning models. The leading attacks -- LiRA (Ca
arXiv:2604.27935v1 Announce Type: new Abstract: This paper presents an expert-guided active-inference-inspired framework for adaptive UAV swarm trajectory planning. The proposed method converts multi-
arXiv:2604.27855v1 Announce Type: cross Abstract: AI inference is becoming a persistent and geographically distributed source of electricity demand. Unlike many traditional electrical loads, inference
arXiv:2610.08021v1 Announce Type: new Abstract: Simulation-based inference (SBI) has become a powerful approach to Bayesian inference in complex scientific models whose likelihoods are difficult or im
ninfer, dwarfstar, Splash, llamAmpere, gufo, etc. We've all seen them popping up, great tok/s, people loving them. Forks of llama.cpp or another engine, or made from scratch. For better or worse, the
arXiv:2603.03146v2 Announce Type: replace-cross Abstract: Integrated communication and computation (IC^2) has emerged as a new paradigm for enabling efficient edge inference in sixth-generation (6G) n
arXiv:2608.27713v1 Announce Type: new Abstract: Edge intelligence systems increasingly require model training and online inference to coexist on resource-constrained devices, while inference demand ca
arXiv:2608.28044v1 Announce Type: cross Abstract: Large language model (LLM) inference serving is priced by tokens, but GPU energy is consumed over inference windows. This accounting mismatch makes to
arXiv:2502.08004v2 Announce Type: replace-cross Abstract: Simulation-based inference (SBI) is a method to perform inference on a variety of complex scientific models with challenging inference (invers
arXiv:2607.21162v1 Announce Type: new Abstract: Outsourced Transformer inference exposes clients to model substitution and incomplete execution, while direct replay removes the computational benefit o
arXiv:2607.20480v1 Announce Type: new Abstract: Accurate distribution system topology is essential for outage localization, voltage analytics, and operation of distribution grids, yet maintaining reli
arXiv:2607.13093v1 Announce Type: cross Abstract: On-device LLM inference faces a trilemma of response latency, limited hardware resources and user privacy. Full cloud inference delivers strong comput
Building and serving models on infrastructure is a strong use case for businesses. In Google Cloud, you have the ability to design your AI infrastructure to suit your workloads. Recently, I experiment
Amazon SageMaker AI shipped 13 inference launches in the first half of 2026 across two deployment paths: fully managed endpoints and Amazon SageMaker HyperPod Inference. This post reviews each launch,
arXiv:2608.24662v1 Announce Type: new Abstract: Large language models (LLMs) are commonly evaluated under the assumption that their observable behavior is primarily determined by model weights, traini
arXiv:2608.15636v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models have demonstrated remarkable capabilities in the field of embodied AI, but their high computational cost and limit
arXiv:2608.15502v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models have emerged as a promising foundation for Embodied AI, but their high inference cost poses significant challenges f
arXiv:2608.05926v1 Announce Type: cross Abstract: Edge inference is a promising paradigm to provide large language model (LLM) inference services in next-generation mobile networks. LLM inference main
arXiv:2608.04428v1 Announce Type: cross Abstract: Vision-language-action (VLA) models have emerged as a key component in embodied AI. Among existing approaches, diffusion-based VLA models achieve supe
arXiv:2606.00829v1 Announce Type: new Abstract: EgoCross evaluates multimodal large language models on egocentric video question answering under substantial domain shift, where test videos come from s
DigitalOcean's Dedicated Inference offering provides cloud infrastructure optimized for running machine learning inference workloads with guaranteed resources and performance isolation. This technical
arXiv:2609.15992v1 Announce Type: new Abstract: Recent advances in large language models (LLMs) have rendered them necessary for Natural Language Processing (NLP) tasks, and their high inference cost
arXiv:2608.30319v1 Announce Type: cross Abstract: Large language models (LLMs) finetuned for specialized domains represent crucial high-impact applications. Inference-time alignment improves safety de
arXiv:2608.28667v1 Announce Type: cross Abstract: The rapid proliferation of Large Language Models (LLMs) has raised concerns about their environmental impact during inference. While Green AI research
arXiv:2510.25977v4 Announce Type: replace Abstract: Emerging AI accelerators have started to gain attention and offer new opportunities for efficient inference of large language models (LLMs). Trainiu
arXiv:2610.16669v1 Announce Type: cross Abstract: Bayesian inference for latent point process intensities often relies on per-dataset sampling or optimization, making repeated inference computationall