Search: “inference”

The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

arXiv:2606.29526v1 Announce Type: new Abstract: Reinforcement learning (RL) has gained growing attention in large language model (LLM) post-training, yet RL training remains fragile and can suffer fro

30 Jun 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Local Ai

EnerInfer: Energy-Aware On-Device LLM Inference

DGX agent

arXiv:2606.23001v1 Announce Type: cross Abstract: On-device LLM inference is increasingly attractive for privacy-preserving, reliable, and cost-effective deployment, yet its energy and thermal costs r

local-aiarxiv-cs-lg

24 Jun 2026

BatchGen: An Architecture for Scalable and Efficient Batch Inference

arXiv:2606.21712v1 Announce Type: cross Abstract: Batch inference has become a central mode of AI computation, yet existing inference engines still rely on execution models designed for interactive se

23 Jun 2026

From Explicit Elements to Implicit Intent: A Predefined Library for Auditable Behavioral Inference

arXiv:2606.11207v1 Announce Type: new Abstract: We present SemantiClean, a modular framework for extracting structured semantic signals from e-commerce session data and driving pluggable inference tar

safetyarxiv-cs-ai

11 Jun 2026

Learning the Universe: Posterior Reliability of Neural Generative Models in High-Dimensional Field-Level Inference of Cosmic Initial Conditions

arXiv:2606.10023v1 Announce Type: cross Abstract: Accurate posterior estimation is central to scientific inference, as uncertainties determine what can be reliably learned from observational data. Whi

applicationsarxiv-cs-lg

10 Jun 2026

PSA: Just added a few thousand chips, including B200s and B300s to our Dedicated Model Inference (http://api.together.ai/endpoints). With De…

model-releasestogether-ai--x

PSA: Just added a few thousand chips, including B200s and B300s to our Dedicated Model Inference (http://api.together.ai/endpoints). With Dedicated Model Inference, you can now on-click deploy our Bla

8 Jun 2026

DriftSched: Adaptive QoS-Aware Scheduling under Runtime Token Drift for Multi-Tenant GPU Inference

arXiv:2606.02982v1 Announce Type: cross Abstract: The rapid growth of large language model (LLM) inference services has increased the demand for efficient multi-tenant GPU scheduling. While modern inf

3 Jun 2026

Impact of Graph Structure on Membership-Inference Risk for Graph Neural Networks

arXiv:2601.17130v2 Announce Type: replace Abstract: Graph neural networks (GNNs) are widely used for tasks such as node classification and link prediction, but their use in sensitive settings raises c

3 Jun 2026

Ideas in Inference-time Scaling can Benefit Generative Pre-training Algorithms

arXiv:2503.07154v3 Announce Type: replace-cross Abstract: Generative pre-training is often framed through a false dichotomy between autoregressive models for discrete signals and diffusion models for

researcharxiv-cs-ai

2 Jun 2026

Noise-Aware Differentially Private Variational Inference

arXiv:2410.19371v3 Announce Type: replace-cross Abstract: Differential privacy (DP) provides robust privacy guarantees for statistical inference, but this can lead to unreliable results and biases in

29 May 2026

Prediction-Powered Inference Across Many Tasks for AI Evaluation & Social Science Research

arXiv:2605.29249v1 Announce Type: cross Abstract: Many applications require statistically valid inference across many related tasks, while using only a handful of high-quality labels per hypothesis. I

applicationsarxiv-cs-lg

29 May 2026

Industry

How We Built DigitalOcean Inference Router

DigitalOcean describes the architecture and engineering approach behind their Inference Router, a system designed to route and manage machine learning inference requests efficiently across distributed

industrydigitalocean

20 May 2026

Breaking the accuracy-resource dilemma: a lightweight adaptive video inference enhancement

arXiv:2601.14568v2 Announce Type: replace-cross Abstract: Existing video inference (VI) enhancement methods typically aim to improve performance by scaling up model sizes and employing sophisticated n

researcharxiv-cs-ai

19 May 2026

p2p ai inference mesh

A peer-to-peer AI inference mesh is a decentralized network architecture where agents connect directly to discover peers and communicate through distributed protocols to share inference workloads . Su

local-air-ollama

13 May 2026

Training-Inference Consistent Segmented Execution for Long-Context LLMs

arXiv:2605.11744v1 Announce Type: new Abstract: Transformer-based large language models face severe scalability challenges in long-context generation due to the computational and memory costs of full-

researcharxiv-cs-cl

13 May 2026

Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts

arXiv:2509.21892v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) models typically fix the number of activated experts k at both training and inference. However, real-world deployment

applicationsarxiv-cs-ai

12 May 2026

Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers

arXiv:2605.03780v1 Announce Type: cross Abstract: Transformers are effective at inferring the latent task from context via two inference modes: recognizing a task seen during training, and adapting to

researcharxiv-cs-cl

6 May 2026

ViLegalNLI: Natural Language Inference for Vietnamese Legal Texts

model-releasesarxiv-cs-cl

arXiv:2605.00116v1 Announce Type: new Abstract: In this article, we introduce ViLegalNLI, the first large-scale Vietnamese Natural Language Inference (NLI) dataset specifically constructed for the leg

4 May 2026

Strait: Perceiving Priority and Interference in ML Inference Serving

arXiv:2604.28175v1 Announce Type: new Abstract: Machine learning (ML) inference serving systems host deep neural network (DNN) models and schedule incoming inference requests across deployed GPUs. How

1 May 2026

Even More Guarantees for Variational Inference in the Presence of Symmetries

arXiv:2604.21407v1 Announce Type: new Abstract: When approximating an intractable density via variational inference (VI) the variational family is typically chosen as a simple parametric family that v

NPU Design for Diffusion Language Model Inference

arXiv:2601.20706v2 Announce Type: replace-cross Abstract: Diffusion-based LLMs (dLLMs) fundamentally depart from traditional autoregressive (AR) LLM inference: they leverage bidirectional attention, b

researcharxiv-cs-ai

SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference

arXiv:2604.21231v1 Announce Type: cross Abstract: Efficient inference for on-device Large Language Models (LLMs) remains challenging due to limited hardware resources and the high cost of the prefill

local-aiarxiv-cs-ai

Subject-level Inference for Realistic Text Anonymization Evaluation

model-releasesarxiv-cs-cl

arXiv:2604.21211v1 Announce Type: new Abstract: Current text anonymization evaluation relies on span-based metrics that fail to capture what an adversary could actually infer, and assumes a single dat

Industry

Amazon SageMaker AI now supports optimized generative AI inference recommendations

Today, Amazon SageMaker AI supports optimized generative AI inference recommendations. By delivering validated, optimal deployment configurations with performance metrics, Amazon SageMaker AI keeps yo

industryaws-ml-blog

22 Apr 2026

Streaming Structured Inference with Flash-SemiCRF

arXiv:2604.18780v1 Announce Type: new Abstract: Semi-Markov Conditional Random Fields (semi-CRFs) assign labels to segments of a sequence rather than to individual positions, enabling exact inference

22 Apr 2026

We are excited to have @baseten as a day 0 launch partner for Kimi K2.6! Their inference stack brings KV-aware routing, NVFP4 on Blackwell, …

We are excited to have @baseten as a day 0 launch partner for Kimi K2.6! Their inference stack brings KV-aware routing, NVFP4 on Blackwell, multi-modal hierarchical caching, and prefill-decode disaggr

hardwarekimi-moonshot--x

20 Apr 2026

Industry

The Inference Cloud Memory Layer: A Technical Dive into DigitalOcean Managed Databases

DigitalOcean's Inference Cloud Memory Layer is a technical architecture component designed to optimize database performance by implementing an in-memory caching layer for faster data access and reduce

industrydigitalocean

17 Apr 2026

Tutorials

Accelerating decode-heavy LLM inference with speculative decoding on AWS Trainium and vLLM

Speculative decoding is a technique used to accelerate the slow, sequential token generation (decode stage) in LLM inference. This method significantly reduces latency and improves hardware utilizatio

tutorialsaws-ml-blog

15 Apr 2026

Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects

arXiv:2604.05546v2 Announce Type: replace Abstract: Large Vision-Language Models (LVLMs) enable sophisticated reasoning over images and videos, yet their inference is hindered by a systemic efficiency

researcharxiv-cs-cl

15 Apr 2026

Unified Multimodal Uncertain Inference

model-releasesarxiv-cs-cv

arXiv:2604.08701v1 Announce Type: new Abstract: We introduce Unified Multimodal Uncertain Inference (UMUI), a multimodal inference task spanning text, audio, and video, where models must produce calib

13 Apr 2026

New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys @lmsysorg and RadixArk @radixark, an…

New course: Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys @lmsysorg and RadixArk @radixark, and taught by Richard Chen @richardczl, a Member of Technical

hardwareandrew-ng--x

9 Apr 2026

Cross-Prompt Inference Mitigates Rare-Class Suppression in Imbalanced VLM Finetuning

arXiv:2610.21470v1 Announce Type: new Abstract: When a vision-language model is finetuned on class-imbalanced data, rare-class recall can fall well below the model's own zero-shot performance. In this

researcharxiv-cs-cv

16 Oct 2026

A Graph-Based Analysis of Comparability, Scale, and Attribution in Private Transformer Inference

arXiv:2610.19607v1 Announce Type: cross Abstract: Rising privacy concerns around large language models have accelerated research in private Transformer inference (PTI). Synthesizing the empirical clai

applicationsarxiv-cs-lg

15 Oct 2026

Seeing at the Edge: Energy Characterization of Vision-Language Model Inference on Edge Platforms

arXiv:2610.20136v1 Announce Type: new Abstract: Vision-language models (VLMs) are moving onto edge devices, but little is known about how much energy they draw on single-board computers with NPU or GP

15 Oct 2026

SKG: A Law-Induced Engine from Inference to Induction

arXiv:2610.19505v1 Announce Type: new Abstract: This paper proposes the SKG (Symplectic-Kernel Gradient) reasoning engine for law-induced inference and induction. Laws are a priori; data provide insta

15 Oct 2026

Agents

How distributed inference keeps large AI models moving

Distributed inference splits a large AI model into sub‑components that run concurrently on multiple compute nodes, allowing each node to process only part of the workload. This parallelism boosts thro

agentsibm-research

12 Oct 2026

Microsoft debuts multichiplet Maia 300 inference accelerator

Microsoft Corp. today pulled back the curtains on Maia 300, its newest internally developed artificial intelligence accelerator. The chip is optimized for inference. Microsoft estimates that it can pr

hardwaresiliconangle

12 Oct 2026

Impressive by @river_ai_inc to beat @deepseek_ai rates - cache in particular is tough! Talented team there, full RL + inference stack should…

model-releasesemad-mostaque--x

Impressive by @river_ai_inc to beat @deepseek_ai rates - cache in particular is tough! Talented team there, full RL + inference stack should be real competition River Inference is live! We're serving

11 Oct 2026

PSI-SINDy: Post-Selection Inference for Sparse Identification of Nonlinear Dynamics

arXiv:2610.11486v1 Announce Type: cross Abstract: Sparse identification of nonlinear dynamics (SINDy) is a data-driven framework for discovering governing dynamics from time-series data by identifying

9 Oct 2026

Evaluating Inference Compute for Generative AI: A Framework for Enterprise Workloads

model-releasesarxiv-cs-lg

arXiv:2610.07094v1 Announce Type: cross Abstract: LLM deployment is shifting from single-turn completion to agentic trajectories in which a model plans, calls tools, reads results and reasons at test

7 Oct 2026

Stepped MoE: Segment-Level Routing with Configurable Inference Complexity

arXiv:2610.07348v1 Announce Type: cross Abstract: Training large language models (LLMs) is resource-intensive, and adapting them for diverse deployment scenarios with varying computational constraints

local-aiarxiv-cs-ai

7 Oct 2026

Antseed launches a decentralized marketplace for AI inference that dramatically lowers developers’ costs

model-releasessiliconangle

Developers now have a much cheaper way to access cutting-edge artificial intelligence inference capabilities with the launch of Antseed today. It’s a new peer-to-peer marketplace that makes it possibl

Benchmarking Generative Trajectory Models for Active-Inference Control

model-releasesarxiv-cs-ro

arXiv:2610.05692v1 Announce Type: new Abstract: Learning from trajectory demonstrations offers a route to active-inference control of complex systems whose dynamics are difficult to model explicitly.

Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs

arXiv:2610.05305v1 Announce Type: cross Abstract: Large Language Model (LLM) inference has become the dominant workload in modern AI systems, requiring serving infrastructures to maximize throughput w

Finite-Sample Distribution Theory and Efficient Large-Scale Inference for Online Quantile Regression

arXiv:2610.05869v1 Announce Type: cross Abstract: This paper studies online quantile regression for large-scale and streaming data using Stochastic SubGradient Descent (SSGD) with constant learning ra

FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices

arXiv:2601.17063v2 Announce Type: replace Abstract: Recently, Mixture-of-Experts (MoE) models have gained attention for efficiently scaling large language models. Although these models are extremely l

local-aiarxiv-cs-lg

Task Inference Beyond Least Squares in Behavioral Foundation Models

arXiv:2610.05350v1 Announce Type: new Abstract: Behavioral Foundation Models (BFMs) aim to solve a wide range of downstream tasks without test-time policy learning by inferring a task vector from the

Q&A with OpenAI VP of Hardware Richard Ho on its Jalapeño inference chip co-designed with Broadcom, using internal OpenAI models to design the chip, and more (Dr. Ian Cutress/More Than Moore)