Search: “inference”

SpiralFovea: Input-Adaptive Foveated Tokenization as a Third Lever of Resource-Adaptive Inference

model-releasesarxiv-cs-cv

arXiv:2607.00780v1 Announce Type: new Abstract: Most adaptive-inference techniques for foundation models change what the model does - early exit, MoE routing, KV-cache compression, dynamic attention s

2 Jul 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

“Agentic kernel optimization is the future of on-device inference” @xenovacom used Fable 5 to write kernels that pushed Gemma 4 to a massive…

DGX agent

“Agentic kernel optimization is the future of on-device inference” @xenovacom used Fable 5 to write kernels that pushed Gemma 4 to a massive 255 tok/s on WebGPU with M4. He shared the demo, so you can

model-releasesclem-delangue--x

1 Jul 2026

Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference

arXiv:2606.31903v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) increasingly process long visual-token sequences, increasing the overall inference computation. Existing acce

researcharxiv-cs-ai

ComplianceGate: Classifier-Gated Multi-Tier LLM Routing for Inference in Regulated Industries

arXiv:2606.31163v1 Announce Type: cross Abstract: Large language models deployed in regulated industries operate under two constraints: compliance enforcement and cost efficiency. Personally identifia

local-aiarxiv-cs-ai

DigitalOcean Evaluations: Production Model and Router Testing for the Inference Stack

DigitalOcean has released Evaluations, a feature for testing production models and routers within their inference stack. This tool enables developers to validate and benchmark their AI/ML models befor

applicationsdigitalocean

Agents

We gave a 2 hr deepdive on how to build inference engines that handle trillion token agentic workloads at @aiDotEngineer. Will drop slides a…

Together AI presented a 2-hour technical deep dive on building inference engines capable of handling trillion-token agentic workloads, covering architecture and optimization strategies for large-scale

agentstogether-ai--x

AI inference startup Etched raised 800M from investors including Jane Street and a TSMC-linked venture firm, and says it has signed sales contracts worth 1B (Dina Bass/Bloomberg)

Dina Bass / Bloomberg: AI inference startup Etched raised 800M from investors including Jane Street and a TSMC-linked venture firm, and says it has signed sales contracts worth 1B — Nvidia rival says

hardwaretechmeme

As open models get stronger, more workloads move into the competitive inference market. That pushes the real fight toward speed, cost, relia…

applicationstogether-ai--x

As open models get stronger, more workloads move into the competitive inference market. That pushes the real fight toward speed, cost, reliability, and control. Together AI is where open models become

Doubly Robust Adaptive Conformal Inference for Causal Effects Under Temporal Dependence

arXiv:2606.30500v1 Announce Type: cross Abstract: We propose doubly robust adaptive conformal inference (DR-ACI), which constructs prediction intervals for doubly robust pseudo-outcomes under temporal

inference reliability has historically been a tax on devs that only large well-funded startups could afford: reserve GPUs in advance, sign a…

applicationsfireworks-ai--x

inference reliability has historically been a tax on devs that only large well-funded startups could afford: reserve GPUs in advance, sign a contract, guess your peak throughput requirements. everyone

Industry

Source: OpenAI engineers earlier this month told some colleagues they had figured out a way to more than halve the cost of inference (Stephanie Palazzolo/The Information)

Stephanie Palazzolo / The Information: Source: OpenAI engineers earlier this month told some colleagues they had figured out a way to more than halve the cost of inference — We closely track efforts b

industrytechmeme

Warm-Starting Iterative Gaussian Processes for Faster Sequential Inference

arXiv:2511.16340v2 Announce Type: replace Abstract: Efficient Gaussian process (GP) inference is critical for sequential decision-making tasks such as active learning, online prediction, and Bayesian

DeepSeek details DSpark, a speculative decoding framework for its V4 models, saying it speeds up AI inference by up to 85% and was tested on Gemma and Qwen (Ben Jiang/South China Morning Post)

Ben Jiang / South China Morning Post: DeepSeek details DSpark, a speculative decoding framework for its V4 models, saying it speeds up AI inference by up to 85% and was tested on Gemma and Qwen — Chin

model-releasestechmeme

Directed Graph Topology Inference via Graph Filter Identification

arXiv:2606.27455v1 Announce Type: cross Abstract: We address the problem of inferring a directed network from nodal measurements generated by linear diffusion dynamics on the sought graph. Observation

Going to be in Paris for @RaiseSummit? Join us and @nvidia to hear from leaders at both companies on the future of AI inference and infrastr…

Going to be in Paris for @RaiseSummit? Join us and @nvidia to hear from leaders at both companies on the future of AI inference and infrastructure. After that: cocktails and a DJ. Oh la la! See you th

hardwarefireworks-ai--x

Hybrid coupling with operator inference and the overlapping Schwarz alternating method

arXiv:2511.20687v3 Announce Type: replace-cross Abstract: This paper presents a novel hybrid approach for coupling subdomain-local non-intrusive Operator Inference (OpInf) reduced order models (ROMs)

local-aiarxiv-cs-ai

Tutorials

On the Effect of Uncertainty on Layer-wise Inference Dynamics

arXiv:2507.06722v2 Announce Type: replace Abstract: Understanding how large language models (LLMs) internally represent and process their predictions is central to detecting uncertainty and preventing

tutorialsarxiv-cs-cl

This is from a popular inference provider GLM-5.2 plus the US banning the most capable new models means open source caught up to SOTA closed…

This is from a popular inference provider GLM-5.2 plus the US banning the most capable new models means open source caught up to SOTA closed source coding models This could be v problematic for Anthro

safetygary-marcus--x

27 Jun 2026

Tutorials

This is very interesting. Coinbase seems to have lowered their token spend ($$) to about half, by 1) routing to cheap inference like GLM 5.2…

This is very interesting. Coinbase seems to have lowered their token spend ($$) to about half, by 1) routing to cheap inference like GLM 5.2 and Kimi 2.7 that are still pretty performant 2) Smart rout

tutorialsgary-marcus--x

27 Jun 2026

Beyond Global Divergences: A Local-Mass Perspective on Bayesian Inference

model-releasesarxiv-cs-ai

arXiv:2606.27090v1 Announce Type: cross Abstract: Global objectives, such as KL divergence and ELBO, are widely used in Bayesian inference for measuring distributional discrepancy. This paper studies

Dataset Usage Inference without Shadow Models or Held-out Data

arXiv:2606.26257v1 Announce Type: new Abstract: How much of my data was used to train a machine learning model? Dataset Usage Inference (DUI) aims to answer this by estimating what fraction of a datas

Fireworks AI is now live on EvoSkill v1.3.0! You can now use @FireworksAI_HQ directly with EvoSkill to run fast inference on open models as …

model-releasesfireworks-ai--x

Fireworks AI is now live on EvoSkill v1.3.0! You can now use @FireworksAI_HQ directly with EvoSkill to run fast inference on open models as both the evolution harness backend and the LLM scorer. Along

Inference-Time Robot Behavior Steering through Physically-Aware Reconfiguration of Task-Structure

arXiv:2606.26588v1 Announce Type: new Abstract: A central challenge in deploying learned robot policies is inference-time behavior steering: redirecting a policy at test time to satisfy user preferenc

safetyarxiv-cs-ro

No Free Lunch: Non-Asymptotic Analysis of Prediction-Powered Inference

arXiv:2505.20178v2 Announce Type: replace-cross Abstract: Prediction-Powered Inference (PPI) is a popular strategy for combining gold-standard and possibly noisy pseudo-labels to perform statistical e

applicationsarxiv-cs-lg

Retrieval-Warmed Energy-Based Reasoning: A Five-Arm Ablation Methodology for Diffusion-as-Inference on Structured Reasoning Tasks

arXiv:2606.26476v1 Announce Type: cross Abstract: Warm-started diffusion samplers accelerate iterative inference, but it is rarely clear which part of the pipeline carries the gain. We study extbf{ret

safetyarxiv-cs-ai

Consistent Zero-Shot Imitation with Contrastive Goal Inference

arXiv:2510.17059v2 Announce Type: replace Abstract: Zero-shot imitation learning requires an agent to reproduce expert behavior from a single demonstration without additional environment interaction o

safetyarxiv-cs-lg

Gaussian Mean Field Variational Inference can Overestimate Predictive Variance

model-releasesarxiv-cs-lg

arXiv:2606.25745v1 Announce Type: cross Abstract: Mean Field Variational Inference (MFVI) is widely understood to underestimate posterior variance. By analysing conjugate Bayesian Linear Regression (B

The Inference-Compute Frontier and a Latency-Efficient Architecture for Limit Order Book Prediction

arXiv:2606.25986v1 Announce Type: new Abstract: We study whether a scaling-law-style inference-compute frontier appears in limit order book prediction. Using FI-2010 and a suite of models ranging from

Variational Inference via Entropic Transport Descent

arXiv:2606.25265v1 Announce Type: new Abstract: Particle-based variational inference (ParVI) methods approximate an intractable target distribution by evolving an ensemble of interacting samples. Exis

OpenAI and Broadcom announce chip designed for LLM inference at scale

OpenAI and Broadcom unveiled Jalapeño, OpenAI's first Intelligence Processor designed specifically for LLM inference , with early testing showing substantially better performance per watt than current

hardwarears-technica

OpenAI and Broadcom unveil Jalapeño, an LLM-optimized inference chip developed from design to manufacturing tape-out in nine months, aided by OpenAI's models (OpenAI)

OpenAI: OpenAI and Broadcom unveil Jalapeño, an LLM-optimized inference chip developed from design to manufacturing tape-out in nine months, aided by OpenAI's models — Loading... - Early testing shows

hardwaretechmeme

Tools

The hard part of reinforcement learning on a frontier model is the infrastructure that keeps training and inference numerically identical: z…

The hard part of reinforcement learning on a frontier model is the infrastructure that keeps training and inference numerically identical: zero KLD, end to end. We've solved this challenge, and are no

toolsfireworks-ai--x

Understanding the Hidden Economics of Disaggregated AI Inference

New research explores how game theory can optimize NVIDIA Dynamo disaggregated serving architectures, revealing critical performance thresholds and reducing worst-case AI inference latency by up to 7.

hardwarevultr

.@cartesia runs one of the hardest inference workloads: real-time voice. Their stack has to keep long-lived streams moving, serve millions o…

.@cartesia runs one of the hardest inference workloads: real-time voice. Their stack has to keep long-lived streams moving, serve millions of audio minutes a day, and hold model latency around 90ms. T

hardwaretogether-ai--x

Efficient Traffic State Prediction With Dynamic Joint Spatio-Temporal Relation Inference

arXiv:2504.08061v2 Announce Type: replace Abstract: Traffic prediction is difficult due to the complex interplay of temporal evolution, spatial interactions, and delayed spatio-temporal propagation ov

local-aiarxiv-cs-cv

Tools

Introducing Modal Auto Endpoints: Optimized inference you actually own

Modal Auto Endpoints is a feature that automatically optimizes and manages inference endpoints, allowing users to deploy machine learning models with reduced operational overhead. The service likely h

toolsmodal-blog

Translating Inference-Time Control to Radiology Vision-Language Models: Activation Steering for Pneumonia Classification on Chest X-rays

model-releasesarxiv-cs-cv

arXiv:2606.20852v1 Announce Type: new Abstract: Inference-time engineering can alter model behavior without fine-tuning. However, its utility for improving diagnostic performance in medical vision-lan

The sharpest questions in AI live at the local–cloud boundary : where should inference run, & where should your data live? In town for AIE? …

The sharpest questions in AI live at the local–cloud boundary : where should inference run, & where should your data live? In town for AIE? Come hash it out with @theoryvc, @lancedb & @ollama. June 30

local-aiollama--x

22 Jun 2026

ALIGNBEAM : Inference-Time Alignment Transfer via Cross-Vocabulary Logit Mixing

arXiv:2606.12342v1 Announce Type: cross Abstract: Domain fine-tuning degrades the safety of large language models: fine-tuned specialists readily comply with harmful prompts framed in domain language.

safetyarxiv-cs-ai

CP4SBI: Local Conformal Calibration of Credible Sets in Simulation-Based Inference

arXiv:2508.17077v3 Announce Type: replace-cross Abstract: Current experimental scientists have been increasingly relying on simulation-based inference (SBI) to invert complex non-linear models with in

local-aiarxiv-cs-lg

CRUMB: Efficient Prior Fitted Network Inference via Distributionally Matched Context Batching

model-releasesarxiv-cs-ai

arXiv:2606.11473v1 Announce Type: cross Abstract: Prior-fitted networks (PFNs) are a promising class of tabular foundation models that perform in-context learning, whereby the entire labelled training

Discovery and inference beyond linearity for epidemiological data by integrating Bayesian regression, tree ensembles and Shapley values

arXiv:2505.00571v3 Announce Type: replace-cross Abstract: Machine Learning (ML) is gaining popularity in epidemiology and healthcare studies for hypothesis-free discovery of risk and protective factor

local-aiarxiv-cs-lg

exttt{Range-Arithmetic}: Verifiable Deep Learning Inference on an Untrusted Party

arXiv:2505.17623v2 Announce Type: replace-cross Abstract: Verifiable computing (VC) has gained prominence in decentralized machine learning systems, where resource-intensive tasks like deep neural net

researcharxiv-cs-ai

GraphInfer-Bench: Benchmarking LLM's Inference Capability on Graphs

model-releasesarxiv-cs-cl

arXiv:2606.11562v1 Announce Type: cross Abstract: Graph analysis underlies many applications whose answers cannot be looked up in a single record or retrieved along a path: laundering rings, drug repu

Litespark Inference For CPUs: Ultra-Fast SIMD Framework for Ternary (1.58-bit) Language Models

arXiv:2605.06485v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have transformed artificial intelligence, but their computational requirements remain prohibitive for most users.

hardwarearxiv-cs-ai

PoQ-Judge: A Multi-Architecture Evaluation Framework for Cost-Aware Proof-of-Quality in Decentralized LLM Inference

arXiv:2606.11196v1 Announce Type: cross Abstract: Decentralized LLM inference networks need lightweight, reference-free quality evaluation for Proof of Quality (PoQ). We present PoQ-Judge, a framework

researcharxiv-cs-ai

To Intervene or Not: Guiding Inference-time Alignment with Probabilistic Model Blending

arXiv:2606.11201v1 Announce Type: cross Abstract: The wide deployment of LLMs has made model alignment necessary to make newly trained models safely and effectively respond to user instructions. Among

safetyarxiv-cs-ai

Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design