Search: “inference”

DCGAN inference on a microcontroller: 12.6M parameters, 512KB SRAM, 26-second generation, pure C [P]

researchr-machinelearning

This post describes implementing DCGAN (Deep Convolutional Generative Adversarial Network) inference on resource-constrained microcontroller hardware, achieving image generation with a 12.6 million pa

25 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

Robust Counterfactual Inference in Markov Decision Processes

DGX agent

arXiv:2502.13731v5 Announce Type: replace Abstract: This paper addresses a key limitation in existing counterfactual inference methods for Markov Decision Processes (MDPs). Current approaches assume a

researcharxiv-cs-ai

25 May 2026

What Does the Server See? Understanding Privacy Leakage from Large Language Models in Split Inference

arXiv:2605.23158v1 Announce Type: cross Abstract: The deployment of large language models (LLMs) on resource-constrained devices remains challenging, spurring interest in split inference, where models

researcharxiv-cs-cl

25 May 2026

extit{BlockFormer} : Transformer-based inference from interaction maps

arXiv:2605.21617v1 Announce Type: new Abstract: Inference from interaction maps, such as centromere identification from genome-wide chromosome conformation capture techniques -- notably Hi-C -- can be

23 May 2026

Harnesses for Inference-Time Alignment over Execution Trajectories

arXiv:2605.21516v1 Announce Type: new Abstract: Harness engineering has emerged as an important inference-time technique for large language model (LLM) agents, aiming to improve long-term performance

23 May 2026

NEW paper worth reading. A full agentic workflow can be distilled into model weights and run at roughly 100x lower inference cost while pres…

NEW paper worth reading. A full agentic workflow can be distilled into model weights and run at roughly 100x lower inference cost while preserving near-frontier task quality. The workflow includes mul

agentsdair-ai--x

22 May 2026

Quant.npu: Enabling Efficient Mobile NPU Inference for on-device LLMs via Fully Static Quantization

arXiv:2605.20295v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed on mobile devices, where Neural Processing Units (NPUs) necessitate fully static quantization for

local-aiarxiv-cs-lg

21 May 2026

SPHERICAL KV: Angle-Domain Attention and Rate-Distortion Retention for Efficient Long-Context Inference

arXiv:2605.18856v1 Announce Type: cross Abstract: Long-context inference is increasingly constrained by the KV cache: resident memory grows with context length, and decoding becomes limited by repeate

researcharxiv-cs-cl

20 May 2026

Charon: A Unified and Fine-Grained Simulator for Large-Scale LLM Training and Inference

arXiv:2605.17164v1 Announce Type: cross Abstract: Deploying large-scale LLM training and inference with optimal performance is exceptionally challenging due to a complex design space of parallelism st

Geometric Scaling of Bayesian Inference in LLMs

arXiv:2512.23752v5 Announce Type: replace-cross Abstract: Recent work has shown that small transformers trained in controlled 'wind-tunnel'' settings can implement exact Bayesian inference, and that t

Goal inference with Rao-Blackwellized Particle Filters

arXiv:2512.09269v2 Announce Type: replace Abstract: Inferring the eventual goal of a mobile agent from noisy observations of its trajectory is a fundamental estimation problem. We initiate the study o

agentsarxiv-cs-lg

Inference-Time Diversity in RL-Trained Lean Theorem Provers: A Diagnostic Study

arXiv:2601.16172v2 Announce Type: replace Abstract: RL-trained Lean theorem provers mode-collapse at inference time: on miniF2F-test with DeepSeek-Prover-V1.5-RL, doubling the i.i.d. sampling budget f

Lightweight Gaussian Process Inference in C++ on Metal and CUDA

arXiv:2605.17898v1 Announce Type: new Abstract: Gaussian process (GP) inference in Python is dominated by libraries such as GPyTorch and GPflow, which are built on deep-learning frameworks and inherit

hardwarearxiv-cs-lg

'One thing that we've been seeing recently is that inference benchmarks don't really match production workloads that well.' - @realDanFu, VP…

model-releasestogether-ai--x

'One thing that we've been seeing recently is that inference benchmarks don't really match production workloads that well.' - @realDanFu, VP of Kernels When you're running dozens of concurrent coding

PACE: Geometry-Aware Bridge Transport for Single-Cell Trajectory Inference

model-releasesarxiv-cs-lg

arXiv:2605.18587v1 Announce Type: cross Abstract: Single-cell trajectory inference from destructive time-course snapshots is fundamentally ill-posed: neither cross-time cell correspondences nor contin

Quantum Sidecar Architectures for Hybrid AI Training and Inference: Stateful Protected Registers, Stateless Reset-and-Reprepare Circuits and Quantum Weight-State Outlook

arXiv:2605.18031v1 Announce Type: cross Abstract: We propose a quantum sidecar architecture family for future hybrid AI training and inference. The central idea is not to store an entire Transformer i

researcharxiv-cs-ai

TaskGround: Structured Executable Task Inference for Full-Scene Household Reasoning

arXiv:2605.18109v1 Announce Type: new Abstract: In real home deployments, household agents must often operate from a complete household scene and a situated household request, rather than from a clean

Trajectory-Aware Adaptive Inference in Object Detection Models

arXiv:2605.16397v1 Announce Type: cross Abstract: The increasing integration of sensors in autonomous maritime navigation has led to large-scale multimodal datasets, raising challenges in achieving ef

agentsarxiv-cs-ai

what? we need more support for local options from inference engines, not less!

Clem Delangue argues for increased support and flexibility for local inference engine options rather than restrictions or consolidation. The post advocates for maintaining diverse, locally-deployed AI

local-aiclem-delangue--x

Going Beyond the Edge: Distributed Inference of Transformer Models on Ultra-Low-Power Wireless Devices

arXiv:2605.15694v1 Announce Type: new Abstract: Transformer models are rapidly becoming a cornerstone of modern Internet of Things (IoT) applications, yet their computational and memory demands far ex

18 May 2026

Industry

ROI now stands for “return on inference” and it’s all that matters for the next 18 months

The post argues that 'return on inference' (ROI) has become the primary metric for evaluating AI investments and business value over the next 18 months, shifting focus from traditional return on inves

industrysonya-huang--x

17 May 2026

Tools

As inference workloads dominate, what if these matmuls could also perform useful work and generate beneficial byproducts!? Similar to all th…

As inference workloads dominate, what if these matmuls could also perform useful work and generate beneficial byproducts!? Similar to all the byproducts we get from crude oil distillation that are not

toolstogether-ai--x

15 May 2026

Training-Free Inference for High-Resolution Sinogram Completion

arXiv:2506.08809v5 Announce Type: replace Abstract: High-resolution sinogram completion is critical for computed tomography reconstruction, as missing projections can introduce severe artifacts. While

local-aiarxiv-cs-cv

15 May 2026

Attention Once Is All You Need: Efficient Streaming Inference with Stateful Transformers

model-releasesarxiv-cs-lg

arXiv:2605.13784v1 Announce Type: new Abstract: Conventional transformer inference engines are request-driven, paying an O(n) prefill cost on every query. In streaming workloads, where data arrives co

14 May 2026

Temper and Tilt Lead to SLOP: Reward Hacking Mitigation with Inference-Time Alignment

arXiv:2605.13537v1 Announce Type: cross Abstract: Inference-time alignment techniques offer a lightweight alternative or complement to costly reinforcement learning, while enabling continual adaptatio

safetyarxiv-cs-ai

14 May 2026

Active inference as a unified model of collision avoidance behavior in human drivers

arXiv:2506.02215v5 Announce Type: replace Abstract: Collision avoidance -- involving a rapid threat detection and quick execution of the appropriate evasive maneuver -- is a critical aspect of driving

researcharxiv-cs-ro

Tutorials

FERMI: Exploiting Relations for Membership Inference Against Tabular Diffusion Models

arXiv:2605.11527v1 Announce Type: new Abstract: Diffusion models are the leading approach for tabular data synthesis and are increasingly used to share sensitive records. Whether they actually protect

tutorialsarxiv-cs-lg

LPDP: Inference-Time Reward Control for Variable-Length DNA Generation with Edit Flows

arXiv:2605.11368v1 Announce Type: new Abstract: We study the application of recent Edit Flows for inference-time reward control for DNA sequence generation. Unlike most reward-guided DNA generation fr

local-aiarxiv-cs-lg

probably correct, from @polynoamial: “with today’s AI models, intelligence is a function of inference compute.” but what about tomorrow’s mo…

probably correct, from @polynoamial: “with today’s AI models, intelligence is a function of inference compute.” but what about tomorrow’s models? never forget that humans are remarkably intelligent (t

safetygary-marcus--x

Why Conclusions Diverge from the Same Observations: Formalizing World-Model Non-Identifiability via an Inference

arXiv:2605.12255v1 Announce Type: cross Abstract: When people share the same documents and observations yet reach different conclusions, the disagreement often shifts into a judgment that the other pa

Active Multiple-Prediction-Powered Inference

arXiv:2605.08429v1 Announce Type: cross Abstract: Post-deployment monitoring of healthcare AI requires statistically valid, label-efficient methods, but gold-standard labels from clinician chart revie

Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts

arXiv:2503.05066v5 Announce Type: replace-cross Abstract: The Mixture of Experts (MoE) is an effective architecture for scaling large language models by leveraging sparse expert activation to balance

Foundations of Reliable Inference: Reliability-Efficiency Co-Design

arXiv:2605.10351v1 Announce Type: new Abstract: Reliable inference requires that artificial intelligence (AI) models provide trustworthy uncertainty estimates, not merely accurate predictions. Recent

Nested Slice Sampling: Vectorized Nested Sampling for GPU-Accelerated Inference

model-releasesarxiv-cs-lg

arXiv:2601.23252v2 Announce Type: replace-cross Abstract: Model comparison and calibrated uncertainty quantification often require integrating over parameters, but scalable inference can be challengin

ORICF -- Open Robotics Inference and Control Framework

arXiv:2605.09656v1 Announce Type: new Abstract: Recent advances in artificial intelligence (AI) have enabled effective perception and language models for robots, but their deployment remains computati

researcharxiv-cs-ro

Reasoning emerges from constrained inference manifolds in large language models

model-releasesarxiv-cs-cl

arXiv:2605.08142v1 Announce Type: cross Abstract: Reasoning in large language models is predominantly evaluated through labeled benchmarks, conflating task performance with the quality of internal inf

Supercharging Bayesian Inference with Reliable AI-Informed Priors

arXiv:2605.09834v1 Announce Type: cross Abstract: Modern predictive systems encode beliefs that can act as useful prior information for statistical inference in data-limited settings. Using them for p

Understand and Accelerate Memory Processing Pipeline for Disaggregated LLM Inference

arXiv:2603.29002v2 Announce Type: replace-cross Abstract: Modern large language models (LLMs) increasingly depends on efficient long-context processing and generation mechanisms, including sparse atte

An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference

arXiv:2605.07719v1 Announce Type: cross Abstract: Long-context inference increasingly operates over CPU-resident KV caches, either because decoding-time KV states exceed GPU memory capacity or because

Bayesian Sensitivity of Causal Inference Estimators under Evidence-Based Priors

arXiv:2605.07993v1 Announce Type: new Abstract: Causal inference, especially in observational studies, relies on untestable assumptions about the true data-generating process. Sensitivity analysis hel

Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation

arXiv:2605.07985v1 Announce Type: cross Abstract: Selecting the optimal LLM inference configuration requires evaluation across hardware, serving engines, attention backends, and model architectures, s

SGD for Variational Inference: Tackling Unbounded Variance via Preconditioning and Dynamic Batching

arXiv:2605.07531v1 Announce Type: new Abstract: Black-Box Variational Inference (BBVI) typically relies on Stochastic Gradient Descent (SGD) to optimize the Evidence Lower Bound (ELBO). However, the s

Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache

arXiv:2605.06763v1 Announce Type: new Abstract: Sparse attention improves LLM inference efficiency by selecting a subset of key-value entries, but at the cost of potential accuracy degradation. In par

hardwarearxiv-cs-lg

Industry

How Superhuman and Databricks built a 200K QPS inference platform together

Superhuman and Databricks collaborated to build a high-performance inference platform capable of handling 200,000 queries per second (QPS), leveraging Databricks' infrastructure and optimization capab

industrydatabricks

8 May 2026

GraphPI: Efficient Protein Inference with Graph Neural Networks

arXiv:2605.04376v1 Announce Type: new Abstract: The integration of deep learning approaches in biomedical research has been transformative, enabling breakthroughs in various applications. Despite thes

7 May 2026

LAWS: Learning from Actual Workloads Symbolically -- A Self-Certifying Parametrized Cache Architecture for Neural Inference, Robotics, and Edge Deployment

arXiv:2605.04069v1 Announce Type: new Abstract: We introduce LAWS (Learning from Actual Workloads Symbolically), a self-certifying inference caching architecture that builds a growing library of certi

agentsarxiv-cs-lg

7 May 2026

Predict-then-Diffuse: Adaptive Response Length for Compute-Budgeted Inference in Diffusion LLMs

arXiv:2605.04215v1 Announce Type: new Abstract: Diffusion-based Large Language Models (D-LLMs) represent a promising frontier in generative AI, offering fully parallel token generation that can lead t

7 May 2026

Accelerating Inference of Discrete Autoregressive Normalizing Flows by Selective Jacobi Decoding

arXiv:2505.24791v2 Announce Type: replace Abstract: Discrete normalizing flows are promising generative models with advantages such as analytical log-likelihood computation and end-to-end training. Ho