Search: “inference”

Denoising Surface: Modeling and Predicting Inference Cost for Diffusion LLM Serving

arXiv:2610.00499v1 Announce Type: new Abstract: As diffusion large language models (dLLMs) become more capable, they are moving from research settings to real-world extit{serving}, where request manag

2 Oct 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

Stochastic Rounding in Low-Precision Transformer Inference: A Variable-Precision Emulation Study of a Small GPT-2

DGX agent

arXiv:2610.01889v1 Announce Type: new Abstract: Should low-precision transformer inference use stochastic rounding (SR) or round-to-nearest (RN)? The answer depends on where in the network you look. W

researcharxiv-cs-lg

2 Oct 2026

A polynomial time algebraic solution to exact marginal inference in Markov Random Field models

arXiv:1709.09051v3 Announce Type: replace-cross Abstract: This paper develops on algebraic grounds a polynomial time exact linear solution to the hard combinatorial problem of marginal inference in Ma

researcharxiv-cs-ai

Tools

Day 2 at @AIconference with my favorite @FireworksAI_HQ team 🎆 come talk to our team about model inference, training, and more 🙌

Sophia Yang posted that she was attending Day 2 of the AIconference with the FireworksAI_HQ team. She highlighted discussions on model inference and training among other AI topics. The tweet received

toolsfireworks-ai--x

Instruction Retrieval at Inference Time for Small Language Models

model-releasesarxiv-cs-ai

arXiv:2510.13935v3 Announce Type: replace-cross Abstract: The facts a language model stores are tied to its parameter count, so small models that fit on edge devices fail on expert problems, which nee

Applications

Taipan: A Query-free Transfer-based Multiple Sensitive Attribute Inference Attack Solely from Auxiliary Graphs

arXiv:2602.06700v2 Announce Type: replace-cross Abstract: Graph-structured data underpin a wide spectrum of modern applications, yet their multiple sensitive attributes are not isolated but deeply cou

applicationsarxiv-cs-lg

Industry

Volantis raises $88M to develop photonic inference systems

Volantis Inc. today announced that it has raised a 88 million funding round led by prominent angel investor Lachy Groom and Abstract Ventures. The Series A deal also drew more than a half-dozen other

industrysiliconangle

Calibrating One-Round Membership Inference with Neighbors

arXiv:2609.36331v1 Announce Type: cross Abstract: The state-of-the-art Membership Inference (MI) methods calibrate their signal separately for each example using reference models, auxiliary models tra

researcharxiv-cs-ai

CipherGenome: Homomorphic Inference for Genomic Mixture-of-Experts

arXiv:2609.35883v1 Announce Type: cross Abstract: Genome foundation models are growing into sparse mixture-of-experts (MoE) networks whose expert weights no longer fit on the machines that hold the se

Privacy-Friendly Cohort Determination: Sealed, CSP-Independent In-Browser ML Inference of Professional Segments for Identity-Less Advertising

arXiv:2609.36153v1 Announce Type: cross Abstract: B2B advertising targets a viewer's professional attributes (employer size and industry, function, seniority) and has obtained them by matching identit

local-aiarxiv-cs-lg

Replay the Curvature: Accurate and Scalable NVFP4 Quantization for Large Language Model Inference

model-releasesarxiv-cs-cl

arXiv:2609.36654v1 Announce Type: cross Abstract: Large language models make weight storage and memory traffic major inference costs, motivating low-precision formats that represent each weight with o

AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs

arXiv:2609.34683v1 Announce Type: new Abstract: The optimization of LLM serving engines, such as vLLM and SGLang, is largely benchmark-driven: optimizations, scheduling policies, hardware and system d

From Pixel Generation to Topological Inference: Structural Dual Super-Resolution for Trustworthy Cross-Physical-Domain Trabecular Morphology Learning

arXiv:2609.34716v1 Announce Type: new Abstract: Clinical CT and UHRCT cannot resolve individual trabeculae, whereas synchrotron radiation microCT (SR{mu}CT) provides 3.2{mu}m high-resolution reference

researcharxiv-cs-cv

Same Probe, Different Numbers: Are Activation Probes Robust to Inference-Time Numerical Non-Determinism?

arXiv:2609.31796v1 Announce Type: new Abstract: Activation probes are increasingly used to monitor LLMs in deployment. A probe is typically trained under one inference configuration, then used under w

Tutorials

When Does Depth Matter For In-Context Learning? Adaptive Inference in Deep Transformers

arXiv:2606.16694v2 Announce Type: replace Abstract: Transformers perform computations through many successive attention and feedforward blocks, allowing them to learn complex correlations between a la

tutorialsarxiv-cs-lg

Tutorials

A Little Guide to Learning Distributed Algorithms for LLMS Training and Inference [D]

tutorialsr-machinelearning

Distributed Training and Inference both involves having a fundamental understanding of how distributed systems work in general Distributed Parallelism Tensor Parallelism Pipeline Parallelism Model Par

26 Sept 2026

I added Qwen-Image 2.1 + LoRA support to TensorSharp (GGUF, local inference)

model-releasesr-localllama

I maintain TensorSharp, an open-source inference engine. It can now run Qwen-Image 2.1 locally for text-to-image generation and image editing, with support for its LoRA adapters. I’ve added configs fo

26 Sept 2026

Safety

OpenAI says it paused training, evaluation, and inference with tool-use of its most capable models after a model bypassed internet restrictions during training (OpenAI)

OpenAI: OpenAI says it paused training, evaluation, and inference with tool-use of its most capable models after a model bypassed internet restrictions during training — Summary — An agent attempting

safetytechmeme

26 Sept 2026

Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inference

model-releasesarxiv-cs-ai

arXiv:2602.08329v2 Announce Type: replace-cross Abstract: A core bottleneck in large language model (LLM) inference is the cost of attending over the ever-growing key-value (KV) cache. Although near-o

25 Sept 2026

COMED: The Missing Middle Between Routing and Collaboration in Multi-LLM Inference

model-releasesarxiv-cs-ai

arXiv:2609.26913v1 Announce Type: cross Abstract: No single Large Language Model (LLM) is uniformly reliable across queries, motivating multi-model inference systems that either route among models or

24 Sept 2026

BOBA: Dynamic Bayesian Optimization through Bayesian Active Inference

arXiv:2609.26021v1 Announce Type: new Abstract: Dynamic black-box optimization presents significant challenges for Bayesian Optimization (BO), as the objective function evolves over time, causing opti

researcharxiv-cs-lg

23 Sept 2026

Conditional Tensor Diffusion: Distributional Counterfactual Learning and Inference

arXiv:2609.25924v1 Announce Type: cross Abstract: Causal inference guides operational and managerial decisions but remains challenging in high-dimensional panel or tensor settings, where decisions may

researcharxiv-cs-lg

23 Sept 2026

Adapting Tree-Structured Speculative Decoding to DeepSeek-V4 for Efficient Inference

model-releasesarxiv-cs-cl

arXiv:2609.24698v1 Announce Type: new Abstract: Repeated execution of the target model during autoregressive decoding is a major source of LLM inference latency. Unlike linear speculation, which follo

Co-occurrence Patterns of LoRA Adapters in Production Diffusion Model Inference Services

arXiv:2609.23321v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) has become a key technology for serving large-scale personalized large language models and diffusion models in the cloud. H

From Articles to Publishers: Aggregating Language Model Predictions for News Source Reliability Inference

arXiv:2609.24219v1 Announce Type: new Abstract: Traditionally, the reliability of news publishers is assessed by expert organisations that evaluate editorial practices, transparency and factual standa

researcharxiv-cs-cl

From Tables to Quantified Statements: Evaluating LLM Inference Generation through Executable Verification

arXiv:2609.23966v1 Announce Type: new Abstract: LLMs can generate fluent descriptions from tables, but their outputs may remain logically unsupported by the structured data. We introduce STAT-TO-TEXT,

researcharxiv-cs-cl

Locally Private Inference for Riemannian Stochastic Optimization

arXiv:2609.22642v1 Announce Type: cross Abstract: We develop inference for manifold-valued population minimizers when each observation belongs to a different participant and only locally private messa

Meet Wally. Our inference stack for open frontier models, built to be the fastest place to run them. Performance snapshot: GLM-5.3 Flash: 38…

model-releasesyohei-nakajima--x

Meet Wally. Our inference stack for open frontier models, built to be the fastest place to run them. Performance snapshot: GLM-5.3 Flash: 380 tok/s. GLM-5.3 Max: 790 tok/s Qwen3.8-27B: 485 tok/s DeepS

21 Sept 2026

Riemannian Simultaneous Inference for Tangent Vector Field Regression

arXiv:2609.21910v1 Announce Type: cross Abstract: We consider nonparametric tangent vector field regression on a Riemannian manifold without boundary. Because responses at different points lie in diff

local-aiarxiv-cs-lg

21 Sept 2026

Automated Membership Inference Attacks (AutoMIA): Discovering MIA Signal Computations using LLM Agents

arXiv:2603.19375v2 Announce Type: replace-cross Abstract: Membership inference attacks (MIAs), which enable adversaries to determine whether specific data points were part of a model's training datase

researcharxiv-cs-lg

Benchmarking LLM Inference at Scale with AIPerf

NVIDIA AIPerf is a benchmarking framework that supersedes GenAI‑Perf with a multiprocess architecture, eliminating client‑side bottlenecks during high‑concurrency LLM testing. It supports more than 15

hardwarenvidia-developer

Hot topic livestream: Learn about Jev A buzzy new model Jev, by @typesafeai, reports up to 200x faster inference and 400x lower cost than co…

model-releasesharrison-chase--x

Hot topic livestream: Learn about Jev A buzzy new model Jev, by @typesafeai, reports up to 200x faster inference and 400x lower cost than comparable LLMs on classification tasks. It's aiming to solve

Tutorials

Learning Submanifolds for Subsequent Inference on Random Dot Product Graphs, Part 1: Theory

arXiv:2609.19357v1 Announce Type: cross Abstract: We propose a framework for restricted inference on random dot product graphs whose latent positions lie on an unknown low-dimensional support manifold

tutorialsarxiv-cs-lg

Applications

The Organization of Inference: Information, Resource Constraints, and AI Production

arXiv:2609.20449v1 Announce Type: new Abstract: The economic value of inference depends on how capacity and task information are distributed across stages of AI production. We study these organization

applicationsarxiv-cs-ai

ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference

arXiv:2609.17943v1 Announce Type: cross Abstract: Long-context LLM inference is bottlenecked by attention, whose repeated KV-cache reads make decoding memory-bound. Self-speculative decoding alleviate

researcharxiv-cs-cl

GroupKV: Hierarchical KV Cache Management for Long-Context Diffusion LLM Inference

arXiv:2609.17573v1 Announce Type: cross Abstract: Diffusion large language models (dLLMs) are emerging as a promising generative paradigm that complements autoregressive decoding. In long-context sett

Industry

Inference hardware startup Delos Data raises $100M+ in funding

Delos Data Inc., a startup that develops hardware for artificial intelligence clusters, has raised more than 100 million in funding. The capital was provided by Matrix, Playground, Socratic Partners,

industrysiliconangle

Visual Perception Engine: Fast and Flexible Multi-Head Inference for Robotic Vision Tasks

arXiv:2508.11584v3 Announce Type: replace-cross Abstract: Deploying multiple machine learning models on resource-constrained robotic platforms for different perception tasks often results in redundant

hardwarearxiv-cs-ai

Confidential Computing keeps your data from being tracked or used in three ways: - End-to-end encrypted inference - Hardware-enforced isolat…

Confidential Computing keeps your data from being tracked or used in three ways: - End-to-end encrypted inference - Hardware-enforced isolation that extends to the GPU with @nvidia Confidential Comput

hardwarecohere--x

Is INT8 Portable? A Cross-Platform Measurement Study of Quantized Inference on Embedded and Automotive Accelerators

arXiv:2609.16085v1 Announce Type: cross Abstract: Eight-bit integer (INT8) post-training quantization is the default recipe for edge deployment, under a widely held assumption: INT8 makes inference fa

StackTok: Accelerating VLMs Inference with Budget-Adaptive Visual Token Selection

arXiv:2609.16841v1 Announce Type: cross Abstract: Increasing image resolution produces ever-longer visual-token sequences in vision-language models (VLMs), substantially raising their inference cost.

local-aiarxiv-cs-ai

VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs

arXiv:2609.16722v1 Announce Type: new Abstract: Scaling Multimodal Large Language Models (MLLMs) to long-form video understanding is bottlenecked by the explosion of visual tokens, which saturates con

researcharxiv-cs-ai

A deep dive into on-device and data center inference for robots, including a primer on robot models, deployments, supply chains, the 'network wall', and more (SemiAnalysis)

SemiAnalysis: A deep dive into on-device and data center inference for robots, including a primer on robot models, deployments, supply chains, the “network wall”, and more — Robot Models, Silicon & DR

local-aitechmeme

Safety

Anthropic and OpenAI cannot become profitable unless they can reduce the ratio of training cost to inference profits. In the long run, their…

Anthropic and OpenAI cannot become profitable unless they can reduce the ratio of training cost to inference profits. In the long run, their survival may depend on outlawing competitive open source mo

safetyyann-lecun--x

Netherlands-based inference chip startup Euclyd raised a €200M+ Series A co-led by Samsung, Somerset Capital, Scaleup Europe Fund, and Innovation Industries (Kai Nicol-Schwarz/CNBC)

Kai Nicol-Schwarz / CNBC: Netherlands-based inference chip startup Euclyd raised a €200M+ Series A co-led by Samsung, Somerset Capital, Scaleup Europe Fund, and Innovation Industries — Samsung has bac

hardwaretechmeme

Shared KV Caching for Replicated 27B Inference: Correctness Failures and Performance Boundaries

arXiv:2609.15021v1 Announce Type: cross Abstract: Shared host-memory caching can avoid repeated prefill when a request moves between inference replicas. Its usefulness depends on both correct state tr

local-aiarxiv-cs-lg

Fundamental Dynamical Units for Physics-Informed Structural Inference from Perturbation Time-Series in Networked Systems

arXiv:2609.11934v1 Announce Type: new Abstract: In networked dynamical systems, the parameter of primary mechanistic interest is signed interaction structure. Recovering this structure from perturbati

14 Sept 2026

Get started with Portable Computer in the Perplexity Windows app. On-device inference requires an NVIDIA RTX GPU with 24GB of VRAM or higher…