Search: “inference”

arXiv:2610.00586v1 Announce Type: cross Abstract: Activation memory, not compute, limits CNN inference on constrained hardware such as microcontrollers. Direct in-place convolution removes the dual-bu

arXiv:2609.40024v1 Announce Type: cross Abstract: We develop a general method for amortized Bayesian inference on multilevel models of arbitrary structure. Given a generative model specified as a dire

arXiv:2609.39801v1 Announce Type: new Abstract: Post-training quantization (PTQ) has become a widely adopted technique for reducing the memory footprint and inference cost of large language models (LL

arXiv:2609.33114v1 Announce Type: cross Abstract: For decades, statistical inference has largely been developed one problem at a time. Given a scientific target, such as a treatment effect or a regres

arXiv:2609.32320v1 Announce Type: new Abstract: Hyperdimensional computing (HDC) is attractive for efficient and robust learning, but conventional inference still encodes every query independently, re

the inference platforms are doing hardcore work, post-training open weight models to solve customer problems: @FireworksAI_HQ ember-1 @fal H3 max more to come;) makes total sense considering their pos

arXiv:2602.05145v2 Announce Type: replace-cross Abstract: Speculative decoding can substantially accelerate LLM inference, but realizing its benefits in practice is challenging due to evolving workloa

arXiv:2609.25537v1 Announce Type: new Abstract: Large language model (LLM) inference is constrained by the quadratic scaling of self-attention and the linear scaling of the KV cache, increasing latenc

arXiv:2609.25484v1 Announce Type: new Abstract: Smart meter (SM) data provides fine-grained visibility into household energy consumption, but also exposes users to privacy risks. Inference attacks, kn

arXiv:2609.24754v1 Announce Type: new Abstract: We investigate next generation reservoir computing (NGRC) as a data-driven approach for inferring unseen components of dynamical systems. We compare NGR

arXiv:2601.20332v3 Announce Type: replace Abstract: Diffusion language models (DLMs) generate text through iterative denoising, but inference requires full-sequence attention at every iteration, resul

arXiv:2609.14138v1 Announce Type: cross Abstract: As LLM agents become integrated into increasingly complex workflows, they must continually acquire new capabilities while retaining competence on prev

arXiv:2609.13205v1 Announce Type: cross Abstract: Sparse long-context inference requires efficient token retrieval in both prefill and decode. Existing methods often use different retrieval strategies

arXiv:2609.10105v1 Announce Type: cross Abstract: Compute governance today is a governance of training: the thresholds, reporting requirements, and frontier-AI regimes now in force attach to training

arXiv:2609.11058v1 Announce Type: new Abstract: Recent advances in multimodal large language mod- els (MLLMs) have opened new opportunities for edge intelligence by enabling reasoning across heterogen

arXiv:2609.10550v1 Announce Type: cross Abstract: AI deployment performance is shaped not by model architecture alone, but by interactions among compression, compiler transformations, and serving poli

Positron AI Inc., a developer of inference appliances that don’t include HBM memory, today announced that it has raised 875 million in funding. NEA, Atreides Management, Valor Equity Partners, Andra C

arXiv:2609.09241v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures have emerged as a powerful paradigm for scaling model capacity while preserving efficient inference in large fo

arXiv:2609.09357v1 Announce Type: cross Abstract: Fully homomorphic encryption (FHE) enables inference on private data without revealing it to the server, but evaluating an entire input under FHE is e

arXiv:2609.10264v1 Announce Type: cross Abstract: Open-source large language models (LLMs) are increasingly competitive with closed-source models while offering transparency and the ability to run inf

arXiv:2609.09662v1 Announce Type: cross Abstract: Deploying Large Language Models (LLMs) directly on mobile platforms at the edge is gaining traction due to a myriad of benefits, such as increased pri

arXiv:2609.07660v1 Announce Type: cross Abstract: The concept of Markov chain Monte Carlo (MCMC) cycles, an analogy to cyclic processes in heat engines, is presented in order to examine Bayesian infer

arXiv:2603.26483v2 Announce Type: replace Abstract: Medical edge-AI systems must operate under a difficult tension: delivering reliable diagnostic inference while running on devices with limited batte

arXiv:2609.03149v1 Announce Type: cross Abstract: Analog in-memory computing (AIMC) speeds up neural-network inference by doing the arithmetic directly inside a memory array, instead of shuttling weig

NVIDIA’s PAIR Virtual Inference Router distributes AI inference requests across multiple local machines—supporting Ollama and LM Studio without any agent or harness changes—to mitigate multi‑agent bot

arXiv:2609.00738v1 Announce Type: new Abstract: Inference-time search with large language models (LLMs) often concentrates on a small set of structurally or semantically similar trajectories, leaving

Today we’re open-sourcing Lily, the local inference engine we built for hybrid compute in Perplexity Computer. Lily is specialized for Qwen3.6-35B-A3B on Apple silicon, built so on-device compute does

arXiv:2609.00372v1 Announce Type: cross Abstract: With quantum sensors, simulators and networks emerging, a future of quantum technology may produce quantum states as data---that is, coherently rather

arXiv:2608.28652v1 Announce Type: new Abstract: Artificial intelligence (AI) models have demonstrated remarkable capabilities across various domains, yet their widespread deployment is impeded by sign

for september, we’re cutting the price of Dedicated Inference on H100s from 5.49/hr to 3.99/hr new + existing deployments get the lower price automatically deploy gemma 4, qwen3/3.5, gpt-oss, llama, n

Inference workloads fall into four use‑case buckets (Chatbots/Copilots, Agents, Content Generation, Translation), each with distinct token patterns that dictate the required GPU memory and compute. Si

arXiv:2512.17452v4 Announce Type: replace-cross Abstract: Long-context LLM inference is bottlenecked by the quadratic attention complexity and linear Key-Value (KV) cache growth. Prior approaches miti

arXiv:2608.28113v1 Announce Type: new Abstract: The NVIDIA Blackwell architecture, with native support for the ultra-fine-grained NVFP4 format, opens new opportunities for accelerating large language

Folks! We're just 50 PRs away from more faster inference. Hopefully by end of year. Experts!, please chip in there. List of Open/Ongoing PRs(and also Discussions) related to CPU/RAM/Disk/Hybrid: [Disc

arXiv:2608.27455v1 Announce Type: new Abstract: Recent advances in inference-time scaling have significantly improved the reasoning performance of large language models (LLMs). However, these methods

NVIDIA TensorRT Model Connect delivers a set of reference implementations for deploying open‑source models from Hugging Face checkpoints to native C++ inference in two steps: a Python CLI builds a dep

arXiv:2608.27384v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models are increasingly promising for robotic manipulation, yet their real-world deployment remains bottlenecked by high in

Serving automatic speech recognition (ASR) models at scale is costly when each request uses only a fraction of a GPU. Learn how NVIDIA CUDA Multi-Process Service (MPS) with NVIDIA Triton Inference Ser

arXiv:2608.23831v1 Announce Type: cross Abstract: While reinforcement learning (RL) allows generalist robot policies to continually improve during deployment, the large model size of modern generalist

arXiv:2608.23923v1 Announce Type: new Abstract: Slicing-Aided Hyper Inference (SAHI) improves small object detection in high-resolution images but often spends substantial compute on background tiles.

arXiv:2608.24338v1 Announce Type: new Abstract: Inference-time decoding methods improve LLM reasoning by exploring multiple candidate trajectories, yet treat each trajectory as atomic: either retainin

AI storage infrastructure is becoming a more consequential planning issue as organizations move from model training toward agentic AI. As agents reason, act and reassess, they build longer contexts an

OpenAI reports Jalapeño delivers 1.5–1.9× more AI work per watt and 1.7–3.6× lower latency vs comparison systems on InferenceX.

Qwen3.8 27B is now available for both fine-tuning and Dedicated Model Inference Fine-tune it on your own data, then deploy that model on dedicated infrastructure for production, without stitching toge

Chipmaker Nvidia Corp. says its dedicated artificial intelligence inference accelerator Groq 3 LPX has now entered full production as it strives to maintain its dominance in the world of AI compute. T

Inference Hours is back with another Kimi K3 webinar! On Wednesday, August 26, Zain Hasan sits down with Kevin Cui and Austin Silveria from Together AI's inference team on how we actually serve K3 in