Knowledge catalogue
arXiv:2607.24585v1 Announce Type: new Abstract: We present ELMOD - Efficient Language Model for On-Device Deployment - a compact (2.7B) German language model designed for efficient inference on resour
arXiv:2607.21927v1 Announce Type: new Abstract: Full self-attention in large language models scales as O(N^2), which limits long-context document analysis to 65,536 tokens and requires costly GPU clus
arXiv:2606.20295v2 Announce Type: replace-cross Abstract: Large model inference optimization serves as a key foundation for supporting the scalable, low-cost, and highly stable operation of large mode
arXiv:2607.21985v1 Announce Type: cross Abstract: The increasing deployment of large language models (LLMs) has magnified the computational and memory bottlenecks of autoregressive decoding, where low
Hey everyone, I have been looking into how people source compute for their Inference workloads (and in general). I wanted to understand some specific pain points here. If you've used online services l
Hi everyone! Over the past five months I've been working on DKV (DifferentialKV), an open-source project exploring KV-cache compression for long-context local LLM inference. The goal is to reduce KV-c
arXiv:2607.20152v1 Announce Type: cross Abstract: Active Inference (AIF) frames adaptive behavior as the minimization of expected free energy (EFE), combining epistemic and pragmatic objectives within
OpenAI has updated its inference platform to give users full control over performance, cost, and quality without building their own stack—models go live in minutes and support multiple deployments beh
arXiv:2510.15824v2 Announce Type: replace-cross Abstract: This article considers an online version of conformal inference, called adaptive conformal inference [ACI] and introduced by Gibbs and Candes
arXiv:2607.13120v1 Announce Type: cross Abstract: Inferring gene regulatory networks (GRNs) from single-cell transcriptomic data is crucial for biological discovery, yet existing approaches suffer fro
arXiv:2607.10183v2 Announce Type: replace-cross Abstract: Running large language models on consumer devices such as laptops and desktops is challenging because model weights often exceed GPU memory ca
arXiv:2607.12659v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models have achieved impressive performance on diverse embodied tasks. However, deploying VLA models on low-power onboard
In this post, we introduce the UI for optimized generative AI inference recommendations in Amazon SageMaker AI Studio, a low-code no-code (LCNC) experience. The API already gives you programmatic acce
arXiv:2411.07447v5 Announce Type: replace-cross Abstract: LLMs are increasingly used world-wide from daily tasks to agentic systems and data analytics, requiring significant GPU resources. While LLM i
arXiv:2607.01417v1 Announce Type: new Abstract: Conditional inference trees (CIT) and conditional inference forests (CIF) reduce split-selection bias by testing features before choosing split threshol
arXiv:2603.14526v2 Announce Type: replace Abstract: The recent success of inference-time scaling in large language models has inspired similar explorations in video diffusion. In particular, motivated
arXiv:2606.27743v1 Announce Type: cross Abstract: Large Language Models (LLMs) inference is typically deployed under a static resource assumption, where models execute a fixed computational graph rega
arXiv:2606.25534v1 Announce Type: new Abstract: Implicit Neural Representation (INR) provides an effective approach for continuous signal modeling, but classical per-pixel inference results in quadrat
who wants to help eyal poke holes in this approach to run LLM inference... in browser? Looking for peer reviewers 🧪 Built an LLM inference engine in pure WebGPU/WGSL. Runs in browser and Node, on-devi
Agentic coding changes what inference engines need to handle. At AI Engineer World’s Fair, Together AI engineers will lead a hands-on workshop on how inference engines work and what it takes to serve
arXiv:2602.18934v2 Announce Type: replace Abstract: Membership inference attacks (MIAs) threaten the privacy of machine learning models by revealing whether a specific data point was used during train
arXiv:2410.14843v4 Announce Type: replace-cross Abstract: Vanilla variational inference finds an optimal approximation to the Bayesian posterior distribution, but even the exact Bayesian posterior is
arXiv:2606.23743v1 Announce Type: cross Abstract: Modern video diffusion models achieve higher generation quality through scaling, but this also increases inference cost. Although many acceleration me
arXiv:2606.21099v1 Announce Type: new Abstract: Variational inference (VI) is a powerful method for principled posterior inference for scientific inverse imaging. VI learns the posterior distribution,
arXiv:2606.09924v1 Announce Type: cross Abstract: Deploying deep neural networks on memory-constrained edge accelerators is bottlenecked by per-inference off-chip weight transfer rather than computati
arXiv:2411.16102v2 Announce Type: replace Abstract: Offline batch inference, which leverages the flexibility of request batching to achieve higher throughput and lower costs, is becoming more popular
arXiv:2606.07841v1 Announce Type: cross Abstract: Black-box variational inference (BBVI) is a methodology for posterior approximation that relies on stochastic optimization. In practice, the stochasti
arXiv:2604.02121v2 Announce Type: replace-cross Abstract: Stochastic kinetic models are ubiquitous in physics, yet inferring their parameters from experimental data remains challenging. For determinis
Many research labs only consider inference efficiency after the fact. Step 3.7 Flash is a 198B sparse MoE VLM designed by @StepFun_ai for inference from the start. 196B language backbone with a 1.8B v
arXiv:2606.02643v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG)-enhanced LLM systems, while powerful, introduce substantial inference costs due to the inclusion of an extra mult
arXiv:2606.00946v1 Announce Type: cross Abstract: Efficiently serving large language model (LLM) inference tasks is crucial both for user-perceived latency such as time-to-first-token (TTFT) and for G
arXiv:2602.07905v2 Announce Type: replace Abstract: Large Language Models (LLMs) have shown strong potential in complex medical reasoning yet face diminishing gains under inference scaling laws. While
arXiv:2605.25123v1 Announce Type: cross Abstract: We study inference-time alignment for diffusion-based generative models, aiming to steer a base model toward high-reward outputs without updating its
arXiv:2605.21805v1 Announce Type: cross Abstract: State-space models (SSMs) are powerful probabilistic tools for modeling time-varying systems with latent dynamics. Inference in SSMs involves the esti
arXiv:2603.26539v2 Announce Type: replace Abstract: How does the extent to which a model is open or closed impact the scientific inferences that can be drawn from research that involves it? In this pa
arXiv:2511.21223v2 Announce Type: replace-cross Abstract: Variational inference (VI) is a cornerstone of modern Bayesian learning, enabling approximate inference in complex models. However, its formul
arXiv:2605.17889v1 Announce Type: new Abstract: The Mixture-of-Experts (MoE) architecture improves computational efficiency via sparse expert activation, but throughput-oriented inference faces substa
arXiv:2504.03158v2 Announce Type: replace-cross Abstract: In this work, we propose a new particle-based variational inference (ParVI) method for accelerating the Energetic Variational Inference with I
arXiv:2605.13778v1 Announce Type: cross Abstract: Diffusion-based vision-language-action models (dVLAs) are promising for embodied intelligence but are fundamentally limited in real-time deployment by
arXiv:2605.11093v1 Announce Type: new Abstract: Today's inference-time workloads increasingly depend on timely access to a model's internal states. We present DMI-Lib, a high-speed deep model inspecto
arXiv:2605.11869v1 Announce Type: new Abstract: While the overall inference latency of Video Diffusion Transformers (DiTs) can be substantially reduced through model distillation, per-step inference l
arXiv:2505.13770v3 Announce Type: replace-cross Abstract: Reliable causal inference is essential for making decisions in high-stakes areas like medicine, economics, and public policy. However, it rema
arXiv:2605.11638v1 Announce Type: cross Abstract: U-statistics play a central role in statistical inference. In many modern applications, however, acquiring the labels required for U-statistics is cos
arXiv:2605.10005v1 Announce Type: cross Abstract: In this paper, we describe early work on a specification inference tool for the Move Prover that combines a weakest-precondition (WP) analysis over Mo
arXiv:2605.10533v1 Announce Type: new Abstract: In causal inference, confounders are variables that influence both treatment decisions and outcomes. However, unlike as in randomized clinical trials, t
arXiv:2605.10124v1 Announce Type: cross Abstract: The recent growth of on-device Large Language Model (LLM) inference has driven significant interest in device-edge collaborative LLM inference. As a p
arXiv:2605.09357v1 Announce Type: cross Abstract: Running deep neural networks on microcontroller units (MCUs) is severely constrained by limited memory resources. While TinyML techniques reduce model
arXiv:2605.08168v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models offer a promising path to generalist robot control, but their inference latency causes observation staleness when
arXiv:2605.05615v2 Announce Type: replace Abstract: Large language models (LLMs) impose rapidly growing energy demands, creating an emerging energy and carbon crisis driven by large-scale inference. S
arXiv:2605.06055v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) inference requires large-scale token exchange across devices, making dispatch and combine major bottlenecks in both p
arXiv:2605.07384v1 Announce Type: new Abstract: Inferring the evolution of high-dimensional and multi-modal (e.g., spatio-temporal) physical fields from irregular sparse measurements in real time is a
Excited to speak at PyCon US @pycon about production LLM inference systems, runtime optimization, and next-gen engine design — including some of the new ideas behind the recently open-sourced project
arXiv:2605.04595v1 Announce Type: new Abstract: The rapid adoption of large language models (LLMs) has created significant challenges for efficient inference at scale. Unlike traditional workloads, LL
arXiv:2605.04930v1 Announce Type: new Abstract: Despite theoretical advantages, causal methods for Gene Regulatory Network (GRN) inference from single-cell RNA-seq data consistently fail to match or o
arXiv:2605.03710v1 Announce Type: cross Abstract: Bayesian predictive inference propagates parameter uncertainty to quantities of interest through the posterior-predictive distribution. In practice, t
We’ve developed our own inference engine Runtime-Optimized Serving Engine (ROSE) to serve models ranging from embeddings to trillion-parameter LLMs. With CuTeDSL integrated into our inference engine,
arXiv:2601.10915v2 Announce Type: replace-cross Abstract: In the emerging paradigm of edge learning, neural networks (NNs) are partitioned across distributed edge devices that collaboratively perform
Mike Wheatley / SiliconANGLE: Inference cloud startup DeepInfra raised a 107M Series B co-led by 500 Global and Georges Harik; it currently supports more than 190 open models — Dedicated inference clo
arXiv:2604.09718v2 Announce Type: cross Abstract: LLM-driven web agents operating through continuous inference loops -- repeatedly querying a model to evaluate browser state and select actions -- exhi
arXiv:2604.27536v1 Announce Type: new Abstract: In black-box large language model (LLM) services, response reliability is often only partially observable at decision time, while stronger inference pat