Knowledge catalogue
This post discusses upgrade recommendations for improving local LLM inference performance on a gaming laptop with an RTX 4080 (12GB VRAM) and 16GB system RAM. Community members likely shared options f
We have greatly expanded the surface area for plugins over the last few weeks, and now you can extend LLM Inference Providers and Gateway Channels with plugins. Want a custom implementation of your fa
Dedicated inference cloud startup Deepinfra Inc. is looking to expand its global capacity after raising 107 million in a Series B round of funding led by 500 Global and Georges Harik, who was one of G
arXiv:2506.06057v2 Announce Type: replace Abstract: Large Language Models (LLMs) rely on massive training datasets, often including proprietary data, which raises concerns about unauthorized usage and
arXiv:2511.06838v4 Announce Type: replace-cross Abstract: The substantial memory bandwidth and computational demands of large language models (LLMs) present critical challenges for efficient inference
arXiv:2511.08303v2 Announce Type: replace-cross Abstract: This study investigates treatment effect estimation in the semi-supervised setting, also can be interpreted as prediction-powered inference. I
arXiv:2605.00519v2 Announce Type: cross Abstract: The operational landscape of local Large Language Model (LLM) inference has shifted from lightweight models to datacenter-class weights exceeding 70B
This Modal blog post describes a performance optimization technique for multimodal AI inference that achieves over 10% improvement through a simple Python dictionary-based approach. The article likely
arXiv:2605.00300v1 Announce Type: cross Abstract: Public inference benchmarks compare AI systems at the model and provider level, but the unit at which deployment decisions are actually made is the en
arXiv:2602.03216v2 Announce Type: replace Abstract: The quadratic complexity of attention remains the central bottleneck in long-context inference for large language models. Prior acceleration methods
arXiv:2505.11329v5 Announce Type: replace-cross Abstract: Distributed inference of large language models (LLMs) using tensor parallelism can introduce communication overheads of 20% even over GPUs con
arXiv:2604.26979v1 Announce Type: cross Abstract: In-memory computing (IMC) enables energy-efficient neural network inference by computing analog matrix-vector multiplications (MVM) in memory crossbar
This article from Latent Space discusses a pivotal shift in AI development toward inference—the computational process of running trained models—suggesting this represents a critical turning point comp
arXiv:2604.26557v1 Announce Type: cross Abstract: The increasing deployment of Large Language Model (LLM) inference on edge AI systems demands efficient execution under tight memory budgets. A key cha
arXiv:2502.14912v2 Announce Type: replace Abstract: We present a framework for generating universal semantic embeddings of chemical elements to advance materials inference and discovery. This framewor
arXiv:2510.14703v2 Announce Type: replace Abstract: Large language models (LLMs) excel at function calling, but inference scaling has been explored mainly for unstructured generation. We propose an in
arXiv:2509.17625v2 Announce Type: replace Abstract: In this paper, we present the first systematic comparison of Data Assimilation (DA) and Likelihood-Based Inference (LBI) in the context of an Agent-
DeepInfra is an inference provider integrated with Hugging Face that enables users to deploy and run machine learning models at scale. This integration allows developers to access DeepInfra's serverle
arXiv:2510.19669v4 Announce Type: replace Abstract: Recent reasoning Large Language Models (LLMs) demonstrate remarkable problem-solving abilities but often generate long thinking traces whose utility
arXiv:2604.24818v1 Announce Type: new Abstract: Bayesian finite mixture models can identify discrete risk clusters (low-risk vs. high-risk equipment), but face three critical bottlenecks: (1) insuffic
arXiv:2604.24647v1 Announce Type: cross Abstract: Long-context reasoning is a critical capability of large language models (LLMs), enabling applications such as long-document understanding, summarizat
arXiv:2604.23350v1 Announce Type: cross Abstract: Deep generative models and neural operators have demonstrated significant potential for 3D aerodynamic inference. However, they often face inherent ch
DigitalOcean describes their optimization and deployment of three large language models (DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B) on their Serverless Inference platform, likely leveraging NVIDI
arXiv:2604.23467v1 Announce Type: cross Abstract: Large Language Models (LLMs) have achieved strong performance across natural language and multimodal tasks, yet their practical deployment remains con
arXiv:2604.23107v1 Announce Type: cross Abstract: Causal effect estimation from observational data requires careful adjustment for confounding. Classical estimators such as inverse probability weighti
arXiv:2509.25003v2 Announce Type: replace-cross Abstract: Membership inference attacks (MIAs) against Diffusion Models (DMs) raise pressing privacy concerns by revealing whether a sample was part of t
Stability Matrix's Inference tab enables users to generate images with a straightforward interface powered by ComfyUI , with seed functionality allowing either random seeds generated each time or cust
Clem Delangue highlights a significant development in local inference technology, noting that a project (referred to as 'whale bros') has released their work as open source. The post emphasizes the im
arXiv:2604.20500v1 Announce Type: new Abstract: Self-consistency boosts inference-time performance by sampling multiple reasoning traces in parallel and voting. However, in constrained domains like ma
arXiv:2604.19936v1 Announce Type: cross Abstract: With the emergence of new evaluation metrics and attack methodologies for Membership Inference Attacks (MIA), it becomes essential to reevaluate previ
arXiv:2604.20174v1 Announce Type: new Abstract: Reinforcement learning (RL) policies are typically trained for fixed objectives, making reuse difficult when task requirements change. We study inferenc
arXiv:2604.20723v1 Announce Type: cross Abstract: The cost of simulator evaluations is a key practical bottleneck for Simulation Based Inference (SBI). In hierarchical settings with shared global para
arXiv:2604.18788v1 Announce Type: new Abstract: Apple Neural Engine (ANE) is a dedicated neural processing unit (NPU) present in every Apple Silicon chip. Mixture-of-Experts (MoE) LLMs improve inferen
Enjoyed the read? If you have deep experience in ML frameworks (training or inference) and love working on problems like these, our team is hiring! ML Systems Engineer, Frameworks & Tooling: https://j
arXiv:2604.19278v1 Announce Type: new Abstract: LLM-based multi-agent systems (MAS) show promise on complex tasks but remain prone to coordination failures such as goal drift, error cascades, and misa
Google LLC introduced two new custom silicon chips for artificial intelligence today at Google Cloud Next 2026, unveiling two distinct Tensor Processor Unit architectures built for training and infere
arXiv:2604.16462v1 Announce Type: new Abstract: High-resolution Multimodal Large Language Models (MLLMs) face prohibitive computational costs during inference due to the explosion of visual tokens. Ex
arXiv:2604.18319v1 Announce Type: cross Abstract: Selection bias arises when the probability that an observation enters a dataset depends on variables related to the quantities of interest, leading to
We are excited to have @FireworksAI_HQ as a day 0 launch partner for Kimi K2.6! Their inference and fine-tuning platform is fast, reliable, and scales well under real production load, making it easy t
arXiv:2604.15750v1 Announce Type: cross Abstract: Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive language generation due to their potential for parallel dec
arXiv:2604.15214v1 Announce Type: cross Abstract: Quantum kernel methods are among the leading candidates for achieving quantum advantage in supervised learning. A key bottleneck is the cost of infere
arXiv:2512.23748v2 Announce Type: replace Abstract: For complex simulation problems, inferring parameters often precludes the use of classical likelihood-based techniques due to intractable likelihood
arXiv:2603.28942v3 Announce Type: replace Abstract: The pervasive deployment of deep learning models across critical domains has concurrently intensified privacy concerns due to their inherent propens
arXiv:2604.13556v1 Announce Type: new Abstract: Cross-layer key-value (KV) compression has been found to be effective in efficient inference of large language models (LLMs). Although they reduce the m
arXiv:2604.12634v1 Announce Type: new Abstract: Large language models (LLMs) face a fundamental trade-off between computational efficiency (e.g., number of parameters) and output quality, especially w
arXiv:2502.18026v3 Announce Type: replace-cross Abstract: Retrieving targeted pathways in biological knowledge bases, particularly when incorporating wet-lab experimental data, remains a challenging t
arXiv:2604.10152v1 Announce Type: new Abstract: The Mixture-of-Experts (MoE) architecture has emerged as a promising approach to mitigate the rising computational costs of large language models (LLMs)
arXiv:2604.11465v1 Announce Type: new Abstract: Large language model (LLM) agents show promise on realistic tool-use tasks, but deploying capable agents on modest hardware remains challenging. We stud
arXiv:2603.27960v2 Announce Type: replace-cross Abstract: Although Large Vision Language Models (LVLMs) have demonstrated impressive multimodal reasoning capabilities, their scalability and deployment
arXiv:2502.02345v2 Announce Type: replace Abstract: Subspace inference for neural networks assumes that a subspace of their parameter space suffices to produce a reliable uncertainty quantification. I
The ASUS UGen300 is a plug-and-play USB AI accelerator featuring a Hailo-10H processor that delivers up to 40 TOPS (INT4) of inference performance and 8GB of dedicated LPDDR4 memory, consuming just...
arXiv:2604.07815v1 Announce Type: new Abstract: Long-context inference in LLMs faces the dual challenges of quadratic attention complexity and prohibitive KV cache memory. While token-level sparse att
arXiv:2604.06036v3 Announce Type: replace-cross Abstract: Video streaming analytics is a crucial workload for vision-language model serving, but the high cost of multimodal inference limits scalabilit
arXiv:2401.00870v5 Announce Type: replace-cross Abstract: State-of-the-art large language models (LLMs) are typically deployed as online services, requiring users to transmit detailed prompts to cloud
arXiv:2604.04344v2 Announce Type: replace Abstract: We establish a computation-substrate-agnostic inference architecture in which domain is an explicit first-class computational parameter. This produc
arXiv:2604.06779v1 Announce Type: new Abstract: We introduce Fleming-Viot Diffusion (FVD), an inference-time alignment method that resolves the diversity collapse commonly observed in Sequential Monte
arXiv:2604.06485v1 Announce Type: cross Abstract: 'Best-of-N' selection is a popular inference-time scaling method for code generation using Large Language Models (LLMs). However, to reliably identify
arXiv:2604.06796v1 Announce Type: cross Abstract: Latent variable models, including variational autoencoders (VAE), remain a central tool in modern deep generative modeling due to their scalability an
arXiv:2507.22025v4 Announce Type: replace Abstract: The emergence of Multimodal Large Language Models (MLLMs) has driven significant advances in Graphical User Interface (GUI) agent capabilities. Neve
Introducing GLM-5.1 from @Zai_org on Together AI. AI natives can now use GLM-5.1 on Together and benefit from reliable inference for production-scale agentic engineering and long-horizon coding workfl