Knowledge catalogue
TorchServe is no longer maintained, leaving teams to own the entire GPU inference stack. The AWS Ray Serve Deep Learning Container is a supported, pre-tested container with the framework, GPU drivers,
While GPUs remain essential for LLM training and inference, enterprise‑scale agentic AI workloads shift the focus to orchestration rather than raw compute. Agents must break down requests into subtask
arXiv:2609.07833v1 Announce Type: cross Abstract: Weak gravitational lensing shear and convergence trace the distribution of baryonic and dark matter across space, making them a powerful probe of cosm
Encode‑Prefill‑Decode (EPD) disaggregation is an inference optimization in NVIDIA Dynamo that separates vision encoding from LLM prefill–decode stages for multimodal requests. On image‑heavy prompts,
arXiv:2608.10601v1 Announce Type: cross Abstract: Two instruments of EU digital law place inference at their centre and mean different things by it. Article 3(1) of the AI Act uses the capability to i
arXiv:2608.10439v1 Announce Type: new Abstract: Streaming video generation holds strong potential for world modeling, where future frames must be inferred online sequentially to form a continuous vide
arXiv:2608.10489v1 Announce Type: new Abstract: Abundant visual information strengthens vision-language model (VLM) perception, yet massive visual tokens raise inference costs. Existing visual token p
DigitalOcean launched Kimi K3 on day 0. It’s already one of the most popular models on the platform and across the market: second most likes on Hugging Face, sixth most traffic on OpenCode. Getting a
Automated lint: 20 errors, 8743 warnings, 3 info
arXiv:2607.12733v1 Announce Type: new Abstract: Large language models (LLMs) excel at pattern recognition and text generation, but their capacity for abductive inference - inferring latent hypotheses
Running frontier AI in production is demanding — accelerators to manage, latency to hold steady across continents, regulated data to keep in-region, and long-context requests to serve reliably. Claude
arXiv:2607.00170v1 Announce Type: cross Abstract: Thermodynamic computing devices based on the Ising model show great promise for low-power AI inference and edge computing, but scalable methods for tr
Protecting sensitive data used with AI is a critical part of our commitment to providing advanced and secure cloud infrastructure. Confidential Computing cryptographically protects data in use in hard
arXiv:2509.08318v2 Announce Type: replace Abstract: Early-exit cascades over a frozen convolutional backbone enable adaptive inference but suffer from three sources of train-inference mismatch: branch
arXiv:2604.23368v1 Announce Type: new Abstract: Event-Based Models (EBMs) infer biomarker progression from cross-sectional data but typically only as ordinal sequences and rely on rigid model assumpti
arXiv:2604.22050v1 Announce Type: cross Abstract: Transformers are mostly relying on softmax attention, which introduces quadratic complexity with respect to sequence length and remains a major bottle
arXiv:2509.22343v2 Announce Type: replace-cross Abstract: Reasoning capability is essential to ensure the factual correctness of the responses of transformer-based Large Language Models (LLMs), and ro
AI is evolving from answering questions to reasoning and taking action. Companies who want to lead in today’s agentic era require computing infrastructure designed and optimized for these new requirem
arXiv:2502.07415v2 Announce Type: replace-cross Abstract: Inferring the mechanical properties of soft tissues from measured deformations is a fundamental challenge in elastography. A rarely examined a
arXiv:2610.18014v1 Announce Type: cross Abstract: Non-prefix KV cache reuse is a promising approach for improving inference efficiency in multi-agent systems as agents share the same context blocks. H
arXiv:2610.16615v1 Announce Type: new Abstract: Visual token compression has emerged as a critical technique for reducing the inference cost of Vision-Language Models. Existing methods select tokens b
arXiv:2610.16293v1 Announce Type: new Abstract: Garment manipulation requires inferring both garment identity and complete configuration from partial, self-occluded observations. This creates a chicke
arXiv:2610.16448v1 Announce Type: new Abstract: Diffusion models, particularly Diffusion Transformers, achieve strong image and video generation quality but remain expensive at inference time due to r
I started Flama in 2018 to take a trained model to a production API with as little ceremony as possible. With 2.0 LLMs became first-class alongside classic models, and together with Miguel Durán-Olive
arXiv:2610.12741v1 Announce Type: new Abstract: We interpret Transformer networks as learned approximations to a hybrid probabilistic-inference procedure that combines max-product Loopy Belief Propaga
arXiv:2610.12582v1 Announce Type: new Abstract: A latent recurrent language model applies a shared block of layers repeatedly, so one checkpoint can trade inference compute for accuracy by changing th
arXiv:2610.07457v1 Announce Type: cross Abstract: Fine-grained mixed-precision quantization promises efficient large language model inference, but local precision choices can conflict with regular GPU
arXiv:2610.04938v1 Announce Type: new Abstract: We propose D-DOIT (Discrete Doob-Oriented Inference-time Transformation), a training-free and efficient adaptation method for discrete diffusion models
arXiv:2610.04396v1 Announce Type: cross Abstract: Split Computing (SC) enables efficient deployment of Deep Neural Networks (DNNs) by partitioning inference between edge devices and cloud servers. How
arXiv:2610.04784v1 Announce Type: cross Abstract: Traffic classification (TC) is crucial to secure Internet of Things (IoT) networks, whose edge nodes often operate under privacy, bandwidth, and energ
arXiv:2610.01847v1 Announce Type: cross Abstract: Model specifications define how large language models (LLMs) should behave, guiding alignment training, inference-time behavior, and evaluation. Yet t
arXiv:2610.00940v1 Announce Type: cross Abstract: We propose ReHoPER, an inference-only, zero-shot method that improves large language models' reasoning by generating and answering intermediate questi
arXiv:2610.00395v1 Announce Type: new Abstract: Inference-time steering enables pretrained diffusion models to satisfy new constraints without full retraining. However, specificity-aware generation is
arXiv:2609.38386v1 Announce Type: new Abstract: Concurrent autoregressive inference creates a fundamental interference problem: prefilling a newly arrived long prompt can delay tokens for requests tha
arXiv:2607.15870v2 Announce Type: replace Abstract: Human label variation in natural language inference is increasingly treated as signal rather than noise, but how much of it formal semantic structur
arXiv:2609.38984v1 Announce Type: cross Abstract: World-action models (WAMs) leverage pretrained video models to improve generalization in robot control by jointly predicting future visual states and
arXiv:2609.37664v1 Announce Type: new Abstract: Causal Normalizing Flows (CNFs) enable causal inference from observational data given the causal structure, but they assume fully observed training data
I think most people are sleeping on this inference engine. I tried multiple llama.cpp forks and none of them comes close to the inference speed of Strata. Initial version had some bugs with kv cache,
arXiv:2609.36636v1 Announce Type: cross Abstract: Looped language models (LoopLMs) increase computational depth through parameter sharing, offering a path to scale inference computation without adding
arXiv:2609.22866v1 Announce Type: new Abstract: We introduce Causilo, a tabular foundation model (TFM) that combines frontier predictive performance with exceptionally fast inference. On TabArena, Cau
arXiv:2609.22204v1 Announce Type: new Abstract: This exploratory pilot study evaluates the scope and perceived accuracy of personal information output from ongoing conversational interactions in gener
NVIDIA TensorRT’s multi‑device inference lets a single TensorRT network span multiple GPUs using NCCL‑backed collectives, preserving all standard optimizations and supported from TensorRT 11.0 onward.
arXiv:2609.18663v1 Announce Type: cross Abstract: Billion-parameter vision--language--action (VLA) policies demand substantial onboard power, while communication delays in remote inference hinder time
arXiv:2609.17556v1 Announce Type: cross Abstract: Edge-deployed AI operate under dynamically changing power budgets, reliability requirements, and input distributions, requiring continuous adaptation.
arXiv:2609.12322v1 Announce Type: cross Abstract: Affective computing has advanced wearable state inference, but on-device reasoning about whether, when, and how to intervene remains challenging. We p
arXiv:2609.10712v1 Announce Type: new Abstract: We study how model post-training and test-time inference design affect natural-language proof generation for hard olympiad mathematics. Starting from Ne
arXiv:2609.10627v1 Announce Type: cross Abstract: Machine learning explanations reveal model behavior beyond predictions, creating attack surfaces for model confidentiality and data privacy. We system
arXiv:2609.09721v1 Announce Type: new Abstract: Low-bit attention accelerates Transformer inference by moving the QK^op and PV matrix multiplications to FP8 or FP4 matrix engines. However, the softmax
I initially saw CED as just an efficiency improvement, but the more I read about it, the more it feels like an inference architecture leap. The encoder/decoder split has some pretty interesting implic
arXiv:2609.01925v1 Announce Type: cross Abstract: The attention prefilling phase of long-context LLM inference scales quadratically, making self-attention a severe computational bottleneck. Traditiona
arXiv:2609.01345v1 Announce Type: new Abstract: Inference cascades cut cost by answering most queries with a cheap model and escalating a hard tail to a frontier model that acts as verifier. A natural
arXiv:2609.01338v1 Announce Type: cross Abstract: On-device mobile Large Language Model (LLM) inference is gaining significant attention. However, mobile devices operate in highly dynamic multitasking
arXiv:2608.29988v1 Announce Type: new Abstract: Large language models (LLMs) achieve strong reasoning performance, which depends critically on inference-time decisions. Yet these decisions are commonl
arXiv:2608.29264v1 Announce Type: new Abstract: Diffusion-based visual generative models deliver strong image and video synthesis quality but incur high inference costs because sequential samplers rep
arXiv:2603.24595v2 Announce Type: replace-cross Abstract: Large language model (LLM) inference systems rely on CUDA kernels for core GPU computations, yet the interface between models and kernels is i
arXiv:2608.28875v1 Announce Type: cross Abstract: Supplying context at inference time to a large multimodal model is an inexpensive lever for adapting speech transcription to a domain, and earlier res
arXiv:2608.27038v1 Announce Type: new Abstract: Although batch prompting makes large language model inference more efficient by processing multiple instances simultaneously, it suffers from unpredicta
Mahadev Konar co‑hosted a 2‑hour live broadcast titled “Amazing deep dive on K3!!” with Together Compute. The session explored scaling the Kimi K3 language model for inference, detailing deployment st
arXiv:2608.19758v1 Announce Type: new Abstract: Long-context modeling is a pivotal capability for Large Language Models, yet the quadratic complexity of attention remains a critical bottleneck, partic
arXiv:2608.18597v1 Announce Type: new Abstract: Protein language models are widely used priors for protein sequence design, and a growing body of work controls them at inference time as an alternative