Search: “inference”

arXiv:2608.14385v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models have been widely adopted in real-time interactive applications such as coding assistants, real-time audio-video intera

arXiv:2608.14205v1 Announce Type: new Abstract: Load imbalance poses a major bottleneck to the efficiency of expert parallelism in distributed inference of Mixture-of-Experts (MoE) models. The most he

arXiv:2608.12385v1 Announce Type: new Abstract: As large language models serve more requests, cumulative inference cost is becoming increasingly important relative to one-time training cost. The two i

Running large language model inference at scale forces a KV cache trade-off: oversized GPU instances or slow time-to-first-token. This post builds a tiered KV cache on Amazon SageMaker HyperPod that e

The past week I've been running DSv4 inference on my laptop by keeping everything RAM-resident except the MXFP4-experts (since expert pool is ~147GB and won't fit) TL;DR - read speed is the limiter mo

Modern Large Language Models achieve impressive reasoning capabilities with long Chain of Thoughts, but they incur substantial computational cost during inference, and this motivates techniques to imp

arXiv:2608.05124v1 Announce Type: cross Abstract: Long context reasoning in large language models (LLMs) is usually constrained by the fact that a single inference trajectory has to simultaneously exp

The Amazon SageMaker Python SDK v3 now exposes generative AI inference recommendations in Amazon SageMaker AI directly in your notebook. Benchmark an endpoint, generate data-driven deployment recommen

I'm hoping that one of you guys has been working on an inference engine or has somehow found improvements to running DSV4F on RDNA4 multi-GPU setups. I am currently building a custom inference engine

arXiv:2608.00311v1 Announce Type: new Abstract: Long-context inference with large language models (LLMs) is costly: self-attention during prefill scales quadratically with sequence length, and the key

arXiv:2608.01880v1 Announce Type: new Abstract: World Action Models generate fixed-horizon action chunks through iterative denoising, creating substantial inference latency that can cause pauses, stal

arXiv:2607.27766v1 Announce Type: new Abstract: On-device in-context learning (ICL) relies on pre-inference retrieval to select demonstrations for useful context before downstream model inference. Thi

This post was not written by a clanker. Hey guys, I'm a comp sci major who wanted to introduce a cool project I built for quantizing models to ternary (1.58 bit) with as minimal of loss as possible, a

arXiv:2607.26410v1 Announce Type: new Abstract: We present Voice Memory, a inference-only scheme for agentic speech recognition: at stream time, a frozen corrector reads a single per-domain memory.md

The root cause: request-level engines never see that a series of LLM calls belongs to one longer workflow. ThunderAgent adds that missing view. It treats each agent workflow as a schedulable program,

arXiv:2607.22568v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly deployed on mobile and embedded devices to improve privacy and reduce network latency. Yet on-device infer

arXiv:2607.23335v1 Announce Type: new Abstract: Auxiliary signal pathways in VLMs are routinely fitted with learnable gates so the optimiser can decide how much of the signal to admit. We find that th

arXiv:2607.22352v1 Announce Type: new Abstract: We introduce time-reversed imaging, a new paradigm that infers what just happened in a scene from fading multimodal traces. Instead of extrapolating or

Cuda and Vulkan Benchmark: TensorSharp vs. llama.cpp I would like to share my latest open source local Unsloth (GGUF) LLM inference engine and applications. It supports many models from Unsloth, like

common: infer the speculative type from the draft repo sidecars (#25989) With -hfd pointing to a repo that ships mtp-/dflash-/eagle3- sidecars and no --spec-type given, the draft resolved to a full mo

arXiv:2607.19515v1 Announce Type: cross Abstract: Continuous surveillance video creates a growing storage, transmission, and inference burden for enterprise video analytics systems. While modern codec

arXiv:2607.19358v1 Announce Type: new Abstract: Recent advances in long chain-of-thought reasoning models such as DeepSeek-R1 have led to increasingly longer inference context lengths under the test-t

arXiv:2607.20357v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) have recently demonstrated strong performance across vision-language tasks. However, their high inference cost,

arXiv:2607.18544v1 Announce Type: new Abstract: Machine olfaction for gas unmixing is an underconstrained inverse problem in which gas compositions must be inferred from low-dimensional, delayed, and

arXiv:2607.13960v1 Announce Type: new Abstract: World Action Models (WAMs) improve robot policy learning by jointly modeling actions and future visual observations, using future scene evolution as den

arXiv:2607.12550v1 Announce Type: cross Abstract: The key-value (KV) cache has become the dominant memory cost of transformer inference. It grows with batch size, context length, and depth, and at lon

How should your security team manage shadow AI? Workloads deployed by developers without formal registration can often evade traditional security scanners, because organizations are reluctant to slow

this is huge: best news I've heard all day..nice work, Hugging Face team. Big unlock for open-source AI inference: Hugging Face Transformers models can now run in vLLM at native speed, often matching

B9932 is a continuous build-tagged release from the llama.cpp project , an open-source C/C++ inference engine for running large language models locally. llama.cpp performs inference on various large l

arXiv:1811.05336v2 Announce Type: replace-cross Abstract: Inference for factor models is often hampered by the lack of tractable and accurate variance estimates, which can materially distort downstrea

arXiv:2607.03509v1 Announce Type: new Abstract: Recent progress in large-scale generative models has substantially advanced video generation, yet existing methods remain constrained by a rigid inferen

arXiv:2607.02612v1 Announce Type: cross Abstract: Vision Transformers achieve strong image classification accuracy but process all image regions with nearly the same computation, even when many region

arXiv:2508.05049v1 Announce Type: cross Abstract: AI-powered medical devices have driven the need for real-time, on-device inference such as biomedical image classification. Deployment of deep learnin

b9891 is a build release of llama.cpp, the open-source C/C++ inference engine for running large language models locally. This release enables LLM inference with minimal setup and state-of-the-art perf

llama.cpp b9876 is a build-tagged release from the llama.cpp project , a pure C/C++ implementation of large language model inference . The project enables LLM inference with minimal setup and state-of

B9862 is a continuous build-tagged release from llama.cpp , the open-source C/C++ LLM inference engine. Llama.cpp is the inference engine that powers most of the local-AI ecosystem, including tools li

arXiv:2512.07843v2 Announce Type: replace-cross Abstract: Scaling inference-time computation has enabled Large Language Models (LLMs) to achieve strong reasoning performance, but their inherently sequ

arXiv:2606.30850v1 Announce Type: new Abstract: Large language models (LLMs) are typically deployed in multi-turn conversations, where each turn provides new evidence that should reduce epistemic unce

arXiv:2606.31063v1 Announce Type: cross Abstract: Gaussian process inference is often limited by cubic computational costs, a challenge that becomes more pronounced in spatio-temporal settings where p

arXiv:2506.22075v2 Announce Type: replace Abstract: Reasoning is a hallmark of human intelligence, enabling adaptive decision-making in complex unfamiliar scenarios. In contrast, machine intelligence

B9787 is a build release of llama.cpp, an open-source C/C++ implementation for LLM inference created by Georgi Gerganov. The llama.cpp project enables large language model inference with minimal setup

arXiv:2606.21501v1 Announce Type: new Abstract: Occluded tasks remain a bottleneck in robot manipulation. Existing solutions either deploy additional physical cameras requiring training-inference came

arXiv:2606.11576v1 Announce Type: cross Abstract: Modern Vision-Language Models (VLMs) benefit from chain-of-thought prompting and test-time scaling, but these gains often come with prohibitive infere

arXiv:2606.10357v1 Announce Type: cross Abstract: Cross-domain recommendation is a core problem in content-to-e-commerce platforms. Its objective is to leverage user interactions with content to infer

b9573 is a build release of llama.cpp, an open-source C/C++ project for LLM inference that aims to enable language model inference with minimal setup and state-of-the-art performance on various hardwa

arXiv:2606.07835v1 Announce Type: new Abstract: A fundamental tension exists in the large-step inference of diffusion models via their deterministic probability flow ordinary differential equation (PF