Search: “inference”

arXiv:2609.01345v1 Announce Type: new Abstract: Inference cascades cut cost by answering most queries with a cheap model and escalating a hard tail to a frontier model that acts as verifier. A natural

arXiv:2609.01338v1 Announce Type: cross Abstract: On-device mobile Large Language Model (LLM) inference is gaining significant attention. However, mobile devices operate in highly dynamic multitasking

arXiv:2608.29988v1 Announce Type: new Abstract: Large language models (LLMs) achieve strong reasoning performance, which depends critically on inference-time decisions. Yet these decisions are commonl

arXiv:2608.29264v1 Announce Type: new Abstract: Diffusion-based visual generative models deliver strong image and video synthesis quality but incur high inference costs because sequential samplers rep

arXiv:2603.24595v2 Announce Type: replace-cross Abstract: Large language model (LLM) inference systems rely on CUDA kernels for core GPU computations, yet the interface between models and kernels is i

arXiv:2608.28875v1 Announce Type: cross Abstract: Supplying context at inference time to a large multimodal model is an inexpensive lever for adapting speech transcription to a domain, and earlier res

arXiv:2608.27038v1 Announce Type: new Abstract: Although batch prompting makes large language model inference more efficient by processing multiple instances simultaneously, it suffers from unpredicta

Mahadev Konar co‑hosted a 2‑hour live broadcast titled “Amazing deep dive on K3!!” with Together Compute. The session explored scaling the Kimi K3 language model for inference, detailing deployment st

arXiv:2608.19758v1 Announce Type: new Abstract: Long-context modeling is a pivotal capability for Large Language Models, yet the quadratic complexity of attention remains a critical bottleneck, partic

arXiv:2608.18597v1 Announce Type: new Abstract: Protein language models are widely used priors for protein sequence design, and a growing body of work controls them at inference time as an alternative

arXiv:2608.17981v1 Announce Type: new Abstract: We describe an inference-time architectural enhancement for off-the-shelf foundation models that markedly reduces perplexity and boosts accuracy across

IIRC, this is the 3rd month in a row that @FireworksAI_HQ is being featured on @tryramp's top software vendor list. This isn't just for inference / model serving (of which we serve 40T+ tokens a day),

arXiv:2608.14490v1 Announce Type: new Abstract: We present a Test-time World-model Inference (Twin) system, in which a frontier coding agent writes an executable world model for completing continual l

arXiv:2608.10362v1 Announce Type: cross Abstract: Speculative decoding accelerates autoregressive large language model (LLM) inference by using a lightweight draft model to speculate multiple tokens,

arXiv:2607.17188v2 Announce Type: replace Abstract: While test-time scaling improves the problem-solving ability of large reasoning models (LRMs) through additional inference-time computation, it can

Back after ~6 months — rebuilding my spiking language model around CPU-first inference Hey everyone. It’s been around six months since I last posted anything about this project here. Some of you might

I’m planning a dedicated home AI server, mainly for local LLM inference, agents/tool use, Docker services, and eventually larger MoE models with CPU offload. My plan is to start with 2x 16GB GPUs = 32

arXiv:2608.03046v1 Announce Type: new Abstract: Text-to-video (T2V) diffusion transformers (DiTs) are trained with detailed video captions, whereas inference often relies on user prompts rewritten by

arXiv:2608.03682v1 Announce Type: new Abstract: Physical AI policies require inference throughout their lifecycle, including model evaluation, cloud reinforcement learning rollout, edge GPU serving, a

arXiv:2608.01979v1 Announce Type: cross Abstract: Visual token pruning reduces the inference cost of multimodal large language models, but a fixed token ratio is poorly matched to text-rich inputs. In

arXiv:2608.01034v1 Announce Type: new Abstract: We introduce Opt.Gear, a foundation model designed for efficient on-device deployment, real-tim inference, and strong task capability. It includes a den

Built my 1st inference machine and have been tweaking models trying to get the most out of my modest hardware. I think I’m at a good place but I’m testing with my own prompts. I’ve looked into some of

arXiv:2607.27574v1 Announce Type: new Abstract: Activation steering has emerged in large language models as a lightweight alternative for dynamically changing a model's behavior at inference time. How

At Google, AI is a soup-to-nuts endeavor. Obviously, we make leading AI models like Gemini and Nano Banana. We incorporate AI into the tools you use every day (think Gmail, BigQuery, AlloyDB, Google C

arXiv:2607.23362v1 Announce Type: new Abstract: Recent work has shown that subword vocabularies can be trained to optimize compression for a specific inference rule rather than relying on greedy heuri

arXiv:2607.22586v1 Announce Type: new Abstract: Key-Value (KV) caching is essential for efficient inference in multimodal large language models (MLLMs), yet its memory footprint grows linearly with co

arXiv:2603.01119v2 Announce Type: replace-cross Abstract: A fundamental challenge in causal inference with observational data is correct specification of a causal model. When there is model uncertaint

b9908 is a build-tagged release from llama.cpp , the open-source C/C++ inference engine for large language models. llama.cpp is an open-source software library that performs inference on various large

arXiv:2607.06216v1 Announce Type: new Abstract: The future of World Models depends not only on scaling model capability, but also on scaling practicality and inference efficiency. High-frame-rate infe

This entry covers the integration of vLLM, a high-performance inference engine, with Hugging Face's transformers library to enable faster language model inference. The native-speed backend allows user

arXiv:2607.05252v1 Announce Type: new Abstract: Simulation-Based Inference (SBI) is critical for scientific discovery, with generative models offering a promising path toward efficient inference. Howe

arXiv:2606.28600v1 Announce Type: cross Abstract: Neuromorphic and edge computing research has focused on reducing the inference cost of neural network controllers, yet in physical closed-loop systems

arXiv:2606.26922v1 Announce Type: cross Abstract: Continuous driver monitoring in automated vehicles requires low-latency inference while avoiding unsafe decisions under uncertain driver states. Large

arXiv:2606.21238v1 Announce Type: cross Abstract: Key-value (KV) caching is a powerful technique for accelerating large language model inference and generation. Inference workloads are large and diver

B9542 is a build/release version of llama.cpp, the C/C++ implementation of large language model inference designed to enable LLM inference with minimal setup and state-of-the-art performance on variou

arXiv:2606.01269v1 Announce Type: new Abstract: Transitive inference is the challenge of inferring that A < C from knowing only adjacent relations (A < B, B < C). It is solved by humans and animals no

arXiv:2604.01802v2 Announce Type: replace Abstract: Real-time inference of inaccessible interior physical fields from sparse boundary observations is a fundamental but unresolved problem in scientific

arXiv:2603.26506v2 Announce Type: replace-cross Abstract: Connectivity structure shapes neural computation, but inferring this structure from population recordings is degenerate: multiple connectivity

arXiv:2601.03471v3 Announce Type: replace-cross Abstract: Reliable epidemiological reasoning requires synthesizing study evidence to infer disease burden, transmission dynamics, and intervention effec

arXiv:2601.01608v2 Announce Type: replace Abstract: Diffusion models deliver high quality in image synthesis but remain expensive during training and inference. Recent works have leveraged the inheren

arXiv:2605.26584v1 Announce Type: new Abstract: Omnimodal large language models enable unified audio video understanding, but long joint token sequences make inference costly, and existing benchmarks

arXiv:2605.26059v1 Announce Type: cross Abstract: Bayesian inverse design provides a principled framework for inferring aerodynamic geometries from sparse flow observations while quantifying uncertain

arXiv:2509.19929v4 Announce Type: replace-cross Abstract: Uncertainty Quantification (UQ) is paramount for inference in engineering. A common inference task is to recover full-field information of phy

B8984 is a build/release version of llama.cpp, an open-source C/C++ library for large language model inference. Llama.cpp enables LLM inference with minimal setup and state-of-the-art performance on a

arXiv:2604.24938v1 Announce Type: cross Abstract: Depth pruning improves the inference efficiency of large language models by removing Transformer blocks. Prior work has focused on importance criteria

Production low-latency autocomplete implementations employ diverse strategies including inference server optimization (tools like vLLM, llama.cpp, NVIDIA Triton), deployment choices (cloud APIs, on-pr