Search: “inference”

AI companies like @cohere build on both sides of the token economy: the models and the products that use them. By moving production workloads to NVIDIA Blackwell, Cohere lowered token costs, increased

arXiv:2609.07264v1 Announce Type: new Abstract: Inference energy per token drives the cost and carbon footprint of deployed transformers. It is dominated by dense matrix products that incur fused mult

A recap of August 2026 launches for AI builders across Amazon Bedrock, Amazon Bedrock AgentCore, and Strands: million-token context for OpenAI models, cross-Region inference, agents that run for up to

Lightbits Labs Ltd. today announced the general availability of Inferra, a software engine designed to improve the economics and performance of artificial intelligence inference by moving key-value ca

arXiv:2609.05320v1 Announce Type: new Abstract: Deepfake detection models often rely on high-quality inputs, fixed inference paths, and computationally expensive architectures, limiting their use in l

arXiv:2609.03502v1 Announce Type: cross Abstract: In low-resource settings, deploying TTS typically requires choosing between a large voice-cloning model with costly inference or a compact fixed-voice

arXiv:2609.03003v1 Announce Type: new Abstract: Causal inference is the practice of estimating the effect of a treatment or intervention from data. It traditionally requires a bespoke pipeline for eve

arXiv:2609.03807v1 Announce Type: cross Abstract: A free pause token gives a language model extra compute to form each next-token prediction (as a pause, or thinking, token does) but carries that comp

arXiv:2609.03343v1 Announce Type: cross Abstract: This paper addresses infinite-dimensional Bayesian inference for inverse problem of partial differential equations with model parameters in infinite-d

arXiv:2609.03158v1 Announce Type: cross Abstract: Visual token pruning reduces the inference cost of vision-language models (VLMs), but most methods only ask which tokens to keep. This retained-token

arXiv:2609.01778v1 Announce Type: new Abstract: Large-scale video retrieval requires embedding models to encode long and diverse videos under tight visual-input and inference budgets. Existing methods

arXiv:2609.01873v1 Announce Type: new Abstract: Multi-agent AI systems improve inference by spawning agents and synthesizing reports. But another agent is not another observation: apparently independe

arXiv:2609.01683v1 Announce Type: new Abstract: Vision models deployed on microcontrollers (MCUs) are quantized to integer-only arithmetic and run in inference-only runtimes that do not carry the mach

arXiv:2609.02689v1 Announce Type: new Abstract: Embedded devices typically lack the resources of GPU-equipped machines, and existing inference methods suffer from either high computational overhead (p

arXiv:2511.05187v2 Announce Type: replace Abstract: We study whether otherwise-idle inference resources could reduce the scarce-GPU cost of training. Our analysis uses a simulated compute ledger in wh

arXiv:2609.02029v1 Announce Type: new Abstract: Long-context inference retains a growing key--value (KV) cache during decoding, which consumes substantial GPU memory and can reduce generation throughp

arXiv:2609.01315v1 Announce Type: new Abstract: Building an omni-modal foundation model means evaluating it across text, image, video, and audio. Excellent evaluation toolkits exist for each modality,

arXiv:2609.00367v1 Announce Type: cross Abstract: Large Language Models are increasingly deployed for sophisticated data engineering tasks such as generating structured queries from natural language,

arXiv:2609.01224v1 Announce Type: new Abstract: Visual token pruning reduces the inference overhead of multimodal large language models (MLLMs) by retaining only a subset of visual tokens. Existing me

arXiv:2609.01004v1 Announce Type: cross Abstract: Despite their strong multimodal understanding ability, multimodal large language models (MLLMs) incur substantial computational overhead when processi

arXiv:2609.00057v1 Announce Type: cross Abstract: Value signals are aggregated user-level moral representations that capture users' inferred value-related tendencies from their online discourse. User

arXiv:2608.30263v1 Announce Type: cross Abstract: Large vision-language models (LVLMs) incur substantial inference costs due to their long and highly redundant visual-token sequences. Diversity-based

arXiv:2608.28696v1 Announce Type: new Abstract: Visual in-context learning (ICL) with multimodal large language models (MLLMs) is effective for fine-grained visual classification, but each retrieved i

Historically, enterprise predictive analytics tasks such as predicting churn, purchase intent, or fraud scoring have meant building custom models using libraries like XGBoost, Random Forest, or Deep N

arXiv:2608.29904v1 Announce Type: new Abstract: Modern video generators routinely fail at physical dynamics: objects float, trajectories violate gravity, contacts vanish. Standard denoising and flow-m

arXiv:2608.29107v1 Announce Type: new Abstract: While modern generative models excel at modeling complex data, precise inference-time control in conditional generation remains a critical challenge. Cl

arXiv:2608.21719v1 Announce Type: cross Abstract: AI inference clusters are increasingly constrained by instantaneous power, not just energy: grid operators condition new capacity on demand response,

arXiv:2608.29843v1 Announce Type: cross Abstract: Posted prices for AI inference have fallen steadily since 2024, yet the measured speed of that fall depends almost entirely on the method of measureme

arXiv:2206.09012v4 Announce Type: replace-cross Abstract: We consider the problem of inferring high-dimensional data mathbf{x} in a model that consists of a prior p(mathbf{x}) and an auxiliary differe

→ rollout throughput: generation speed and GPU utilization → weight sync: fresh weights on the rollout deployment every step → correctness: trainer and inference engine agreeing on the same model Cofo

I absolutely love Ollama for the majority of my inference needs. Starting to roll out Rehex to friends, and I've gotten requests to support OpenCode Go and Codex. Frankly, I'm really excited to add th

People who moved from older version to this new one, how much improvements do you see on inference? Ex: llama.cpp performance? This new version comes with Linux Kernel 7.0. Hope AMD cards gonna enjoy

Hey! I forked NInfer (a from-scratch C++20/CUDA inference engine for Qwen models) and added two things: tensor-parallel across two GPUs, and YaRN ×4 rope scaling. Together they let Qwen3.8-27B NVFP4 r

arXiv:2608.26926v1 Announce Type: new Abstract: Small language models (sLLMs) are nowadays hosted on devices with limited memory and computational budget. In an autoregressive setup, inference is memo

As in actual physical heat. How are people coping with the heat that running a decent inference rig pumps out? I've got dual 5060 Ti GPUs and a relatively modest CPU (Intel 14 Core Ultra 5 245KF Deskt

arXiv:2608.25727v1 Announce Type: new Abstract: Large language models (LLMs) have recently advanced graph neural networks (GNNs) by enriching node representations with semantic information, giving ris

arXiv:2509.06346v3 Announce Type: replace Abstract: Sparse Mixture-of-Experts (MoE) has become a key architecture for scaling large language models (LLMs) efficiently. Recent fine-grained MoE designs

arXiv:2608.24904v1 Announce Type: new Abstract: Inertial sensors at multiple body locations can improve activity recognition, but requiring every sensor at inference increases the deployment burden. W

arXiv:2608.24905v1 Announce Type: cross Abstract: Service robots may encounter ambiguous user requests that require context-aware inference. Users may also have unique preferences with certain tasks w

arXiv:2608.23641v1 Announce Type: new Abstract: Model welfare research infers what a model prefers from the answers returned to prompts written to elicit preferences. Keeling et al. (2024), Mazeika et

Recent advancements in large language models have intensified the need for efficient and deployable models within limited inference budgets. Structured pruning pipelines have shown promise in token ef

arXiv:2608.23880v1 Announce Type: new Abstract: Inferring the collective emotional state of a group of people from a single image, a task known as group emotion recognition (GER), requires integrating

arXiv:2608.20975v1 Announce Type: new Abstract: Effective social interaction requires agents to translate mental state inferences into coordinated behavioral signals across verbal and nonverbal channe

If you’re trying to run local LLMs on a normal laptop, an older desktop, integrated graphics, limited VRAM, or simply the hardware you already own, r/LowEndLocalAI is meant for you. The idea is simple

arXiv:2508.15813v2 Announce Type: replace-cross Abstract: A big issue in modern LLM applications is they tend to feed long context to LLM, which results in high inference cost and latency, and may exc

AirLLM dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card — without quantization, distillation, or pruning. You can even run 405B Llama 3.1 on