Search: “inference”

When to Ask a Question: Understanding Communication Strategies in Generative AI Tools

arXiv:2605.11240v1 Announce Type: cross Abstract: Generative AI models differ from traditional machine learning tools in that they allow users to provide as much or as little information as they choos

safetyarxiv-cs-lg

13 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

💡 Why did @togethercompute choose NVIDIA Blackwell to serve DeepSeek-V4? Because NVIDIA Blackwell is built for the bottlenecks that matter …

DGX agent

💡 Why did @togethercompute choose NVIDIA Blackwell to serve DeepSeek-V4? Because NVIDIA Blackwell is built for the bottlenecks that matter most in long-context inference: → KV-cache pressure during de

model-releasestogether-ai--x

13 May 2026

b9124

B9124 is a build release from the llama.cpp project, created on May 12, 2026 . Llama.cpp is an open source software library that performs inference on various large language models such as Llama, deve

Belief or Circuitry? Causal Evidence for In-Context Graph Learning

arXiv:2605.08405v1 Announce Type: new Abstract: How do LLMs learn in-context? Is it by pattern-matching recent tokens, or by inferring latent structure? We probe this question using a toy graph random

local-aiarxiv-cs-ai

Breaking the Reward Barrier: Accelerating Tree-of-Thought Reasoning via Speculative Exploration

arXiv:2605.10195v1 Announce Type: new Abstract: Tree-of-Thought (ToT) reasoning structures Large Language Model (LLM) inference as a tree-based search, demonstrating strong potential for solving compl

researcharxiv-cs-lg

Compute Where it Counts: Self Optimizing Language Models

arXiv:2605.10875v1 Announce Type: cross Abstract: Efficient LLM inference research has largely focused on reducing the cost of each decoding step (e.g., using quantization, pruning, or sparse attentio

safetyarxiv-cs-cl

Deterministic Differentiable Structured Pruning for Large Language Models

arXiv:2603.08065v2 Announce Type: replace-cross Abstract: Structured pruning reduces LLM inference cost by removing low-importance architectural components. This can be viewed as learning a multiplica

researcharxiv-cs-cl

Agents

Enhancing Consistency Models for Multi-Agent Trajectory Prediction

arXiv:2605.08572v1 Announce Type: new Abstract: Diffusion models for multi-agent trajectory prediction are limited by iterative denoising, which causes inference latency that hinders their use in time

agentsarxiv-cs-cv

Extended Wasserstein-GAN Approach to Causal Distribution Learning: Density-Free Estimation and Minimax Optimality

arXiv:2605.10206v1 Announce Type: cross Abstract: Distributional causal inference requires estimating not only average treatment effects but also interventional outcome distributions, including quanti

safetyarxiv-cs-lg

FLARE: One-Shot PE-Level Fault Localization in Systolic Arrays via Algebraic Test Vectors

arXiv:2605.08594v1 Announce Type: cross Abstract: Systolic arrays are the dominant compute fabric for neural network inference. Prior work has addressed column-level fault detection efficiently with u

local-aiarxiv-cs-lg

Guided Streaming Stochastic Interpolant Policy

arXiv:2605.10051v1 Announce Type: cross Abstract: Inference-time guidance is essential for steering generative robot policies toward dynamic objectives without retraining, yet existing methods are lar

safetyarxiv-cs-ai

NoisyCoconut: Counterfactual Consensus via Latent Space Reasoning

arXiv:2605.08221v1 Announce Type: cross Abstract: This paper presents NoisyCoconut, a novel inference-time method that enhances large language model (LLM) reliability by manipulating internal represen

Hardware

Non-Monotonic Latency in Apple MPS Decoding: KV Cache Interactions and Execution Regimes

arXiv:2605.08913v1 Announce Type: cross Abstract: Autoregressive inference is typically assumed to scale predictably with decoding length, and key-value (KV) caching is widely regarded as a universall

hardwarearxiv-cs-cl

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Planner

arXiv:2506.01301v2 Announce Type: replace Abstract: Theory-of-Mind (ToM) enables humans to infer mental states-such as beliefs, desires, and intentions-forming the foundation of social cognition. Howe

PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decoding

arXiv:2605.08632v1 Announce Type: cross Abstract: Speculative decoding accelerates Large Language Models (LLMs) inference by using a lightweight draft model to propose candidate tokens that are verifi

PRIM: Meta-Learned Bayesian Root Cause Analysis

model-releasesarxiv-cs-lg

arXiv:2605.08786v1 Announce Type: new Abstract: Root cause analysis (RCA) in complex systems is challenging due to error propagation across multiple variables, the need for structural causal knowledge

RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache

arXiv:2605.08317v1 Announce Type: cross Abstract: Large language models (LLMs) have shown strong performance across diverse tasks, but their inference with long input contexts is bottlenecked by memor

Recursive Language Models

arXiv:2512.24601v3 Announce Type: replace Abstract: We study allowing large language models (LLMs) to process arbitrarily long prompts through the lens of inference-time scaling. We propose Recursive

RubricRefine: Improving Tool-Use Agent Reliability with Training-Free Pre-Execution Refinement

model-releasesarxiv-cs-lg

arXiv:2605.09730v1 Announce Type: new Abstract: Iterative self-refinement is a popular inference-time reliability technique, but its effectiveness in code-mode tool use depends heavily on the structur

SciVQR: A Multidisciplinary Multimodal Benchmark for Advanced Scientific Reasoning Evaluation

model-releasesarxiv-cs-cv

arXiv:2605.10187v1 Announce Type: new Abstract: Scientific reasoning is a key aspect of human intelligence, requiring the integration of multimodal inputs, domain expertise, and multi-step inference a

TacoMAS: Test-Time Co-Evolution of Topology and Capability in LLM-based Multi-Agent Systems

model-releasesarxiv-cs-cl

arXiv:2605.09539v1 Announce Type: new Abstract: Multi-agent systems (MAS) have emerged as a promising paradigm for solving complex tasks. Recent work has explored self-evolving MAS that automatically

Test-Time Speculation

model-releasesarxiv-cs-cl

arXiv:2605.09329v1 Announce Type: new Abstract: Speculative decoding accelerates LLM inference by using a fast draft model to generate tokens and a more accurate target model to verify them. Its perfo

Training with Harnesses: On-Policy Harness Self-Distillation for Complex Reasoning

arXiv:2605.08741v1 Announce Type: new Abstract: Inference-time harnesses substantially improve large language models on complex reasoning tasks. However, the intrinsic capabilities of the underlying m

safetyarxiv-cs-cl

Agents

When Independent Sampling Outperforms Agentic Reasoning

arXiv:2605.08478v1 Announce Type: new Abstract: We study how to allocate inference-time compute for competitive programming under fixed budgets. Evaluating 216 Codeforces problems across Divisions 1-3

agentsarxiv-cs-lg

WindINR: Latent-State INR for Fast Local Wind Query and Correction in Complex Terrain

arXiv:2605.09511v1 Announce Type: new Abstract: Many downstream decisions in complex terrain require fast wind estimates at a small number of user-specified locations and heights for a given forecast

AdaCorrection: Adaptive Offset Cache Correction for Accurate Diffusion Transformers

arXiv:2602.13357v2 Announce Type: replace-cross Abstract: Diffusion Transformers (DiTs) achieve state-of-the-art performance in high-fidelity image and video generation but suffer from expensive infer

EditTransfer++: Toward Faithful and Efficient Visual-Prompt-Guided Image Editing

arXiv:2605.07455v1 Announce Type: new Abstract: Visual-prompt-guided edit transfer aims to learn image transformations directly from example pairs, offering more precise and controllable editing than

safetyarxiv-cs-cv

HARMONY: Bridging the Personalization-Generalization Gap by Mitigating Representation Skew in Heterogeneous Split Federated Learning

arXiv:2605.07211v1 Announce Type: cross Abstract: Mobile devices face diverse resource constraints and non-IID data class distributions, requiring fast on-device inference for local in-distribution (I

local-aiarxiv-cs-ai

Not All Tokens Need 40 Steps: Heterogeneous Step Allocation in Diffusion Transformers for Efficient Video Generation

arXiv:2605.06892v1 Announce Type: new Abstract: Diffusion Transformers (DiTs) have achieved state-of-the-art video generation quality, but they incur immense computational cost because standard infere

researcharxiv-cs-cv

Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compression

arXiv:2502.01941v3 Announce Type: replace-cross Abstract: While Key-Value (KV) cache compression is essential for efficient LLM inference, current evaluations disproportionately focus on sparse retrie

Setting-Matched and Semantics-Scaled Benchmarking of One-Step Generative Models Against Multistep Diffusion and Flow Models

model-releasesarxiv-cs-cv

arXiv:2603.14186v4 Announce Type: replace Abstract: State-of-the-art text-to-image models produce high-quality images, but inference remains expensive as generation requires several sequential ODE or

Hardware

🧵 Slime: The Most Elegant & Comfortable RL Training Framework Ever A deep dive into why Slime redefines LLM RL training with clean architec…

🧵 Slime: The Most Elegant & Comfortable RL Training Framework Ever A deep dive into why Slime redefines LLM RL training with clean architecture & production-grade engineering ✨ Insights from Zhihu con

hardwarezhipu-ai--x

Hardware

SOCKET: SOft Collision Kernel EsTimator for Sparse Attention

arXiv:2602.06283v2 Announce Type: replace Abstract: Exploiting sparsity during long-context inference is key to scaling large language models, as attention dominates the cost of autoregressive decodin

hardwarearxiv-cs-lg

SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting

arXiv:2605.07243v1 Announce Type: new Abstract: Speculative decoding accelerates LLM inference by drafting a tree of candidate continuations and verifying it in one target forward. Existing drafters f

researcharxiv-cs-cl

Agents

Switchcraft: AI Model Router for Agentic Tool Calling

arXiv:2605.07112v1 Announce Type: new Abstract: Agentic AI systems that invoke external tools are powerful but costly, leading developers to default to large models and overspend inference budgets. Mo

agentsarxiv-cs-ai

Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers

model-releasesarxiv-cs-lg

arXiv:2605.07772v1 Announce Type: new Abstract: Transformers perform inference by iteratively transforming token representations across layers. This layerwise computation has been studied empirically,

WiCER: Wiki-memory Compile, Evaluate, Refine Iterative Knowledge Compilation for LLM Wiki Systems

arXiv:2605.07068v1 Announce Type: cross Abstract: The LLM Wiki pattern, to compile and provide domain knowledge into a persistent artifact and serve it to LLMs via KV cache inference, promises context

b9084

b9084 is a release build of llama.cpp, an open-source C/C++ library for LLM inference that enables running large language models on consumer hardware. llama.cpp is a software library that performs inf

b9085

B9085 is a release tag from the llama.cpp project, which provides LLM inference in C/C++. As a commit-based release identifier from an actively developed project, this release likely contains bug fixe

b9088

b9088 is a release of llama.cpp, a C/C++ implementation for LLM inference . The project follows a rapid release cycle where multiple releases can be published in a single day , using commit hash ident

b9090

B9090 is a release from llama.cpp, a C/C++ project for LLM inference . The project uses a rapid release cycle with multiple releases published in a single day . This particular release likely includes

b9093

b9093 is a release of llama.cpp, a project for LLM inference in C/C++ . Based on the release series, this likely contains build updates, bug fixes, and/or feature improvements to the llama.cpp inferen

A Fast Model Counting Algorithm for Two-Variable Logic with Counting and Modulo Counting Quantifiers

arXiv:2605.03391v1 Announce Type: cross Abstract: Weighted first-order model counting (WFOMC) is a central task in lifted probabilistic inference: It asks for the weighted sum of all models of a first

b9055

B9055 is the latest version of llama.cpp, released on May 7, 2026. Llama.cpp is an LLM inference framework implemented in C/C++ that enables efficient large language model execution with broad hardwar

b9056

b9056 is a release of llama.cpp, a C/C++ implementation for LLM inference . The project enables users to run LLaMA models on consumer hardware without expensive GPUs or cloud infrastructure . This rel

b9061

Release b9061 is a build version of llama.cpp, a C/C++ implementation designed to enable LLM inference with minimal setup and state-of-the-art performance across various hardware platforms . As a spec

b9066

llama.cpp b9066 is a release of a C/C++ implementation for large language model (LLM) inference . Based on the release repository structure, b9066 represents a specific build or commit version of the

Exact Dual Geometry of SOC-ICNN Value Functions

arXiv:2605.04722v1 Announce Type: new Abstract: Input Convex Neural Networks (ICNNs) are commonly used in a two-stage manner: one first trains a convex network and then minimizes it over its input in

local-aiarxiv-cs-lg