Search: “inference”

Accelerating LLM Inference with Prompt Caching for Open‑Source Models on Databricks

This article describes techniques for using prompt caching to improve the inference speed and efficiency of open-source large language models when deployed on Databricks' platform. Prompt caching redu

industrydatabricks

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Agents

AMD CEO Lisa Su projects the CPU market will grow over 35% annually through 2031, up from 3% to 4% historically, driven by AI inference and agentic AI demand (Cheng Ting-Fang/Nikkei Asia)

DGX agent

Cheng Ting-Fang / Nikkei Asia: AMD CEO Lisa Su projects the CPU market will grow over 35% annually through 2031, up from 3% to 4% historically, driven by AI inference and agentic AI demand — TAIPEI —

agentstechmeme

22 May 2026

Boundary-targeted Membership Inference Attacks on Safety Classifiers

arXiv:2605.22373v1 Announce Type: cross Abstract: Safety classifiers are essential safeguards within generative AI systems, filtering harmful content or identifying at-risk users when interacting with

local-aiarxiv-cs-cl

Applications

ELSA: An ELastic SNN Inference Architecture for Efficient Neuromorphic Computing

arXiv:2605.20802v1 Announce Type: cross Abstract: Spiking neural networks (SNNs) exploit event-driven and addition-only computation to substantially improve efficiency for intelligent computation. A k

applicationsarxiv-cs-ai

MTP means Multi Token Prediction. It's a speculative decoding technique that can result in large inference speedups in many cases. 1. Update…

model-releaseslm-studio--x

MTP means Multi Token Prediction. It's a speculative decoding technique that can result in large inference speedups in many cases. 1. Update to LM Studio 0.4.14 2. Download a model that supports MTP l

Physiology and Anatomy Aware Inverse Inference of Myocardial Infarction for Cardiac Digital Twin

model-releasesarxiv-cs-cv

arXiv:2605.22044v1 Announce Type: new Abstract: Accurate localization of myocardial infarction is essential for risk stratification. While LGE-MRI remains the gold standard, it is resource-intensive.

Temporal Coding as a Substrate for Sensorimotor Object Inference: A Spiking Reinterpretation of Thousand Brains Architecture

model-releasesarxiv-cs-ro

arXiv:2605.22206v1 Announce Type: cross Abstract: The Thousand Brains Theory (TBT) and its open-source Monty framework model object recognition through sensorimotor inference -- identifying objects by

Everywhere Valid Bounds on False Discovery Proportions in Conformal Inference

arXiv:2605.20726v1 Announce Type: cross Abstract: Modern applications of conformal inference to multiple testing problems, such as outlier detection and candidate selection, often involve selecting te

researcharxiv-cs-lg

21 May 2026

Theoretical guidelines for annealed Langevin dynamics in compositional simulation-based inference

arXiv:2605.21253v1 Announce Type: cross Abstract: Compositional score-based approaches to simulation-based inference (SBI) approximate the posterior over a shared parameter given n independent observa

21 May 2026

Agents

LLMs are stateless (every time you reply to an LLM, you re-inject the entire conversation to a fresh inference) the purpose of memory is to …

LLMs are stateless (every time you reply to an LLM, you re-inject the entire conversation to a fresh inference) the purpose of memory is to provide continuity (in games we'd call it a 'persistent worl

agentsyohei-nakajima--x

Rotation-Aligned Key Channel Pruning for Efficient Vision-Language Model Inference

arXiv:2605.19218v1 Announce Type: cross Abstract: Vision-Language Models suffer severe KV cache pressure at inference, as a single image often encodes into thousands of tokens. Most existing methods e

Stochastic Gradient Variational Inference with Price's Gradient Estimator from Bures-Wasserstein to Parameter Space

arXiv:2602.18718v2 Announce Type: replace-cross Abstract: For approximating a target distribution given only its unnormalized log-density, stochastic gradient-based variational inference (VI) algorith

TIDE: Efficient and Lossless MoE Diffusion LLM Inference with I/O-aware Expert Offload

arXiv:2605.20179v1 Announce Type: new Abstract: Diffusion Large Language Models (dLLMs) have emerged as a competitive alternative to autoregressive (AR) models, offering better hardware utilization an

hardwarearxiv-cs-cl

Safety

Adaptive Generate-Rank-Verify: Inference-Time Search with Costly Verification

arXiv:2605.17609v1 Announce Type: new Abstract: Many inference-time language-model pipelines combine a cheap reward signal with an expensive verifier, such as exact answer checking in mathematical rea

safetyarxiv-cs-lg

Benchmarking inference at scale: coding agents

model-releasestogether-ai-blog

This article presents benchmarking results for AI coding agents evaluated at scale, likely comparing performance metrics such as code generation accuracy, execution success rates, and inference effici

Beyond Inference-Time Search: Reinforcement Learning Synthesizes Reusable Solvers

arXiv:2605.18374v1 Announce Type: cross Abstract: Large language models (LLMs) typically approach combinatorial optimization as an inference-time procedure, solving each instance separately through sa

local-aiarxiv-cs-ai

HDTree: Generative Modeling of Cellular Hierarchies for Robust Lineage Inference

arXiv:2506.23287v3 Announce Type: replace Abstract: In single-cell research, tracing and analyzing high-throughput single-cell differentiation trajectories is crucial for understanding biological proc

researcharxiv-cs-lg

JDCNet: Confidence-Gated Privileged-Modality Distillation for Cost-Preserving X-ray Inference

model-releasesarxiv-cs-cv

arXiv:2603.29167v2 Announce Type: replace Abstract: We study a systems-level visual inference problem: using an expensive privileged modality during training while preserving a fixed-cost, single-moda

OxyGen: Unified KV Cache Management for VLA Inference under Multi-Task Parallelism

arXiv:2603.14371v2 Announce Type: replace-cross Abstract: Embodied AI agents increasingly require parallel execution of multiple tasks, such as manipulation, conversation, and memory construction, fro

local-aiarxiv-cs-ai

ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inference

arXiv:2605.16360v1 Announce Type: cross Abstract: Efficient long-context inference in Large Language Models (LLMs) is severely constrained by the Key-Value (KV) cache memory wall, yet existing pruning

Sources: Zyphra, which trains and runs inference for its open-weight models on AMD hardware, is raising a 500M Series B at a valuation of at least 5B (Anna Tong/Forbes)

Anna Tong / Forbes: Sources: Zyphra, which trains and runs inference for its open-weight models on AMD hardware, is raising a 500M Series B at a valuation of at least 5B — The Series B round, which ch

hardwaretechmeme

Stabilizing Temporal Inference Dynamics for Online Surgical Phase Recognition

arXiv:2605.16387v1 Announce Type: cross Abstract: Online Surgical Phase Recognition (SPR) models can reach high frame-wise accuracy, yet their predictions often lack temporal stability, fragmenting wo

TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks

arXiv:2605.17170v1 Announce Type: new Abstract: Agentic workloads have emerged as a major workload for LLM inference. They differ significantly from chat-only workloads, requiring long-context process

⚡️After weeks of hard work, we're thrilled to fully open-source HY World 2.0 today -- full inference code and all models! Build, explore, an…

⚡️After weeks of hard work, we're thrilled to fully open-source HY World 2.0 today -- full inference code and all models! Build, explore, and create your own interactive worlds with us.👇 https://githu

industryemad-mostaque--x

Safety

Opponent State Inference Under Partial Observability: An HMM-POMDP Framework for 2026 Formula 1 Energy Strategy

arXiv:2603.01290v3 Announce Type: replace Abstract: The 2026 Formula 1 technical regulations introduce a fundamental change to energy strategy: under a 50/50 internal combustion engine / battery power

safetyarxiv-cs-ai

Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints

arXiv:2504.11320v3 Announce Type: replace-cross Abstract: Large language models now serve millions of users daily, with providers incurring costs exceeding $700,000 per day. Each request requires toke

Agents

Teams like @modal are already using Devin Auto-Triage for incidents on their inference team. “Devin Automations feels like a step forward fr…

Teams like @modal are already using Devin Auto-Triage for incidents on their inference team. “Devin Automations feels like a step forward from other auto-triage tools we’ve tried. It monitors our chan

agentscognition-ai--x

Tools

Heading to hashtag#MLSys2026? Come unwind with the Together AI team at Inference After Dark. Drinks, bites, shuffleboard, and a room full of…

Heading to hashtag#MLSys2026? Come unwind with the Together AI team at Inference After Dark. Drinks, bites, shuffleboard, and a room full of researchers and AI-native builders. 🟠 Tuesday, May 19 🟠 7:3

toolstogether-ai--x

16 May 2026

Dual-Dimensional Consistency: Balancing Budget and Quality in Adaptive Inference-Time Scaling

arXiv:2605.15100v1 Announce Type: new Abstract: Large Language Models (LLMs) have demonstrated remarkable abilities in reasoning. However, maximizing their potential through inference-time scaling fac

EnergyLens: Predictive Energy-Aware Exploration for Multi-GPU LLM Inference Optimization

arXiv:2605.14249v1 Announce Type: new Abstract: We present EnergyLens, an end-to-end framework for energy-aware large language model (LLM) inference optimization. As LLMs scale, predicting and reducin

Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inference

arXiv:2605.13915v1 Announce Type: cross Abstract: Quantization is essential for efficient large language model (LLM) inference, yet the dequantization step-converting low-bit weights back to high-prec

Optimizing PyTorch Inference with LLM-Based Multi-Agent Systems

arXiv:2511.16964v2 Announce Type: replace-cross Abstract: Maximizing performance on available GPU hardware is an ongoing challenge for modern AI inference systems. Traditional approaches include writi

Tools

Pearl generates proofs from matrix multiplications that already happen during training and inference computations. Those proofs help secure …

Pearl generates proofs from matrix multiplications that already happen during training and inference computations. Those proofs help secure Pearl Network, and the future value of Pearl emissions helps

toolstogether-ai--x

To discretize continually: Mean shift interacting particle systems for Bayesian inference

arXiv:2605.14142v1 Announce Type: cross Abstract: Integration against a probability distribution given its unnormalized density is a central task in Bayesian inference and other fields. We introduce n

XFP: Quality-Targeted Adaptive Codebook Quantization with Sparse Outlier Separation for LLM Inference

arXiv:2605.14844v1 Announce Type: cross Abstract: We introduce XFP, a dynamic weight quantizer for LLM inference that inverts the conventional workflow: the operator specifies reconstruction quality f

hardwarearxiv-cs-ai

GLASS: Global-Local Aggregation for Inference-time Sparsification of LLMs

arXiv:2508.14302v2 Announce Type: replace-cross Abstract: Inference-time sparsification is a promising path to deploy large language models (LLMs) on resource-constrained devices, yet existing trainin

local-aiarxiv-cs-ai

Agents

great agents need great infrastructure. proud to be @LangChain's Deep Agents Inference Partner at Interrupt 2026 in SF. great to spend time …

great agents need great infrastructure. proud to be @LangChain's Deep Agents Inference Partner at Interrupt 2026 in SF. great to spend time with builders at our day 1 happy hour. excited for day 2 tod

agentsharrison-chase--x

Inference-Time Machine Unlearning via Gated Activation Redirection

arXiv:2605.12765v1 Announce Type: new Abstract: Large Language Models memorize vast amounts of training data, raising concerns regarding privacy, copyright infringement, and safety. Machine unlearning

Plan Before You Trade: Inference-Time Optimization for RL Trading Agents

arXiv:2605.12653v1 Announce Type: cross Abstract: Reinforcement learning agents for portfolio management are typically trained and deployed as static policies, with no mechanism for using price foreca

Training a diffusion LLM from scratch is challenging, and diffusion inference advantages don't show up at training. So we convert instead. B…

Training a diffusion LLM from scratch is challenging, and diffusion inference advantages don't show up at training. So we convert instead. Building on the TiDAR recipe, we take ZAYA1-8B-base and run d

industryemad-mostaque--x

ChunkFlow: Communication-Aware Chunked Prefetching for Layerwise Offloading in Distributed Diffusion Transformer Inference

arXiv:2605.11335v1 Announce Type: cross Abstract: Layerwise offloading reduces the GPU memory footprint of large diffusion transformer (DiT) inference by prefetching upcoming layers from host memory,

Fast MoE Inference via Predictive Prefetching and Expert Replication

arXiv:2605.11537v1 Announce Type: new Abstract: The Mixture of Experts (MoE) architecture has become a fundamental building block in state-of-the-art large language models (LLMs), improving domain-spe

Nebius snaps up Clarifai’s compute orchestration tech and talent to enhance AI inference

Dutch artificial intelligence infrastructure giant Nebius Group N.V. said today it’s recruiting the core engineering team from AI orchestration software firm Clarifai Inc. in an effort to boost its ma

industrysiliconangle

UK chip startup Fractile raised a $220M Series B led by Factorial Funds, Accel and Founders Fund to make specialized logic and memory chips for inference (Robbie Whelan/Wall Street Journal)

Robbie Whelan / Wall Street Journal: UK chip startup Fractile raised a $220M Series B led by Factorial Funds, Accel and Founders Fund to make specialized logic and memory chips for inference — Factori

industrytechmeme

VERDI: Single-Call Confidence Estimation for Verification-Based LLM Judges via Decomposed Inference

model-releasesarxiv-cs-cl

arXiv:2605.11334v1 Announce Type: cross Abstract: LLM-as-Judge systems are widely deployed for automated evaluation, yet practitioners lack reliable methods to know when a judge's verdict should be tr

What we find most useful about TST is the decoupling. The training-time efficiency is fully separated from the inference-time architecture, …

What we find most useful about TST is the decoupling. The training-time efficiency is fully separated from the inference-time architecture, which makes TST a clean addition on top of other pretraining

researchnous-research--x

CLEAR: Context-Aware Learning with End-to-End Mask-Free Inference for Adaptive Video Subtitle Removal

arXiv:2603.21901v2 Announce Type: replace Abstract: Video subtitle removal aims to distinguish text overlays from background content while preserving temporal coherence. Existing diffusion-based metho

researcharxiv-cs-cv

12 May 2026

Safety

Reward Auditor: Inference on Reward Modeling Suitability in Real-World Perturbed Scenarios