Search: “inference”

arXiv:2607.28418v1 Announce Type: cross Abstract: Pruning is a promising approach for improving the efficiency of LLMs. Existing static structured pruning methods are hardware-friendly and can deliver

applicationsarxiv-cs-cl

31 Jul 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Applications

AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding

DGX agent

arXiv:2607.25852v1 Announce Type: new Abstract: Speculative decoding accelerates large language model inference without changing the target distribution, but no single drafting structure performs best

applicationsarxiv-cs-cl

29 Jul 2026

Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference

arXiv:2607.25018v1 Announce Type: new Abstract: Large language model (LLM) cascades reduce inference cost by routing easy queries to a small model and deferring hard queries to a larger one. Productio

29 Jul 2026

DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference

arXiv:2607.24434v1 Announce Type: cross Abstract: Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, bu

28 Jul 2026

Inference-Time Consensus for Mitigating Hidden Behaviors from LLM Fine-Tuning

arXiv:2607.23394v1 Announce Type: new Abstract: Recent work shows that fine-tuning language models on even a small amount of poisoned data can install targeted misbehavior, and ostensibly benign data

28 Jul 2026

MIITA: Memory-Induced Inference-Time Adaptation for Continual Learning with Small Language Models

arXiv:2607.22556v1 Announce Type: new Abstract: Continual learning (CL) is essential for small language models (SLMs) to adapt to evolving real-world needs in resource-constrained deployments. However

28 Jul 2026

InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents

arXiv:2607.20468v1 Announce Type: new Abstract: AI agents are increasingly used to automate research and development tasks, yet existing benchmarks typically evaluate them on prescribed workflows or n

24 Jul 2026

BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators

arXiv:2607.19438v1 Announce Type: cross Abstract: Apple's M5 generation introduces a redesigned GPU architecture in which every core carries a dedicated Neural Accelerator: on-die matrix units exposed

23 Jul 2026

Benchmarking Confidential GPU Inference on NVIDIA H100 under Intel TDX

arXiv:2607.19353v1 Announce Type: new Abstract: Confidential computing is becoming a practical deployment requirement for AI inference workloads that process sensitive inputs or protect proprietary mo

23 Jul 2026

How Inference Compute Shapes Frontier LLM Evaluation

arXiv:2606.17930v2 Announce Type: replace Abstract: AI evaluations are shifting toward harder tasks that benefit from longer trajectories involving tool use and iterative problem solving. As a result,

15 Jul 2026

We Hebben Een Serieus Translatie: Modeling Intercomprehension as Probabilistic Inference

arXiv:2607.12169v1 Announce Type: new Abstract: Intercomprehension refers to partial intelligibility of an unfamiliar language (L2) by a speaker of a related language (L1). How is this zero-shot cross

safetyarxiv-cs-cl

15 Jul 2026

Diffusion Models in Simulation-Based Inference: A Tutorial Review

arXiv:2512.20685v3 Announce Type: replace-cross Abstract: Diffusion models have recently emerged as powerful learners for simulation-based inference (SBI), enabling fast and accurate estimation of lat

9 Jul 2026

EdgeCompress: Coupling Multidimensional Model Compression and Dynamic Inference for EdgeAI

arXiv:2607.06982v1 Announce Type: new Abstract: Convolutional neural networks (CNNs) have demonstrated encouraging results in image classification tasks. However, the prohibitive computational cost of

researcharxiv-cs-cv

9 Jul 2026

Tutorials

Blind Quality Enhancement of Compressed Video via Fine-Grained Degradation-Guided Sequential Inference

arXiv:2511.16137v2 Announce Type: replace Abstract: Existing studies on quality enhancement for compressed video (QECV) predominantly rely on known quantization parameters (QPs), training separate enh

tutorialsarxiv-cs-cv

8 Jul 2026

FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference

arXiv:2607.06519v1 Announce Type: new Abstract: Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specif

8 Jul 2026

UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation

arXiv:2607.06306v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated growing competence in web page generation. However, existing text-driven approaches rely on complex pro

8 Jul 2026

Adaptive Inference Batching using Policy Gradients

arXiv:2607.05272v1 Announce Type: cross Abstract: Inference serving systems must balance throughput and latency under bursty, heterogeneous workloads, yet the industry standard remains static batching

HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference

arXiv:2607.04302v1 Announce Type: cross Abstract: We present HiFA4, a post-training operator-level design that executes both QK^T and PV in FlashAttention as 4-bit HIF4 Cube GEMMs for LLM inference on

Applications

One Framework for All: Cross-Modal Membership Inference for Generative Models

arXiv:2607.04339v1 Announce Type: cross Abstract: Large generative models across text-to-text, text-to-image, and image-to-text modalities have been shown to pose significant privacy risks. One fundam

applicationsarxiv-cs-ai

Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters

arXiv:2504.08791v3 Announce Type: replace-cross Abstract: On-device inference offers privacy, offline use, and instant response, but consumer hardware restricts large language models (LLMs) to low thr

R^2PO: Decoupling Rollout and Inference Policies for LLM Reasoning

arXiv:2601.11960v3 Announce Type: replace-cross Abstract: Existing reinforcement learning methods for LLM reasoning implicitly assume that the policy generating training trajectories should coincide w

Vision Token Manipulation Attacks on Cloud-Edge Inference of Large Vision-Language Models

arXiv:2607.02819v1 Announce Type: cross Abstract: Cloud-edge Large Vision-Language Model (LVLM) inference enables efficient deployment by splitting computation between edge devices and cloud servers.

Hardware

An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation

arXiv:2607.02119v1 Announce Type: cross Abstract: While Large Multimodal Models excel in comprehension, high-throughput inference engines lack native support for multimodal generation. This is severe

hardwarearxiv-cs-ai

3 Jul 2026

Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots

model-releasesarxiv-cs-ro

arXiv:2607.02501v1 Announce Type: new Abstract: Embodied AI models now span vision-language-action (VLA) models and world-action models (WAMs), but practical deployment remains fragmented across model

3 Jul 2026

LLM-Guided ODE Discovery and Parameter Inference from Small-Cohort Aggregate Data

arXiv:2607.00733v1 Announce Type: cross Abstract: Mechanistic modeling via ordinary differential equations (ODEs) provides interpretable descriptions of complex dynamics and enables inference of under

2 Jul 2026

Reading Order Inference for Complex Document Layouts

arXiv:2607.01018v1 Announce Type: cross Abstract: Reading order inference remains a critical bottleneck in the digitization of complex historical manuscripts, where pages contain multiple spatially in

2 Jul 2026

Hierarchical Clustering As a Novel Solution to the Notorious Multicollinearity Problem in Observational Causal Inference

arXiv:2606.30992v1 Announce Type: cross Abstract: Multicollinearity is a long lasting challenge in observational causal inference, especially in regressions -- highly correlated independent variables

researcharxiv-cs-lg

1 Jul 2026

RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

arXiv:2606.31519v1 Announce Type: cross Abstract: Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often

researcharxiv-cs-cl

1 Jul 2026

Behavior Uncloning: Distilling Mode Redirection into Policy Weights without Inference-Time Steering

arXiv:2606.29201v1 Announce Type: cross Abstract: Behavior-cloned policies often learn multiple behavior modes from demonstration datasets, including modes that are unsafe or otherwise undesired at de

Local Ai

KernelSight-LM: A Kernel-Level LLM Inference Simulator

arXiv:2606.28565v1 Announce Type: cross Abstract: As large language models (LLMs) move into production serving, practitioners must rapidly evaluate inference performance across diverse hardware, model

local-aiarxiv-cs-ai

Spatial Deconfounder: Interference-Aware Deconfounding for Spatial Causal Inference

arXiv:2510.08762v2 Announce Type: replace Abstract: Causal inference in spatial domains faces two intertwined challenges: (1) unmeasured spatial factors, such as weather, air pollution, or mobility, t

W4A4 Quantization for Inference on Wan2.2-I2V-A14B

arXiv:2606.29337v1 Announce Type: new Abstract: We summarize our submission to Sub-Challenge 1: W4A4 Quantization for Inference (HiF4 / MXFP4) of the ICME 2026 Low-Bit-width Large-Model Quantization C

researcharxiv-cs-cv

A Survey on Federated Causal Discovery and Inference

arXiv:2606.23741v1 Announce Type: cross Abstract: Causal reasoning, which encompasses the discovery of causal structures and the inference of causal effects, is fundamental to data-driven decision mak

CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference

arXiv:2606.24467v1 Announce Type: new Abstract: Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiti

Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study

arXiv:2606.21428v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) language models are often described as ideal for resource-constrained inference. Each token activates only a small su

Enabling Robust Cloth Manipulation via Inference-Time Simulator-in-the-Loop Refinement

arXiv:2606.24552v1 Announce Type: new Abstract: Simulator-in-the-loop optimization offers a promising inference-time mechanism for robot manipulation. It uses a physical simulator as a backend rollout

safetyarxiv-cs-ro

Hardware

Hardware-Oriented Inference Complexity of Kolmogorov-Arnold Networks

arXiv:2604.03345v2 Announce Type: replace Abstract: Kolmogorov-Arnold Networks (KANs) have recently emerged as a powerful architecture for various machine learning applications. However, their unique

hardwarearxiv-cs-lg

MGI: Member vs Generated Inference

arXiv:2606.23872v1 Announce Type: cross Abstract: As generative models increasingly produce samples that are indistinguishable from human-created content, it becomes difficult to determine whether a g

Efficient Network Inference via Hardware-Aware Architecture Search, Model Pruning & Quantization

arXiv:2606.23210v1 Announce Type: new Abstract: Embedded global navigation satellite system (GNSS) interference monitoring requires fast and memory-efficient inference to process large volumes of raw

researcharxiv-cs-lg

Tutorials

MAVRL: Learning Reward Functions from Multiple Feedback Types with Amortized Variational Inference

arXiv:2602.15206v2 Announce Type: replace Abstract: Reward learning typically relies on a single feedback type or combines multiple feedback types using manually weighted loss terms. Currently, it rem

tutorialsarxiv-cs-lg

PG-MAP: Joint MAP Optimization for Inference-Time Alignment of Diffusion and Flow-Matching Models

arXiv:2606.22958v1 Announce Type: cross Abstract: Inference-time alignment of pretrained text-to-image models is typically performed along a single control axis, such as classifier-free guidance, atte

safetyarxiv-cs-cv

Statistical Inference for Misspecified Contextual Bandits

arXiv:2606.22639v1 Announce Type: cross Abstract: Contextual bandit algorithms have transformed modern experimentation by enabling real-time adaptation for personalized treatment. Yet these advantages

safetyarxiv-cs-lg

Local Ai

Compiler-First State Space Duality and Portable O(1) Autoregressive Caching for Inference

arXiv:2603.09555v2 Announce Type: replace-cross Abstract: High-throughput Mamba-2 inference is usually tied to fused CUDA and Triton kernels, limiting portability across accelerator backends. We show

local-aiarxiv-cs-ai

11 Jun 2026

Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scaling

arXiv:2509.20241v2 Announce Type: replace Abstract: As AI inference scales to billions of queries, estimates of per-query energy use are increasingly important for capacity planning, efficiency interv

safetyarxiv-cs-lg

11 Jun 2026

Hardware

Accounting for AI Inference in Corporate GHG Inventories: A Four-Tier Methodology for Scope 3 Category 1 Reporting

arXiv:2606.10660v1 Announce Type: cross Abstract: AI inference services -- API subscriptions, enterprise chat tools, and SaaS products with embedded AI features -- fall unambiguously within Scope 3 Ca

hardwarearxiv-cs-ai

10 Jun 2026

Disjoint or Overlapping? Inference Windowing for Reconstruction-Based Time Series Anomaly Detection

arXiv:2606.09874v1 Announce Type: new Abstract: Reconstruction-based methods are widely used for time series anomaly detection, where models are trained to reconstruct subsequences, and anomalies are

10 Jun 2026

Spiking Neural Network inference on FPGAs with hls4ml

arXiv:2606.10008v1 Announce Type: cross Abstract: Spiking Neural Networks (SNNs) provide a naturally temporal machine-learning framework. Their neurons maintain an internal state and propagate informa

researcharxiv-cs-lg

10 Jun 2026

AgentCompile: An LLM-Guided Compiler for Direct CUDA Inference

arXiv:2606.07665v1 Announce Type: cross Abstract: Transformer inference increasingly depends on specialized compiler and runtime support, but real model graphs still require semantic decisions about w