Search: “inference”

BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators

arXiv:2607.19438v1 Announce Type: cross Abstract: Apple's M5 generation introduces a redesigned GPU architecture in which every core carries a dedicated Neural Accelerator: on-die matrix units exposed

23 Jul 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Benchmarking Confidential GPU Inference on NVIDIA H100 under Intel TDX

DGX agent

arXiv:2607.19353v1 Announce Type: new Abstract: Confidential computing is becoming a practical deployment requirement for AI inference workloads that process sensitive inputs or protect proprietary mo

model-releasesarxiv-cs-ai

23 Jul 2026

Kimi K3 is coming to Together AI on day zero, July 27. Start building with it the moment it launches, powered by Together AI inference for c…

model-releasestogether-ai--x

Kimi K3 is coming to Together AI on day zero, July 27. Start building with it the moment it launches, powered by Together AI inference for coding, agents, and production workloads. https://www.togethe

23 Jul 2026

How Inference Compute Shapes Frontier LLM Evaluation

arXiv:2606.17930v2 Announce Type: replace Abstract: AI evaluations are shifting toward harder tasks that benefit from longer trajectories involving tool use and iterative problem solving. As a result,

15 Jul 2026

We Hebben Een Serieus Translatie: Modeling Intercomprehension as Probabilistic Inference

arXiv:2607.12169v1 Announce Type: new Abstract: Intercomprehension refers to partial intelligibility of an unfamiliar language (L2) by a speaker of a related language (L1). How is this zero-shot cross

safetyarxiv-cs-cl

15 Jul 2026

Diffusion Models in Simulation-Based Inference: A Tutorial Review

model-releasesarxiv-cs-lg

arXiv:2512.20685v3 Announce Type: replace-cross Abstract: Diffusion models have recently emerged as powerful learners for simulation-based inference (SBI), enabling fast and accurate estimation of lat

9 Jul 2026

EdgeCompress: Coupling Multidimensional Model Compression and Dynamic Inference for EdgeAI

arXiv:2607.06982v1 Announce Type: new Abstract: Convolutional neural networks (CNNs) have demonstrated encouraging results in image classification tasks. However, the prohibitive computational cost of

researcharxiv-cs-cv

9 Jul 2026

Agents

Fun conversation with @swyx on our journey building the cloud for true elastic inference, sandboxes, and more. And of course, how we're evol…

Fun conversation with @swyx on our journey building the cloud for true elastic inference, sandboxes, and more. And of course, how we're evolving Modal's dev experience to be better for agents. Modal's

agentsswyx--x

9 Jul 2026

Tutorials

Blind Quality Enhancement of Compressed Video via Fine-Grained Degradation-Guided Sequential Inference

arXiv:2511.16137v2 Announce Type: replace Abstract: Existing studies on quality enhancement for compressed video (QECV) predominantly rely on known quantization parameters (QPs), training separate enh

tutorialsarxiv-cs-cv

FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference

arXiv:2607.06519v1 Announce Type: new Abstract: Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specif

Agents

Solidigm targets the intelligence layer as agentic inference pushes storage to center stage

The shift from model training to agentic inference is forcing a fundamental rethink of how artificial intelligence infrastructure is built and which components carry the most strategic weight. What wa

agentssiliconangle

Tensordyne targets AI inference market with logarithmic math and Juniper-derived rack architecture

The race to serve AI inference faster and cheaper is exposing the hard limits of conventional chip architecture. As demand for real-time AI responses accelerates, the industry’s standard response — st

hardwaresiliconangle

UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation

arXiv:2607.06306v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated growing competence in web page generation. However, existing text-driven approaches rely on complex pro

Industry

We're still just scratching the surface of how much better, faster, cheaper we can make inference for open-source models! So super excited t…

We're still just scratching the surface of how much better, faster, cheaper we can make inference for open-source models! So super excited to see companies like @zml_ai by @steeve come out of stealth

industryclem-delangue--x

Adaptive Inference Batching using Policy Gradients

arXiv:2607.05272v1 Announce Type: cross Abstract: Inference serving systems must balance throughput and latency under bursty, heterogeneous workloads, yet the industry standard remains static batching

HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference

arXiv:2607.04302v1 Announce Type: cross Abstract: We present HiFA4, a post-training operator-level design that executes both QK^T and PV in FlashAttention as 4-bit HIF4 Cube GEMMs for LLM inference on

Applications

One Framework for All: Cross-Modal Membership Inference for Generative Models

arXiv:2607.04339v1 Announce Type: cross Abstract: Large generative models across text-to-text, text-to-image, and image-to-text modalities have been shown to pose significant privacy risks. One fundam

applicationsarxiv-cs-ai

Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters

arXiv:2504.08791v3 Announce Type: replace-cross Abstract: On-device inference offers privacy, offline use, and instant response, but consumer hardware restricts large language models (LLMs) to low thr

R^2PO: Decoupling Rollout and Inference Policies for LLM Reasoning

arXiv:2601.11960v3 Announce Type: replace-cross Abstract: Existing reinforcement learning methods for LLM reasoning implicitly assume that the policy generating training trajectories should coincide w

Report: China’s DeepSeek follows OpenAI in developing its own custom inference chips

model-releasessiliconangle

DeepSeek Ltd., one of China’s most visible artificial intelligence companies, is pushing to design its own, in-house silicon aimed at inference workloads, according to a report by Reuters today. The r

Vision Token Manipulation Attacks on Cloud-Edge Inference of Large Vision-Language Models

arXiv:2607.02819v1 Announce Type: cross Abstract: Cloud-edge Large Vision-Language Model (LVLM) inference enables efficient deployment by splitting computation between edge devices and cloud servers.

Local Ai

Most models run on whatever inference engine they ship with. That's rarely the fastest option. We built our own, AGI-RUN. On a phone it beat…

Most models run on whatever inference engine they ship with. That's rarely the fastest option. We built our own, AGI-RUN. On a phone it beat every model's own native engine, and ran up to 9.9x faster

local-aidiv-garg--x

6 Jul 2026

An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation

arXiv:2607.02119v1 Announce Type: cross Abstract: While Large Multimodal Models excel in comprehension, high-throughput inference engines lack native support for multimodal generation. This is severe

hardwarearxiv-cs-ai

3 Jul 2026

Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots

model-releasesarxiv-cs-ro

arXiv:2607.02501v1 Announce Type: new Abstract: Embodied AI models now span vision-language-action (VLA) models and world-action models (WAMs), but practical deployment remains fragmented across model

3 Jul 2026

We're releasing the full slides for our 2 hr deepdive session from the AI Engineer World's Fair. We covered how we build inference engines t…

We're releasing the full slides for our 2 hr deepdive session from the AI Engineer World's Fair. We covered how we build inference engines to serve agentic workloads at trillion token production scale

hardwaretogether-ai--x

3 Jul 2026

LLM-Guided ODE Discovery and Parameter Inference from Small-Cohort Aggregate Data

arXiv:2607.00733v1 Announce Type: cross Abstract: Mechanistic modeling via ordinary differential equations (ODEs) provides interpretable descriptions of complex dynamics and enables inference of under

2 Jul 2026

Reading Order Inference for Complex Document Layouts

arXiv:2607.01018v1 Announce Type: cross Abstract: Reading order inference remains a critical bottleneck in the digitization of complex historical manuscripts, where pages contain multiple spatially in

2 Jul 2026

Hierarchical Clustering As a Novel Solution to the Notorious Multicollinearity Problem in Observational Causal Inference

arXiv:2606.30992v1 Announce Type: cross Abstract: Multicollinearity is a long lasting challenge in observational causal inference, especially in regressions -- highly correlated independent variables

researcharxiv-cs-lg

1 Jul 2026

RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

arXiv:2606.31519v1 Announce Type: cross Abstract: Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often

researcharxiv-cs-cl

1 Jul 2026

Behavior Uncloning: Distilling Mode Redirection into Policy Weights without Inference-Time Steering

arXiv:2606.29201v1 Announce Type: cross Abstract: Behavior-cloned policies often learn multiple behavior modes from demonstration datasets, including modes that are unsafe or otherwise undesired at de

Inference chip startup Etched launches with $800M in funding

model-releasessiliconangle

Etched Inc., a developer of artificial intelligence inference chips, launched today with 800 million in funding. The startup raised the capital over multiple rounds. The most recent investment, which

Local Ai

KernelSight-LM: A Kernel-Level LLM Inference Simulator

arXiv:2606.28565v1 Announce Type: cross Abstract: As large language models (LLMs) move into production serving, practitioners must rapidly evaluate inference performance across diverse hardware, model

local-aiarxiv-cs-ai

Spatial Deconfounder: Interference-Aware Deconfounding for Spatial Causal Inference

model-releasesarxiv-cs-lg

arXiv:2510.08762v2 Announce Type: replace Abstract: Causal inference in spatial domains faces two intertwined challenges: (1) unmeasured spatial factors, such as weather, air pollution, or mobility, t

Tools

Special new @latentspacepod live recording added to the Expo schedule on Thursday, with @swyx and @robertwachen alongside his Inference trac…

Special new @latentspacepod live recording added to the Expo schedule on Thursday, with @swyx and @robertwachen alongside his Inference track appearance! congrats to Etched on a massive milestone! We'

toolsswyx--x

W4A4 Quantization for Inference on Wan2.2-I2V-A14B

arXiv:2606.29337v1 Announce Type: new Abstract: We summarize our submission to Sub-Challenge 1: W4A4 Quantization for Inference (HiF4 / MXFP4) of the ICME 2026 Low-Bit-width Large-Model Quantization C

researcharxiv-cs-cv

A Survey on Federated Causal Discovery and Inference

arXiv:2606.23741v1 Announce Type: cross Abstract: Causal reasoning, which encompasses the discovery of causal structures and the inference of causal effects, is fundamental to data-driven decision mak

Tools

Achieve state-of-the-art inference latencies with speculative decoding

Speculative decoding is a technique for reducing inference latency in large language models by using a smaller, faster model to generate candidate tokens that are then verified by a larger model in pa

toolsmodal-blog

CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference

arXiv:2606.24467v1 Announce Type: new Abstract: Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiti

Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study

arXiv:2606.21428v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) language models are often described as ideal for resource-constrained inference. Each token activates only a small su

Enabling Robust Cloth Manipulation via Inference-Time Simulator-in-the-Loop Refinement

arXiv:2606.24552v1 Announce Type: new Abstract: Simulator-in-the-loop optimization offers a promising inference-time mechanism for robot manipulation. It uses a physical simulator as a backend rollout

safetyarxiv-cs-ro

Hardware-Oriented Inference Complexity of Kolmogorov-Arnold Networks

arXiv:2604.03345v2 Announce Type: replace Abstract: Kolmogorov-Arnold Networks (KANs) have recently emerged as a powerful architecture for various machine learning applications. However, their unique

hardwarearxiv-cs-lg

MGI: Member vs Generated Inference

arXiv:2606.23872v1 Announce Type: cross Abstract: As generative models increasingly produce samples that are indistinguishable from human-created content, it becomes difficult to determine whether a g

OpenAI and Broadcom unveil LLM-optimized inference chip

OpenAI and Broadcom announced a collaboration on a specialized inference chip designed to optimize the performance and efficiency of large language model deployments. The chip, referred to as 'Jalapen

hardwareopenai

Upbound open-sources Modelplane to optimize inference clusters

model-releasessiliconangle

Upbound Inc. today released Modelplane, a new open-source tool for managing artificial intelligence inference clusters. San Francisco-based Upbound is backed by $69 million from Alphabet Inc.’s GV fun

Boost Inference Performance up to 15x on NVIDIA Blackwell Using DFlash Speculative Decoding

DFlash is an open source block diffusion model for speculative decoding that significantly accelerates LLM inference on NVIDIA Blackwell GPUs by drafting entire token blocks in parallel and verifying

hardwarenvidia-developer

Efficient Network Inference via Hardware-Aware Architecture Search, Model Pruning & Quantization

arXiv:2606.23210v1 Announce Type: new Abstract: Embedded global navigation satellite system (GNSS) interference monitoring requires fast and memory-efficient inference to process large volumes of raw

researcharxiv-cs-lg

Tutorials

MAVRL: Learning Reward Functions from Multiple Feedback Types with Amortized Variational Inference

arXiv:2602.15206v2 Announce Type: replace Abstract: Reward learning typically relies on a single feedback type or combines multiple feedback types using manually weighted loss terms. Currently, it rem

tutorialsarxiv-cs-lg

PG-MAP: Joint MAP Optimization for Inference-Time Alignment of Diffusion and Flow-Matching Models

arXiv:2606.22958v1 Announce Type: cross Abstract: Inference-time alignment of pretrained text-to-image models is typically performed along a single control axis, such as classifier-free guidance, atte

safetyarxiv-cs-cv