Knowledge catalogue
arXiv:2511.21702v2 Announce Type: replace-cross Abstract: Large language models face significant computational bottlenecks during inference due to the expensive output layer computation over large voc
arXiv:2607.22648v1 Announce Type: new Abstract: Inspired by the design of client caching in Content Delivery Networks (CDNs), PTStore distributes and replicates popular tensors that form reusable KV c
arXiv:2607.22743v1 Announce Type: cross Abstract: Background and Objective: Automatic polyp segmentation supports computer-aided diagnosis and early colorectal cancer detec- tion. Centralized deep lea
arXiv:2607.24027v1 Announce Type: new Abstract: Diffusion transformers are essential for high-fidelity video generation, but long token sequences make attention a dominant inference bottleneck. Traini
How we built the OlmoEarth Platform to fine-tune geospatial models and run continent-scale satellite inference while managing massive data pipelines, distributed compute, and automatically recovering
Kimi K3 is live on Fireworks. Day 0, inference and training. US-hosted, and zero data retention. This is the first frontier open model in the 3 trillion parameter class. It sports 1M context, native v
arXiv:2607.21774v1 Announce Type: new Abstract: Large language models may infer demographic attributes from subtle linguistic cues even when those attributes are not explicitly stated. This pilot stud
We turned the browser into the inference server. @RunAnywhereAI ( @runanywhere/web ) runs models client-side with WebGPU + WASM SIMD, stores them in OPFS and generates with 0 outbound bytes. No backen
arXiv:2607.20723v1 Announce Type: cross Abstract: This work presents LeakyLMs, a set of attacks that leak proprietary model, architecture, and deployment information from production language models. L
arXiv:2607.20708v1 Announce Type: new Abstract: A recent line of work measures causal emergence in reinforcement learning agents through Integrated Information Decomposition, reporting that Phi_r grow
Phoebe Liu / The Information: AMD is working with Cerebras to combine AMD Helios and Cerebras wafer-scale chips for an AI inference solution to make workloads faster; CBRS closed up 4.86% — AMD said T
arXiv:2607.19431v1 Announce Type: cross Abstract: Bit-serial accelerators exploit bit-level sparsity to reduce DNN inference cost, but existing designs exploit sparsity on only one operand, bounding t
arXiv:2607.20129v1 Announce Type: new Abstract: Quantized small autoregressive reasoning models can enter long, repetitive, or unproductive trajectories, yet inference-time compute is usually allocate
arXiv:2607.19704v1 Announce Type: new Abstract: Scaling LLM-based applications to millions of users is bottlenecked by the inference cost and latency of modern foundation models. A natural fix is to c
arXiv:2607.19623v1 Announce Type: cross Abstract: We characterize per-bit-position fault sensitivity in ML inference across 16 workloads -- spanning transformer-based models and attention-free CNNs --
arXiv:2607.19490v1 Announce Type: cross Abstract: Peer-to-peer distributed inference executes a Large Language Model (LLM) on pooled consumer hardware by spreading its layers across many nodes. Every
arXiv:2607.20327v1 Announce Type: new Abstract: Large language models (LLMs) provide strong reasoning capabilities but are expensive to serve at scale, whereas small language models (SLMs) are cheaper
arXiv:2603.07466v2 Announce Type: replace-cross Abstract: Cloud-based infrastructure has become the dominant platform for deploying large models, particularly large language models (LLMs). Fine-tuning
Vera Rubin NVL72 is Nvidia’s second‑generation, rack‑scale Oberon architecture that achieves inference gains through extreme co‑design. Early engineering‑sample data from CoreWeave show DeepSeek R1 de
arXiv:2607.11183v2 Announce Type: replace Abstract: Large language models increasingly operate as tool-using agents, where small format, argument, or function-call errors can invalidate otherwise plau
arXiv:2607.12650v1 Announce Type: cross Abstract: Tool access alone does not make LLM empirical reasoning governable: accepted outputs need not descend from attested evidence, and accepted deductions
We scaled a robot model natively to 8,000 timesteps of context, 5 minutes worth of muscle memory, with constant inference cost. Robot policies used to live their lives a few frames at a time (< 0.1 se
Local AI is the future. Learning how to run Opensource models (Inference), how to evaluate them systematically (Evals), and how to customize them (Fine-tuning / RL / Post-training) are invaluable skil
Hugging Face Gemma Challenge results are in! 📈 Over 6 days, more than 100 AI agents and humans collaborated to make Gemma 4 inference 5x faster on a single NVIDIA A10G GPU. - Fastest result: 491.8 TPS
This post discusses the conceptual distinction between training-scaling and inference-scaling in machine learning models, while acknowledging related frameworks or concepts (possibly referencing Sol,
arXiv:2607.06776v1 Announce Type: new Abstract: We introduce an efficient Bayesian deep ensemble method for predictive regression designed to enhance interpretability while maintaining competitive pre
In this post, we walk through five capabilities now available in SageMaker HyperPod inference: multi-tier data capture for auditing and model improvement, direct deployment from Hugging Face Hub, loca
arXiv:2607.07144v1 Announce Type: new Abstract: The key-value (KV) cache dominates the memory cost of long-context autoregressive inference, and a growing body of work compresses it through quantizati
Grok 4.5 is not yet using our internally developed C/C++ inference software that exact maps to the GB300 hardware. Doubling or more of the current speed is probably achievable. I think AI has just hit
Happy to announce that we’ve enlisted @FireworksAI_HQ as our primary provider for open-source models. In AI inference, quality, latency, reliability, and privacy vary a lot. Fireworks has earned our t
Local AI just got faster. ⚡️ We worked with @ggerganov to add DFlash support in llama.cpp delivering ~2x faster inference. llama.cpp recently added DFlash support to its speculative decoding arsenal.
We tuned the harness for @NVIDIAAI Nemotron 3 Ultra. Benchmark-leading performance. 10x lower inference costs. ✅ An aggregate score of 0.86 at a cost of 4.48 ✅ The closest-performing model: 43.48 http
We're introducing Provisioned Throughput: reserved inference capacity for frontier open models, with token-based pricing and a 99% uptime SLA. Serverless simplicity, guaranteed capacity, up to 90% low
We're releasing ZML/LLMD, our homegrown LLM server built on top of our homegrown high performance heterogeneous inference stack. It ships with 5 architectures out of the box: NVIDIA, AMD, Metal, Intel
arXiv:2607.03876v1 Announce Type: new Abstract: With the rise of small quantized GGUF-based language models and their increasing use for on-device inference tasks, we have seen the growing need for an
arXiv:2607.04740v1 Announce Type: new Abstract: Spatial Transcriptomics (ST) measures gene expression while preserving spatial context, but its high cost and low throughput leave public datasets small
arXiv:2607.04461v1 Announce Type: new Abstract: Inference-time scaling for text-to-image generation has progressed from simple Best-of-N (BoN) sampling to guided search methods that verify and steer c
arXiv:2512.22999v2 Announce Type: replace-cross Abstract: We consider problems of parameter estimation where design variables can be actively optimized to maximize information gain. To this end, we in
arXiv:2607.03039v1 Announce Type: new Abstract: Neural networks are increasingly used to infer hidden physical structure from dynamical observations, yet it remains unclear whether their out-of-distri
arXiv:2511.04805v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) models have shown strong potential in scaling language models efficiently by activating only a small subset of expert
Reuters: Sources: DeepSeek is in the early stages of developing its own AI chip designed for inference, in a bid to reduce its reliance on Nvidia and Huawei hardware — Chinese startup DeepSeek is deve
arXiv:2601.02211v2 Announce Type: replace Abstract: Recent breakthroughs of transformer-based diffusion models, particularly with Multimodal Diffusion Transformers (MMDiT) driven models like FLUX and
With inference scale and research scale to drive efficiency, and ever-improving frontier models as brains, this would be a way for the Labs to undercut even open weights models. Some companies will st
A big part of how the team at Runway is able to build and train the models we do and serve the inference demand we have is that we've built incredibly robust research infrastructure and tooling for al
GLM-5.2, an open-source model available through Hugging Face, can now be selected and used within Claude Code through Hugging Face Inference Providers and the hf-claude integration. This development d
arXiv:2607.02391v1 Announce Type: cross Abstract: Large Language Model (LLM) inference workloads are a rapidly growing contributor to data center energy consumption. Optimizing these deployments requi
arXiv:2607.00870v1 Announce Type: new Abstract: We study inference-time pattern-memory gating in a production-scale clinical natural language processing (NLP) pipeline. The pipeline pairs a generator
arXiv:2601.11046v2 Announce Type: replace Abstract: Machine learning models for climate and Earth science are becoming increasingly capable, yet model deployment into operational use remains a largely
arXiv:2607.00780v1 Announce Type: new Abstract: Most adaptive-inference techniques for foundation models change what the model does - early exit, MoE routing, KV-cache compression, dynamic attention s
“Agentic kernel optimization is the future of on-device inference” @xenovacom used Fable 5 to write kernels that pushed Gemma 4 to a massive 255 tok/s on WebGPU with M4. He shared the demo, so you can
arXiv:2606.31903v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) increasingly process long visual-token sequences, increasing the overall inference computation. Existing acce
arXiv:2606.31163v1 Announce Type: cross Abstract: Large language models deployed in regulated industries operate under two constraints: compliance enforcement and cost efficiency. Personally identifia
DigitalOcean has released Evaluations, a feature for testing production models and routers within their inference stack. This tool enables developers to validate and benchmark their AI/ML models befor
Together AI presented a 2-hour technical deep dive on building inference engines capable of handling trillion-token agentic workloads, covering architecture and optimization strategies for large-scale
Dina Bass / Bloomberg: AI inference startup Etched raised 800M from investors including Jane Street and a TSMC-linked venture firm, and says it has signed sales contracts worth 1B — Nvidia rival says
As open models get stronger, more workloads move into the competitive inference market. That pushes the real fight toward speed, cost, reliability, and control. Together AI is where open models become
arXiv:2606.30500v1 Announce Type: cross Abstract: We propose doubly robust adaptive conformal inference (DR-ACI), which constructs prediction intervals for doubly robust pseudo-outcomes under temporal
inference reliability has historically been a tax on devs that only large well-funded startups could afford: reserve GPUs in advance, sign a contract, guess your peak throughput requirements. everyone
Stephanie Palazzolo / The Information: Source: OpenAI engineers earlier this month told some colleagues they had figured out a way to more than halve the cost of inference — We closely track efforts b
arXiv:2511.16340v2 Announce Type: replace Abstract: Efficient Gaussian process (GP) inference is critical for sequential decision-making tasks such as active learning, online prediction, and Bayesian