Knowledge catalogue
arXiv:2609.17863v1 Announce Type: new Abstract: LLM inference optimizations report speedups on different models, GPUs, prompts, and quality metrics, making them hard to compare or combine. We build a
arXiv:2609.17008v1 Announce Type: new Abstract: Large language model (LLM) inference is often constrained by both computation and memory, especially in offloading-based deployments where model weights
Disclaimer: no AI was used whatsoever to write this post Cautionary tale about chasing cheap tokens. exposé: https://kendell.dev/blog/crofaifalse/ reaction by nahcrof, announcing the shutdown of the s
arXiv:2609.14144v1 Announce Type: new Abstract: A transformer language model is trained to respond to any prompt, but each deployment asks only a narrow range of questions: a support assistant sees de
arXiv:2609.14850v1 Announce Type: new Abstract: Large language models (LLMs) demonstrate impressive capabilities, but their deployment presents significant efficiency challenges. Autoregressive decodi
arXiv:2609.12448v1 Announce Type: new Abstract: Sensitive attributes such as age, income, and occupation can be inferred from user-generated content by aggregating indirect cues across many ordinary p
**Summary for AI knowledge base** Vera Rubin NVL72 is a co‑designed platform combining the Rubin GPU, Vera CPU, NVLink 6 Switch, ConnectX‑9, BlueField‑4, and Spectrum‑6, targeting agentic inference wo
Spoke with many providers who lurk in this sub, plus met folks who work in inference. For a 10k monthly revenue, a provider was able to retain only 200 dollars in profit due to GPU costs. Their custom
arXiv:2609.08082v1 Announce Type: new Abstract: Preference-based fine-tuning methods such as RLHF and DPO require substantial compute and large preference datasets. They also need direct access to the
arXiv:2609.09981v1 Announce Type: cross Abstract: We study offline inference for the optimal value in reinforcement learning. Two new nuisances are derived as fixed points of a self-induced Bellman eq
Amazon SageMaker HyperPod now supports model caching for inference, which pre-loads model weights and container images onto cluster nodes so pods read from local NVMe storage instead of downloading ov
Ari Levy / CNBC: Qualcomm signs a deal with Amazon to provide “multiple generations of customized silicon” to help build AWS' AI infrastructure for inference; QCOM jumps ~5% — Qualcomm shares rose 4%
arXiv:2609.05073v1 Announce Type: new Abstract: Conformal counterfactual inference enables network operators to use logged telemetry to reliably answer 'what-if' questions about network operation. The
arXiv:2609.05262v1 Announce Type: new Abstract: Computational phylogenetics has become an essential tool in historical linguistics, yet its application at a global scale remains constrained by two fac
arXiv:2609.01679v1 Announce Type: new Abstract: The ability of AI systems to improve their behavior during deployment is becoming increasingly important. As inference moves beyond the static execution
arXiv:2609.02160v1 Announce Type: cross Abstract: Diffusion models achieve high sample quality but remain expensive at inference time because sampling requires many sequential neural function evaluati
Australian teams can now access OpenAI GPT-5.6 Sol, Terra, and Luna models on Amazon Bedrock with global cross-Region inference from the Asia Pacific (Sydney) and Asia Pacific (Melbourne) Regions. Thi
arXiv:2602.03708v3 Announce Type: replace Abstract: Large Language Models (LLMs) achieve strong performance across many tasks but suffer from high inference latency due to autoregressive decoding. The
Speculative decoding speeds LLM inference by letting a lightweight draft model propose multiple tokens that the larger target model verifies in parallel, reducing decoding iterations while preserving
arXiv:2609.00407v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models enable efficient scaling of large language model (LLM) inference but suffer from substantial data-movement overhead wh
arXiv:2609.01024v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) architectures scale Large Language Model (LLM) capacity efficiently by activating a sparse subset of experts per token. However
arXiv:2609.01232v1 Announce Type: cross Abstract: Vision Transformers (ViTs) are increasingly used in split-inference systems, where edge devices transmit intermediate token representations to a remot
Hi, everyone, I’ve just released VoxGen, a lightweight native inference engine for VoxCPM2, written in Rust and using Vulkan compute instead of Python/PyTorch/CUDA. Why VoxGen? The main reason I start
arXiv:2608.30295v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated strong capabilities in handling long-context tasks, but processing such long contexts remains challengi
arXiv:2512.16391v2 Announce Type: replace-cross Abstract: Attention is the dominant source of latency during long-context LLM inference, an increasingly popular workload with reasoning models and RAG.
arXiv:2608.28327v1 Announce Type: cross Abstract: Practitioners defend large language models (LLMs) by stacking defenses, assuming the layers compound. A stack is an ensemble, and ensembles compound o
arXiv:2608.26324v1 Announce Type: new Abstract: Best-of-N (BoN) sampling is the simplest and most widely deployed inference-time alignment strategy, but it suffers from two distinct problems: reward h
Learn how Salesforce used Amazon SageMaker AI Inference Component placement (the SchedulingConfig parameter) to distribute model copies across multiple Availability Zones, meeting their Multi-AZ high
Together AI announced a broadcast titled “Serving Kimi K3 at scale: An inference deep dive” scheduled for 7 pm on August 26, 2026 (UTC‑adjusted time shown via X’s timestamp formatting). The streamed s
arXiv:2608.22115v1 Announce Type: new Abstract: Collaborative multimodal inference improves edge perception by combining observations from distributed sensing devices, but transmitting high-dimensiona
arXiv:2509.04719v3 Announce Type: replace-cross Abstract: The widespread adoption of diffusion models for image generation necessitates efficient parallel inference to manage their substantial computa
arXiv:2608.22179v1 Announce Type: cross Abstract: Membership inference asks whether a text was used to train a language model, whereas AI-generated text detection asks whether it was generated by a la
arXiv:2608.21070v1 Announce Type: cross Abstract: Inferring continuous system evolution from sparse temporal snapshots is a key challenge in generative modeling and single-cell omics. While Optimal Tr
arXiv:2608.20123v1 Announce Type: cross Abstract: We study inference-time control for text generation in discrete diffusion language models, where the goal is to steer sampling toward sequence-level r
arXiv:2608.18374v1 Announce Type: cross Abstract: We address two open questions in streaming PCA via Oja's algorithm: sharp operator-norm convergence for general rank under sub-Gaussian data, and dist
arXiv:2608.19115v1 Announce Type: new Abstract: Modern pretrained encoders make representations from heterogeneous views increasingly reusable, but the procedure that determines view utility and combi
arXiv:2608.18884v1 Announce Type: new Abstract: Reinforcement-learning training of reasoning LLMs (e.g., GRPO) is expensive and requires a controllable environment, committing every contribution to a
Counterpoint: “If alignment issues are becoming big enough that OpenAI is willing to commit 20% of research inference compute to chain-of-thought monitoring [suggesting] that suggests that alignment i
Robbie Whelan / Wall Street Journal: AI inference chip startup Etched raised 700M led by its new server rack customer Jane Street at a 21B valuation, up from 10.3B after raising 300M in July — Etched,
arXiv:2608.15127v1 Announce Type: cross Abstract: Agentic applications are shifting AI serving from isolated model inference to long-running workloads in which LLMs coordinate tools, environments, and
arXiv:2608.15402v1 Announce Type: new Abstract: Generative model alignment has received broad interest, and significant progress has been made in supervised fine-tuning and inference-time computation.
Qwen3.8-Max is live on DigitalOcean Serverless Inference. Launching side by side with DigitalOcean as our Day 0 launch partner. Big model. Smooth sailing. Now on DigitalOcean.🌊🏄♀️ @digitalocean Now a
arXiv:2608.13076v1 Announce Type: new Abstract: Large Language Models (LLMs) have achieved remarkable success in natural language understanding and generation, but their deployment is constrained by h
arXiv:2608.11840v1 Announce Type: cross Abstract: Growing demand for artificial intelligence (AI) inference services requires scalable infrastructure, yet centralized serving costs rise with demand. W
arXiv:2608.10506v1 Announce Type: cross Abstract: Accurate pre-deployment estimation of CNN inference cost--energy, latency, and peak memory--is increasingly critical as models are deployed on resourc
arXiv:2601.20819v2 Announce Type: replace-cross Abstract: Machine learning predictions are increasingly used to supplement incomplete or costly-to-measure outcomes in fields such as biomedical researc
arXiv:2608.09444v1 Announce Type: cross Abstract: A main promise of looped language models (LMs) is depth-adaptive inference. By iterating a block of shared layers a variable number of times, the mode
arXiv:2502.11027v5 Announce Type: replace Abstract: Large language model (LLM) scaling inference is key to unlocking greater performance, and leveraging diversity has proven an effective way to enhanc
Together AI is teaming up with @IBM and @nvidia to bring enterprise-grade AI inference to IBM Cloud. A dedicated NVIDIA B300 cluster. Spectrum-X networking. First of its kind on IBM Cloud, powered by
arXiv:2608.06752v1 Announce Type: new Abstract: This paper proposes DKG-MTI, a dual knowledge graph framework for unified multi-task user intent inference from online travel reviews. Existing approach
Together Serverless Inference gives developers a high-throughput, managed production path for running Muse Glimmer across agentic and multimodal workloads. Start building: https://www.together.ai/mode
arXiv:2006.04156v2 Announce Type: replace Abstract: Our inferences in the real world are rarely naive - we acquire experiences through our lifetime that can help us more quickly understand the structu
Congrats to @mattrubens and the Roomote team on the launch. Builders can use Together AI as an inference provider in Roomote and assign different open models to coding, planning, vision, and review ac
arXiv:2608.04255v1 Announce Type: cross Abstract: Graph inference over relational data can expose sensitive edge information, and this risk becomes more severe in dynamic graphs, where repeated model
99.9% uptime changes what your inference architecture has to survive. At Together AI, it means multi-data-center deployment, live traffic across both facilities, and enough capacity to absorb a full d
.@Kimi_Moonshot benchmarked K3 endpoints across major inference providers. Together AI leads or ties for #1 on 3 of 4 benchmarks: OCRBench, MMMU Pro Vision, and DeepSWE. Open models like Kimi K3 have
arXiv:2608.01536v1 Announce Type: cross Abstract: Large Language Models (LLMs) increasingly rely on sparsity to reduce inference cost, but most prior work targets a single sparsity source-either weigh
arXiv:2608.00577v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models have become a dominant architecture for large-scale AI services, yet deploying them over geo-distributed heterogeneous
Disclosure: I’m the author and maintainer of QuarkStar. I built QuarkStar, a small native inference engine inspired by Antirez’s DwarfStar. QuarkStar currently supports: Qwen3.6-35B-A3B, using the sam
arXiv:2407.20432v3 Announce Type: replace Abstract: Bayesian inference methods such as Markov Chain Monte Carlo (MCMC) typically require repeated computations of the likelihood function, but in some s