Knowledge catalogue
arXiv:2608.30567v1 Announce Type: new Abstract: We present Turing-20B-A2B, a 20B-parameter Mixture-of-Experts language model that activates approximately 2B parameters per token, designed for long-con
arXiv:2608.28467v1 Announce Type: new Abstract: Persona-based dialogue systems are usually conditioned on speaker biography, but dialogues involve at least two participants, and who has access to whos
I’ve been experimenting with MiniMax H3 and finally have video generation working in TensorSharp. TensorSharp started primarily as a local GGUF/LLM inference engine, so getting a video-generation pipe
arXiv:2608.24938v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models scale capacity for strong quality while keeping per-token compute bounded through sparse expert activation. Yet low-late
Amazon Bedrock now supports the OpenAI GPT-5.6 models, Terra and Luna, in India with India geographic cross-Region inference. If you have local data processing requirements, you can now use these mode
arXiv:2608.21621v1 Announce Type: new Abstract: In on-device NLP tasks, limited resources of embedded hardware, such as the Raspberry Pi 5, require efficient inference strategies. This paper introduce
Be sure to tune in this Wednesday! Inference Hours is back with another Kimi K3 webinar! On Wednesday, August 26, Zain Hasan sits down with Kevin Cui and Austin Silveria from Together AI's inference t
NVIDIA’s new GPUs in the Vera Rubin and Blackwell families set a new benchmark for agentic‑AI performance per watt on AgentX, an open‑source inference test that models realistic multi‑step, tool‑using
arXiv:2608.20316v1 Announce Type: new Abstract: Heterogeneous AI systems composed of multiple models, architectures, harnesses, or inference-time settings can improve quality and efficiency by routing
arXiv:2608.18656v1 Announce Type: new Abstract: Inference with transformer models on CPUs is increasingly important, especially for Small Language Models (SLMs), where vector architectures are emergin
arXiv:2608.18410v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models process long multimodal token sequences, making inference expensive in both memory and computation. Existing efficie
arXiv:2608.15065v1 Announce Type: new Abstract: Large Reasoning Models produce diverse, sometimes inconsistent answers across repeated queries on the same problem, so multi-sample inference is a prere
Every data team knows the moment. Someone opens a table, sees a column called cust_seg_flg, and has to go ask around to find out what it means, whether it's safe to use, and whether anyone has already
Two days ago I released a hyper-optimized Qwen3.8-27B inference engine for an RTX 3090 (82 tps single request, 672 peak) - yesterday's update took that to 99 tps single-user / ~1,000 tps at 64 concurr
arXiv:2608.15043v1 Announce Type: new Abstract: Video world models are increasingly used as simulators for planning and embodied decision making, yet improving them at inference time introduces a subt
arXiv:2608.14385v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models have been widely adopted in real-time interactive applications such as coding assistants, real-time audio-video intera
arXiv:2608.14205v1 Announce Type: new Abstract: Load imbalance poses a major bottleneck to the efficiency of expert parallelism in distributed inference of Mixture-of-Experts (MoE) models. The most he
Bloomberg reports Stripe agreed to buy model router OpenRouter for over $7B; Stripe has not confirmed the deal.
arXiv:2608.12385v1 Announce Type: new Abstract: As large language models serve more requests, cumulative inference cost is becoming increasingly important relative to one-time training cost. The two i
Running large language model inference at scale forces a KV cache trade-off: oversized GPU instances or slow time-to-first-token. This post builds a tiered KV cache on Amazon SageMaker HyperPod that e
The past week I've been running DSv4 inference on my laptop by keeping everything RAM-resident except the MXFP4-experts (since expert pool is ~147GB and won't fit) TL;DR - read speed is the limiter mo
Modern Large Language Models achieve impressive reasoning capabilities with long Chain of Thoughts, but they incur substantial computational cost during inference, and this motivates techniques to imp
arXiv:2608.05124v1 Announce Type: cross Abstract: Long context reasoning in large language models (LLMs) is usually constrained by the fact that a single inference trajectory has to simultaneously exp
The Amazon SageMaker Python SDK v3 now exposes generative AI inference recommendations in Amazon SageMaker AI directly in your notebook. Benchmark an endpoint, generate data-driven deployment recommen
I'm hoping that one of you guys has been working on an inference engine or has somehow found improvements to running DSV4F on RDNA4 multi-GPU setups. I am currently building a custom inference engine
arXiv:2608.00311v1 Announce Type: new Abstract: Long-context inference with large language models (LLMs) is costly: self-attention during prefill scales quadratically with sequence length, and the key
arXiv:2608.01880v1 Announce Type: new Abstract: World Action Models generate fixed-horizon action chunks through iterative denoising, creating substantial inference latency that can cause pauses, stal
arXiv:2607.27766v1 Announce Type: new Abstract: On-device in-context learning (ICL) relies on pre-inference retrieval to select demonstrations for useful context before downstream model inference. Thi
This post was not written by a clanker. Hey guys, I'm a comp sci major who wanted to introduce a cool project I built for quantizing models to ternary (1.58 bit) with as minimal of loss as possible, a
arXiv:2607.26410v1 Announce Type: new Abstract: We present Voice Memory, a inference-only scheme for agentic speech recognition: at stream time, a frozen corrector reads a single per-domain memory.md
The root cause: request-level engines never see that a series of LLM calls belongs to one longer workflow. ThunderAgent adds that missing view. It treats each agent workflow as a schedulable program,
arXiv:2607.22568v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly deployed on mobile and embedded devices to improve privacy and reduce network latency. Yet on-device infer
arXiv:2607.23335v1 Announce Type: new Abstract: Auxiliary signal pathways in VLMs are routinely fitted with learnable gates so the optimiser can decide how much of the signal to admit. We find that th
arXiv:2607.22352v1 Announce Type: new Abstract: We introduce time-reversed imaging, a new paradigm that infers what just happened in a scene from fading multimodal traces. Instead of extrapolating or
Cuda and Vulkan Benchmark: TensorSharp vs. llama.cpp I would like to share my latest open source local Unsloth (GGUF) LLM inference engine and applications. It supports many models from Unsloth, like
common: infer the speculative type from the draft repo sidecars (#25989) With -hfd pointing to a repo that ships mtp-/dflash-/eagle3- sidecars and no --spec-type given, the draft resolved to a full mo
arXiv:2607.19515v1 Announce Type: cross Abstract: Continuous surveillance video creates a growing storage, transmission, and inference burden for enterprise video analytics systems. While modern codec
arXiv:2607.19358v1 Announce Type: new Abstract: Recent advances in long chain-of-thought reasoning models such as DeepSeek-R1 have led to increasingly longer inference context lengths under the test-t
arXiv:2607.20357v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) have recently demonstrated strong performance across vision-language tasks. However, their high inference cost,
arXiv:2607.18544v1 Announce Type: new Abstract: Machine olfaction for gas unmixing is an underconstrained inverse problem in which gas compositions must be inferred from low-dimensional, delayed, and
arXiv:2607.13960v1 Announce Type: new Abstract: World Action Models (WAMs) improve robot policy learning by jointly modeling actions and future visual observations, using future scene evolution as den
arXiv:2607.12550v1 Announce Type: cross Abstract: The key-value (KV) cache has become the dominant memory cost of transformer inference. It grows with batch size, context length, and depth, and at lon
How should your security team manage shadow AI? Workloads deployed by developers without formal registration can often evade traditional security scanners, because organizations are reluctant to slow
this is huge: best news I've heard all day..nice work, Hugging Face team. Big unlock for open-source AI inference: Hugging Face Transformers models can now run in vLLM at native speed, often matching
B9932 is a continuous build-tagged release from the llama.cpp project , an open-source C/C++ inference engine for running large language models locally. llama.cpp performs inference on various large l
arXiv:1811.05336v2 Announce Type: replace-cross Abstract: Inference for factor models is often hampered by the lack of tractable and accurate variance estimates, which can materially distort downstrea
arXiv:2607.03509v1 Announce Type: new Abstract: Recent progress in large-scale generative models has substantially advanced video generation, yet existing methods remain constrained by a rigid inferen
arXiv:2607.02612v1 Announce Type: cross Abstract: Vision Transformers achieve strong image classification accuracy but process all image regions with nearly the same computation, even when many region
arXiv:2508.05049v1 Announce Type: cross Abstract: AI-powered medical devices have driven the need for real-time, on-device inference such as biomedical image classification. Deployment of deep learnin
b9891 is a build release of llama.cpp, the open-source C/C++ inference engine for running large language models locally. This release enables LLM inference with minimal setup and state-of-the-art perf
llama.cpp b9876 is a build-tagged release from the llama.cpp project , a pure C/C++ implementation of large language model inference . The project enables LLM inference with minimal setup and state-of
B9862 is a continuous build-tagged release from llama.cpp , the open-source C/C++ LLM inference engine. Llama.cpp is the inference engine that powers most of the local-AI ecosystem, including tools li
arXiv:2512.07843v2 Announce Type: replace-cross Abstract: Scaling inference-time computation has enabled Large Language Models (LLMs) to achieve strong reasoning performance, but their inherently sequ
arXiv:2606.30850v1 Announce Type: new Abstract: Large language models (LLMs) are typically deployed in multi-turn conversations, where each turn provides new evidence that should reduce epistemic unce
arXiv:2606.31063v1 Announce Type: cross Abstract: Gaussian process inference is often limited by cubic computational costs, a challenge that becomes more pronounced in spatio-temporal settings where p
arXiv:2506.22075v2 Announce Type: replace Abstract: Reasoning is a hallmark of human intelligence, enabling adaptive decision-making in complex unfamiliar scenarios. In contrast, machine intelligence
B9787 is a build release of llama.cpp, an open-source C/C++ implementation for LLM inference created by Georgi Gerganov. The llama.cpp project enables large language model inference with minimal setup
arXiv:2606.21501v1 Announce Type: new Abstract: Occluded tasks remain a bottleneck in robot manipulation. Existing solutions either deploy additional physical cameras requiring training-inference came
arXiv:2606.11576v1 Announce Type: cross Abstract: Modern Vision-Language Models (VLMs) benefit from chain-of-thought prompting and test-time scaling, but these gains often come with prohibitive infere
arXiv:2606.10357v1 Announce Type: cross Abstract: Cross-domain recommendation is a core problem in content-to-e-commerce platforms. Its objective is to leverage user interactions with content to infer