4x 6000 Pro for the company
Here is how i serve 40 people (10 of them developers) with local AI services. n8n for automatisation whisper and Vexa AI, a bot joins Teams and makes transcripts, n8n later extracts decisions and todo
Knowledge catalogue
Here is how i serve 40 people (10 of them developers) with local AI services. n8n for automatisation whisper and Vexa AI, a bot joins Teams and makes transcripts, n8n later extracts decisions and todo
Follow-up to last week's post about streaming MoE experts from disk in a browser tab. That ran on hand-written engines for two models. This week the same disk tier sits under llama.cpp's own WebGPU ba
hexagon: extend scalar DMA broadcasting support for binary ops (#30245) tests: cover weighted scalar broadcasts Assisted-by: OpenCode ggml : support extended scalar DMA binary ops Assisted-by: OpenCod
models : remove redundant ggml_cont calls now supported by all unary kernels (#30198) models: remove redundant ggml_cont calls now supported by CUDA unary kernels models: restore cont_2d after sigmoid
ggml-zdnn : introduce Q8_0 mat mul (#29836) ggml-zdnn: initial prototype of q8_0 matmul Signed-off-by: Aaron Teo aaron.teo1@ibm.com docs: update zdnn docs Signed-off-by: Aaron Teo aaron.teo1@ibm.com S
graph : avoid reallocations in the 'no cache' path (#30439) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/55354972 macOS/iOS: macOS Apple Silicon (arm64)
Banger paper from Google DeepMind on co-evolving an agent's harness and its weights. (bookmark it) If you already log your harness changes and results, those logs can become training data for the mode
Dynamic vision, in action Basalt is an inference engine for 3.8 Flash Next, forked from Strata and heavily tuned for Blackwell. It supports vision, and for this to work you need to have the image enco
My project is optimising qwen flash next on Mac Mini m5 64gb through SSD streaming. I recently asked LocalLLama for guidance on how to optimise token output. Today a popular method to do this is a loo
I set up a dedicated workstation for local AI experimentation with various models and tools. But every single one of them wants all the available VRAM. To fix this, I built a request dispatching layer
I built a desktop app harness that keeps local meeting notes and can use them to autocomplete my typing as well as complement my dictation with additional context via reading my screen or using the pr
@cjav_dev @aiDotEngineer @ricmac @latentspacepod our workshop writeup is live now! https://x.com/latentspacepod/status/2109713951604826153?s=20 At AI Engineer New York, Anthropic's @cjav_dev & @Harris
GPT-4 was the undisputed best model in the world for a full year and we all debated whether anyone could catch up or if OpenAI had some secret formula that nobody else knew about. Now there are are 19
Grok @Bot memory transfer NEW GROK BOT! AI MEMORY TRANSFER! Your old AI knows you. Your new one shouldn't have to start from zero. 🧠➡️ Meet Memory Transfer, a @Grok @Bot that brings over what Claude o
Grok @Bot rocks I've been using Grok @Bot as my daily driver for a few weeks now and: - it has replaced 100% of my search-like queries on chatgpt / claude - it has replaced 100% of my Hermes usage & a
So I wanted to try the horsey challenge myself on the latest engine i been testing which is a strata fork by sergqwer. https://github.com/sergqwer/strata-nvfp4. I seen a post earlier and seen i believ
A lot of projects have come up recently consisting of specific hardware config specialized forks of existing conferencing engines, but they all seem to be made using Claude or OpenAI's biggest frontie
Lipwise is a [yet another] dictation app for Windows, macOS and Linux. You hold a hotkey, talk, and it types into whatever app has focus but with a twist. Since most of us that speak to our computers
Look up! Mistral models are in orbit 🌍. Proud to join @orbitworksspace and @LoftOrbital's first-of-its-kind AI satellite constellation with @NVIDIAAIInfra. Just a few weeks ago, @orbitworksspace launc
When we talk about agents, that doesn't necessarily mean we're talking about Python. In today's modern ecosystem, an agent might run as a high-concurrency Go microservice on Cloud Run, or live inside
i'm the maintainer of dodaios (formerly personal jarvis), an open-source ai agent app for the desktop. apache 2.0, runs on windows, macos and linux, no account and no cloud of ours between you and you
Hi guys! I got a problem with one of my V100 SXM2 Modules. In 'nvidia-smi' its shows with 'Pwr:Usage/Cap' as 'N/A / N/A'. When I try to use it i get the following error in dmesg: NVRM: Xid (PCI:0000:0
Hi, I see that vllm released d3 (a family of decision models) and llama.cpp already has support for them. I want to try them out. Are decision models sensitive to quantization? I know that for autocom
I’ve been building SonicBoom, a native tensor runtime written in C++/CUDA with a C ABI and no Python anywhere in the stack. I just tagged v0.0.1, and the first target is local LLM inference on consume
This is today’s edition of The Download, our weekday newsletter that provides a daily dose of what’s going on in the world of technology. Open models have real benefits. Do they outweigh the risks? An
total black eye for Anthropic’s efforts at alignment. Anthropic admits Houthis used Claude AI instead of weapons engineers to make rockets and ballistic missiles that correct themselves after missing
This is an update to my previous post which can be found here: https://www.reddit.com/r/LocalLLaMA/s/n7sgRZZc4z Since my last post I have continued working on the conversion process and also convertin
I came across this blog post from OrcaRouter: https://www.orcarouter.ai/blog/qwen-3-8-next-vs-qwen-3-8 Not only is the blog post nonsensical in some parts, it's also completely without any value. It b
A decade ago, Google released Site Reliability Engineering: How Google Runs Production Systems with O'Reilly Media. What began as a collection of our practices for managing large-scale infrastructure
arXiv:2610.13703v1 Announce Type: new Abstract: The fast pace in which protocols, frameworks, and technologies are being proposed for LLM-based agents results in a complex scenario for anyone aiming t
arXiv:2610.12728v1 Announce Type: new Abstract: Parameter-efficient fine-tuning has hundreds of methods, usually compared by parameter count and benchmark score. We compare them as maps. A pretrained
Disclaimer: I work for NuMind, the company behind this plugin. TLDR: Text/image (simili-)embeddings from the LLM you already have loaded :-) I've been working on making it easier to obtain embeddings
I am an independent researcher. Due to budget constraints, I shifted my 21B MoE pre-training—which I had originally planned to run on 64 H100s—to a single NVIDIA GH200 ($2.29 per hour), and I have com
arXiv:2501.17499v1 Announce Type: cross Abstract: A variety of complex biological, natural and man-made systems exhibit non-Markovian dynamics that can be modeled through fractional order differential
arXiv:2610.13480v1 Announce Type: cross Abstract: Nonlinear model predictive control (NMPC) can be applied to large nonlinear systems with complex objectives while enforcing constraints, but the cost
arXiv:2610.13184v1 Announce Type: new Abstract: While reinforcement learning with verifiable reward (RLVR) enhances reasoning in large language models, its efficacy in long-horizon multimodal tasks re
arXiv:2610.13708v1 Announce Type: new Abstract: Long-context serving runs out of KV memory before the hardware reaches its useful throughput. HiSparse stores history in host memory and fetches only th
arXiv:2610.13361v1 Announce Type: cross Abstract: As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their us
arXiv:2610.13335v1 Announce Type: new Abstract: Tool-using agents increasingly require a common substrate for building, training, evaluating, distilling, and maintaining agentic systems, yet existing
I designed a node to align any pair of images trying to correct the pixel drift on Qwen Image 2.1 https://github.com/zuanfilm/Qalign workflow This node don't need any model or text encoder Qalign is a
arXiv:2610.12494v1 Announce Type: new Abstract: In this article, we propose algebraic integer number formats (AINF) as a framework for highly configurable and accurate compressed computation. A key pr
arXiv:2610.13354v1 Announce Type: new Abstract: The NIST Assembly Task Board #1 remains a challenging benchmark for learning-based methods in robotic manipulation. Over short horizons, it demands prec
arXiv:2610.13627v1 Announce Type: new Abstract: Evaluating large language models (LLMs) on AMLworld's smallest test splits is costly because each of over a million targets requires a separate graph pr
arXiv:2610.13724v1 Announce Type: cross Abstract: Counterfactual inference asks how an observed outcome might have differed under different circumstances, such as whether a patient would have recovere
arXiv:2610.12571v1 Announce Type: new Abstract: Most systems trained by equilibrium propagation or coupled learning are in the same class as multi-layer perceptrons: while they may be universal functi
arXiv:2610.12496v1 Announce Type: cross Abstract: Sequence models with a fixed-size state avoid the quadratic cost of attention, but face a design choice. An input-dependent state transition gives con
Anthropic: Anthropic says new measures like preserved thinking in Fable 5.1 and Opus 5.5 have reduced data harvesting through distillation attacks by about 90% to 99% — In recent months, we've made su
arXiv:2610.13623v1 Announce Type: cross Abstract: Professional agents increasingly work through command-line interfaces and multiple tools over shared workspaces containing files, messages, and busine
At AI Engineer New York, Anthropic's @cjav_dev & @HarrisonStall from @ClaudeDevs drew on lessons from Unix and Linux to explain how agents can tackle bigger projects without overwhelming developers. h
arXiv:2610.13078v1 Announce Type: new Abstract: Safety-aligned large language models (LLMs) remain vulnerable to jailbreaks and can over-refuse benign requests. Understanding these failures requires d
arXiv:2610.13600v1 Announce Type: cross Abstract: Large Audio Language Models (LALMs) achieve strong performance on speech-text tasks, but standard audio encoders produce hundreds of tokens for even s
arXiv:2610.13643v1 Announce Type: new Abstract: Automating biological laboratories requires robots to perform dexterous tool interactions under realistic visual conditions while handling dynamic exper
hexagon: add IM2COL_3D (#30305) hexagon: add IM2COL_3D support Assisted-by: OpenCode hexagon: add DMA paths for IM2COL_3D Assisted-by: OpenCode hexagon: avoid scalar fp16 conversions in IM2COL Assiste
vulkan: fix buffer logging on 32-bit platforms after source split (#30326) Move the vk::Buffer stream operator to the shared header so it is visible to ggml-vulkan-debug.cpp. Website: https://llama.ap
ggml: Change default visibility of symbols to hidden (#30014) See discussion in #29997. This prevents a handful of internal symbols leaking via GGML_API. Website: https://llama.app Attestations: https
vulkan: use ternary expressions to avoid switch lowering for CM2 quant block size (#30164) vulkan: specialize quant block size for CM2 matmul vulkan: use ternary expressions to avoid switch lowering f
sycl: guard DMMV reads at row tails (#29338) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/54988744 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Sil
llama : report compute graph reallocations during inference (#30329) Add an optional ggml_backend_sched_realloc_callback that user code registers with ggml_backend_sched_set_realloc_callback to be not
metal : fix mul_mat_id with duplicate ids (#30373) metal : fix mul_mat_id limits and kernel name metal : fix mul_mat_id with duplicate ids kernel_mul_mm_id_map0 assumed at most one match per token and
tests : link each test against only the libraries it uses (#30299) Give llama_build() and llama_build_and_test() a LINK keyword (default: llama-common) so call sites can declare what a test actually n