Search: “chat”

A community-built, privacy-focused security scanning tool shared on r/ollama that uses four specialized AI agents running locally via Ollama to analyze code or systems for vulnerabilities, aggregating

A Reddit user exploring Ollama Cloud noted that, at the time of their post, only the Kimi model supported image (vision/multimodal) inputs among the available cloud models. Kimi K2.5 is a native multi

So I created this simple test (for myself) to see how good the new models are in analytic output. This is not asking them to draw a SVG (which just tests how much SVG or HTML they saw) but creating a

Defiance puts DavidAU's Turbo-Brilliance 'AI Construct' modes on HauhauCS's uncensored Qwen3.8-27B. You pick a mode (Joker, Dr. House, a 20-agent Einstein panel, a Socratic tutor) per request, and the

Beam is an open model built by Reflection. Beam takes text as input and generates text as output. This release includes the instruction-tuned model in BF16, FP8, and NVFP4 numerics. Beam uses a sparse

https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is a vi

I've been using ChatGPT Voice for years, particularly during my commute to work. Until recently, I've almost exclusively used Standard Voice Mode, but a recent issue has made it practically unusable.

I built BitRun because I wanted to run 1-bit ternary LLMs on ordinary laptops without melting my battery, spinning up a dedicated GPU, or wrestling with complex build environments. Right now, BitRun g

I've been working on an inference engine called jitllm. The idea came from my background in operating systems and JIT compilers: generate kernels for whatever hardware is there, and treat layers a bit

[UPDATE / RETRACTION: The '100% GTO' claim had a prompt-leakage bug] We patched the repo to pass strictly neutral states, here are the real, honest numbers: • Latency: Still ~2ms (ONNX + Absorbing Mar

My project is optimising qwen flash next on Mac Mini m5 64gb through SSD streaming. I recently asked LocalLLama for guidance on how to optimise token output. Today a popular method to do this is a loo

I started Flama in 2018 to take a trained model to a production API with as little ceremony as possible. With 2.0 LLMs became first-class alongside classic models, and together with Miguel Durán-Olive

tl;dr: running Qwen3.8FN Q4 locally is cheaper for me than every cloud model except Luna and the newer Haiku at mid-peak and peak electricity pricing where I live, and becomes the absolute cheapest mo

Quartermaster (my local inference manager: one server that swaps llama.cpp, image, audio and video models in and out of VRAM) can now help you server and manage decision models: Auto-discovered. Drop

Hi, I see that vllm released d3 (a family of decision models) and llama.cpp already has support for them. I want to try them out. Are decision models sensitive to quantization? I know that for autocom

I’ve been building SonicBoom, a native tensor runtime written in C++/CUDA with a C ABI and no Python anywhere in the stack. I just tagged v0.0.1, and the first target is local LLM inference on consume

We're launching Precision Numerics by releasing Boole-1.7B, a 1-bit version of Qwen3-1.7B. 247 MB packed weights: 13.9× smaller than BF16. 85.9% of the BF16 model’s average score across our 10-task ev

My Jetson Orin–optimized engine, little-gemma V1.0, substantially outperforms llama.cpp. Even after exhausting every practical GGUF option, however, it still falls short of the ideal performance level

I run local LLMs pretty much around the clock, but I also enjoy gaming on the same PC. Having models sitting in VRAM while gaming isn't exactly ideal, and constantly unloading and reloading them manua

No more excuses for GPU-poor folks not to start LLMing! Full 27B-class reasoning in ternary transformer weights, for llama.cpp (CUDA, Metal, CPU) ~9.3x smaller than FP16 (ideal) | 98.2% of FP16 intell

Basalt: Blackwell inference engine Hi all! So over the last week I've been working on a Strata fork that's heavily tuned and can achieve throughput up to 2.6x what Strata usually does on the same weig

Spent the past month tweaking and experimenting with many different numbers to achieve 30tps. Hardware: -Rx6700xt 12gb vram (AMD) -2x16 ddr4 3200 ram -r5 5600x -llama.cpp vulkan sdk -window11 (no wsl

Disclosure: I work at H2O.ai. We released H2O-Lightning-4B, an open-weight (Apache-2.0) model for the 'decisions API' style of inference that Jev made popular: you send a state plus typed questions (p

Hi everyone! So far in 2026 I have been iterating and evolving a research system I designed to help me sift through things that LLMs all agree on that may not be correct, and to give me actual confide

All anyone ever talks in this sub is about x new model having x benchmark scores and how well it can code. Or some new tech to increase t/s for qwen. So I wanted to go back to the roots of this sub an

Preamble and disclaimer, Sample size: 8, at any size of form this research is just screwing around being writing it down. This was just a little,(well... big), curiosity test I wanted to run to see wh

The other week I posted about Jev vs. Kev compared and since then, OpenAI released the decisions endpoint, Cloudflare released Clef and many here asked about Laya as well. This time we compared six po

Hi all, having searched google and asked several AIs without success, maybe s.o. can help. I have 2 x dgx spark in a cluster. deepseek-flash is running successful. Now I want to test qwen3.8-flash-nex

Disclosure: This unit was provided free of charge by Orange Pi for testing. No editorial review, no preconditions, no script. All data, bottlenecks, and thermal behavior are reported directly from har

Standard distillation usually means burning weeks of compute and billions of tokens hoping the student model eventually mimics the teacher. We wanted to see what happens if you skip backprop entirely

With llama.cpp using a customized SYCL backend on Intel B70 (32GB), overclocking my DDR5 memory gave modest gains for MoE models which do not fit in VRAM. Both PP and TG increased after overclocking D

On my two DGX Sparks, Swift 1.5 (UkisAI's reasoning-efficient fine-tune of Qwen3.8-Flash-Next) running on veloGB10 (https://github.com/sf-stav/veloGB10, sf-stav's Rust/CUDA engine built only for GB10)

For the last few months, I ran DeepSeek v4.0 flash (NVFP4). First 0731, then visionexp because it was a free improvement. I got around 65 tps decode and almost 2k prefill, and ran 4-5 agents in parall

First of all, thank you for reading. I trained a small MoE model completely from scratch (no external base weights) and wanted to share the results + a couple of lessons. Apex-2 - Architecture: Decode

Reupload: I'm adding this again, since previous post had a totally stupid title which made people assume this app is some kind of cloud/proprietary models UI - this is totally not the case here. In ve

We built an engine, Kyojin, on top of ExLlamaV3 for Strix Halo (gfx1151, ROCm), and packed two 300B-class MoE models so each fits one 128 GB machine. First release, all measured on Ryzen AI Max+ 395.

Somebody just released a fully open-source, open-weights decision model that beats Jev! Just kidding, it's me and I this is my first time releasing a public model, recipe and dataset so I am looking f

Row-Bot 5.0 is a major rebuild of the local-first AI assistant. The NiceGUI interface is gone, replaced by one React app across desktop, browsers, phones and tablets. The biggest change is how work is

Something keeps happening in frontier AI, and the explanation we are given is always one size larger than the event. An OpenAI research model, running in a supposedly offline training environment, hit

I have been building a somewhat unusual local inference machine around two Huawei Atlas 300I Duo cards. They are relatively inexpensive, passive, dual-accelerator PCIe cards with 96 GB of device memor

Hey guys, Looking for an alternative for Claude Cowork: Project Work / Documents Integrations like Notion, Gmail, etc. Tools like Websearch, PDF creation, etc. Came over Eigent (https://github.com/eig

I've run the benchmark on a RTX 3090, 1024 tokens in / 256 out, concurrency 1–32. If the model fits on vRAM (Gemma-4-26B-A4B, byte-identical GGUF on both engines): llama.cpp has 2.2–3.2× the throughpu

A few days ago I released Jeff-Qwen3.5-0.8B, a small 'System 1' model that picks between options you define and returns a calibrated probability for each, in one forward pass. Speed was great on my M4

I put an old RTX 3090 to work as a local coding agent with Qwen3.8-27B, llama.cpp, and OpenCode. Here are the setup and results, including what failed. This is a summary of my own blog post, linked be

Last update: https://www.reddit.com/r/LocalLLaMA/comments/1wu9ksu/update_yandexaliceai_80ba3b_fine_tune_progress/ - basically, an instruct fine tune on the base model using a synthetic distilled data

I run local models through Ollama and kept hitting the same quiet problems, so I wrote a scanner for them. llm-doctor checks your model stores (Ollama, LM Studio, Hugging Face, MLX) for duplicate weig