All entries

Results up front: I had to prioritize prefill or token generation - there was no happy medium. Using UD-Q4_K_XL, q8 kv cache, and 96k max context: focus on generation (MTP = 2): 350 pp and 36 tg @ 12k

Yesterday I downloaded Qwen3.8 as MLX from the Ollama library from the commandline on a MAC. Simple 'Ollama Pull xxx' This is the answer I get when I ask it what model it is... I swear to god this is

Im running a Ryzen 7 5700x, a 3080ti (12GB) with 64GB of RAM. I’m still new to Local AI, and I’ve tried it in the past but none of the previous generations of AI have been good enough for my specific

After about a year of building, testing and constantly finding things to improve, Nyx AI is finally out.available in 240 market in 6 languages Nyx AI is a local first AI workspace for Windows for chat

Two days ago I released a hyper-optimized Qwen3.8-27B inference engine for an RTX 3090 (82 tps single request, 672 peak) - yesterday's update took that to 99 tps single-user / ~1,000 tps at 64 concurr

Here's the DFlash2 announcement, and I was pretty excited for this after trying out DSpark on llama.cpp a few days ago and being somewhat disappointed that it wasn't really working. Anyways, I spent a

Hey everyone, I am trying to make a LLM interface optimized for AI-cowork experience. Heres what I've thought of and tried to fix: 1)You shouldn't have to copy and paste context for the AI model. 2)Yo

I was checking out benchmarks of this model and apparantly better than Qwen3.5 9b reasoning across the bench on artificial analysis. I have used the 9b model for variety of stuff and it has been amazi

Finally llama.cpp now officially supports Ling-3.0! (Starting from build b10472+) If you want to run them locally, bartowski has already released the GGUF imatrix quantizations for both models: - Ling

I have a lot of content in my personal Obsidian vault and would feel much more comfortable using LLMs with an offline model. I've experimented with Ollama and various small 3b/8b local models (given c

I gave three AI models the same brief: build a premium Three.js fragrance launch site from the same Git baseline, independently and with no collaboration. Three very different results. Here’s the full

This took one prompt to build, and another follow up prompt to fix two issues (player got stuck with the bomb and broken enemies path-finding), this is only html, css and js, no external assets, all d

The AI wants to run a higher risk action. Nyx AI stops it here. Nyx AI deliberately invokes a native Windows confirmation before certain higher risk actions can continue. The model cannot approve this

This is invisible with llama.cpp or derivatives, but ninfer helpfully logs the sampler settings on each request and auto-configures the correct ones for the model. Basically, OpenCode will always send

This post was made with AI. I tried to remove as much slop as possible and keep it straight to the point to save your time as I know how annoying AI slop posts can be, but I still wanted to retain all

I've been experimenting with Qwen3.8-27B using DeepSeek Harness. It's a monster at long-horizon tasks, and the results were pretty wild. DeepSeek Harness ran on my Windows PC and connected over LAN to

Hey Guys, What amazing weeks it has been for open source releases. I was really impresssed by DS flash final checkpoint and i have been playing around with it until qwen 3.8 released. I checked the be

It's a small experiment of mine to check thinking effort on Qwen and I do have to say xhigh does overthink but I'm not sure if it's bad because the result is rather amazing. Although the prompt was ve

Qwen released the 2.4T Max weights and I was curious how well it can re-create COD in one prompt I ran the model on a rented B200 cluster and used roughly 1.1M output tokens over a 5 hour time span Re

When Qwen3.8-27B dropped on Thursday the obvious question came up for us : does a 27B dense model actually fit on the 24GB Mac Mini machines? Ran it properly over the weekend on an M4 Pro (24GB unifie

I managed to run the 143–144 GiB DeepSeek-V4-Flash-0731 UD-Q4_K_XL GGUF on four RTX 3060 12GB cards while keeping a 360k–376k context window. Hardware: CPU: Intel Core i9-10920X, 12C/24T RAM: 128 GB D

i was having ollama cloud in last two month and it was great with generous amount of ds v4 pro and glm5.2 usage and actually i built so much with it without worrying about costs or limits. and i was u

Overview UI-Mate-27B is an open-weight foundation GUI agent for long-horizon work across applications and operating systems. It observes live screenshots, reasons over the visible state, and produces

Hi everyone, I use AI agents to code and manage my personal files heavily. However, I am worried that the agents may accidentally delete some important files on my machine, outside of the defined scop

Theoretical use case: I have a compressed version of wikipedia, and some function or tool fooRAG. Then I set my system prompt (or make a skill, or hook, etc.) to include 'You can search wikipedia with

Qwen 27b Q3_K_M 2x rx 580 8gb (~50 each in my country, edge cases 60 per gpu) gives us 16gb vram We used it on an old already existing ddr3 motherboard with 2 gpu slots(you can buy it ror around 200$

Following up on my previous post about my budget server setup (Intel N100 + RTX 5060 Ti 16GB), a few of you asked for a deeper dive into my actual inference config and real-world agentic performance.

Hello all, I'm fairly new to the whole Llm space and could really use some guidance from people who've done similar work. Im building a pipeline to extract structured data from large-batch of invoices

If you're running Ollama, llama.cpp, or vLLM behind nginx or HAProxy for more than a single user, you've probably hit this: nginx treats a 10-token prompt and a 10k-token prompt as identical 'one requ

Guys, could someone smarter than me explain what makes Deepseek Harness so efficient? I run it with local Qwen 3.8 (Q6). I tried Opencode/Openchamber (my favourite so far), Pi agent and Hermes. New Qw

Note: I use a Mac Studio M3U with 512 GB RAM, so this is not for everyone. I have been using antirez/ds4 with DS4 Flash for a few weeks now. Top quality, I am really impressed. This combination just d

EXL3 is an alternative to llama.cpp. And while there is extensive tooling for llama.cpp, EXL3's primary deployment (TabbyAPI), has a OpenAI compatible API so it shouldn't matter. Why won't this tool m

Do you think today’s GPU-poor systems running Qwen 3.5 9B can outperform the ones we had with ChatGPT-4o? I mean, when 4o disappeared, people felt like they’d lost a great model, and now Qwen 3.5 9B Q

Trying to get a feel for where I stand. If you can list your relevant hardware and model used, that would be awesome. Here's mine: Model: Qwen3.8-27B-heretic-ara, Q5_K_M GGUF T/s: ~30-32 t/sec (I thin

Original Article - https://x.com/p_nawrot/status/2089315591010079034 I've spent the last few years working on efficient attention and KV Cache Compression. I've read many papers, dug deep into referen

I created 3 SVG prompts, each one rather hard. Perspective: A animated drone view perspective on a park. Beauty: A beach scene with an evil cat Composition: An AGI breaking out of a virtual sandbox pr

I'm back. Yesterday I released the first version of hyper-optimized Qwen3.8-27B inference engine for a RTX 3090, reaching 82 tps on single request and 672 peak. Over the last 24 hours I've been explor

Support for the new ling 3.0 models has been merged into llama.cpp: https://github.com/ggml-org/llama.cpp/pull/26608#event-29549472828 Ling tiny 8b1b - https://huggingface.co/inclusionAI/Ling-3.0-tiny

In medium reasoning mode, it both scores higher than the 3.6 version, AND is very much more efficient (almost half requests needed, and a third less tokens generated) - at DeepSeek v4 Flash 3107 MXFP4

Hi all! I finally just got around to testing out Qwen 3.8 27b. I'm using Unsloth's UD-Q8_K_XL quant as a sit-in replacement to Qwen 3.6 27b, same quant size. Wow -- this thing isn't messing around. I

for those looking for something small AND powerful, there is a new 1B (they claim, it looks more like 1.7B ...) model that claims to beat qwen 3.5 0.8B & 2B and gemma 4 E2B on a range of benchmarks. t

Now that OpenCode plan isn't really worth it anymore, I'm looking for other options. I'm trying to figure out the Ollama Cloud subscription, specifically the $20 tier, because that's my budget for mon

I’m trying to compare Ollama Pro with OpenCode Go, specifically in terms of how much real-world usage each plan provides. Ollama Pro costs 20/month, while OpenCode Go costs approximately 10/month. Bas

I run locally Ollama + Qwen (tried multiples versions) and I connected it with MCP into my local SQLite database. When I ask a question, 90% of times I see it thinking and just stop and never answer t

pagedMark removes AI provenance from content you generated yourself. Two different things, and it is worth separating them. The first is metadata: C2PA Content Credentials, EXIF, XMP, IPTC, the genera

We introduce Mobius-v0, an architecture that comprises a globally shared Memory (FFN) that stores knowledge vectors and multiple Reasoners (Self-Attn) that iteratively achieve compositional reasoning.