Search: “models”

I've been out of the loop for some time. Is there already an official way to offload ngram to RAM or SSD in something like Unsloth Studio? Interested in running Qwen3.8-Flash-Next on 72GB VRAM, but na

Title says it all. https://github.com/gjabdelnoor/Day1DeepseekV4.1-CPU The goal is pretty simple, I like having infinite slow tokens from the bioinformatics machine in the lab to run overnight or over

In v0.10 we achieved significant speedups over v0.9. The video shows the maximum speedups we measured for specific groups of models. Among LLMs, the biggest gain came from Qwen3 0.6B, which runs over

Anyone else play with vllm-mlx? There was an interesting paper that came out that compared this version of vllm that was originally designed for multi modal inference performance gains, but ended up s

Running the full deepseek-ai/DeepSeek-V4-Flash-Vision-Exp on consumer Ampere — 10-12x RTX 3090, SM86-compatible vLLM build. 285B MoE, FP4 experts + FP8 attention, 157 GB weights. Highlights: - **60+ t

I shared ThoughtDAG here a while ago. I've now recorded how I use it: select a phrase in an answer, explore it on a branch, and connect the parts I want in the next question. I'm the developer. Though

I was curious about how AIs talk to each other and how a conversation changes as it goes on, so I built AI Arena Lite. It connects to Ollama and lets you assign models to different agent roles, follow

https://www.reuters.com/technology/us-accuses-chinese-ai-firms-industrial-scale-theft-ai-technology-2026-09-08/ https://www.nbcnews.com/tech/tech-news/us-accuses-china-ai-developers-deepseek-alibaba-c

This is from the https://huggingface.co/IFM/K2-Horizon-0.9B benchmark image Tried searching for it but couldn't find it, I am guessing is https://huggingface.co/openbmb/MiniCPM5-1B but the numbers do

I only just found ollama, but I ran into a problem described in this conversation from years back: https://www.reddit.com/r/ollama/s/YZ88XkHmIS Has nothing changed? The storage seems a mess. I had to

hey guys. so I should start with, I'm not s programmer at all. but I'm having a great time letting qwen3.8 27b donit for me. 😃 so far I've just been doing Linux scripts and things for my PC. but my ki

I’m currently running llama.cpp on two RTX 2060 12GB cards, so 24GB total VRAM. With Qwen3.8 27B IQ4_XS at 131k context I’m getting around 45 tok/s, which is actually pretty good for this setup. I fou

I was inspired by Bijan Bowen video - Subway FPS https://youtu.be/6kjXzTVmT58?t=1035 Wondered how far I can push Qwen 3.8 27b so I used a plan made by Fable 5.1 DESIGN.md which has 267 KB! ( 26K of de

Gemma 4 12B runs on an RTX PRO 4500 Blackwell. Gemma 4 E2B run on a Jetson Orin NX 16GB; similar performance is expected on a Jetson Orin Nano Super 8GB. Both systems use a reSpeaker Flex 4-mic array

Please, Oh gentle soul, please beautiful soul, please kindhearted soul, with serious hardware and time to kill, please benchmark Unsloth Qwen3.8-27B-Q8_K_XL against Qwen3.8-Next-Flash-Q4_K_XL and prov

Our research team has been exploring an alternative approach to achieving interactivity and better responsiveness with LLM systems. One of the team members wrote up a post about it: https://research.y

I used an LLM to iteratively evolve an optimization algorithm rather than solve the packing directly. Starting from a simple seed solver, the LLM proposes algorithmic changes guided by a scoreboard of

I just tried to run Qwen 3.8 27B 4bit qwant on 5060ti 16GB and got terrible results. With Qwen 3.6 I got something around 15-20 tok/s. Are there any combination of parameters or tweeks how to push the

Was considering choosing a 4xRadeon AI Pro R9700 build for a office AI server. Curious to know what results you guys are getting? Mainly considering DeepSeek V4 flash and Qwen 3.8 Flash with offloadin

Hello, I'm a bit obsessed with cache management on local LLMs. For the last few days I've been working on cache management on vLLM with my 2x DGX Spark and DeepSeek v4 Flash 0731. I felt something was

You need Blender 5.x with MCP extension. I use pi so I also need: pi install npm:pi-mcp-adapter and then: $ cat .mcp.json { 'mcpServers': { 'blender': { 'command': '/home/jacek/.local/bin/uvx', 'args'

Hi everyone I'm running four rtx 4090, 64GB ram, on a threadripper pro motherboard so all PCIe x16 ports, as a homelab machine for coding. I was migrating from vllm + qwen 3.8 27B (fp8+256k kv cache)

Time for another update! We have been busy and managed to improve the gains substantially (mostly from exploring existing llama cpp PRs and adopting relevant things). Among other things the README.md

I have a 5070 Ti and two 5060 Ti (all 16Gb cards). I planned to add another 5070 Ti giving me two pairs of 32Gb each but the NVidia prices have just jumped by 25% where I am and I've found a 3090 Foun

OK, you guys want to start with local vibecoding but don't know how. In the discussions, you recommended that I use Godot. I downloaded it without knowing what it was. I was expecting a library, but i

Used qwen3.8-27b in Opencode to make this silly mini-game because I'm not sober: ``` We are going to play a game, it will be the Wikipedia game. The Wikipedia game has the following rules: You will ha

I was running into a lot of posts that praised both the Fixed template and the Sharp template in comparison to the stock one, so I put them to the test. It's not as extensive as it should be for a pap

I've been using Deepseek V4 Flash 0731 for a few weeks now and while I havent thrown it anything very hard, im quite happy with it. Using through antirez's great ds4 project. They've added support for

Everyone is talking about how the new credit system gives significantly less usage than the old Ollama Pro limits, but there's another part of the pricing that seems a bit crazy to me. For example, I

There is a video stream running on my desktop right now. It has sound, it has never repeated itself, and it will not stop. I point VLC at a local URL and it plays. One RTX 5090 does all of it — no clo

https://preview.redd.it/6e9xb57a42nh1.png?width=787&format=png&auto=webp&s=ffae7996bbf8ab00498cc62c733e7597dc550f24 I'm not sure how many people care about Android Studio, but I think it's cool that G

Two of the ways that exist (im sure there are more) to train a LoRA on H3 well are on two different trainers. Fizgig's is Optimised Likeness Learning: I've found the stable core of H3's identity lives

I've said this in another post, but again started with Claude Code, loved OpenCode, but I felt chained and tied to them both for the longest time (especially CC, just the worst). I felt less productiv

More new massive updates from turboderp: - CPU offload of MoE experts - Qwen-3.8-Flash-Next ngram disk offload - GLM-5.3-Flash - New self-calibrated optimization technique - Countless other optimizati

Kaitchup just posted results of his benchmarks for Qwen3.8 27B for quants from different labs, Q4 to Q1, . All the details are hidden behind the paywall, but high level result is visible and looks lik

Hello! I've just uploaded a recent checkpoint of my model trained from scratch: https://huggingface.co/piotr-ai/polanka_3.7b_exp_wip_260901 It was pre-trained, mid-trained, and fine-tuned on a single

I only started using some vLLM forks recently in a 4 x 48GB 4090 system. DS4F - ~5000pp/180tg (DSpark) Qwen3.8 Flash next - ~7500pp/135tg (MTP) This is amazing, like having the API in my house. But it

I am absolutely blown away. Yes my setup is crap but the fact that it managed to do this in a single take is unbelievable (and I'm a developer). Hardware used: - Windows PC with 4070ti (12GB VRAM, 32G

if you are on Qwen Flash Next make sure to update your build often https://preview.redd.it/zoq3j9gnjvmh1.jpg?width=900&format=pjpg&auto=webp&s=bb21bc405696df94fd5ce38c22f4ca6dc596135f merged already (

I posted a new gfx906 based llama.cpp fork a few days ago. One of the main points of critique was that i did not provide sufficient numbers for the gains to be achieved. -- TL;DR: After switching our

When using kvarn quants at low context depth tg speed is similar to llama.cpp on and equivalent qx_x quant. However, as context depth grows kvarn tanks your tg speed. This fork optimises kvarn to have

TL;DR - yeah it works, no wild skills or patching required. 37-50 tg/s when generating text, spiking over 60tg/s when writing code and mtp hits hard. Your results can be better if you have a better mo

With b10726, the default --lazy-mode change keeps the 51B-parameter PLE n-gram embedding table of Qwen 3.8 Flash Next on disk: it is mmap'd and its rows are read on demand during inference, even with

Hi All, I am currently contemplating an upgrade from my 5060ti 16gb. I am getting ~40t/s with 130k context on Qwen 3.8 IQ3_S HF quant. I am running llama.cpp on linux. Objective is to increase context

Leo @Synthwavedd 🚨 SCOOP: Astra recently graduated from the dogfood stage, and is now being made available to select OpenAI partners under the name 'ultima-alpha' If feedback over this weekend is posi

I’ve been experimenting with MiniMax H3 and finally have video generation working in TensorSharp. TensorSharp started primarily as a local GGUF/LLM inference engine, so getting a video-generation pipe