All entries

Tested on Laptop with RTX 5060 8gb vram and 16gb ram Claude code - autocompact = false Ollama - Flash attention = 1, kv cache = q8_0 Gemma4 12B 64K - https://ollama.com/SetneufPT/Gemma4-12B-IT-QAT_Q4_

Hi everyone, I was interested in learning LoRA fine-tuning, and ended up building CodeFinetuner over the past few months, a full pipeline that fine-tunes a small code autocomplete model (e.g. Qwen2.5-

Hey everyone, If you run local models via Ollama in production or personal projects, you've probably run into the hallucination problem: how do you know when a model is hallucinating without burning e

I wanted to see how my fully local home voice assistant compared to the latest GPT Live, so I tested it using the same conversation used in their 'Improved Intelligence' demo. In this video they ask t

A huge favor to ask llama maintainers - please implement this feature. Even with one 3090 card there will be tangible improvements in decode speed on MOE models with moderate number of active paramete

AI-generated because I’m currently way too high to write this coherently. I have HPPD (Hallucinogen Persisting Perception Disorder), which basically means visual effects from past psychedelic use can

Hello Guys, Just sharing this harness I've created, this runs entirely on chromium based browser. No installation is required, it has its limitations and by no means is supposed to replace better agen

I just watched a YouTube from Luke’s Dev Lab where he literally just plugged a RTX 2000 ADA Into the side of the Zima Board 2’s PCIE socket and it just friggin worked and had great token speed despite

Hey gang, im wondering if you in theory could use ngrams as seen with Qwen 3.8 Flash or DS4.1 in order to dynamically train the model? Normally the ngram embeddings behave similar to a lookup table of

I've been out of the loop for some time. Is there already an official way to offload ngram to RAM or SSD in something like Unsloth Studio? Interested in running Qwen3.8-Flash-Next on 72GB VRAM, but na

I haven't seen this mentioned yet, so I thought it deserves a post. I was trying out OpenWhispr when this model came up as the recommendation. So I don't have personal experience yet, but it's suppose

I made this because I was getting genuinely annoyed at trying to have a normal conversation with LLMs. Even with prompting and various tricks, most models I've tried still have this 'AI assistant' vib

I can't stand kv cache quantization. Even at q8_0, I can feel the difference. But realistically, when running Qwen3.8-27B-UD-Q4_K_XL on my 32 GB GPU, I only have room for ~170k tokens (with mtp and mm

I’ve been building 'Spomin', a router that replaces context with summaries directly in the KV cache. The goal is to keep long running sessions going without repeatedly stopping for full compaction and

Some people says terminal bench reflects model intelligence better than the intelligent index. From the look of it, the ranking does seem to reflect how people feel about the open and closed models. F

That's amazing. I've been using the pay-as-you-go plan, and because DeepSeek 4 Flash wasn't great, I was using GLM 5.3 Flash instead. But now that I'm using the high-performing DeepSeek 4.1 Flash, the

Astra's launch has produced a strange discourse. The reporting that broke the story framed the model's use of recurrent depth primarily as a safety regression, because it means the model reveals less

i've been thinking about this more after trying different agent setups. the model isn't the only thing that determines how well an agent performs. The harness around the model matters a lot too. With

I wish I could spend $15k on my own homelab hardware, but no this is for work lol. Like the title says, we're looking to run DSV4 Flash (and similar tier models) locally at good speeds, both for token

Looking for tools to build/run an engine to do STT-LLM-TTS basically. I know how to run all of them separately, and if need be I can build one myself, but I was curious if there is already an open sou

I'm currently building my system around 3060s, but I might be able to get a 4060 for a nice deal. At first it seemed like a no brainer, but turns out the 4060 has lower memory bandwidth. In a system t

Spoke with many providers who lurk in this sub, plus met folks who work in inference. For a 10k monthly revenue, a provider was able to retain only 200 dollars in profit due to GPU costs. Their custom

Like many of you, I have seen many posts and tweets in the last weeks complaining about Artificial Analysis being 'broken', 'meaningless', and 'bought out.' People who say this have done no research a

OpenAI has decided to fully shut down a protein design project I'm working on for a client. Needless to say, open weight models are the only way forward. submitted by /u/Terminator857 [link] [comments

Title says it all. https://github.com/gjabdelnoor/Day1DeepseekV4.1-CPU The goal is pretty simple, I like having infinite slow tokens from the bioinformatics machine in the lab to run overnight or over

The downside of uncensoring a model is that it is known to potentially damage it, but CyberTiel is an even more capable software engineer than its censored TielCoder base, while allowing offensive sec

Here we go again, DeepSeek is back again with a new model V4-1 Flash A multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters and support for contexts of up to one million tokens Mark

People keep on getting confused about this, so I looked at the safetensors on hf. To be precise, the main model about 551.566B parameters with 40 layers. The FFN experts total to 543.582B parameters,

I like to benchmark new models that come out on motion videos. So here's a test I did for deepseek v4.1 flash. And I have to say flash has probably graduated from being a Luna class model to nearly an

Original Source from DeepSeek WeChat Official Account: https://mp.weixin.qq.com/s/qg0NU3NNUbp1co2PdkAPAg Today we're officially releasing the DeepSeek V4.1 Flash model. It is the smallest model in our

I believe I may currently hold the record for memory constrained inference for Qwen3.8–Flash-Next on Apple Silicon — needing only about 21GB of allocations. Introducing Cherenkov, an inference engine

Hey y'all! We've released a new model in our lineup: GigaChat-3.5 Reasoning. It's a 432B-A28B MoE with Gated DeltaNet for long-context efficiency. We trained domain experts (code, math, general, etc.)

https://preview.redd.it/4c6otbeg1qoh1.png?width=1139&format=png&auto=webp&s=2d127bd3bc83d5e6ba65c3a4bd909529f34e9a3c Total beginner here, i've just downloaded Ollama and Gemma4:26b, but when it answer

So I was playingw ith embedding models I saw that when you upgrade from model A to B, you face a very big backfilling cost Ie, suppose you have a 1b vectors from model A, and then you want to use mode

First off, I'm super thankful for the value Ollama has given me Got me setup with my first local LLM setup (early OpenClaw days amirite!?) Then got me into their 20 plan and what really sold me here w

I’ve been working on perfecting my setup ever since 3.8 hit, but no longer pay for subscriptions. I had a coding interview today and setup openrouter ahead of time as another option. I figured I could

After noticing that it is ranked among MUCH larger frontier models in the EQ-Bench Creative Writing benchmark and the Hemingway-bench, I decided to give it a try and was very impressed. I didn't do ve

Spent about two weeks convinced I had a prompting problem. My local Qwen3 agent would just stop. No error, no crash, finish_reason came back as 'stop' like the model had decided it was done. Except it

are very strict... if 20 calls to kimi are more than my 5 hour limit why even bother? im gonna cancel my sub for sure https://preview.redd.it/gbqkci93gnoh1.png?width=1080&format=png&auto=webp&s=6fac5a

Both good and bad things have come from a subreddit that was lot more niche than for example r/flashlight rapidly transforming into the largest online forum about an increasingly core part of the infr

so i saw that openui.com released OUI-1, a model fine-tuned on DiffusionGemma. the training dataset uses OpenUI-Lang, a custom DSL (domain-specific language), instead of plain HTML, Markdown, or React

Github Repo. Blog post. 💡 TL;DR (from the Github Readme) Spend less without making the agent do less useful work. SoL-Pi is a standalone extension for Pi that packages four reusable efficiency mechani

Part 4 of the same box. Part 1 was 17 -> 25-29 t/s with the expert cache PR, part 2 was 37-41 t/s after switching to UD-Q4_K_XL and stacking MTP on the cache, part 3 was the top-k fallback that was so

In v0.10 we achieved significant speedups over v0.9. The video shows the maximum speedups we measured for specific groups of models. Among LLMs, the biggest gain came from Qwen3 0.6B, which runs over

I am just sharing my config for Qwen 3.8 27b that fits on a 5060TI, what is cool about this is that you can even get vision! and a 85K context (I have 1.5gb of headroom for more context or a better qu

I initially saw CED as just an efficiency improvement, but the more I read about it, the more it feels like an inference architecture leap. The encoder/decoder split has some pretty interesting implic