Search: “context”

Been working on this backend migration for like two weeks and my chat with ChatGPT is one of those endless scrolls. Last Tuesday it confidently told me we'd 'decided' on the wrong database. We had lit

Hi everyone, I’ve been experimenting with local LLMs on my 48 GB RAM MacBook Pro and I’m still trying to figure out what the best setup is. Ollama feels very fast, which I really like. The problem for

Hey community, Jovan from UkisAI here, We're the team behind Swift Qwen3.8 27B, the Qwen model with token usage and overthinking error improvements Our estimate is that we can make a great improvement

I'm trying to find out whether back-and-forth between two different LLMs improves task success beyond strong alternatives under a controlled resource budget — and I'd rather hear that it's already set

I let AI run my phone 60 real tasks, back to back, on the OnePlus I use every day - Best text model still failed 43% of them - Peak chip temp 98.2 C - 69% of the battery gone The benchmark is AndroidL

Salut, je cherche de l'aide pour un projet d'IA locale. J'ai un Dell OptiPlex 7070 Micro sous ubuntu server avec un i5-9500T, 32 Go de RAM et aucun GPU (j'en ajouterai un à l'avenir). Je sais que la m

UPDATE https://github.com/ggml-org/llama.cpp/pull/29009 MERGED More Gen AI coverage and frameworks integrations to minimize code changes New models supported: On CPU: Gemma-3n On CPU, GPU: Kokoro-82M,

Hey Reddit. This is a post asking for advice / user experience. The goal is simple: deploy a small private server for a developer to run a harness that rivals/beats Claude Opus (in perf/intelligence,

I spent the first week of September building a personal assistant over Gemma 4 in Ollama on a 16 GB M1 Pro: Node backend, React web UI, Expo phone client, two-model routing, cross-session memory, RAG

Hey everyone, Jovan from UkisAI here, a small lab building the tech to make tiny frontier LLMs possible (and doing it open-source!) The purpose of this post is simply to thank the community for all th

Anyone else trying out the new Union stealth model on openrouter? It’s pretty good for a free model, and of course I had to run tokenizer tests and training cutoff tests which established it’s the 99%

We've been tuning vLLM/SGLang/llama.cpp setups for a long time and got tired of the profiling part: nsys trace, open the GUI, squint, change a flag, repeat. The profilers assume a human is looking at

A follow-up to my earlier post about small models on phones. Last time it was fixed tasks in a single browser. This is something else. Ministral 3 3B Q4_K_M in llama.cpp on a Samsung Galaxy S21, four

Achieved steady 15 tokens/second output and 100-120 promp processing per second with 12GB GPU (RTX 5070 SFF) with Qwen3.8-Flash-Next-GSQ-RCO-GGUF at 3 bpw (IQ3_XXS which maches BF16 on AIME25). Around

Note - This is translated from the actual blog link right at the bottom. A few days ago, DeepSeek v4.1 was released. It raised the ability of small models to a new level. AI is improving much faster t

I'm currently using Qwen3.5 4B as the brain of my local AI assistant because my hardware is relatively limited. One thing I really like about it is the speed. On my system, I'm getting around 40–50 to

Hi everyone, Inspired by tools like Chessvision.ai, I wanted to take a different architectural approach and build a browser extension (ChessInsights AI) that performs chessboard detection and piece re

I'm starting a PhD where I will be doing Graph ML (somewhere along the lines of graph signal processing/ graph deep learning.) My eventual goal is research scientist at big tech, or whichever company

Hi everybody, we post-trained Qwen 3.8 27B to be more efficient by figuring out which tokens were linked to overthinking and penalizing them without 'attacking' the reasoning length directly then fixe

I finished my home inference server. First I tried Lenovo p620 workstation and while it’s a good value overall it pissed me off with a ton of proprietary Lenovo shit to deal with and I return it in th

It’s always bothered me that after fine-tuning a model for a project, there isn’t a particularly easy way to host it without either running it locally and keeping a GPU on 24/7 or paying for an entire

I've seen a lot of Super Mario showcases, but never a mashup or derivative work. So I took a Super Mario generation prompt from ChopSticksPlease's Post here - and added a Matrix twist - and found myse

I wanted to see how my fully local home voice assistant compared to the latest GPT Live, so I tested it using the same conversation used in their 'Improved Intelligence' demo. In this video they ask t

I've been out of the loop for some time. Is there already an official way to offload ngram to RAM or SSD in something like Unsloth Studio? Interested in running Qwen3.8-Flash-Next on 72GB VRAM, but na

I made this because I was getting genuinely annoyed at trying to have a normal conversation with LLMs. Even with prompting and various tricks, most models I've tried still have this 'AI assistant' vib

I wish I could spend $15k on my own homelab hardware, but no this is for work lol. Like the title says, we're looking to run DSV4 Flash (and similar tier models) locally at good speeds, both for token

Original Source from DeepSeek WeChat Official Account: https://mp.weixin.qq.com/s/qg0NU3NNUbp1co2PdkAPAg Today we're officially releasing the DeepSeek V4.1 Flash model. It is the smallest model in our

so i saw that openui.com released OUI-1, a model fine-tuned on DiffusionGemma. the training dataset uses OpenUI-Lang, a custom DSL (domain-specific language), instead of plain HTML, Markdown, or React

In july I posted here about aletha codex and I got a lot of support, and a decent amount of people started using it. I also got a lot of criticism too, which really helped me funnel this version 1.5 t

The fix: run the server detached/headless instead of keeping it attached to a console window. RTX 5090, ~27B NVFP4 model via ninfer: Terminal focused: 130–200 tok/s Terminal unfocused: 50–60 tok/s Cli

I found a lot of room on the table for these cards so I decided to make a specialized build to squeeze all I could. first The results: qwen 3.8 next Q3_K_XL: 920tk/s pp8192 (2 cards, ram offload), 24/

I was inspired by Bijan Bowen video - Subway FPS https://youtu.be/6kjXzTVmT58?t=1035 Wondered how far I can push Qwen 3.8 27b so I used a plan made by Fable 5.1 DESIGN.md which has 267 KB! ( 26K of de

I’ve been thinking about this after using more agentic/local coding models. A lot of the newer models are surprisingly good at continuing on their own. But sometimes that seems like the problem. If a

Setup, so the question makes sense: my coding agent's memory now lives in a folder of Markdown — entities, episodes, and procedures (workflows with a per-step success/fail record). Nothing needs an ac

What can I do to get more from MTP if anything? It am I sitting at the expected range. Thanks /opt/llama.cpp/build/bin/llama-server --host 10.210.44.126 --port 11434 --model /opt/models/Qwen3.6-35B-A3

I moght get shit on for posting this but, I feel like i don't see this being talked enough and it feels like such a waste of a good piece of software. Exl3 is incredible, albeit only if you have NVIDI

I ran a side-by-side ML text-processing and model-training workflow using Fable 5.1 vs. Astra (both on xhigh), and the results could not have been more different. Warning, long post. TL;DR -- Astra co

I just spent the last 2 hours of my life on a Friday night debugging a strange error in a prod CLI app. EF core was receive a readonlyspan during a Contains query. Normally, this query converted to a

Anyone who has a 3D printer and get use of it finds it incredibly useful for those odd jobs around the house, a missing bracket, a cable router, steam deck holder and so on. In the past if I was missi

A month ago I posted about Gemma sometimes solving the reasoning problems inside my translation data instead of translating them. A few people suggested two very fixes, which is to use a proper transl

Being a person obsessed with testing new models that come out, times are really insane for me. Tested different kinds of TTS and voice cloning models but none of them gets it right in terms of emotion

Ling-3.0-flash-Fin is the first finance-enhanced model in the Ant Ling family. Developed by Ant Group with leading financial institutions and domain experts, it extends Ling 3.0 flash through continue

I've tried both and been having this debate with myself for the last few days, on two Asus Ascent GX10s (effectively the same as 2x DGX Spark): DeepSeek-V4-Flash-0731 (official weights) GLM-5.3-Flash

It started as a fork of llama-swap, but I have been building it out for myself since then as a convenient tool for all my local AI needs, and by now it has drifted far enough to be its own thing. The

A potato can create a very cool RPG in 24 minutes Laptop I3 8gb RAM 0gb VRAM, Windows 11 llama-server.exe --host 0.0.0.0 --port 8080 -m modelsqwen3.6-35b-InstructQwen_Qwen3.6-35B-A3B-IQ2_XXS.gguf -c 8

ik_llama.cpp merged qwen4exp MTP support yesterday (PR #2369, mine, reviewed and tested by four other people on their own hardware). It's on main now, no fork or patch needed. Posting since the last c