Search: “inference”

I have been running the same midrange card for a couple of years now, and every model release sends me back to the price sheets. The performance gains are real but they keep landing next to price tags

Full disclaimer: I've leaned heavily on Fable to develop this, but I've tested it thoroughly on my own library for a couple of months before putting it on GitHub. I live in Thailand, and it started as

**DISCLAIMER** THE PREFILLING TPS SHOWN ON THE PHONE IS COMPUTED ONLY FOR THE LAYERS IT HOLDS. ALREADY FIXING IT TO SHOW END-TO-END PREFILL RATE. NUMBERS BELOW ARE ACCURATE FOR E2E PREFILL RATE. Every

I put an old RTX 3090 to work as a local coding agent with Qwen3.8-27B, llama.cpp, and OpenCode. Here are the setup and results, including what failed. This is a summary of my own blog post, linked be

Hi folks, I've spent the last couple of months experimenting with Strix Halo and previously I released a proof of concept I called llama-halo-hybrid. I've continued updating it and it now performs ver

TL;DR: I'm a solo dev who wanted a simple, private way to have local LLMs watch my screen and do simple logging/notifying. After a year of building, I released v3.0.0 and my mom was able to use it for

Model Developer: NVIDIA Corporation Model Development: Fine-tuned from NVIDIA-Nemotron-3-Ultra-550B-A55B What is Nemotron? NVIDIA Nemotron™ is a family of open models with open weights, training data,

Been doing a bunch of vibe coding lately. Had my agents host a private WoW server for me, then built out a web browser client so you can play without installing the game and it has mobile controls. Af

On my M5 Pro 64GB I can comfortably work in an agentic setup with the Qwen3.8 27B model in good quality (Unsloth UD-Q4_K_XL) at a decent speed of 50 t/s. Splash combines optimized kernels, excellent s

Every rent-vs-buy thread I read has confident people on both sides, but not many actually show the numbers. So I finally ran the numbers for our own decision. Posting the working here in case it is us

I've been playing around with various models on the M5 Ultra 256GB 80-core Mac Studio. These are the results over many rounds of agentic inferencing. I'm happy with the performance. Glad to have the l

I’ve been experimenting with whether Qwen3.8-Flash-Next’s pretrained PLE n-gram memory can improve a much smaller Qwen3.5-0.8B model. I trained the 0.8B setup with limited resources, mostly using free

Been dealing with the same thing a lot of people here probably are - RAM prices keep climbing and buying a single machine with enough memory for the models I actually want to run just isn't realistic

I have a tool that uses Gemini Flash with the lowest thinking budget to do summarization work. It's very fast, 0.9-1.2s in most cases. But I have a user experience problem where people make the wrong

I’m releasing OpenJudgement-4B-Preview, an experimental Qwen-based model fine-tuned on custom datasets for classification, scoring and true/false judgments. It scores answer options directly, and Pyth

Model Overview Ling-3.0-flash-VL inherits the language, reasoning, and long-context capabilities of Ling-3.0-flash, while extending them with native image and video understanding. The model has 124B t

Hey there folks! Aritra here from Hugging Face. I wanted to update you all about the latest changes in `transformers`. We now natively support GGUFs (llama cpp quants). You can use it like so: from tr

I keep seeing Jev presented as some new class of decision model, but most of what’s being advertised is just normal classifier behavior with modern zero-shot capabilities. It outputs probabilities ove

MiMo-V2.6-Pro has an insanely high score of 46 on AA, putting it at the head of the opensource models available. It also costs pennies. Flash is not out on AA yet, but it costs less than half on datac

Saw a new open 27B model release that looks like it could be interesting for the Ollama crowd: VeriLoop E2. It is a Qwen3.8-27B post-train aimed at code, mathematics, scientific reasoning, and agentic

Dynamic Quantiser TLDR - Makes dynamic and/or custom quants of ggufs. Doesn't need data, just gguf file + llama.cpp. Minimises cosine deviation - highly correlated with minimising KLD. Fast, much bett

There's quite a lot of public anti-AI sentiment at the moment, but a lot of the most common concerns and objections have to do with the politics, economics and ethics of datacenter-based AI. So, when

Hey everyone! It has been quite a while since the last SupraLabs model - but today we've something special for y'all: Supra2-IMG It's a 100M parameter DiT text-to-image model trained entirely from scr

Sorry for the pretentious name, I know, I know.. It just contains all the pieces I would like to see a AGI model to have, and I can't stand the temptation. Before throwing rocks at me, please take a g

So this jev thingy is getting kind of big... tbh it seems overhyped by a large margin, but here we are. Not that its bad, just feels like we usually ignored larger things... Anyway to the point of thi

i have been running local models as browser agents on an m5 macbook pro, 128gb, all through mlx, no cloud. first i just wanted to know which one was faster at real browser work, so i gave gemma 4 31b

Meet Qwen-Image-2.1, the most balanced and cost-effective image generation model in the Qwen-Image series! Now open weights! 🎨 A unified model for both generation and editing, delivering top-tier qual

I am trying to learn more about long-term memory and how people are handling that nowadays. Historically, I have used a mixture of paid cloud AI services (ChatGPT, Gemini, OpenRouter API) and my own l

I needed a relatively simple but acceptable level of AI for working on one project. I didn't have any heavy requests, I just needed to give the AI access to the project files so it could search throug

Play it right in your browser, no download. Some of the GLB’s are messy, but overall I’ve enjoyed playing with it, and the last iteration made was the gun/weapon placement. Contributors are welcome, w

I improved the TPS of Gemma 4 31B. Improving TPS and performing optimisations requires understanding of the model architecture, and I had to fork VLLM and apply my own patch to break into making a con

Given the recent interest in tools like Jev, I wanted to share ProgramAsWeights (PAW), an open-source research project I'm working on at the University of Waterloo. You describe a text function in Eng

Since OpenAI retired SWE-bench Verified in February (every frontier model tested could reproduce reference fixes for some tasks; underspecified tests rewarded knowing the intended fix), I've been tryi

I have always posted about budget builds on here, and often asked how we are going to run the next big models. Often Plenty of downvotes too or folks telling me that it's not running if I'm getting 5t

Hi there, I'm Yuesong, a researcher at Video Rebirth. We open-sourced HyperFlow, an 8-step LoRA for MiniMax-H3. The core idea: data-free flow self-distillation. Diffusers' default sampler does 49 mode

Saw this little dude in a Forbes article (https://www.forbes.com/sites/johnkoetsier/2026/08/18/american-humanoid-robot-launches-for-just-1688-delivery-this-fall-made-in-san-francisco/) This is definit

Hi everyone, I’ve been experimenting with local LLMs on my 48 GB RAM MacBook Pro and I’m still trying to figure out what the best setup is. Ollama feels very fast, which I really like. The problem for

Hey all, Henry from Cactus Compute here, I kinda wanted to share our latest model and get feedback from the family :) Needle 3 is a small foundation model for automation: you give it the functions you

UPDATE https://github.com/ggml-org/llama.cpp/pull/29009 MERGED More Gen AI coverage and frameworks integrations to minimize code changes New models supported: On CPU: Gemma-3n On CPU, GPU: Kokoro-82M,

Hi Ollama Team, I’m writing regarding the announced retirement of GLM-5.1 from Ollama Cloud on 25 September 2026. I wanted to provide some fairly detailed technical feedback because GLM-5.1 is not sim

Another question asked 1 thousand time, but I can't find actual threads. Sorry in case of mistake. Due to interstellar price of hardware like high-end GPU (more than 5k euros for a 5090 and similar pr

I find another new model at HF: https://huggingface.co/XingChen-AGI/Xing4.0-29B-A4B 'Xing4.0-29B-A4B is a next-generation large language model in the Xing series (formerly TeleChat), developed by Chin

Apple is considering offering an AI server built around 'M8' series chips and has discussed incorporating Nvidia networking hardware, The Information reports. The system would be sold to outside custo

GitHub: https://github.com/pfekin/LARA I've been working on LARA (Lightweight Additive Residual Adaptation), a research project on making post-training modular for frozen language models. I've also de

https://preview.redd.it/e5wn8eyh7vph1.png?width=1080&format=png&auto=webp&s=5faeec866acc9e8eca8dc9e84a6661bee7bcf954 https://preview.redd.it/8ov5gl8j7vph1.png?width=1080&format=png&auto=webp&s=440320c

I've been working on something for this community (and other self-hosted communities) for almost 5 months now. Personally, I would grade it as probably the closest open source equivalent of Manus and