Search: “llm”

Check out Chimera Arena (https://chimeraarena.com/static/showcase/index.html?lang=en)! Here’s my slightly scrappy AI-generated teaser for Chimera Arena, my card game combining AI-generated video battl

So this is maybe a niche problem, but at my job I work on a huge Python codebase and every time I change some shared function I'm basically playing roulette. grep tells who mentions it in the code bas

Hi everyone, TL/DR I created NInfer 4080 to run ISTA-DASLab-Qwen-3.8-27B-GSQ at 100k context on an RTX 4080 16GB GPU using way more of the hardware capabilities (max overall: 2720 tok/s prefill, 262 t

Disclosure: I am the creator of this project. After days of lurking and building up enough karma, I can finally post here. Every time I tried running local RAG on my own machine, I hit the exact same

I wanted to see how far I could push a fairly ordinary laptop with a huge MoE model. Turns out, Qwen3.8 Flash Next 176B can run on: RTX 3080 Laptop — 16GB VRAM 32GB system RAM SSD No 128GB/256GB RAM w

Hey local AI community, I've been working on this for a while and finally feel ok sharing it. It's a cyber benchmark where the model gets a shell in an isolated docker box and has to find the exact fl

Hi all! I am a former SWE who has recently transitioned into inference engineering. I launched a side hustle a few months back and I've just taken it full time due to excessive demand. Its been such a

https://www.percepta.ai/blog/can-llms-grow-their-own-capabilities https://www.percepta.ai/blog/spotlight-memory 'Our new architecture, Spotlight, replaces attention with a memory that escapes this tra

I've been using my free time during weeknights and weekends for the last 11 months working on and refining a small domain-specific language model. It specializes on information about World War II. The

Follow-up to my post from about two weeks ago. A lot has changed since then, so I wanted to post an update on where the backend is now. Where the green architectures stand When I posted last time, 7 a

Running things locally already means you care more about control and reproducibility than most. Worth extending that to your prompts too. There's a live workshop on Oct 3 covering exactly that. Serj S

The collection includes kernels for more than 200 common ML operations, all of which can run entirely locally in your browser on WebGPU. We're also working to upstream these optimizations to Transform

TL;DR: I'm a solo dev who wanted a simple, private way to have local LLMs watch my screen and do simple logging/notifying. After a year of building, I released v3.0.0 and my mom was able to use it for

Some context first. I am process improvement / business consultant and had worked with Fortune 500 companies on improving their processes around refunds, returns, customer support, etc. This entire th

I've been working on Mica, a 4B decision model, and wanted to see how far it could get in actual Minecraft, not a sim. New world, empty inventory, and the goal was a diamond pickaxe. Last time I poste

For instance whenever a model comes out, what’s the best engine to run it, the best harness and absolute minimum you need to get same or near same re results that the benchmark of that model claims. A

Sometimes it’s easy to forget that this sub and others like it are probably the extreme minority when it comes to this hobby. Most people, I would think, don’t use or can’t afford one good GPU, let al

I have a tool that uses Gemini Flash with the lowest thinking budget to do summarization work. It's very fast, 0.9-1.2s in most cases. But I have a user experience problem where people make the wrong

Hey r/Ollama, I've been working on Presentation Forge, an open-source presentation/report generator that supports Ollama for local inference. I originally made it because I'm a university student and

Hey everyone, I’m new to running AI models locally and I’d like some advice. My laptop specs are: HP OMEN 15 CPU: Intel i5-8300H RAM: 16 GB GPU: NVIDIA GTX 1050 Ti 4 GB VRAM OS: Windows I’m planning t

Dynamic Quantiser TLDR - Makes dynamic and/or custom quants of ggufs. Doesn't need data, just gguf file + llama.cpp. Minimises cosine deviation - highly correlated with minimising KLD. Fast, much bett

Retrieval benchmarks sometimes feel benchmaxxed by models, so we wanted to find a way to tie it as close as possible to my objective: finding the article that answers a product question right. We work

# the What An engine to run Gemma 4 31B on blackwell under massive concurrency and rather specific workload patterns. I've been waiting for someone to do ninfer but for gemma, and, well, ended up havi

Small, free finding. I use local Qwen models for typed decisions: a state plus a question with fixed allowed answers, and I read the probability of each answer from the logits of one forward pass inst

The headlines right now are full of stories about AI models 'escaping their sandboxes' and literally killing all humans, lol. I've even heard several commentators and writers say that AI escaped an 'A

This weekend, I did a fun experiment to create something similar to Jev. I LoRA fine-tuned Qwen3.5 4B using a mix of publicly available datasets and synthetic data. For the synthetic data, I used Deep

So this jev thingy is getting kind of big... tbh it seems overhyped by a large margin, but here we are. Not that its bad, just feels like we usually ignored larger things... Anyway to the point of thi

Hey everyone, After wrestling with hardware and engine issues for days, I finally got Qwen 3.8 Next running properly on my multi-GPU rig. Thought I’d share the setup journey, benchmarks, and thermal r

we're a small lab (switzerland + south africa) and just open-sourced our first model: a 27b fine-tune of qwen3.8-27b specialized for creative writing and short-form personal text (stories, dialogue, r

I gave Jev, Laya, a finetuned ModernCE-base-nli and a finetuned Qwen3.5-4B the controls to Doom. Thanks to TypeSafe AI for Jev access. The video lines up the starts of four separate games with the sam

First time trying something like this. I asked chatgpt and it told me there are mcp server plugins for both ghidra and ida pro as well as RE-MCP which has support for both. I could run that on my low

Behold, workflow whisperers! YuE2 was released a week ago and I gotta say it's splendid for its size (3B). My favourite local music generation model at the moment, no doubt. The good news is, I've fou

full disclosure, i am the developer of locally uncensored :) i use ollama myself and honestly ollama was never really the problem. the problem was everything around it. one app for local chat, another

Okay, this one seems probably really odd (or maybe not), but most harnesses operate within token spaces. That is, every tool output, even batching of tool outputs, go back up into inference calls. Lea

Took me a while since I'm on a family trip and have limited hardware, but here it is! Von: Open-source 'System One' drop-in replacement for TypeSafe's JEV. https://github.com/wfzyx/von https://hugging

So I know that Ollama is built on top of llama.cpp, I decided to run a model on both and applied settings on llama.cpp like flash attention and q8/q4 kv cache. On Ollama (using the Python package) I w

Hi everyone, I’ve been experimenting with local LLMs on my 48 GB RAM MacBook Pro and I’m still trying to figure out what the best setup is. Ollama feels very fast, which I really like. The problem for

I let AI run my phone 60 real tasks, back to back, on the OnePlus I use every day - Best text model still failed 43% of them - Peak chip temp 98.2 C - 69% of the battery gone The benchmark is AndroidL

Hey all, Henry from Cactus Compute here, I kinda wanted to share our latest model and get feedback from the family :) Needle 3 is a small foundation model for automation: you give it the functions you

We've been tuning vLLM/SGLang/llama.cpp setups for a long time and got tired of the profiling part: nsys trace, open the GUI, squint, change a flag, repeat. The profilers assume a human is looking at

Hey experts! I genuinely have this question and would love a general consensus from people actually using these models. LLMs have advanced a lot in benchmarks and in practical usage. Even GPT-4o and G

I've been working on something that started as the training backend for my Hierarchos architecture, but it has grown into a much broader project: https://github.com/necat101/Hierarchos-Native Hierarch

Follow up post of https://www.reddit.com/r/LocalLLaMA/s/nGMyKswrch. Thanks everyone who replied. I didn't change the specs. Might be loosing some of the memory bandwidth but will scale in future if I

I have a full model, it's ready to train. It's ~9b parameters. 9.4b to be more exact. That includes a 1/2/3 Engram table, Moonshot's AttnRes modeling, and RoPE / NoPE layering at 3:1 as more or less v

I'll start by saying I'm not talking about the models themselves, I'm aware that I can't come close to something like Fable's intelligence locally. Just wanted to get that out of the way. Basically. O

Same 44-token prompt: low = 202 output tokens in 15s, max = 4,955 in 35s, unset = 10,579 in 68s. I'm using it via claude code with this: CLAUDE_CODE_MAX_CONTEXT_TOKENS=1048576 ollama launch claude --m