Search: “hardware”

I saw someone here posted about getting a Tenstorrent QuietBox 2, and I wanted to look into it the hardware. It's very difficult to find any benchmarks, but I managed to find some from an employee. Th

Got the new Qwen3.8-Flash-Next (176B total, 6B active MoE) running locally and figured I'd share for anyone else wondering if their setup can handle it. **My hardware:** 192GB DDR5 RAM, RTX 5070 Ti (1

specs hardware: M4 Max 128GB Studio inference engine: oMLX & lllama.cpp insights it still very early, so had to disable oMLX K/V caching, qwen4_exp architectureis not yet supported + the obvious n-gra

With hopes of a Qwen3.8-35B-A3B release now mostly dashed, many people including myself are looking at fine-tunes and other variants of Qwen3.6-35B-A3B to run on VRAM-limited hardware. I decided to tr

I’ve been experimenting with a custom inference/runtime research project called CRANE V2, mostly because I wanted to answer a stupid question: How far can you push absurdly large MoE models on an ordi

A month i setup a proxmox server using spare parts like a z270 i7 7700k and 2x8 2666 but since i don't have any hardware acceleration i was planning to pass down my rx 580 8GB to my server but i heard

For the GPU poor among us: I'm curious what results you're getting with low quants of Qwen 27B 3.8. My main inference hardware is limited (Mac mini M4 24 GB), but I'm getting great results with Unslot

Try it here: https://ocean.blackbeardlabs.dev/ Model: Qwen 3.8 27b Q8_X_KL Unsloth Hardware: 3 x RTX3090 Harness: DeepSeek Harness Prompt: /goal I want you to create a **JavaScript + Node.js WebGL pro

Qwen3.8-27B is amazing, but it’s slow. A stronger 35B-A3B Mixture of Experts-coder that can run and solve real codebase issues fast (even on constrained hardware) is a valuable addition to the arsenal

Hey r/Ollama, If you run local models on mid-range hardware, you probably know how fast VRAM disappears when you try to set up document memory or local RAG pipelines. I built Hillock (https://github.c

This is one more vibe slopped custom optimization for, in this case, my hardware (m2 ultra 60 cores, 192gb). It is just a fork from llama.cpp with a few changes, it achieves: - DeepSeek V4 Flash, no k

Our devs got their hands on it a few days ago. One wired it into Codex to compare with GPT Luna, our usual workhorse right now for its cost effectiveness. Another tried it out on one of our OCR pipeli

Here's a brand new Blackwell-native, prefill-optimized 4-bit quant that runs 50% faster on compatible hardware than a Q4 quant of the same memory footprint. And it runs 4-7% faster than other NVFP4 qu

QwQ was genuine next-gen performance usable on local hardware, but the massive required context (it's reasoning style was akin to 'if I say every possible word, I'll notice the right one!') kinda made

I have been searching for suitable model to run on my 8GB RAM toy, NVIDIA Orin Nano Super 8GB. This little toy was priced at 249 earlier this year (not any more), and pulls very little power when idle

I managed to run the 143–144 GiB DeepSeek-V4-Flash-0731 UD-Q4_K_XL GGUF on four RTX 3060 12GB cards while keeping a 360k–376k context window. Hardware: CPU: Intel Core i9-10920X, 12C/24T RAM: 128 GB D

Including the rationalisation for the data below - this is a more robust version of an earlier post I did similar to this - explaining below: How I chose the comparisons The basic question I’m trying

Hello; I'm a semi-beginner at local AI. I've been experimenting with this tech for a while, and I still haven't found a proper harness that fits my models, hardware, and needs. My use case is pretty s

Overview ggml_cpu_fp16_to_fp32 leverages hardware F16C intrinsics (AVX-512, AVX2, etc.), faster than the software-only ggml_fp16_to_fp32_row, bringing 17-31% gain in prompt processing rate for a small

Having a ‘Killer Application’ that everyone wants to use helps sell hardware, plain and simple. DeepSeek V4 Flash 0731 isn’t an app of course, but I think it’s going to be the major catalyst for getti

Hey everyone, I could use some advice on setting up speculative decoding correctly with llama-server. My Hardware: GPUs: RTX 4090 + RTX 6000 Pro (120GB total VRAM) RAM: 32GB I am currently testing the

I'm mosty interested in 128-192GB VRAM with 128-256GB RAM to spare, so SSD streaming is basically not even necessary. Seems only FP4 is supported, so older hardware will likely be slow - no Unsloth GG

We run a local model instance in our company that the dev we hired built for us. We're a trade business and we want to further use our on hand hardware for it. The specs given we have is a 5090 gpu wi

I love the progress happening on open models but I feel like it is kind of getting clear that hardware to run good sized models is completely unaffordable for me right now. I know that you all love Qw

Built my 1st inference machine and have been tweaking models trying to get the most out of my modest hardware. I think I’m at a good place but I’m testing with my own prompts. I’ve looked into some of

Here is my experience, which I would like to share with you and I also would like to hear your thoughts and valuable tips&tricks. Hardware: Mac Mini M4 (32GB Unified Memory) Model Server: Ollama Orche

I keep seeing the 'use a big model via API as the architect, run local small/mid models as workers' pattern recommended for people with modest local hardware. I've been running it myself (orchestrator

I can run the mediums, but sometimes I want a faster option that's smarter than Qwen 27B/35B. On my hardware I get like 500 to 800 tok/s prefill and 16 to 22 tok/s gen on ~120B class models, which is

I've always found it a little strange that AI is everywhere, but the codecs we use in practice are the traditional hand-engineered systems like h.264, h.265, av1. Alexnet started the wave of neural ne

Greetings Community! 👋 I hope everyone is doing well! I'm Tauhid — Senior EEE student from a Bangladeshi University Today I'd like to share an open-source project I’ve been developing called P.A.I. (P

I was never a huge AI guy but I was starting to wonder if my ISP was really providing the speeds I was paying for, and it turns out they were, but I had a bottleneck somewhere in my hardware chain. I

Running a 70B parameter model in full 16-bit precision requires roughly 140GB of memory , which is beyond most consumer hardware. Professional-tier GPUs like the RTX PRO 6000 with 96GB GDDR7 enable fu

A Reddit discussion seeking Stable Diffusion model recommendations for faster and cleaner image outputs, addressing the reality that no single best model exists as the right choice depends on hardware

This post describes implementing DCGAN (Deep Convolutional Generative Adversarial Network) inference on resource-constrained microcontroller hardware, achieving image generation with a 12.6 million pa

This Reddit thread likely discusses hardware recommendations and considerations for purchasing a new laptop capable of running Ollama, a tool that enables running large language models locally. The di

The Qwen 3.6-27B and 35B models are designed as open-weight models that developers can run locally on their own hardware without requiring cloud services. Alibaba released a separate cloud-only produc

Flux 2-Klein-9B NVFP4 is a quantized image generation model that runs on mid-range GPUs like the RTX 3050. On this hardware, the model produces 1024-resolution images but with relatively slow inferenc

This r/StableDiffusion Reddit thread discusses community recommendations for photorealistic image generation models that can run within a 16GB VRAM constraint, a common hardware limit for consumer GPU

LogLens with local AI 'Ollama' is a Reddit post on r/ollama discussing the integration of LogLens — a log analysis tool — with Ollama to perform AI-powered log inspection entirely on local hardware, w

This r/ollama Reddit thread addresses a common question among the local AI community about whether reliable, self-hosted AI systems are genuinely achievable. Community members in this space typically

KIV is a project shared on r/MachineLearning presenting a drop-in replacement for HuggingFace's `DynamicCache` that enables up to 1 million token context windows on consumer hardware with only 12GB of

A Reddit thread on r/StableDiffusion where a user reports poor output quality when using the ACE-Step 1.5 XL SFT model variant for AI music generation. The SFT (Supervised Fine-Tuning) variant of ACE-

Ollama Cloud offers Free, Pro ($20/month), and Max ($100/month) subscription tiers for cloud-hosted inference, but token generation speed depends on model size, architecture, and hardware optimiza...

[ I'm posting this on behalf of u/LodestoneRock since for some reason his post on this sub keep getting auto removed by Reddit. ] Hey everyone, First of all, thank you to everyone for supporting this

Alright, I learend a few valuable lessons since I posted about a week ago, and I've made more strides with the DDR4x Ascend 96GB 310P cards I purchased, so here we go, meet the new friendly and less v

Basalt: Blackwell inference engine Hi all! So over the last week I've been working on a Strata fork that's heavily tuned and can achieve throughput up to 2.6x what Strata usually does on the same weig