Search: “hardware”

Hi all! I am a former SWE who has recently transitioned into inference engineering. I launched a side hustle a few months back and I've just taken it full time due to excessive demand. Its been such a

Sometimes it’s easy to forget that this sub and others like it are probably the extreme minority when it comes to this hobby. Most people, I would think, don’t use or can’t afford one good GPU, let al

Every rent-vs-buy thread I read has confident people on both sides, but not many actually show the numbers. So I finally ran the numbers for our own decision. Posting the working here in case it is us

Privacy have been concern of many of us to have their own hardware to run llms, and here's another reason why: two mathematicians spent a year cracking one of the hardest problems in math and fed ever

Four Tesla V100s from 2017 matched my RTX 5090 on single-request Qwen 3.8 decode. Repo: https://github.com/dnv2003/v100-skinny https://i.redd.it/5ws2ak3uqckh1.gif The 5090 was not being held back. It

Been benchmarking DSv4 Flash 0731 on a Flow Z13 (Ryzen AI MAX+ 395, Radeon 8060S / gfx1151, 128GB LPDDR5X) for the past week. Figured I'd share what actually works and what doesn't — there are a lot o

A discussion on r/ollama exploring which high-performance LLM models can be effectively run locally on standard laptop hardware , likely covering model size comparisons, hardware requirements, and per

This table to help anyone looking to build a budget Data Center homelab. I copied the bulk of value based, mid level, decent speed results GPUs and feed it to AI or SI and here are the recommended res

[INFO] Prompt executed in 00:30:49 Full 1080p vid on https://www.youtube.com/watch?v=ER_5AOteE-8 because Reddit cramps everything to 720p max. From my previous post, I got a message if I could use my

Hi everyone, TL/DR I created NInfer 4080 to run ISTA-DASLab-Qwen-3.8-27B-GSQ at 100k context on an RTX 4080 16GB GPU using way more of the hardware capabilities (max overall: 2720 tok/s prefill, 262 t

Hey everyone, I'm getting into running LLMs locally with Ollama and would like some recommendations on which models I should realistically run with my hardware. My PC specs GPU: RTX 5060 Ti 16GB VRAM

My prompt is: Epic high-fantasy concept art, ultra-detailed. A road crossing golden plains leading toward a colossal dwarven mountain hall entrance recessed into themouth of a giant cave. The whole en

Lately because of the current hardware shortage, unfortunately or fortunately, we can’t just throw infinite cloud compute at our problems, but we’re forced to actually care about what’s happening unde

I've been out of these forums for awhile but I figured I would provide a formal update on how this has been going now that it has some operation time under its belt, just to put the information out th

Ran DeepSeek V4-Flash-0731 — the full official checkpoint, not a re-quant — on commodity used hardware. Sharing because I couldn't find anyone else publishing Ampere results for this engine. Why bothe

This Reddit discussion in the StableDiffusion community addresses recommendations for affordable, used or refurbished laptops equipped with GPUs suitable for running open-source image generation model

This Reddit post from r/ollama discusses a user's experience running Cline (an AI coding agent) with Ollama on a high-end local hardware setup consisting of an NVIDIA RTX 4090 with 24GB VRAM and an In

This Reddit thread from r/StableDiffusion discusses a performance issue where Roop Unleashed version 4.3.1 — a face-swapping tool commonly used alongside Stable Diffusion — fails to fully utilize the

A Reddit post from r/ollama in which a user shares their experience running a 31B parameter model locally using Ollama, reflecting on the surprisingly demanding hardware and infrastructure requirement

This Reddit thread from r/ollama discusses community recommendations for purchasing or renting GPU servers to run Ollama and local LLMs. It likely covers options ranging from dedicated GPU server prov

Nvidia reportedly halts GeForce RTX 5090 production in favor of AI data center and professional GPUs — impending supply drought expected to drive up prices, RTX 5080 24GB rumored as new gaming flagshi

I’ve been pushing TinyStories-style models downward in size, and this is the smallest one so far: MacroStories — 19,969 parameters, 81 KB FP32 https://huggingface.co/raincandy-u/MacroStories For scale

Disclosure: This unit was provided free of charge by Orange Pi for testing. No editorial review, no preconditions, no script. All data, bottlenecks, and thermal behavior are reported directly from har

Someone who has always loved looking at the space between different AI techniques, this time I went a little deeper into the memory trade-offs between RNNs, Transformers and SSMs. I found it interesti

Hi folks, LIVE PROJECT PAGE TLDR: Moral of the story. You need better CPU to do actual agentic coding doing real work... I've been on a mission to make my RTX5090 go brrr for past 2 months so much so

I've been experimenting with running larger models locally with Ollama, mainly because I'm interested in using them for agentic coding. I kept seeing benchmarks expressed as something like: 40 tokens/

Hi, i got QFN running on my single r9700 but im not sure if i did everything right to get the best quality and speed out of this setup. Dont want to annoy anybody, maybe someone with the same card can

Hi again! This is a update to my previous post MageTrail V0.25 where I continued this tech demo toy model thingy called MageTrail, a Danbooru/E621 proof of concept Full-Finetune of Microsoft's MageFlo

I'm part of the Lokutor team that built this. Model: NVIDIA Conformer-CTC Small (13M params, int8). It runs on an ESP32-S3 with 8 MB PSRAM, no GPU or NPU. LibriSpeech WER is 3.7 / 8.2, versus 6.3 / 15

Other than the obvious self promotion, is there a practical reason people do this that I am missing? There's dozens of llamacpp forks with silly names that are supposedly 'optimized' for this or that

you read about gpu's like the Tesla P40, V100, AMD M150 etc that people pick up for cheap. probably many others as well. Most are either server gpu's being phased out or mining discards, right? The pr

Ok, so the Mac M5 Ultra (256GB) hit the market, but the only publicly available benchmarks material are flashy YouTube 'clown influencers' videos. We need serious numbers to evaluate whether Apple’s s

Kijai put his magic to good use to improve your MiniMax H3 workflows. The video VAE now encodes up to ~2.2x faster and decodes ~1.4-2.7x faster on Nvidia GPU. Checkout the blog post how to use it: htt

I didn't know my set up was outperforming nearly everyone until reading another discussion where people were struggling getting half of that speed with half the context on the same hardware. I benchma

I’ve been working on making small models more capable at agentic coding and work, because most people in the world don’t have the sort of hardware needed to run 3.8-27B, or even 35B-A3B or 9B dense, a

Took me a while since I'm on a family trip and have limited hardware, but here it is! Von: Open-source 'System One' drop-in replacement for TypeSafe's JEV. https://github.com/wfzyx/von https://hugging

CPU : epyc 7262 MB : ROMED8-2T GPU : V100 16GB PCIE x6 I have built a server with six V100 GPUs. I am now testing it and plan to eventually run Qwen3.8-Next-Flash configured with TP2 and PP3. Because

Apple is considering offering an AI server built around 'M8' series chips and has discussed incorporating Nvidia networking hardware, The Information reports. The system would be sold to outside custo

I've been working on something that started as the training backend for my Hierarchos architecture, but it has grown into a much broader project: https://github.com/necat101/Hierarchos-Native Hierarch

Hi, For single-GPU training, I’m using Hugging Face SFTTrainer with auto_find_batch_size=True, which automatically reduces the batch size after a CUDA OOM until it finds a batch size that works. I wou

Spoke with many providers who lurk in this sub, plus met folks who work in inference. For a 10k monthly revenue, a provider was able to retain only 200 dollars in profit due to GPU costs. Their custom

I believe I may currently hold the record for memory constrained inference for Qwen3.8–Flash-Next on Apple Silicon — needing only about 21GB of allocations. Introducing Cherenkov, an inference engine

It seems to use a 96-bit LPDDR5X memory bus, instead of the previous 64-bit wide busses. Considering it's on 2nm, that's expensive silicon. That should result in around 115 GB/s memory bandwidth. A20

Most of the new ~30B (and now ~10B, thankfully for my GPU) models we see score really high on benchmarks, but I feel like they don't push back on dumb ideas enough. I think most people don't being lik

I ran a 112-case MiniMax H3 quality sweep on an RTX 3090 last night trying to find the best combos . Kept the prompt, seed, reference image, RefMods, Sage setup, sampler, and scheduler stayed constant

Hi, I use hermes as a harness, and I am pleased with it, but sometimes Hermes's context size is a bit too much for my system, so I wanted to delegate or straight up use a lighter option for coding tas