Local Ai
Hardware Roofline Inference Calculator
Hello everyone, I made a calculator for the theoretical HW roofline for decoding / prefill based on several parameters (LLM model architecture, quants, GPU, Memory, ..). It still a theoretical bound,
Hello everyone, I made a calculator for the theoretical HW roofline for decoding / prefill based on several parameters (LLM model architecture, quants, GPU, Memory, ..). It still a theoretical bound, but helpful as a step-0 check to understand what fits (would fit) in your hardware, and understand the effects of the contributing knots. I hope it helps. You can access it from here: https://www.ai-leaderboard.dev/ (click HW Roofline) submitted by /u/Informal-Trouble2183 [link] [comments]
Related
- The idea: on a CPU the decode speed depends on the active params per token, not the total. My objective is trying to run a 10B at 100tok/s on a mid level PC (No GPU).
- I pushed Qwen3.8-27B to 2.000 prefill per second and 132 decode per second on A RTX 3090.
Source: r/LocalLLaMA | 2026-09-27