How to run big models on old hardware 30B at 22 tok/s on 6GB GPU and 16GB RAM

I have been working on this tool for months and there are a lot of new functionalities and tests that are going to be released in the next few weeks! The goal of the tool is to allow community members to run big models on old hardware optimizing every bit at its max for your specific purpose. I’m going in the direction of building not only the full model pipeline but also a series of tools that will help anyone to get the most out of the models. Quantprobe is an open-source memory allocation optimization framework and deployment assistant designed to run massive Local Large Language Models (LLMs) on resource-constrained consumer hardware. The tool specializes in projecting local LLM inference speeds and calculating optimal, layer-by-layer quantization placement to maximize performance without crashing your system. The project demonstrate the implementation recipes capable of running a massive 110B parameter model (GLM-4.5-Air) on a standard 16GB RAM consumer machine. Core Features & Functionality Performance Prediction: It accurately estimates a model's generation speed (tokens per second) on your specific hardware profile before you waste time and network bandwidth downloading massive model weights. Layer-by-Layer Placement Optimization: Instead of uniformly squeezing an entire model to a single low bit-rate, Quantprobe acts as an intelligent placement solver. It evaluates system limits and assigns varying quantization levels to individual model layers depending on their depth and computational importance. Resource Balancing: It dynamically balances quantization levels against VRAM/RAM constraints, cleanly splitting weights between your CPU and GPU to avoid Out-Of-Memory (OOM) errors. The "Four Laws" Framework: The tool relies on a mathematically rigorous, measurement-based methodology (dubbed the "four laws" of token economy) to accurately trade inference speed for memory. How it Differs from Standard Tools Traditional backends like llama.cpp often apply uniform quantization across the board or rely on naive offloading. Users have reported that utilizing Quantprobe's optimized depth-aware recipes can practically double generation speeds compared to a default, uncalibrated setup (e.g., jumping from ~10 tok/s to over 22 tok/s on older hardware like a GTX 1060 setup) submitted by /u/Ok_Brush_3449 [link] [comments]

Source: r/ollama | 2026-08-03