Fewer Bits, More Tries: Recovering Capability Under Memory Constraints

arXiv:2610.15372v1 Announce Type: new Abstract: Large language models are generally more capable than smaller ones, but their memory requirements can make local deployment difficult. We ask whether part of this capability gap can instead be recovered by running a smaller or more compressed model multiple times and selecting among its outputs using public or model-generated tests. Single-user local inference can leave hardware underutilized, allowing parallel candidate generation to trade additional test-time computation for accuracy at modest latency cost. Across four code-generation benchmarks, verified retry recovers substantial accuracy lost to model size or compression. A 7B Qwen model with eight parallel attempts essentially matches a 32B model on MBPP+ and exceeds it on the other three benchmarks, while using about 4imes less model memory, running up to 6imes faster, and using less energy, as measured on Nvidia GPU. We observe similar tradeoff across model sizes, compression levels and hardware choices, although exact benefit depends on the hardware profile and problem setting. These results show that model capacity can be traded for verified test-time computation when correct solutions remain reachable within a modest retry budget.

Related

• Unlocking the Edge deployment and ondevice acceleration of multi-LoRA enabled one-for-all foundational LLM
• Soro: A Lightweight Foundation Model and Chatbot for Tajik
• GAMMA: Global Bit Allocation for Mixed-Precision Models under Arbitrary Budgets
• Enhancing Software Engineering Through Closed-Loop Memory Optimization
• TriSP: Tri-Signal Structured Pruning for Large Language Models

Source: arXiv cs.AI | 2026-10-15