Local Ai
One Simple Trick for Improving the Performance of Energy-Limited Local Inference and Training
arXiv:2609.11936v1 Announce Type: cross Abstract: Energy supply and heat dissipation are two of the main challenges with modern GPU deployments. While typically discussed in the context of new datacen
arXiv:2609.11936v1 Announce Type: cross Abstract: Energy supply and heat dissipation are two of the main challenges with modern GPU deployments. While typically discussed in the context of new datacenter constructions, the same constraints also apply to small form-factor consumer devices, such as the DGX spark. In workloads characterized by alternating compute-intensive tasks such as matmuls with memory-bound operations such as norms or cross-entropy, the compute-intensive parts might hit power and/or thermal limits and start throttling. In this short paper, we show that chunking the workload into smaller parts that alternate compute and memory in higher frequencies, these power and temperature spikes can be smoothed out, preventing throttling and resulting in considerably faster wall-clock time and reduced total energy consumption. We present several scenarios in which this effect can be exploited on a DGX Spark with up to 2% performance and energy improvements, and demonstrate that the same phenomenon also happens on less constrained systems, such as a multi-GPU server, albeit at significantly reduced effect size of 1-2%.
Related
- EcoFair: Energy-Efficient Inference Routing for Edge AI under Data Degradation
- The Battery Price of edge AI: A study of the Environmental Impact of LLM Inference on Mobile Devices
- The Model Parking Tax: Quantifying the Hidden Energy Cost of Always-On GPU Model Deployment
Source: arXiv cs.LG | 2026-09-14