The first thing we learned building autoscaling for dedicated inference is that CPU-style metrics don't tell the whole picture. A GPU can re…

The first thing we learned building autoscaling for dedicated inference is that CPU-style metrics don't tell the whole picture. A GPU can read 60% busy while the engine's queue is already backing up, because utilization measures arithmetic intensity, not pressure. @soyoung_park's deep dive covers what to do about it: -> Which of the eight scaling metrics to pick, and why in-flight requests is the default -> How to set the up and down windows, including the sawtooth replica count everyone hits first -> What a cold start actually costs, phase by phase, with measured numbers on 1xH100 Plus an experiment replaying one load profile under three policies, where two of them never scaled.

Source: Together AI (X) | 2026-07-31