A Brain Too Big to Carry — On-Device vs Datacenter Inference

Robotics forces AI models to run on‑device because real‑time control loops cannot tolerate the latency or capital cost of sending data to a datacenter, so manufacturers must embed compute within each unit. As a result, robot “brains” are deliberately kept small—often far below the multi‑trillion‑parameter size of frontier language models—to fit affordable, low‑latency silicon such as Jetson Thor or B300. The field remains nascent because decisions on hardware and model size must balance timing constraints, unit economics, and the embodied‑AI challenge where the machine’s physical embodiment dictates the feasible computation.

Source: SemiAnalysis | 2026-09-14