Hardware
Introducing Amazon SageMaker HyperPod Inference Gateway
Amazon SageMaker HyperPod Inference Gateway is a Kubernetes-native, GPU-aware routing add-on for Amazon EKS. It uses real-time GPU signals to send each inference request to the best-suited pod, cuttin
Amazon SageMaker HyperPod Inference Gateway is a Kubernetes-native, GPU-aware routing add-on for Amazon EKS. It uses real-time GPU signals to send each inference request to the best-suited pod, cutting first-token latency by up to 82% with no changes to your model servers or client applications.
Related
- Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine
- Build a Physical AI model factory with NVIDIA Cosmos 3 on SageMaker HyperPod
- Best practices to run inference on Amazon SageMaker HyperPod
Source: AWS ML Blog | 2026-09-18