Model Releases
Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference
Amazon SageMaker Inference now offers prefix-aware routing, a routing strategy that sends requests sharing the same prompt prefix to the same instance so the KV cache stays warm. In benchmarks on Llam
Amazon SageMaker Inference now offers prefix-aware routing, a routing strategy that sends requests sharing the same prompt prefix to the same instance so the KV cache stays warm. In benchmarks on Llama 3.1 70B, it reduced P50 time-to-first-token by up to 77% and raised KV cache hit rates from about 25% to over 80%.
Related
- Benchmarking small LLM inference on SageMaker AI: G7 vs G5 and G6
- LLM optimization integration for Amazon SageMaker Python SDK
- Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference
Source: AWS ML Blog | 2026-09-10