Search: “routing”

Prompt caching delivers significant efficiency gains at a single replica, but under standard round-robin load balancing, a request with an identical prefix has only a 1/N chance of hitting the repl...