Learning to Route and Schedule LLMs from User Retrials via Contextual Queueing Bandits
DGX agentarXiv:2602.02061v2 Announce Type: replace Abstract: Explosive demands for LLMs often cause user queries to accumulate in server queues, requiring efficient routing (query-LLM matching) and scheduling