Newest papers

RouterWorld: Scaling Conversational AI with Latency-Aware LLM Routing

Jinyang Li, Denis Emelin, Simone Tedeschi, Weiwei Cheng, Cristina Morariu, Haytham Assem, Jens Lehmann, Emine Yılmaz, Angus Addlesee · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26511 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

基于大语言数据模型(LLMs)构建的交谈人工智能系统需要平衡响应质量和响应延迟的路由机制。然而,现有的LLM路由基准假设所有候选模型结果的完全可观察性,低强调延迟作为一流指标,以及与现实世界设置不一致的异质协议的片段评估。我们引入了RouterWorld,该基准是谈话 GenAI中的LLM路由的综合基准。RouterWorld统一了371k单转回响应和76k多转回代理轨迹,采用了与标准化的LLM-as-a-judge协议的自然用户交互,并通过硬件为基础的延迟公告解除了过渡定价的效率。

Machine translation (Meta M2M100); the English original follows.

Abstract

Conversational AI systems built on Large Language Models (LLMs) require routing mechanisms that balance answer quality and response latency at scale. However, existing LLM routing benchmarks assume full observability of all candidate model outcomes, under-emphasize latency as a first-class metric, and fragment evaluation across heterogeneous protocols misaligned with real-world settings. We introduce RouterWorld, a comprehensive benchmark for LLM routing in conversational GenAI. RouterWorld unifies 371k single-turn responses and 76k multi-turn agentic trajectories, adopts natural user interactions with a standardized LLM-as-a-judge protocol, and decouples efficiency from transient pricing via hardware-grounded latency annotations. Crucially, RouterWorld formalizes routing under partial observability through two setups: Cold-Start, which restricts learning to sparse, static logs, and Voyager, which models continuous data streams where routing decisions shape future training distributions. Evaluating 13 routing baselines, we find that most routers over-emphasize accuracy while ignoring latency under partial observability. Code and benchmark data are publicly available.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Observability, Latency (audio), Static routing, Benchmark (surveying), Link-state routing protocol

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange