Visual Summary | LLMROUTER: UNIFIED INFRASTRUCTURE FOR DEVELOPING, EVALUATING, AND DEPLOYING LLM ROUTERS

Summary (Overview)

LLMRouter is a unified infrastructure for developing, evaluating, and deploying LLM routers, addressing the lack of standardized foundations in the field by providing a common formulation, automated evaluation pipeline, and open-source library.
The paper introduces a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals, organizing existing methods into three families (single-turn, multi-turn, and personalized routing).
xRouteBench is a new multi-scenario benchmark spanning generic LLM tasks, memory-augmented, vision (image and video), time-series, and personalized routing scenarios, comprising 4,767 instances evaluated under a unified cost-aware protocol.
Key empirical findings: (i) no single router dominates across all tasks; (ii) learned routers achieve a 14.6% relative improvement over the strongest fixed-model baseline; (iii) multi-turn routing does not consistently outperform single-turn routing; (iv) personalization pays off but only when user context is modeled well.
The library implements more than 16 representative routers spanning all three families, with deployment support for real users via OpenClaw (Slack/Discord) and multi-agent systems.

Introduction and Theoretical Foundation

The paper addresses a critical problem in the LLM ecosystem: no single large language model is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment. The authors identify two key obstacles hindering progress:

Diverse formalisms and incompatible implementations — existing routers (binary quality predictors, cost-aware cascades, graph-based routers, agentic routers) are developed under distinct formalisms with incompatible interfaces, making it difficult to isolate design elements that drive performance.
Lack of standardized evaluation pipeline — evaluating a router requires running every candidate model on every benchmark query and scoring each response, which is substantially more demanding than evaluating a single model.

The theoretical foundation is a unified formulation of routing as a sequential decision process. At step $t$ , the router observes a state $s_t = (q, u, h_t)$ consisting of the query $q$ , optional user context $u$ , and interaction history $h_t$ , and takes an action $a_t \in \mathcal{M} \cup \{\perp\}$ . The goal is to find an optimal policy:

\pi^{\star} = \arg \max_{\pi} \mathbb{E}_{q,\tau \sim \pi} \left[ \operatorname{perf}(y \mid q) - \lambda \cdot c(\tau) \right],\tag{1}

where $\operatorname{perf}(y \mid q)$ aggregates task-specific quality metrics, $c(\tau)$ sums the monetary/token cost of every call in the routing trajectory, and $\lambda \geqslant 0$ controls the performance–cost trade-off.

Methodology

Unified Formulation Components

A router is characterized by five components:

Context encoder ( $E_q$ ): Maps routing state to a representation. Two forms:
- Embedding-based: state as a vector (e.g., kNN-style routers use off-the-shelf sentence embeddings; discriminative routers train lightweight encoders)
- Text-based: state kept in natural language (e.g., cascades append draft responses; fine-tuned LM routers verbalize the state in the prompt)
Model encoder ( $E_m$ ): Encodes each candidate model via:
- Static metadata (model size, capability description, pricing)
- Historical profiles (past behavior, Elo ratings, latent factors)
- Learned embeddings (jointly trained with context encoder)
- Verbalized description (named directly in prompt)
Scoring function ( $g$ ): Measures compatibility between encoded state and each candidate (embedding similarity, bilinear products, classification heads, message passing over graphs, next-token logits).
Decision rule ( $d$ ): Converts scores to actions (greedy arg max, cost-aware thresholds, accept/escalate in cascades, sampling for exploration).
Learning signal ( $\mathcal{L}$ ): Fits components toward the optimal policy (non-parametric, supervised pointwise, preference-based pairwise, or trajectory-level RL rewards).

Three Router Families

Family	State $s$	Encoders $E_q, E_m$	Routing action (scoring $g$ , decision $d$ )	Learning signal $\mathcal{L}$
Single-turn	$(q)$	$E_q(q), E_m(m)$	$a = \arg \max_{m \in \mathcal{M}} g(E_q(q), E_m(m))$	fit $g$ to per-candidate reward $\text{perf}(y_m \mid q) - \lambda c_m$
Multi-turn	$(q, h_t)$	$E_q(q, h_t), E_m(m)$	$a_t \sim d(\{g(E_q(q, h_t), E_m(m))\}_m)$	maximize episode return $\mathbb{E}_\tau[\text{perf}(y \mid q) - \lambda c(\tau)]$
Personalized	$(q, u, h_t)$	$E_q(q, u, h_t), E_m(m)$	$a = \arg \max_{m \in \mathcal{M}} g(E_q(q, u, h_t), E_m(m))$	fit $g$ to comparisons $m^+ >_u m^-$ observing $\text{perf}_u$

Automated Pipeline

LLMRouter automates supervision construction and evaluation via three stages:

Query Curation: queries sampled from source benchmarks, normalized into a unified schema, split into train/test
Response Collection: each query dispatched to every candidate in the pool (18 models, 7B–671B parameters)
Metric Scoring and Pricing: every response scored with task metrics and priced from token counts

xRouteBench Design

All tasks share a common query schema, supervision format, and evaluation protocol. Non-text assets are converted to self-contained textual queries with optional pointers to source images/videos/time series, separating routing from perception.

Empirical Validation / Results

Main Results (Performance-First Setting, $(\alpha, \beta) = (1.0, 0.0)$ )

Router	Generic LLM Tasks	LoCoMo	LongMemEval	Geometry3K	MathVista	Video	TimeSeries	Avg
Smallest-LLM	57.55	25.44	36.77	27.87	35.00	33.33	49.61	37.94
Largest-LLM	70.29	26.59	35.57	37.70	33.00	22.22	45.67	38.72
kNNRouter	71.37	25.24	38.74	31.15	41.00	29.63	51.97	41.30
SVMRouter	74.21	27.64	38.68	42.62	47.00	29.63	55.91	45.10
MLPRouter	68.12	26.78	32.27	27.87	34.00	29.63	56.69	39.34
EloRouter	64.15	25.70	37.27	45.90	50.00	25.93	63.78	44.68
GraphRouter	80.54	25.94	33.93	42.62	50.00	22.22	62.99	45.46
RouterDC	80.56	24.93	36.77	16.39	24.00	25.93	45.67	36.32
Router-R1	35.64	24.60	17.28	14.75	18.00	22.22	23.62	22.30

Personalized Track Results

Router	Acc.	Router	Acc.
GMTRouter	68.78	RouterDC	56.44
PersonalizedRouter	67.86	MFRouter	54.39
EloRouter	66.40	MLPRouter	52.93
GraphRouter	65.23	kNNRouter	51.76
SVMRouter	65.08	CausalLM	46.78
Largest-LLM	58.05	Router-R1	45.46
Hybrid LLM	57.91	Smallest-LLM	42.53

Real-User Deployment Results

Router	Acc.	Router	Acc.
PersonalizedRouter	83.05	RouterDC	65.25
EloRouter	82.20	kNNRouter	60.17
MLPRouter	78.81	kNN-MultiRound	60.17
SVMRouter	77.12	Smallest-LLM	55.08
Hybrid LLM	73.73	MFRouter	51.69
GMTRouter	70.70	Largest-LLM	41.53
GraphRouter	67.17	CausalLM	27.97

Multi-Agent System Results

Router	Star	Tree	Graph	Chain	Plan-Exec-Sum	Avg
Largest-LLM	69.00	67.00	77.20	69.00	75.20	71.48
kNNRouter	74.80	78.60	78.60	76.60	71.80	76.08
SVMRouter	76.20	75.60	80.00	74.40	75.20	76.28
MLPRouter	75.40	76.60	76.80	78.00	71.40	75.64
MFRouter	75.40	74.20	81.00	78.60	73.20	76.48
EloRouter	73.80	72.40	78.60	76.60	75.20	75.32
GraphRouter	68.20	70.80	66.20	72.00	69.00	69.24
RouterDC	77.60	79.60	74.20	72.00	76.20	75.92

Key Findings

No single router dominates: The best router varies across tasks and cost budgets; strong average performance reflects consistency rather than dominance.
Learned routing beats fixed-model baselines: Learned routers achieve a 14.6% relative improvement over the strongest fixed-model baseline, as always selecting the largest model incurs the highest cost yet delivers only mediocre performance.
Multi-turn routing does not consistently outperform single-turn: Additional rounds of decomposition and aggregation often add cost and redundant information; performance hinges on the capability of the base model.
Personalization pays off but depends on modeling: GMTRouter ranks first under persona judge (68.78), while PersonalizedRouter leads on real human preferences (83.05), showing the two settings favor different designs.
Router rankings reverse under tighter cost constraints: MLPRouter sits near the bottom under quality-first settings but becomes the best choice for every $\beta \geqslant 0.4$ in Vision.

Theoretical and Practical Implications

Theoretical Contributions

Provides a unified formulation that reconciles seemingly incompatible router designs (binary predictors, cascades, graph-based, agentic, personalized) under a single sequential decision process framework.
Establishes a standardized evaluation protocol that jointly measures response quality and inference cost, enabling fair comparison across router families.
The formulation reveals that the three router families differ only in which portion of the state the context encoder reads, enabling component-level ablations and cross-family transfer.

Practical Implications

Reduced implementation burden: Adding a new router requires only implementing a routing method and a loss function; data construction, training, inference, and evaluation apply unchanged.
Configuration-driven experimentation: Swapping routers, candidate pools, or training objectives requires only a configuration change rather than reimplementation.
Deployment readiness: Routers can be exposed as OpenAI-compatible servers for messaging platforms (Slack, Discord via OpenClaw) or through ComfyUI-based visual interfaces for code-free prototyping.
Multi-agent system routing: Treating model choice as a per-agent decision improves performance across five coordination topologies (Star, Tree, Graph, Chain, Plan-Exec-Sum), with six of seven learned routers beating always selecting the largest model.

Conclusion

LLMRouter provides a unified foundation for LLM routing by casting single-turn, multi-turn, and personalized routing as instances of a common sequential decision process. The system includes:

An automatic pipeline for constructing routing supervision and evaluation for new tasks and candidate pools
xRouteBench, a multi-scenario benchmark spanning five tracks under one protocol
An open-source library implementing more than 16 routers behind a unified interface with deployment support

Key takeaways and future directions:

The empirical study reveals that router rankings are highly sensitive to cost constraints, suggesting practitioners should select routers matching their deployment's performance–cost requirements.
Multi-turn routing needs better sufficiency estimation, early stopping, and more effective decomposition and aggregation to justify its additional computational overhead.
Personalized routing should be validated against real user feedback, as simulated persona judges and real human preferences favor different designs.
The framework enables future work on component-level ablations and cross-family router designs that were previously difficult to explore due to incompatible implementations.