Summary (Overview)
- BUBBLETEA is the first colocation system that leverages expert-parallel bubbles (GPU idle time during MoE inference) for fine-tuning LLMs with LoRA, achieving 1.9–6.2× faster fine-tuning throughput than LLMSTATION, a state-of-the-art colocation system.
- The system predicts each GPU's bubble duration from the MoE router's token assignment with a median error of 8.9 µs (~7% of median bubble), enabling per-GPU, per-layer scheduling of fine-tuning work.
- BUBBLETEA introduces a compiler that automatically derives a fine-tuning runtime from the served LLM as-is, without porting model architectures to a separate training engine or duplicating weights.
- On production traces, BUBBLETEA trains 3.3× faster than LLMSTATION and 14× faster than uncoordinated colocation, while keeping p90 inference latency within 7% of serving alone.
- The system keeps p90 TTFT and TPOT within 10% of serving alone in 50 of 63 workloads, compared to 44 for LLMSTATION.
Introduction and Theoretical Foundation
Background and Motivation
GPU resources dominate the cost of serving large language models (LLMs). Latency-critical inference must meet strict service-level objectives (SLOs) but rarely uses every GPU resource at every instant. Colocation systems reclaim idle capacity by running best-effort work (e.g., fine-tuning) alongside serving.
Mixture-of-Experts (MoE) Architecture
Many frontier LLMs use MoE architecture, where each layer contains multiple feed-forward experts and a router sends each token to only a few of them. For large models, experts are spread across GPUs via expert parallelism (EP).
The Problem: Expert-Parallel Bubbles
Under EP, token routing is input-dependent, causing uneven expert work across GPUs. Lightly loaded GPUs finish early and wait for the most-loaded GPU—this waiting interval is defined as a bubble:
A bubble is the interval between completion of a GPU's local expert computation and the dependency that allows the computation to proceed.
Key findings from characterization:
- Bubbles survive load balancing: Even with optimizations (EPLB, LPLB, WATERFILL), bubbles still account for 8–13% of prefill time
- Bubbles are predictable: They can be estimated from the router's token assignment
- Existing colocation systems miss them: They make one scheduling decision per serving iteration, but bubbles require per-GPU planning inside each MoE layer
Methodology
System Architecture
Each GPU runs two processes: a serving worker and a fine-tuning worker, sharing the same copy of base model weights via CUDA IPC.
Offline Phase: Deriving Fine-Tuning Work
The compiler transforms the serving LLM into a fine-tuning runtime through three innovations:
-
Differentiate the serving LLM in place: Replaces inference-only operations with differentiable equivalents over the same serving weights, using AOTAutograd to derive the backward graph while preserving TP/EP layout.
-
Partition backward into bubble-sized work: Makes cross-GPU exchanges explicit and partitions compute into chunks of at most 50 µs using a roofline cost model, splitting operators like grouped expert GEMMs when necessary. Chunks are compiled with Inductor and captured in CUDA graphs.
-
Allow uneven progress across GPUs: Uses one-sided mailbox messages so GPUs wait only for data they need, allowing GPUs with larger bubbles to run ahead.
Online Phase: Predicting and Admitting Work
Bubble prediction: The predictor estimates bubble duration from the router's expert assignment:
where:
- = sum of over experts on GPU (tile counts)
- = number of tokens routed to expert
- = tile height (fixed block size)
- = execution time of a single tile (derived from profiling)
Admission mechanism: A device-side gate flag controls when fine-tuning chunks can execute:
- Gate opens when a GPU finishes its local expert computation
- Gate closes at: , where margin
- The most-loaded GPU (with no bubble) never admits work
- Bubble filling is disabled for prompts below 4k tokens (bubbles too small to justify overhead)
Paired chains: Two independent fine-tuning chains run in flight—while one is blocked on a cross-GPU exchange, the other can execute. They run concurrently only when prefill takes <70% of the last second.
Empirical Validation / Results
Experimental Setup
- Hardware: 4× NVIDIA A100-SXM4-80GB GPUs, serving with vLLM in BF16
- Models: QWEN3-30B-A3B, QWEN1.5-MOE-A2.7B, DEEPSEEK-V2-LITE
- Baselines: LLMSTATION (iteration-level colocation), Uncoordinated (stock LoRA trainer with CUDA MPS)
Key Results
End-to-End Performance
| Metric | BUBBLETEA | LLMSTATION | Uncoordinated |
|---|---|---|---|
| Fine-tuning throughput (steady-state) | 1.9–6.2× faster than LLMSTATION | Baseline | 2.7–12.4× slower than BUBBLETEA |
| Production trace throughput | 3.3× faster | Baseline | 14× slower |
| p90 TTFT within 10% (63 workloads) | 50/63 | 44/63 | 22/63 |
Sources of Throughput
- Fine-grained scheduling (TEA vs. decode colocation): 36–60% improvement
- Bubble filling (BUBBLETEA vs. TEA): +24–27% at medium/high load; −3% at low load (overhead)
Latency Protection
Ablation study (4k prompts) showing the importance of scheduling constraints:
| Configuration | Throughput (samples/s) | p90 TTFT (× serving alone) | p90 TPOT |
|---|---|---|---|
| TEA | 0.73 | — | — |
| Unconstrained | 2.11 | 5.03× | 1.19× |
| Constrained | 1.33 | 1.03× | 1.13× |
| BUBBLETEA | 1.00 | 1.01× | 1.04× |
Composition with Load Balancing
With EPLB enabled, bubble filling still adds 12–23% throughput at high load (compared to 27–37% without EPLB), showing the two approaches are complementary.
Theoretical and Practical Implications
Key Insights
-
Bubbles are a predictable resource: The router's token assignment provides sufficient information to predict per-GPU idle time accurately, enabling safe scheduling without affecting inference latency.
-
Fine-grained scheduling matters: Splitting training into 50 µs chunks (vs. layer-granularity) improves throughput by 36–60%, even outside bubbles.
-
Per-GPU admission is essential: A single per-iteration headroom budget misses 99% of bubble capacity; per-GPU planning from router state captures 43–45%.
-
Compiler-based approach eliminates porting burden: Deriving backward passes from the serving engine's forward pass means new architectures require no hand-written training kernels.
Practical Implications
- Cost reduction: BUBBLETEA turns otherwise-wasted GPU idle time into useful fine-tuning work, improving accelerator ROI.
- Complementary to load balancing: Works best when serving is busiest—exactly when conventional headroom runs out.
- Production-ready: Maintains SLOs on bursty production traces with minimal latency impact.
Conclusion
BUBBLETEA demonstrates that expert-parallel bubbles—a byproduct of MoE inference—can be systematically harvested for fine-tuning. The system's key contributions are:
- Characterization of expert-parallel bubbles: they persist after load balancing, grow with prompt length, and are predictable from router state
- A novel colocation architecture that schedules per-GPU, per-layer fine-tuning work inside predicted bubbles
- A compiler that automatically derives bubble-compatible training code from the served LLM
Future Directions
- Extending bubble filling to other best-effort workloads beyond LoRA fine-tuning
- Exploring larger-scale deployments with more GPUs and models
- Adaptive policies for when to use bubble filling vs. conventional headroom (e.g., based on load conditions)
- Integration with additional load-balancing optimizations as they emerge
The authors note that bubble filling pays off most when serving is busiest, suggesting future systems should dynamically decide when to leverage bubbles based on available conventional headroom.
Related papers
- Do LLM Agents Still Need to Be Told How to Self-Improve?
Newer LLM generations spontaneously construct their own self-improvement algorithms from primitive tools alone, transferring learned strategies across tasks and models without prescribed learning loops.
- Rethinking Fixed-Horizon Latent Reasoning
Adaptive latent inference that stops reasoning when progress stalls improves accuracy by up to 7.2 points while cutting computation by 29%, as completing fixed latent horizons can degrade performance.
- Late Entry in Looped Language Models
Late entry into deep context caches outperforms early exit in looped language models, yielding up to 8.4x exact and 12.5x lossy decoding speedups.