# Harvesting Expert-Parallel Bubbles for Fine-Tuning

> BUBBLETEA predicts expert-parallel bubbles during MoE inference and schedules LoRA fine-tuning inside them, achieving 1.9-6.2x faster training throughput than state-of-the-art colocation systems.

- **Source:** [arXiv](https://arxiv.org/abs/2611.24496)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/DfE20a
- **Whiteboard:** https://picx.dev/p/DfE20a/image

## Summary

## Summary (Overview)

- **BUBBLETEA** is the first colocation system that leverages expert-parallel bubbles (GPU idle time during MoE inference) for fine-tuning LLMs with LoRA, achieving **1.9–6.2× faster fine-tuning throughput** than LLMSTATION, a state-of-the-art colocation system.
- The system predicts each GPU's bubble duration from the MoE router's token assignment with a **median error of 8.9 µs (~7% of median bubble)**, enabling per-GPU, per-layer scheduling of fine-tuning work.
- BUBBLETEA introduces a **compiler that automatically derives a fine-tuning runtime from the served LLM as-is**, without porting model architectures to a separate training engine or duplicating weights.
- On production traces, BUBBLETEA trains **3.3× faster than LLMSTATION** and **14× faster than uncoordinated colocation**, while keeping p90 inference latency within 7% of serving alone.
- The system keeps p90 TTFT and TPOT within 10% of serving alone in **50 of 63 workloads**, compared to 44 for LLMSTATION.

---

## Introduction and Theoretical Foundation

### Background and Motivation

GPU resources dominate the cost of serving large language models (LLMs). Latency-critical inference must meet strict service-level objectives (SLOs) but rarely uses every GPU resource at every instant. **Colocation systems** reclaim idle capacity by running best-effort work (e.g., fine-tuning) alongside serving.

### Mixture-of-Experts (MoE) Architecture

Many frontier LLMs use MoE architecture, where each layer contains multiple feed-forward experts and a router sends each token to only a few of them. For large models, experts are spread across GPUs via **expert parallelism (EP)**.

### The Problem: Expert-Parallel Bubbles

Under EP, token routing is input-dependent, causing **uneven expert work across GPUs**. Lightly loaded GPUs finish early and wait for the most-loaded GPU—this waiting interval is defined as a **bubble**:

> A bubble is the interval between completion of a GPU's local expert computation and the dependency that allows the computation to proceed.

Key findings from characterization:
- **Bubbles survive load balancing**: Even with optimizations (EPLB, LPLB, WATERFILL), bubbles still account for **8–13% of prefill time**
- **Bubbles are predictable**: They can be estimated from the router's token assignment
- **Existing colocation systems miss them**: They make one scheduling decision per serving iteration, but bubbles require **per-GPU planning inside each MoE layer**

---

## Methodology

### System Architecture

Each GPU runs two processes: a **serving worker** and a **fine-tuning worker**, sharing the same copy of base model weights via CUDA IPC.

### Offline Phase: Deriving Fine-Tuning Work

The compiler transforms the serving LLM into a fine-tuning runtime through three innovations:

1. **Differentiate the serving LLM in place**: Replaces inference-only operations with differentiable equivalents over the same serving weights, using AOTAutograd to derive the backward graph while preserving TP/EP layout.

2. **Partition backward into bubble-sized work**: Makes cross-GPU exchanges explicit and partitions compute into chunks of at most **50 µs** using a roofline cost model, splitting operators like grouped expert GEMMs when necessary. Chunks are compiled with Inductor and captured in CUDA graphs.

3. **Allow uneven progress across GPUs**: Uses one-sided mailbox messages so GPUs wait only for data they need, allowing GPUs with larger bubbles to run ahead.

### Online Phase: Predicting and Admitting Work

**Bubble prediction**: The predictor estimates bubble duration from the router's expert assignment:

$$bubble_g = \alpha(\max_j tiles_j - tiles_g)$$

where:
- $tiles_g$ = sum of $\lceil n_e/M \rceil$ over experts on GPU $g$ (tile counts)
- $n_e$ = number of tokens routed to expert $e$
- $M$ = tile height (fixed block size)
- $\alpha$ = execution time of a single tile (derived from profiling)

**Admission mechanism**: A device-side gate flag controls when fine-tuning chunks can execute:
- Gate opens when a GPU finishes its local expert computation
- Gate closes at: $t_{close} = t_{open} + bubble_g - m$, where margin $m = 60\mu s$
- The most-loaded GPU (with no bubble) never admits work
- Bubble filling is disabled for prompts below **4k tokens** (bubbles too small to justify overhead)

**Paired chains**: Two independent fine-tuning chains run in flight—while one is blocked on a cross-GPU exchange, the other can execute. They run concurrently only when prefill takes <70% of the last second.

---

## Empirical Validation / Results

### Experimental Setup

- **Hardware**: 4× NVIDIA A100-SXM4-80GB GPUs, serving with vLLM in BF16
- **Models**: QWEN3-30B-A3B, QWEN1.5-MOE-A2.7B, DEEPSEEK-V2-LITE
- **Baselines**: LLMSTATION (iteration-level colocation), Uncoordinated (stock LoRA trainer with CUDA MPS)

### Key Results

#### End-to-End Performance

| Metric | BUBBLETEA | LLMSTATION | Uncoordinated |
|--------|-----------|------------|---------------|
| Fine-tuning throughput (steady-state) | 1.9–6.2× faster than LLMSTATION | Baseline | 2.7–12.4× slower than BUBBLETEA |
| Production trace throughput | 3.3× faster | Baseline | 14× slower |
| p90 TTFT within 10% (63 workloads) | **50/63** | 44/63 | 22/63 |

#### Sources of Throughput

- **Fine-grained scheduling (TEA vs. decode colocation)**: 36–60% improvement
- **Bubble filling (BUBBLETEA vs. TEA)**: +24–27% at medium/high load; −3% at low load (overhead)

#### Latency Protection

Ablation study (4k prompts) showing the importance of scheduling constraints:

| Configuration | Throughput (samples/s) | p90 TTFT (× serving alone) | p90 TPOT |
|--------------|----------------------|---------------------------|----------|
| TEA | 0.73 | — | — |
| Unconstrained | 2.11 | **5.03×** | 1.19× |
| Constrained | 1.33 | 1.03× | 1.13× |
| BUBBLETEA | 1.00 | **1.01×** | **1.04×** |

#### Composition with Load Balancing

With EPLB enabled, bubble filling still adds **12–23% throughput** at high load (compared to 27–37% without EPLB), showing the two approaches are complementary.

---

## Theoretical and Practical Implications

### Key Insights

1. **Bubbles are a predictable resource**: The router's token assignment provides sufficient information to predict per-GPU idle time accurately, enabling safe scheduling without affecting inference latency.

2. **Fine-grained scheduling matters**: Splitting training into 50 µs chunks (vs. layer-granularity) improves throughput by 36–60%, even outside bubbles.

3. **Per-GPU admission is essential**: A single per-iteration headroom budget misses 99% of bubble capacity; per-GPU planning from router state captures 43–45%.

4. **Compiler-based approach eliminates porting burden**: Deriving backward passes from the serving engine's forward pass means new architectures require no hand-written training kernels.

### Practical Implications

- **Cost reduction**: BUBBLETEA turns otherwise-wasted GPU idle time into useful fine-tuning work, improving accelerator ROI.
- **Complementary to load balancing**: Works best when serving is busiest—exactly when conventional headroom runs out.
- **Production-ready**: Maintains SLOs on bursty production traces with minimal latency impact.

---

## Conclusion

BUBBLETEA demonstrates that expert-parallel bubbles—a byproduct of MoE inference—can be systematically harvested for fine-tuning. The system's key contributions are:

1. **Characterization** of expert-parallel bubbles: they persist after load balancing, grow with prompt length, and are predictable from router state
2. **A novel colocation architecture** that schedules per-GPU, per-layer fine-tuning work inside predicted bubbles
3. **A compiler** that automatically derives bubble-compatible training code from the served LLM

### Future Directions

- Extending bubble filling to other best-effort workloads beyond LoRA fine-tuning
- Exploring larger-scale deployments with more GPUs and models
- Adaptive policies for when to use bubble filling vs. conventional headroom (e.g., based on load conditions)
- Integration with additional load-balancing optimizations as they emerge

The authors note that bubble filling pays off most when serving is busiest, suggesting future systems should dynamically decide when to leverage bubbles based on available conventional headroom.

---

_Markdown view of https://picx.dev/p/DfE20a, served by PicX — AI-generated visual whiteboard summaries of research papers._
