Summary (Overview)

• Problem: Modern LLM serving systems use static memory isolation between model weights (especially MoE experts) and attention states, leading to memory underutilization under dynamic workloads—either limiting request concurrency during bursts or wasting memory during light loads.

• Key contribution: Rimuru introduces unified memory management via a shared page pool with tensor-specific "memory views," allowing heterogeneous tensors (expert weights, KV cache, SSM states, convolution states) to elastically share device memory.

• Technical innovations: A multi-level page allocator with huge-page alignment, a page-table-aware fused MoE kernel that accesses non-contiguous expert weights directly (avoiding copies), and a workload-aware expert cache adjuster that dynamically resizes expert residency.

• Results: Rimuru achieves up to 19.03× baseline throughput and reduces TTFT by 98.1% on average across diverse hardware configurations (single, dual, and eight-GPU setups).

• Implementation: Built on top of vLLM (~4,400 lines of Python, ~800 lines of C++), compatible with modern LLMs including hybrid attention architectures (Mamba, linear attention).

---

Introduction and Theoretical Foundation

Background and Motivation

Modern LLM serving in resource-constrained environments faces a critical bottleneck: limited device memory. Conventionally, serving systems keep complete model weights on devices, which exhausts available memory. The problem is compounded by:

• Growing context lengths (up to 1M tokens) requiring substantial KV cache memory for batching
• MoE architectures that provide strong performance but incur substantial memory demands (e.g., DeepSeek-V4-Flash on eight A100 GPUs leaves only ~7 GB/GPU for attention states, supporting just 2 concurrent requests with 1M-token contexts)

The Fundamental Mismatch

Existing systems manage memory in statically isolated regions:

• PagedAttention manages KV caches in a dedicated memory region
• Expert offloading systems manage expert caches separately with fixed offloading ratios

This creates a fundamental mismatch between dynamic memory demand (workloads vary over time, as shown in Figure 3 with bursty request arrivals and variable sequence lengths) and static memory isolation (fixed boundaries that cannot adapt).

Tensor Heterogeneity

Modern LLM architectures introduce heterogeneous tensors with diverse layouts and lifetimes:

Case Study Evidence

A case study serving Qwen1.5-MoE-14B on one RTX 5090 showed:

• vLLM (static weights): peak concurrency of 34 requests, ~2100 forward steps
• Expert offloading (fixed cache): peak concurrency of 74, ~1300 steps
• Rimuru (elastic sharing): peak concurrency of 90, ~1000 steps

This demonstrates that elastic memory sharing reduces total workload completion steps by ~30% compared to fixed-boundary expert offloading.

---

Methodology

1. Multi-view Memory Layout

Rimuru allocates contiguous device memory as a shared page pool and exposes tensor-specific memory views over this storage. Each view defines its own page size and page table.

Huge-page alignment: The huge page size is defined as the least common multiple of expert and SSM page sizes:

$$P_h = \mathrm{lcm}(P_e, P_s)$$

For models without SSM states, simply $P_h = P_e$. KV cache and convolution state page sizes must divide $P_h$.

For convolution states of shape $[m, d]$, Rimuru uses a row of shape $[1, d]$ as the page allocation unit, allocating $m$ consecutive pages per state.

2. Multi-level Page Allocation

The allocator maintains a hierarchy:

• L1 pages: huge pages (top level)
• L2 pages: expert pages and SSM pages
• L3 pages: KV cache pages and conv pages

Allocation proceeds via on-demand splitting—L1 pages split into L2, which can further split into L3. This avoids direct L1→L3 splitting (which would cause fragmentation).

Reclamation follows the reverse path: L3 pages merge back into L2 when fully freed, and L2 pages merge into L1 when all their children are reclaimed.

3. Page-table-aware Fused MoE Kernel

Conventional fused MoE kernels require contiguous expert weights. Rimuru's kernel accesses weights directly from cache pages or loading buffers via metadata.

For the joint $W_1/W_3$ projection, with local hidden dimension $d$ and intermediate dimension $f$, the addressing scheme is:

$$\begin{array}{c} p_e(n) = \begin{cases} p_{e,1}, & n < f, \\ p_{e,3}, & n \geq f, \end{cases} \\ \mathrm{addr}_e(n, t) = B_\mathrm{P} + p_e(n) \cdot f d + (n \bmod f) d + t, \end{array}\tag{1}$$

where $B_\mathrm{P}$ is the page-pool base address, $p_{e,1}$ and $p_{e,3}$ are projection-page identifiers, and offsets count stored weight elements.

Dual-buffer design: Two expert buffers alternate roles—one supplies weights for the current layer while the other receives prefetched weights for the next layer, avoiding read-write conflicts without local DtoD copies.

4. Workload-aware Expert Cache Adjuster

The adjuster uses offline hotset profiles (expert orderings with empirical routing probabilities) as a prefix-based cache policy.

CPU-side policy (Algorithm 1):

• Shrink when $R_b > 0$ (scheduler requires additional pages): $N_\mathrm{shrink} = \lceil (R_b + 1)/3 \rceil$ experts
• Shrink one expert if free blocks $F_b < 2$
• Expand when $F_b > 9$: $N_\mathrm{expand} = \min(5, \lfloor \max(0, F_b - 9)/3 \rfloor)$ experts

The gap between shrink threshold (2) and expand threshold (9) reduces frequent cache reversal.

GPU-side application: Expanded experts merge into the prefetch queue with higher priority; page table updates only after copy completion. Shrunk experts remain visible during the current forward pass and are reclaimed afterward.

---

Empirical Validation / Results

Setup

• Hardware: 8× RTX 5090 (32 GB) with PCIe 5.0; 2× RTX 4090D (24 GB) with PCIe 4.0
• Models: Qwen3-30B-A3B, Qwen3.5-35B-A3B (dual GPU), Qwen3.5-122B-A10B (8 GPU)
• Workload: ShareGPT real user interactions
• Baselines: vLLM, SGLang (serving systems); KTransformers, Fate, llama.cpp (offloading systems)

End-to-end Performance vs. Offloading Systems

Rimuru achieves up to 19.03× throughput and reduces TTFT by up to 99% versus baselines:

Key insight: KTransformers performs well on CPUs with AMX instructions but degrades significantly without them. Rimuru's advantage comes from dynamic cache adjustment enabling higher request concurrency.

End-to-end Performance vs. Serving Systems

Rimuru achieves 1.1×–2.88× throughput versus SGLang and vLLM, with TTFT reduced by 63%–99%. Under low workload, Rimuru shows limited degradation by dynamically increasing cached experts to reduce I/O.

Ablation Study

FusedMoE Kernel Efficiency

• Average overhead: 0.83% vs. contiguous-weight baseline
• Large token batches ($M \geq 1024$): overhead ≤ 1.31%
• Single-token case: max 9.63% increase but only 3.38 μs absolute latency
• Mixed expert placement (50%–100% page-resident): overhead varies by ≤ 0.22%

Sensitivity Analysis (expert_offload_limit)

• Minimum EOL degenerates to fixed offloading (like Fate) with low throughput
• Slightly increasing EOL (by ~10 experts) significantly improves throughput
• Peak throughput achieved in a moderate EOL range
• TPOT remains largely unaffected across EOL settings due to hotset-based caching

---

Theoretical and Practical Implications

Theoretical Contributions

1. Breaking memory isolation: Rimuru demonstrates that static boundaries between tensor types fundamentally limit serving performance under dynamic workloads. The concept of tensor-specific memory views over a shared pool generalizes PagedAttention to heterogeneous tensor types.

2. Elastic capacity reuse: The multi-level page allocator shows that cross-type memory reuse is achievable without prohibitive fragmentation, provided page sizes are aligned via least-common-multiple relationships.

3. Execution-memory co-design: The page-table-aware fused MoE kernel proves that non-contiguous weight layouts can be consumed during computation with near-zero overhead, eliminating the traditional requirement for dense, contiguous weight packing.

Practical Implications

1. Resource-constrained deployment: Rimuru makes it feasible to serve large MoE models on limited hardware while maintaining competitive throughput, important for SMEs seeking on-premises deployment.

2. Adaptive serving: The workload-aware adjuster provides a practical mechanism for balancing throughput (more concurrency) against latency (more cached experts), with hysteresis to prevent cache oscillation.

3. Extensibility: The memory view abstraction accommodates new tensor types and weight formats (BF16, INT8) with minimal changes, future-proofing the design as LLM architectures evolve.

4. Complementarity: Expert prediction and prefetching strategies from other systems (e.g., Fate) remain orthogonal and can be integrated, allowing the community to build on Rimuru's unified memory substrate.

---

Conclusion

Rimuru addresses a fundamental mismatch in modern LLM serving: dynamic memory demands versus static memory isolation. By unifying memory management through tensor-specific memory views over a shared page pool, multi-level page allocation, and page-table-aware expert execution, Rimuru enables elastic expert offloading that adapts to workload dynamics.

The workload-aware cache adjuster dynamically balances request concurrency against weight-loading overhead, while the page-table-aware fused MoE kernel preserves execution efficiency despite non-contiguous weights. The system achieves up to 19.03× baseline throughput and reduces TTFT by 98.1% on average.

Future directions include:

• Integration with more sophisticated expert prediction strategies
• Extension to disaggregated and distributed serving architectures
• Support for additional tensor types as LLM architectures evolve
• Exploring optimal expert_offload_limit determination that varies with model architecture, GPU capacity, and workload characteristics