# Knowledge–Geometry Decoupling: Refreshable Pretrained Transfer for Streaming Recommendation

> KGD decouples behavioral knowledge from task geometry via one-way interfaces, improving streaming recommendation by 4–12% and boosting Shopee's GMV per user by 1.75%.

- **Source:** [arXiv](https://arxiv.org/abs/2608.02738)
- **Published:** 2026-08-06
- **Permalink:** https://picx.dev/p/n4eRsD
- **Whiteboard:** https://picx.dev/p/n4eRsD/image

## Summary

## Summary (Overview)

- **Problem Identification**: The paper identifies two critical limitations in the pretrain-then-transfer paradigm for streaming recommendation: (1) next-token prediction treats behavioral adjacency as dependency, encoding noisy transitions across unrelated sessions; (2) pretrained knowledge and task-specific geometry impose conflicting optimization demands on shared parameters, preventing effective continual refresh.
- **Core Contribution**: Introduces **Knowledge–Geometry Decoupling (KGD)**, a framework that separates behavioral knowledge (owned by a refreshable encoder) from task-specific geometry (owned by a task learner) through two one-way interfaces: read-only cross-attention and Anchored Calibration Residual (ACR).
- **Novel Pretraining Objective**: Proposes **Behavioral Multi-Token Prediction (BMTP)**, which filters future transitions by collaborative and semantic relevance rather than raw adjacency, yielding cleaner and more transferable behavioral representations.
- **Empirical Results**: KGD outperforms strong pretrain–transfer baselines by 4–12% on eight public benchmarks and sustains its advantage over a 90-day production stream where baselines show no gains.
- **Deployment Impact**: Fully deployed on Shopee Homepage Search, KGD increases GMV per user by 1.75% and advertising revenue by 1.53% in a live A/B test, with manageable training cost (2 hours daily on A100 GPUs).

## Introduction and Theoretical Foundation

The paper addresses the pretrain-then-transfer paradigm in industrial recommendation systems, where Transformers are pretrained on user behavior sequences with next-token objectives and then transferred to downstream tasks like CTR estimation and ranking. Unlike language and vision domains where models are pretrained once, recommendation systems face **continuous behavioral distribution drift** due to two compounding factors:

- **Item pool turnover**: New content, listings, and campaigns constantly enter and age out.
- **Audience turnover**: Users arrive unevenly and return at irregular intervals, causing behavior sequences to be concatenations of separate interest sessions rather than continuous trajectories of intent.

This segmentation corrupts the supervision on which pretraining relies—across session boundaries, adjacency no longer implies dependency. The paper identifies two fundamental questions for streaming recommendation:

1. **What to learn**: Sequential pretraining adopts the autoregressive objective of language modeling, treating adjacency as dependency. However, adjacency does not imply dependency in recommenders (e.g., a user sequence mixing "bicycle gear" and "pants" sessions). This noise corrupts the learned representations.

2. **How to transfer**: As shown in the paper's analysis (Fig. 1(b)), pretrained knowledge and task-specific geometry pull the same parameters in conflicting directions (evidenced by near-zero cosine similarity of gradients). Existing approaches fail in distinct ways:
   - **Full fine-tuning**: Task gradients overwrite pretrained parameters, causing knowledge erosion during refresh.
   - **Frozen transfer**: Protects knowledge but forbids task from reshaping geometry or injecting task-required knowledge.
   - **Adapters/LoRA**: Leave too little freedom to reshape geometry.

The paper formalizes the problem using the GPSD framework, which frames transfer along two axes: which parameters to carry across (dense parts vs. embeddings) and whether to freeze them. The authors argue that streaming recommenders raise additional dimensions not covered by this framework.

## Methodology

### Behavioral Multi-Token Prediction (BMTP)

BMTP addresses the "what to learn" question by filtering which co-occurrences are worth encoding. Instead of supervising every adjacent pair, BMTP retains only two kinds of transitions:

- **Collaborative transitions**: Items frequently co-consumed even when dissimilar (e.g., beer-and-diapers), measured by proximity on a pretrained item co-occurrence graph.
- **Semantic transitions**: Items close in content, measured by cosine similarity between precomputed text embeddings.

For each position $t$ and each axis $a$, BMTP retains the nearest subsequent items whose similarity exceeds a threshold:

$$
\begin{array}{c} \mathcal{S}_t^a = \big\{i_j: j = \min\{j' > t, \text{sim}_a(i_t, i_{j'}) \geq \tau_a\} \big\}, \quad a \in \{\text{col}, \text{sem}\}, \\ \mathcal{L}_{\text{pre}} = -\sum_{t=1}^{T-1} \sum_{a \in \{\text{col}, \text{sem}\}} \sum_{i^+ \in \mathcal{S}_t^a} \log \frac{\exp \langle h_t, e_{i^+} \rangle}{\sum_{i' \in \mathcal{I}} \exp \langle h_t, e_{i'} \rangle}. \end{array}
$$

Both similarities are computed offline and cached, adding no per-sequence model inference to pretraining.

### Decoupled Read–Write Ownership

KGD addresses the "how to transfer" question by assigning behavioral knowledge and task geometry to separate parameter sets through two interfaces:

**Anchored Calibration Residual (ACR)**: The task learner does not alter the pretrained embedding but forms its own task-adapted embedding by superposing a task-owned residual onto a detached copy. The residual is constrained to be **orthogonal** to the pretrained embedding:

$$
\begin{array}{rl} & \tilde{e}_i^{(k)} = s_k \cdot \text{sg}\big(e_i^{\text{pre}}\big) + \Delta e_i^{(k)}, \\ & \quad s_k = 1 + \text{ReLU}(\tilde{s}_k), \quad \Delta e_i^{(k)} = \big(Z^{(k)} B^{(k)^\top}\big)_i, \quad \Delta e_i^{(k)} \perp e_i^{\text{pre}}, \end{array}
$$

where $k$ indexes downstream tasks, $\text{sg}(\cdot)$ is stop-gradient, and $Z^{(k)} \in \mathbb{R}^{|\mathcal{I}| \times r}, B^{(k)} \in \mathbb{R}^{d \times r}$ are task-owned low-rank factors $(r < d)$ kept orthogonal to $e_i^{\text{pre}}$ by regularization.

**Read-only cross-attention**: The task transformer reads the encoder's hidden states through:

$$
Q = W_Q \rho, \quad K = W_K \text{sg}(\tilde{H}), \quad V = W_V \text{sg}(\tilde{H}),
$$

where $\rho$ is a task-side reader token and $W_Q, W_K, W_V$ are owned by the task transformer. The stop-gradient makes the interface strictly one-way.

### Training under Daily Stream

Each day: (1) refresh the encoder on new data for one pass; (2) freeze encoder parameters and run the combined encoder-learner graph, updating only the learner. Training cost is approximately twice an encoder-only pass (about 2 hours on A100 GPUs).

## Empirical Validation / Results

### Public Benchmark Results

Experiments were conducted on 8 Amazon-2023 Reviews datasets using ManCAR as the backbone. Key results from Table 1:

| Strategy | Pretrain | Arts N@50 | Arts R@50 | Beauty N@50 | Beauty R@50 | CDs N@50 | CDs R@50 | Phones N@50 | Phones R@50 |
|----------|----------|-----------|-----------|-------------|-------------|----------|----------|-------------|-------------|
| TE&FE | NTP | 0.0353 | 0.1011 | 0.0261 | 0.0725 | 0.0637 | 0.1734 | 0.0321 | 0.0879 |
| TE&FE | BMTP | **0.0376** | **0.1069** | **0.0270** | **0.0755** | **0.0690** | **0.1893** | **0.0330** | **0.0933** |
| TA&FE | NTP | 0.0340 | 0.0987 | 0.0256 | 0.0709 | 0.0649 | 0.1780 | 0.0308 | 0.0855 |
| TA&FE | BMTP | **0.0364** | **0.1045** | **0.0273** | **0.0763** | **0.0691** | **0.1901** | **0.0327** | **0.0947** |
| KGD | NTP | 0.0354 | 0.1024 | 0.0262 | 0.0734 | 0.0666 | 0.1844 | 0.0317 | 0.0865 |
| KGD | BMTP | **0.0380** | **0.1099** | **0.0292** | **0.0803** | **0.0701** | **0.1930** | **0.0337** | **0.0951** |

KGD with BMTP achieves the best score on every dataset, improving over the strongest published baseline by 4–12%.

### Industrial Data Results (28-day stream)

Results from Table 2 on Shopee Homepage Search data:

| Strategy | Pretrain | click AUC | click GAUC | order AUC | order GAUC |
|----------|----------|-----------|------------|-----------|------------|
| Scratch (S1) | - | 0.7806 | 0.7759 | 0.8941 | 0.8410 |
| TA&FT (S2) | BMTP | 0.7852 | 0.7801 | 0.8993 | 0.8447 |
| TA&FE (S2) | BMTP | 0.7835 | 0.7789 | 0.8991 | 0.8434 |
| TA&FT (S3) | BMTP | 0.7837 | 0.7793 | 0.8979 | 0.8433 |
| TA&FE (S3) | BMTP | 0.7841 | 0.7792 | 0.8989 | 0.8446 |
| **KGD (S3)** | **BMTP** | **0.7867** | **0.7826** | **0.9015** | **0.8477** |

Key findings:
- Under S3 (refresh daily), KGD reaches 0.7867 by decoupling ownership, while removing its interfaces collapses to 0.7785 (below Scratch).
- Frozen transfer (TA&FE) erodes over 90 days, while KGD holds throughout.
- Buffer replay degrades monotonically; LoRA (0.7818) leaves too little freedom.

### Ablation Study (Table 3)

| Setting | Office N@50 | Office R@50 | Software N@50 | Software R@50 | click AUC | click GAUC |
|---------|-------------|-------------|---------------|---------------|-----------|------------|
| w/o semantic BMTP | 0.0337 | 0.0886 | 0.1374 | 0.3748 | 0.7859 | 0.7818 |
| w/o collaborative BMTP | 0.0338 | 0.0908 | 0.1363 | 0.3731 | 0.7854 | 0.7810 |
| w/o ACR | 0.0350 | 0.0932 | 0.1369 | 0.3753 | 0.7829 | 0.7790 |
| ACR + share-param + freeze dense | 0.0279 | 0.0771 | 0.1259 | 0.3568 | 0.7833 | 0.7797 |
| ACR + share-param + finetune all | 0.0331 | 0.0884 | 0.1344 | 0.3659 | 0.7847 | 0.7811 |
| Align capacity with KGD | 0.0267 | 0.0608 | 0.1224 | 0.3240 | 0.7839 | 0.7802 |
| **KGD full** | **0.0352** | **0.0938** | **0.1381** | **0.3775** | **0.7867** | **0.7826** |

## Theoretical and Practical Implications

### Theoretical Implications

1. **Knowledge–Geometry Separation Principle**: The paper establishes that pretrained behavioral knowledge and task-specific geometry can coexist as two orthogonal layers under separate read-write ownership, rather than competing for one representation. This is theoretically grounded in the observation that task gradients require direction-dependent deformations that are orthogonal to the pretrained subspace.

2. **Orthogonality as Non-Conflicting Adaptation**: The paper proves that constraining task residuals to be orthogonal to pretrained embeddings preserves the pretrained geometry while enabling task-specific deformation. This is formalized through the stationarity conditions showing that the ideal case (temperature scaling) is a special case of the orthogonal residual framework.

3. **Refreshability without Invalidation**: Because task geometry is anchored as a residual relative to the pretrained embedding (not at absolute coordinates), refreshing the encoder moves the base beneath a stable residual rather than invalidating it. This provides a theoretical primitive for continual pretrained transfer.

### Practical Implications

1. **Industrial Deployment**: KGD has been fully deployed in Shopee Homepage Search, serving billion-scale users. The live A/B test shows:
   - GMV per user: +1.75%
   - Advertising revenue per user: +1.53%
   - CTR: +0.95%
   - CVR: +0.72%

2. **Cost Efficiency**: Daily offline training rises from one to two hours on the same A100 count (on par with GPSD), and serving latency stays at 120 ms on A30 GPUs.

3. **Scalability**: Because ownership is decoupled, capacity can be allocated independently to the encoder (for knowledge) and task learner (for geometry), enabling targeted scaling aligned with deployment budgets.

4. **Robustness to Drift**: The 90-day production study demonstrates that KGD maintains its advantage while frozen transfer erodes and buffer replay degrades monotonically, confirming the practical value of refreshable pretrained transfer.

## Conclusion

The paper revisits the pretrain-then-transfer recipe for recommendation under streaming production updates, where pretraining must be refreshed rather than handed off once. It identifies two key limitations: the next-token objective compresses noise from session boundaries, and pretrained knowledge conflicts with task-specific geometry on shared parameters.

**Knowledge–Geometry Decoupling (KGD)** addresses both:
- **BMTP** builds a clean, refreshable base knowledge by filtering transitions by collaborative and semantic relevance.
- **Read-only cross-attention** and **Anchored Calibration Residual (ACR)** let the task learner read the encoder's knowledge without overwriting it, while writing task-specific geometry orthogonally.

The framework achieves 4–12% improvement over strong baselines on eight public benchmarks and demonstrates sustained advantage over a 90-day production stream. In a live A/B test on Shopee Homepage Search, it increases GMV per user by 1.75% and advertising revenue by 1.53%.

**Future directions** implied by this work include extending the decoupled ownership principle to other domains with continuous distribution drift, exploring more sophisticated forms of orthogonal task residuals, and investigating how the framework scales to multi-task and multi-domain settings where multiple downstream tasks share one refreshable encoder.

---

_Markdown view of https://picx.dev/p/n4eRsD, served by PicX — AI-generated visual whiteboard summaries of research papers._