Summary (Overview)

• Problem: Distilling knowledge from proprietary LLMs to open-source models for agentic search is hindered by (1) hidden logits and mismatched tokenizers preventing token-level alignment, and (2) natural language trajectory imitation causing style drift and hallucination.
• Solution: Proposes Multi-Agent Protocol Distillation (MAPD), a joint distillation and reinforcement learning (RL) framework that uses a structured JSON protocol as a style-normalized intermediate representation between proprietary teachers and open-source students.
• Method: An offline multi-agent system (MAS) generates protocols containing task type, reasoning plan, extractive grounding facts, and answer verification. During training, the protocol is provided as privileged information to a conditioned branch of the student policy, enabling dense self-distillation alongside sparse outcome-based RL (GRPO).
• Results: MAPD achieves average success rates of 39.4% on Qwen3-1.7B and 44.4% on Qwen3-4B across seven QA benchmarks, consistently outperforming competitive baselines (SDAR, GRPO+OPSD, etc.). Improvements are more pronounced on complex multi-hop tasks.
• Key properties: The framework generalizes robustly across different proprietary teachers (Claude-Opus-4.6, GPT-5.5, Gemini-3.1-Pro) without retuning, and mitigates style drift and verbosity collapse.

Introduction and Theoretical Foundation

Agentic search enables LLMs to solve knowledge-intensive tasks by interleaving multi-step reasoning with retrieval. Optimizing such systems with outcome-based RL (e.g., GRPO) provides only sparse supervision (final answer correctness). Knowledge distillation can supply denser guidance, and advanced proprietary models are promising teachers. However, two major bottlenecks exist:

1. Heterogeneous tokenizers and hidden logits: Traditional online policy distillation (OPD) relies on token-level KL divergence alignment, which is undefined when teacher logits are inaccessible and tokenizer vocabularies differ.
2. Style drift from trajectory imitation: Free-form natural language trajectories from proprietary models contain verbose, idiosyncratic patterns. Forcing open-source students to mimic these causes severe style drift, hallucination, and poor generalization.

To address these, MAPD introduces a structured, style-normalized intermediate representation—a JSON protocol—that decouples core cognitive strategies from the teacher's linguistic shell. The protocol is synthesized offline by a MAS and used during training as privileged information, enabling dense self-distillation without requiring access to the teacher's logits or tokenizer.

Methodology

Structured JSON Protocol

The protocol $z$ encapsulates five essential dimensions:

• Task Type: single_hop, multi_hop, comparison, others
• Reasoning Plan: Ordered array of actionable sub-goals
• Grounding Facts: Extractive evidence from retrieved passages (verbatim substrings)
• Partial Findings (optional): Intermediate discoveries when search fails
• Answer Verification: Final answer + boolean answer_grounded flag

The privileged information is defined as $p = f(z)$ where $f$ is a descriptive template.

Multi-Agent System (MAS) Generation Pipeline

Three-stage offline pipeline:

1. Stage A – Multi-Agent Collaborative Search: An Orchestrator agent decomposes the query into sub-tasks. Searcher agents issue retrieval queries to a local Wikipedia corpus. If exact match (EM) with ground truth fails, a Repair agent diagnoses and retries (up to $R$ iterations). The output is an exploration log.
2. Stage B – Protocol Generation: A Protocolizer agent converts the log into a Succeeded Protocol (if answer found) or Evidence Protocol (if search failed, with neutral summary of partial findings). Constraints: no hindsight, extractive grounding.
3. Stage C – Quality Gate: Four automated checks: schema validation, EM consistency (for succeeded protocols), grounding verification, and leak detection. Passing rate: 99.33% on 3,000 instances.

Joint Training Objective

The student policy $\pi_\theta$ is trained with two complementary objectives:

• OPSD (Online Policy Self-Distillation): A privileged branch conditions on $s_t^+ = (x, p, y_{<t})$ while the student branch conditions on $s_t = (x, y_{<t})$. Both share the same parameters. The per-token reverse KL divergence provides dense supervision:

$$\mathcal{L}_{\text{OPSD}}^{(t)} = \mathbb{D}_{\text{KL}}\left( \pi_{\theta_k}(\cdot | s_t) \;\|\; \pi_{\theta_k}(\cdot | s_t^+) \right).$$

• GRPO (Group Relative Policy Optimization): Provides sparse outcome-based RL signal:

$$\mathcal{L}_{\text{GRPO}} = -\frac{1}{G} \sum_{i=1}^G \frac{1}{|y^{(i)}|} \sum_{t=1}^{|y^{(i)}|} \left[ J_t^{(i)} - \beta \mathbb{D}_{\text{KL}}\left( \pi_\theta \|\pi_{\text{ref}} \right) \right],$$

where $J_t^{(i)} = \min\left( \rho_t^{(i)} \hat{A}^{(i)}, \text{clip}(\rho_t^{(i)}, 1-\epsilon, 1+\epsilon) \hat{A}^{(i)} \right)$.

The overall objective combines both:

$$\mathcal{L}(\theta) = \lambda_{\text{OPSD}} \cdot \mathcal{L}_{\text{OPSD}}(\theta) + \mathcal{L}_{\text{GRPO}}(\theta).$$

The hyperparameter $\lambda_{\text{OPSD}}$ controls the trade-off (optimal value $\lambda=0.05$).

Empirical Validation / Results

Setup

• Student models: Qwen3-1.7B and Qwen3-4B (pretrained base checkpoints).
• Training data: NQ and HotpotQA train splits only.
• Evaluation: 7 QA benchmarks (3 single-hop: NQ, TriviaQA, PopQA; 4 multi-hop: HotpotQA, 2WikiMultihopQA, MuSiQue, Bamboogle). 5 are out-of-distribution.
• Baselines: Vanilla, OPSD (self-distillation only), GRPO (RL only), GRPO+OPSD, SDAR (advanced hybrid).
• Teacher: Claude-Opus-4.6 (default), GPT-5.5, Gemini-3.1-Pro.

Main Results

Table 1: Success rates (%) on 7 QA benchmarks. MAPD consistently outperforms all baselines, with larger gains on multi-hop tasks.

Ablation Study

Table 2: Component ablation. PM = Proprietary Model, SP = Structured Protocol, MAS = Multi-Agent System. Both SP and MAS are essential for best performance.

Key Findings

• Structured protocol is essential: Raw trajectory distillation (w/o SP&MAS) performs worse than no proprietary model (GRPO+OPSD). Replacing raw text with protocol gives +7.0 (1.7B) and +5.6 (4B) gains.
• MAS improves protocol quality: Full MAPD (MAS + SP) outperforms single-agent protocol (w/o MAS).
• Excessive distillation pressure ($\lambda=0.1$): Induces “retrieve-without-reasoning” shortcut, improving single-hop but harming multi-hop (e.g., 2Wiki drops from 45.7% to 38.4% on 4B).
• Weak distillation ($\lambda=0.01$): Signal collapses early but still improves over GRPO alone.
• Transfer across teachers: Narrow variance (<2 points) across Claude-Opus-4.6, GPT-5.5, Gemini-3.1-Pro without retuning.
• Cost: One-time generation cost ~$1,454 for 25,600 instances ($0.057 per instance).

Theoretical and Practical Implications

• Theoretical: MAPD provides a principled solution to the heterogeneous distillation problem by decoupling semantic reasoning from surface linguistic artifacts. The structured JSON protocol acts as a bridge, enabling dense self-distillation without requiring access to proprietary logits or cross-tokenizer alignment. This addresses a fundamental limitation of prior OPD and trajectory imitation methods.
• Practical: The framework is cost-effective (one-time offline protocol synthesis, zero inference overhead) and generalizes across different proprietary teachers. Practitioners can swap APIs to optimize cost-latency tradeoffs without retuning. The approach reduces style drift and verbosity collapse, which are critical for deploying open-source models in real-time agentic search applications.
• Implications for open-source development: MAPD shows that open-source models can effectively leverage proprietary capabilities without being exposed to harmful stylistic patterns, paving the way for safer and more efficient knowledge transfer.

Conclusion

MAPD introduces a joint distillation and RL framework that uses a structured JSON protocol as an intermediate representation to bridge the distribution gap between proprietary and open-source models in agentic search. By combining offline multi-agent protocol synthesis with online self-distillation and outcome-based RL, MAPD consistently outperforms existing methods on seven QA benchmarks, with larger gains on complex multi-hop tasks. The framework generalizes across different proprietary teachers without retuning and avoids style drift and verbosity collapse.

Future work could explore extending MAPD to other agentic tasks beyond search, integrating with process-level rewards, or developing fully autonomous self-improving frameworks that reduce reliance on proprietary models.