Summary of "DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learning"

Summary (Overview)

• Proposes DVAO: A novel method for multi-reward Reinforcement Learning (RL) in Large Language Model (LLM) alignment that dynamically adjusts combination weights based on the empirical variance of each reward objective within a rollout group.
• Addresses Key Limitations: Solves the training instability of Reward Combination (RC) caused by large advantage magnitudes and the suboptimal trade-offs of Advantage Combination (AC) due to its reliance on static weights and ignorance of cross-objective correlations.
• Provides Theoretical Guarantees: Mathematically proves that DVAO maintains bounded advantage magnitudes for stable training and introduces a self-adaptive cross-objective regularization mechanism.
• Demonstrates Superior Performance: Extensive experiments on mathematical reasoning and tool-use benchmarks show DVAO outperforms baselines (GRPO, RC, AC, GDPO), achieving a superior multi-objective Pareto frontier and robust training stability.
• Enables Synergistic Optimization: The method's dynamic weighting allows it to up-weight objectives with stronger learning signals (higher variance) and suppress noisy ones, promoting holistic policy improvement.

Introduction and Theoretical Foundation

Reinforcement Learning, particularly Group Relative Policy Optimization (GRPO) and its variants, has become a standard paradigm for aligning LLMs. However, real-world applications require optimizing multiple objectives simultaneously (e.g., accuracy, length constraints, low hallucination rates, correct tool-calling format). The standard scalarization practices for multi-reward GRPO are:

1. Reward Combination (RC): Linearly combines raw rewards before advantage calculation.
2. Advantage Combination (AC): Independently normalizes rewards to advantages and then linearly combines them.

Both methods have significant drawbacks. RC frequently generates advantages with excessively large squared magnitudes, leading to erratic policy gradients and training instability. AC, while normalizing magnitudes, relies on static hyperparameters and isolates objectives during normalization, failing to capture intricate cross-objective correlations and leading to suboptimal trade-offs.

This work proposes Dynamic Variance-adaptive Advantage Optimization (DVAO) to bridge the gap. DVAO dynamically adjusts combination weights based on the empirical reward variance of each objective within the rollout group. This data-driven method up-weights objectives with higher variance (indicating a stronger learning signal) and suppresses noisy, low-variance ones.

Formal Preliminaries: Given a policy model $\pi_\theta$, for a query $x_i$ and response $y_j$, there are $n$ reward functions $r_1, r_2, \dots, r_n$. The standard convex combination for scalarization is:

$$r^{(i,j)}_{sum} = \sum_k w_k r^{(i,j)}_k, \quad \sum_k w_k = 1, w_k \in [0,1]$$

In GRPO, the relative advantage for a group of $G$ rollouts is:

$$A^{(i,j)}_{sum} = \frac{ r^{(i,j)}_{sum} - \text{mean}(\{r^{(i,j)}_{sum}\}_{j=1}^G) }{ \text{std}(\{r^{(i,j)}_{sum}\}_{j=1}^G) }$$

The policy optimization objective is:

$$J_{GRPO}(\theta) = \mathbb{E}_{x_i \sim \mathcal{D}, \{y_j\}_{j=1}^G \sim \pi_\theta(\cdot|x_i)} \left[ \frac{1}{G} \sum_{j=1}^G \frac{1}{|y_j|} \sum_{t=1}^{|y_j|} \min\left( s_{j,t}(\theta) A^{(i,j)}_{sum}, \text{clip}(s_{j,t}(\theta), 1-\epsilon, 1+\epsilon) A^{(i,j)}_{sum} \right) \right]$$

where $s_{j,t}(\theta) = \frac{\pi_\theta(y_{j,t}|x_i, y_{j,<t})}{\pi_{\theta_{old}}(y_{j,t}|x_i, y_{j,<t})}$.

Methodology

DVAO replaces the fixed combination weights $w_k$ with dynamic variance-adaptive weights:

$$\tilde{w}_k = \frac{w_k \sigma^i_k}{\sum_l w_l \sigma^i_l}$$

where $\sigma^i_k = \text{std}(\{r^{(i,j)}_k\}_{j=1}^G)$ is the group standard deviation for objective $k$. The DVAO advantage is computed as:

$$A^{(i,j)}_{DVAO} = \sum_k \tilde{w}_k A^{(i,j)}_k = \frac{\sum_k w_k \sigma^i_k A^{(i,j)}_k}{\sum_l w_l \sigma^i_l}$$

where $A^{(i,j)}_k$ is the normalized advantage for objective $k$ (as in AC).

Key Theoretical Properties:

1. Bounded Advantage Magnitude (Proposition 2): For a fixed query and rollout group, DVAO produces a pointwise smaller or equal advantage magnitude compared to RC: $$A^{(i,j)}_{DVAO} \leq A^{(i,j)}_{sum}, \quad \forall j \in \{1,2,\dots,G\}$$ Equality holds only if all reward pairs are perfectly positively correlated. This mitigates the training instability of RC.
2. Cross-Objective Regularization (Proposition 3): The sensitivity of the DVAO advantage to a raw reward $r^{(i,j)}_k$ is: $$\frac{\partial A^{(i,j)}_{DVAO}}{\partial r^{(i,j)}_k} = \frac{\tilde{w}_k}{\sigma^i_k} \left[ 1 - \frac{1}{G-1} G A^{(i,j)}_{DVAO} A^{(i,j)}_k \right]$$ In contrast, for AC it is: $\frac{\partial A^{(i,j)}}{\partial r^{(i,j)}_k} = \frac{w_k}{\sigma^i_k} \left[ 1 - \frac{1}{G-1} G (A^{(i,j)}_k)^2 \right]$. This shows DVAO's gradient contribution for an objective is modulated by the cross-term $A^{(i,j)}_{DVAO} A^{(i,j)}_k$, which depends on the model's overall multi-objective performance, introducing an implicit cross-objective regularization. AC's gradient depends only on the isolated objective's performance $(A^{(i,j)}_k)^2$.

Empirical Validation / Results

Experiments were conducted on mathematical reasoning (AIME-2024, AIME-2025, MATH500, OlympiadBench, AMC23) and tool-use (Berkeley Function Call Leaderboard - BFCL-v4) benchmarks using Qwen3 and Qwen2.5 models. Objectives were accuracy vs. length compliance (math) and accuracy vs. format compliance (tool-use).

Main Results (Key Tables):

Table 1: Performance on Mathematical Reasoning Tasks

Table 2: Performance on Tool-Use Task (BFCL-v4)

Key Findings:

• DVAO achieves the highest average accuracy and near-perfect auxiliary compliance (length/format) across both tasks and model scales.
• Baselines typically sacrifice one dimension for the other (e.g., RC/AC trade accuracy for compliance, GDPO achieves near-perfect compliance at the cost of very low accuracy).
• DVAO's advantage is consistent despite all methods sharing equal-weight initialization.

Training Dynamics & Pareto Frontiers:

• Training Stability: DVAO consistently achieves the highest accuracy reward with the lowest variance (standard deviation), and drives the length/format reward closest to the target (1.0) with the most dramatic variance collapse, confirming its bounded advantage property.
• Pareto Superiority: When sweeping the accuracy weight, DVAO's Pareto frontier dominates those of all baselines (RC, AC, GDPO), maintaining high auxiliary compliance across the entire accuracy range. Baselines exhibit saturation, instability, or incoherent fluctuations.

Theoretical and Practical Implications

• Theoretical: The paper provides a rigorous mathematical analysis of multi-reward GRPO scalarization, proving the instability of RC (Proposition 1) and the superior properties of DVAO (Propositions 2 & 3). DVAO formally guarantees bounded advantage magnitudes and introduces a novel, data-driven cross-objective regularization mechanism.
• Practical: DVAO offers a hyperparameter-free, dynamic weighting scheme that significantly improves the stability and effectiveness of multi-objective RLHF for LLMs. It enables the training of models that excel at primary tasks (e.g., reasoning accuracy) while robustly adhering to critical auxiliary constraints (e.g., response length, tool-calling format), which is essential for real-world deployment.

Conclusion

The paper identifies fundamental flaws in standard multi-reward scalarization techniques for GRPO: Reward Combination leads to training instability due to large advantage magnitudes, and Advantage Combination relies on static weights and ignores cross-objective correlations. To address these, the authors propose Dynamic Variance-adaptive Advantage Optimization (DVAO), which dynamically adjusts combination weights based on the empirical variance of each objective's rewards. DVAO is proven to maintain bounded advantage magnitudes and introduce an implicit cross-objective regularization. Extensive experiments demonstrate that DVAO achieves a superior Pareto optimal policy, effectively balancing multiple objectives without manual hyperparameter tuning. Future work will explore scaling DVAO to environments with more reward functions and extending its mechanism to broader alignment paradigms.