# Progress Reward Modeling for Robotic Learning: A Comprehensive Survey

> This survey unifies progress reward modeling via a three-part framework: interface, construction, and data/evaluation protocols.

- **Source:** [arXiv](https://arxiv.org/abs/2607.21655)
- **Published:** 2026-07-29
- **Permalink:** https://picx.dev/p/MlZ6z3
- **Whiteboard:** https://picx.dev/p/MlZ6z3/image

## Summary

# Progress Reward Modeling for Robotic Learning: A Comprehensive Survey

## Summary (Overview)

- **Unified framework**: The survey provides a three-part organizational structure for progress reward models in robotic learning, covering (1) the input–output interface, (2) internal construction mechanisms, and (3) data pipelines and evaluation protocols.
- **Four construction paradigms**: Existing methods are categorized into frozen foundation-model scoring, learning from temporal/relative supervision, instruction-tuned progress prediction, and programmatic reward generation.
- **Interface taxonomy**: The input interface is characterized by task-state representation (single observation, temporal context, relational comparison, or state-access APIs) and goal specification (language, vision, or structured/programmatic), while the output interface includes state-wise scores, progress deltas, rankings, and executable reward functions.
- **Data and evaluation taxonomy**: Progress supervision is categorized by degree of human involvement (human-driven, human-in-the-loop, fully automated), and benchmarks are grouped into progress fidelity, robustness/generalization, and downstream utility, clarifying what different evaluations actually validate.
- **Key limitations identified**: Current models lack fine-grained sensitivity, adaptive temporal modeling, efficient online inference, and reliable long-horizon memory, pointing to critical future directions.

## Introduction and Theoretical Foundation

Robotic learning operates in dynamic, sequential, high-dimensional environments. A robot must interact over many steps, and similar actions can have different effects depending on state and goal. The same task can be completed via different action sequences, creating a large behavior space.

Most systems rely on a **terminal success signal**—a binary indicator of whether the task was eventually completed. This signal provides no information about intermediate execution: it cannot distinguish progress, stagnation, or regression. The problem worsens in long-horizon tasks where successful outcomes are rare.

**Progress rewards** provide a richer signal by describing how task execution changes over time. They enable:

- Dense feedback before task completion, improving credit assignment
- Distinguishing useful behavior from irrelevant motion
- Detecting stagnation, regression, or failure before the final outcome
- Supporting online monitoring, trajectory reranking, data filtering, planning, action selection, and failure recovery

However, estimating progress is harder than binary success detection. Progress is a **latent, history-dependent, task-conditioned state**—not a directly observable property of a single frame. The model must infer which parts of the task have been completed, what is currently happening, and how far execution remains from the goal. The same observation may represent different stages under different tasks, and progress may depend on earlier actions and whether previous progress was undone.

The paper notes that the literature is fragmented: methods described under similar terms solve different problems (e.g., predicting success probability from a single image vs. estimating progress percentage from a video vs. comparing two trajectories). Evaluations are similarly heterogeneous, making direct comparison difficult.

## Methodology

The survey organizes progress reward modeling into three connected steps, corresponding to the main sections of the paper.

### 1. Interface of Progress Models (Section 2)

The interface determines what the model receives and what it produces. It is organized around three questions:

**Input – Task-State Representation:**
- **Single observation**: Current state + task condition → reward/success/value. Assumes the current state contains enough evidence. Low latency but ignores execution history.
- **Temporal context**: Sequence of observations (recent window or full trajectory). Stronger temporal evidence but weaker online usability.
- **Relational comparison**: Before–after, initial–current, or goal-conditioned pairs. Progress becomes a relational judgment; local comparisons may miss long-horizon dependencies.
- **State-access interfaces**: Environment code, simulator APIs, proprioceptive features. Effective in simulation but unrealistic for open-ended real-world scenes.

**Input – Task Goal Specification:**
- **Language-conditioned**: Easy to encode, compatible with VLMs, but underspecifies physical details for long/compositional tasks.
- **Vision-conditioned**: Goal images, demonstrations, or visual references. Richer grounding but requires curated data and is sensitive to visual variations.
- **Structured/programmatic**: Predicates, object states, constraints, or generated objective functions. Precise but requires explicit state variables.

**Output – Model Prediction Form:**
- **State-wise scalar score**: Maps a state to a number (progress, success likelihood, value, similarity). Most direct but meaning depends on calibration.
- **Progress delta**: Predicts change between two observations (e.g., before–after). Naturally aligned with RL but needs accumulation over time.
- **Ranking**: Orders states, clips, or trajectories. Easier to supervise but only defines progress relatively.
- **Reward function**: Executable program over state variables. Interpretable but relies on reliable state access and task decomposition.

### 2. Construction of Progress Rewards (Section 3)

Four main paradigms are identified:

**Frozen Foundation-Model Scoring (Section 3.1):**
- Uses pretrained models (CLIP, VLMs) as zero-shot reward scorers.
- Examples: CLIP image-text similarity, TOPReward (prompting VLM with task-completion question), GVL/OpenGVL (temporal progress from video).
- **Strength**: Zero-shot usability. **Weakness**: Outputs are semantic priors, not calibrated progress rewards; requires normalization, thresholding, or prompt engineering.

**Learning from Temporal/Relative Supervision (Section 3.2):**
- Learns reward geometry from demonstration time ordering or pairwise preferences.
- Temporal order: goal-proximity imitation learning, VIP, perceptual distance.
- Preferences: RL-VLM-F (generates preferences from frozen VLM), PEARL (transfer across tasks), Rank2Reward.
- **Strength**: Easier supervision than scalar scores. **Weakness**: Does not directly provide scalar rewards; must convert comparisons.

**Instruction-Tuned Progress Prediction (Section 3.3):**
- Formulates progress estimation as an explicit instruction-following task for foundation models.
- Models are fine-tuned on prompts + target responses: absolute progress, success, relative improvement, preferences, reasoning-grounded scores.
- Examples: RoboReward, Robometer, Robo-Dopamine, VLAC, Video-Language Critic, ProgressLM.
- **Key idea**: Progress is a dedicated capability, not an implicit by-product of representation learning.

**Programmatic Reward Construction (Section 3.4):**
- Represents rewards as executable programs (code, predicates, feature functions).
- Examples: Text2Reward, Eureka, Code as Reward, ELEMENTAL.
- **Strength**: Interpretable, flexible, effective when tasks decompose into explicit subgoals. **Weakness**: Shifts challenge to task decomposition, feature design, and reliable state access; may be misaligned if APIs fail to capture intended behavior.

### 3. Data and Benchmarks (Section 4)

**Data Construction (Section 4.1):**
- **Human-driven**: Teleoperation, manual annotation (scalar scores, success cutoffs, keyframes, reward sketches). Captures task semantics but costly and subjective.
- **Human-in-the-loop**: Sparse human anchors + automated interpolation; active learning; human verification of model-generated labels. Balances semantic control with scale.
- **Fully automated**: Temporal structure (later frames = more progress), simulator states, ground-truth rewards, foundation-model annotations, data augmentation (truncation, reversal, perturbation). Scalable but assumes temporal correlation or capture only expressed properties.

**Benchmarking (Sections 4.2–4.4):**
- **Progress Fidelity**: Scalar calibration (regression error, correlation), temporal consistency (ordering accuracy, rank correlation), relative progress (pairwise ordering), task grounding (counterfactual instructions), answerability/uncertainty (selective prediction, confidence calibration).
- **Robustness & Generalization**: Task generalization (unseen tasks/instructions), viewpoint/scene robustness, cross-embodiment transfer, non-monotonic/imperfect execution (failures, reversals, plateaus).
- **Downstream Utility**: Online policy learning (success rate, sample efficiency, stability), offline learning/reward relabeling, retrieval/filtering/reranking, planning/action selection.

## Empirical Validation / Results

The paper is a survey, not an original empirical study. It does not present new experiments. Instead, it synthesizes and organizes the results of many prior works. The empirical findings reported in the literature are classified by the evaluation dimensions above.

Key observations from the literature synthesis:

- Frozen foundation models (CLIP, VLMs) provide zero-shot rewards but often require normalization, thresholding, or differencing to be effective.
- Temporal/relative learning methods (VIP, reward sketching) can produce smooth reward landscapes but may fail on non-monotonic execution.
- Instruction-tuned progress models (ProgressLM, Robometer) achieve better calibration and task grounding but require expensive fine-tuning data.
- Programmatic methods (Eureka, Text2Reward) achieve high precision in simulation where state variables are accessible, but struggle in real-world settings with noisy observations.
- Cross-embodiment and cross-task generalization remains challenging; most methods show degraded performance on unseen tasks, viewpoints, or embodiments.
- Downstream policy improvement is often reported, but the survey cautions that improved policy performance does not by itself demonstrate faithful progress estimation.

## Theoretical and Practical Implications

**Theoretical implications:**
- The survey provides a unified language for describing progress models, enabling clearer comparison of methods that were previously treated as separate subfields.
- It reveals that progress estimation is a fundamentally different problem from success detection: progress is latent, history-dependent, and task-conditioned, requiring explicit modeling of temporal structure and task understanding.
- The taxonomy of construction paradigms clarifies the assumptions behind each approach (e.g., temporal order assumes monotonic progress; frozen models assume semantic priors are sufficient; programmatic methods assume state variables are accessible).

**Practical implications:**
- For practitioners, the interface-based view helps in selecting the right model for a given deployment: single-observation models for low-latency online use, temporal context for accuracy, relational comparisons for preference-based learning.
- The data construction taxonomy guides resource allocation: human-driven for semantic accuracy, automated for scale, human-in-the-loop for a balance.
- The benchmark taxonomy clarifies what different evaluation metrics actually measure, preventing misinterpretation of results (e.g., temporal consistency does not imply absolute calibration; downstream policy improvement does not validate progress fidelity).
- Identified limitations (coarse granularity, fixed-rate assumptions, inference latency, lack of long-horizon memory) directly inform engineering priorities for real-time robotic learning systems.

## Conclusion

The survey provides a comprehensive, unified view of progress reward modeling for robotic learning. It organizes the field around three perspectives: the interface of progress models (what they receive and produce), the internal construction mechanisms (how they produce progress signals), and the data and evaluation pipelines (how supervision is obtained and how quality is validated). This three-part framework connects what a progress model is, how it is built, and how its quality is assessed.

**Main takeaways:**
- Progress reward modeling is a rapidly growing research area with diverse methods that can now be systematically compared under the proposed unified framework.
- Four construction paradigms exist, each with distinct strengths and weaknesses: frozen models are zero-shot but uncalibrated; temporal/relative learning is scalable but assumes monotonicity; instruction-tuned models are accurate but expensive; programmatic methods are precise but environment-dependent.
- Data construction and evaluation must be carefully matched: different benchmarks validate different properties, and no single metric captures all aspects of progress fidelity.

**Future directions:**
1. **Fine-grained sensitivity**: Combine visual evidence with proprioception, force, tactile sensing to capture subtle physical changes (precise alignment, unstable grasping, early contact failure).
2. **Adaptive temporal modeling**: Replace fixed-rate time assumptions with event-driven progress estimation that handles sudden jumps, long plateaus, stagnation, and regression.
3. **Efficient online inference**: Use streaming encoders, cached task representations, model distillation, and hierarchical reward querying (lightweight model for frequent updates, larger model at key transitions).
4. **Long-horizon memory**: Maintain compact task memory that records completed, pending, and invalidated subgoals to resolve ambiguity when visually similar states correspond to different progress stages.

The survey concludes that progress reward modeling is a critical direction for improving robotic learning in dynamic, long-horizon tasks, and that addressing the identified limitations will be necessary for real-time deployment in open-world settings.

---

_Markdown view of https://picx.dev/p/MlZ6z3, served by PicX — AI-generated visual whiteboard summaries of research papers._