# Envisioning Agentic Robotics: A New Era of Robotic Systems

> Agentic robotics replaces task-specific policies with a foundation model plus embodied harness as the decision-making core, enabling broad generalization but facing capped scaling and latency-safety tradeoffs.

- **Source:** [arXiv](https://arxiv.org/abs/2611.18359)
- **Published:** 2026-11-17
- **Permalink:** https://picx.dev/p/x2t38c
- **Whiteboard:** https://picx.dev/p/x2t38c/image

## Summary

# Summary of "Envisioning Agentic Robotics: A New Era of Robotic Systems"

## Summary (Overview)

- **Paradigm shift**: The paper proposes a shift from policy-centric robotics to **agentic robotics**, where a foundation model (e.g., GPT-6 Astra) with an embodied harness serves as the decision-making core (System 2) rather than training task-specific policies.
- **Key framework**: Introduces a three-system reference architecture—**System 2 (S2)** for reasoning/orchestration/verification, **System 1 (S1)** for executable skills, and **System 0 (S0)** for real-time body control—with S2 realized by a foundation model plus an embodied harness.
- **Evaluation framework**: Proposes a 9-dimension, 53-subdimension evaluation framework distinguishing model-level capabilities from closed-loop system performance, with 9 reporting principles including world-clock execution and independent verification.
- **Physical constraints**: Identifies **capped scaling** (improving capability within fixed decision deadlines) as the key challenge, contrasting with the monotonic scaling of frontier models; documents that latency becomes a safety variable in live execution.
- **Research roadmap**: Proposes a **"Ladder of Delegable Work"** with four levels—L1 (primitives), L2 (tasks), L3 (workflows), L4 (roles)—and identifies four research directions: embodied reasoning under constraints, persistent verifiable control, efficient adaptation, and continual improvement from physical experience.

## Introduction and Theoretical Foundation

### Background and Motivation

The paper begins by tracing the evolution of robotics from hand-engineered modular architectures (Nilsson, 1984) to deep reinforcement learning policies for locomotion (Hwangbo et al., 2019), and more recently to vision-language-action (VLA) models like the π series (Black et al., 2025a,b) and world-action models (WAMs). Despite these advances, **reliable generalization** remains a central challenge—policy performance deteriorates when scenes or instructions change (Zhou et al., 2025c).

The release of **GPT-6 Astra** (OpenAI, 2026a,b) offered a different perspective: a general-purpose model outperformed trained policies on many robotic benchmarks without task-specific fine-tuning (Zhang et al., 2026c). This motivates the paradigm shift from policy-centric to agentic robotics.

### Theoretical Foundation: Embodied System 2

The paper formally defines:

> **Definition (Embodied System 2).** Embodied System 2 (Embodied S2) is the functional layer of an agentic robot that interprets intent, reasons about goals and constraints to make decisions, orchestrates skills and tools accordingly, and verifies outcomes to adapt subsequent decisions toward the intended goals.

S2 is identified by two kinds of decisions:
1. **Decisions on how to proceed**: interpreting intent and choosing next subgoals, skills, or whether to continue/retry/replan/ask/stop
2. **Decisions on outcomes**: judging from evidence whether previous steps achieved their goals

### The Agentic Loop

The paper traces six logical steps forming a feedback cycle:
1. **Interpret Intent** — combine user request with context
2. **Decide** — reason about goals and constraints
3. **Invoke** — call S1 skill, S0 interface, or external tool
4. **Execute** — the invoked capability runs under its contract
5. **Verify** — distinguish four questions: was the request accepted, has execution terminated, does the physical effect hold, does it satisfy user intent
6. **Continue, Replan, Ask, or Report** — adapt based on verification

## Methodology

### Reference Architecture

The paper defines three functional roles plus a real-time event monitor:

| System | Role | Examples |
|--------|------|----------|
| **S2** | Intent, reasoning, orchestration, verification | Foundation model + embodied harness |
| **S1** | Skills and callable modules | VLA policies, perception, motion planning |
| **S0** | Body interface and local real-time control | Whole-body tracking, IK, gait control |

**Key architectural components of the embodied harness:**
- **Context manager**: assembles world, user, and body state
- **Tool interface**: exposes S1 skills, S0 calls, code execution
- **Planner**: maintains and revises task plans
- **Memory**: stores/retrieves/updates information beyond context
- **Verifier**: judges whether steps achieved subgoals

**Three execution paths** (Figure 4):
1. **Layered orchestration**: S2 invokes S1 skills → S0
2. **Direct control**: S2 controls S0 directly through action interface
3. **Code as policy**: S2 writes code executed by runtime interpreter

### Evaluation Framework

The paper proposes evaluating at **two levels**:

**Model level**: foundation model answers alone, single-turn, without tools—isolates knowledge, understanding, model-side reasoning and safety.

**System level**: model acts through harness, invokes S1/S0—measures closed-loop reasoning, tool use, memory, cross-embodiment, safe execution, efficiency, cost.

**Nine evaluation dimensions** with 53 subdimensions:
1. Knowledge (world, embodied, programming)
2. Understanding (multimodal perception, spatial relations, physical states, affordances, temporal structure, embodiment, progress, uncertainty)
3. Reasoning (intent inference, causal prediction, hierarchical planning, skill selection, executable action inference, failure recovery, interruption handling)
4. Tool Use (perception, motion planning, action, code tools)
5. Memory (object, spatial, sequential, capacity, body, user)
6. Cross-Embodiment (task transfer, body understanding, experience reuse)
7. Safety (8 axes from risk recognition to coordinated interruption)
8. Efficiency (latency, model calls, observation age)
9. Cost (tokens, API charges, task time, robot occupancy)

### Key Evaluation Principles

The paper proposes 9 principles for evaluation:
1. Separate model from system (fix one, vary the other)
2. Treat action interface as controlled variable
3. **Run on a world clock** (don't pause simulation during deliberation)
4. Disclose observation contract
5. Verify completion independently
6. Quantify uncertainty and reliability
7. Control and sweep budgets
8. Validate on real robots
9. Fix conditions of level claims in advance

The **observation age** is formally defined as:

$$
\Delta t_{\mathrm{obs}} = t_{\mathrm{effect}} - t_{\mathrm{capture}},\tag{1}
$$

where $t_{\mathrm{capture}}$ is when the observation behind a decision was captured and $t_{\mathrm{effect}}$ when the resulting action takes effect.

## Empirical Validation / Results

### Current Results with GPT-6 Astra as S2 Foundation Model

| Benchmark | S2 Foundation Model | S1 Components | Result |
|-----------|-------------------|---------------|--------|
| RoboDojo-Sim | GPT-6 Astra | None | 22.48%; above all 40 public policies |
| RoboDojo-Sim (official) | GPT-6 Astra | Agent-written tools | 46.9%; first on leaderboard |
| RoboLab-120 | GPT-6 Astra | None | 66.3% direct / 74.8% agentic |
| LIBERO-PRO | GPT-6 Astra | Motion primitives + π₀.₅ in code | 94.0% vs π₀.₅: 53.0% |
| RoboMME | GPT-6 Astra | None | 92.1%; best leaderboard: 60.8% |
| LADDER | GPT-6 Astra | None | 62.0%; learned policies: ≤29.6% |
| RobotWorld | GPT-6 Astra | None | 19.0%; 63 tasks unsolved by all models |
| RoboQuest | GPT-6 Astra | None | 23.2% |
| HumanCLAW | Best of 9 VLMs | Whole-body skills | 16.8% |
| RT-Safe (hard routes) | Eight VLMs | None | Safe success 19.8% (paused) vs 0.7% (live) |

### Seven Key Conclusions from Evidence

1. **General models with harnesses are competitive on narrower simulated suites** — but often received privileged information (e.g., written task recipes)

2. **Broad benchmarks remain far from solved** — best models reach only 16.8–23.2% on broad/exploratory benchmarks

3. **Current weak points are physical and evidential** — failures cluster on precise manipulation, contact-rich interaction, dynamic control, and gathering/judging evidence. GPT-6 Astra solved **none** of the 11 locomotion tasks in RobotWorld.

4. **Harness and protocol move results substantially** — with model fixed, harness changes moved success by 9–35 percentage points; interface changes alone moved results by up to 52 points

5. **Tools add capability only beyond what the model can already do** — skill-only interface raised GPT-5.6-Sol from 11% to 63% but lowered GPT-6 Astra from 92% to 61%

6. **Paused worlds hide latency, cost, and risk** — live execution raised collisions 12.3-fold on RT-Safe; agentic Astra sessions used 30.9M tokens ($36.40) per successful episode vs 160k tokens ($1.96) for direct prompting

7. **Safety learned in text transfers only partially to action** — refusal fell from 72% of text requests to 18% of real-robot trials

### Latency Comparison Across Layers

| Layer | Rate | Time per decision |
|-------|------|-------------------|
| S0 loops | 50 Hz – 1 kHz | 1–20 ms |
| S1 control skills | 5–200 Hz | 5–200 ms |
| GPT-6 Astra (S2) | — | 7.7–39.86 s per decision |

## Theoretical and Practical Implications

### Digital vs. Physical Agents

The paper identifies **four enabling conditions** of digital agents that fail in the physical world:

| Condition | Digital (Coding) Agent | Physical Agent |
|-----------|----------------------|----------------|
| **Native interface** | Directly readable files, logs | Partial, noisy, occluded sensing; state must be constructed |
| **Verifiable outcomes** | Fast automated tests | Costly, slow, hard to reproduce |
| **Abundant data** | Decades of public code | No Internet-scale corpus; robot data produced by operating robots |
| **Cheap environments** | Sandboxes copy/reset/parallel | Simulators faithful only for rigid-body motion; real resets need time/people |

**Five principles that transfer:**
1. Close the loop on evidence
2. Treat tools as contracts
3. Manage context explicitly, keep durable state outside it
4. Separate doing from judging
5. Package capabilities as reusable skills

**Three additional constraints from the physical world:**
- **Deliberation is not free** — world evolves while robot thinks
- **Errors cannot be undone** — no version control for physical actions
- **Bodies do not stay fixed** — motors heat, wear, drift; same command produces different motion over time

### The Ladder of Delegable Work

| Level | Name | Description | Typical Duration |
|-------|------|-------------|-----------------|
| **L1** | Primitive | One operation; skills + local checks | Seconds |
| **L2** | Task | One complete task; verification + recovery | Minutes |
| **L3** | Workflow | Scheduling + progress tracking | Hours |
| **L4** | Role | Sustained responsibility; memory + initiative + safe improvement | Days and longer |

**Current status**: Systems can execute primitives and tasks (L1-L2); first workflow-level results are single-company demonstrations; no robot system in unstructured environments has been reported to hold a role (L4) without human repair.

### Four Future Research Directions

1. **Embodied Reasoning Under Physical Constraints** — maximize decision quality subject to response deadlines; make capped scaling an explicit training objective

2. **Persistent and Verifiable Control Across Tasks and Spaces** — state representation spanning rooms/floors/days; workflow coordination; division of judgment between model and execution system

3. **Efficient Adaptation Across Environments and Embodiments** — capability cards, skill libraries, in-context adaptation; conditioning policies on commanded actions and resulting motions

4. **Continual Improvement from Physical Experience** — distinguish adaptation, continual improvement, and recursive self-improvement (RSI); requires held-out task gains, no regression, and separated contributions

## Conclusion

### Key Takeaways

1. **Agentic robotics is a paradigm shift**: from learned action policies to agents that pursue goals by coordinating skills, evaluating outcomes, and adapting decisions.

2. **Embodied S2 is the decision-making core**: realized by a foundation model plus embodied harness, it must translate general-purpose reasoning into effective, verifiable actions under physical constraints.

3. **Evaluation must distinguish model from system**: nine dimensions with 53 subdimensions, evaluated at both model and system levels, with transparent reporting via harness cards.

4. **Physical constraints reshape the problem**: capped scaling, interactivity, deployability, and fresh observations require new approaches beyond answer accuracy.

### Community Priorities

The paper calls for three priorities:
1. **Develop reliable evaluation frameworks** that complement simulation with real-robot testing and extend coverage to higher responsibility levels
2. **Improve reasoning efficiency** for time-sensitive, safety-critical, and highly dynamic settings
3. **Standardize reporting** of model, harness, capabilities, evaluation conditions, inference budgets, response times, and human assistance

### Open Questions

- How to construct benchmarks for workflows and roles at building/site scale over days rather than minutes?
- Whether rankings of S2 foundation models in simulation hold on real robots?
- How to obtain statistically reliable evidence as evaluation extends to larger spaces and longer time periods?
- How to divide judgment between model and execution system for continuous control compatibility?

The paper concludes: "We envision a new era of robotic systems that understand human intent, adapt to changing circumstances, and reliably take on broader responsibilities over longer time horizons. Agentic robotics offers a system-level path toward this vision, bringing the general-purpose capabilities of foundation models into systems that people can trust with sustained work in the physical world."

---

_Markdown view of https://picx.dev/p/x2t38c, served by PicX — AI-generated visual whiteboard summaries of research papers._
