# ABot-AgentOS: A General Robotic Agent OS with Lifelong Multi-modal Memory

> ABot-AgentOS improves embodied reasoning and execution by integrating a deliberative agent layer with universal graph memory and self-evolution.

- **Source:** [arXiv](https://arxiv.org/abs/2607.10350)
- **Published:** 2026-07-15
- **Permalink:** https://picx.dev/p/mmBNvc
- **Whiteboard:** https://picx.dev/p/mmBNvc/image

## Summary

## Summary (Overview)

- **ABot-AgentOS** is a general robotic Agent Operating System that provides a deliberative layer connecting VLM/VLA models with physical robot execution, including reasoning, memory, tool use, verification, and cross-embodiment execution.
- **EmbodiedWorldBench** is introduced as an executable benchmark with 16 indoor/outdoor/hybrid scenes, 4 difficulty levels, and over 200 tasks covering navigation, object search, NPC dialogue, dynamic events, and trace-grounded scoring.
- **Universal Multi-modal Graph Memory** stores embodied experience as typed, source-grounded nodes and edges, supporting traceable retrieval and auditable answering.
- A **failure-driven self-evolution loop** converts diagnosed memory failures into gated runtime evo-assets, improving later evaluation splits without current-split ground-truth leakage.
- On memory benchmarks, ABot-AgentOS Static achieves 87.5 on LoCoMo, 59.9 on OpenEQA EM-EQA, 88.6 on Mem-Gallery, 76.5 on NExT-QA, and 65.4 on EgoLife; self-evolution further improves these scores.

## Introduction and Theoretical Foundation

The paper addresses three key gaps in current embodied AI systems:

1. **Reasoning-execution gap**: Most foundation-model-based robotic systems lack an intermediate agent layer for task decomposition, tool invocation, skill delegation, verification, recovery, and multi-step reasoning. They either map model outputs directly to actions or use monolithic pipelines.

2. **Embodiment-generalization gap**: Robotic agent systems are often tightly coupled to specific hardware, control APIs, or environment assumptions, making cross-body transfer costly.

3. **Persistent embodied-memory gap**: Short-term buffers, text-only caches, or task-specific memory modules cannot reliably preserve multi-modal, relational, source-grounded experience across long-term interactions.

The theoretical basis draws on **Dual-System Theory** (fast perception-action vs. slow deliberative reasoning), and prior work on agent systems (ReAct, Toolformer, Voyager, AutoGen) and long-term memory (MemoryBank, MemGPT, Mem0). The paper argues that a general Agent OS layer above low-level controllers can unify cognitive reasoning with physical execution, providing a reusable foundation for embodied agents.

## Methodology

### Agent Architecture (Agent Harness)

ABot-AgentOS uses a hierarchical LLM agent framework with three roles:

- **Main LLM**: Performs scene-conditioned task planning, forms high-level plans with explicit completion conditions, and delegates subtasks.
- **Skill Runner**: A skill-level subagent that manages procedural execution in an isolated local context, handling repeated observations, actions, recovery, and returning compressed outcomes.
- **Verifier**: Supervises execution at runtime, skill-level, and finish-time, checking consistency between agent belief and environment-grounded facts.

The system uses an **edge-cloud collaborative routing** layer: a lightweight Tiny LLM on the edge handles routine tasks, escalating to a cloud-based Large LLM for complex reasoning.

### Memory System

The memory is represented as a typed graph:

$$G = (V, E)$$

where each node $v \in V$ denotes an entity or evidence unit, and each edge $e \in E$ denotes temporal, semantic, spatial, identity, interaction, or provenance relations.

**Multi-modal Memory Updating**: Raw observations (dialogue, video, images) are converted into compact, source-grounded graph records. The writer prioritizes information likely to support future tasks (identities, object locations, state changes, etc.). Near-duplicate nodes are merged, and stale state facts are superseded via temporal edges.

**Retrieval and Grounded Answering**: Given a query $q$, the retriever computes a hybrid score:

$$s(q, v) = \lambda_{\text{sem}} s_{\text{sem}}(q, v) + \lambda_{\text{lex}} s_{\text{lex}}(q, v) + \lambda_{\text{meta}} s_{\text{meta}}(q, v) + \lambda_{\text{type}} s_{\text{type}}(q, v)$$

Top-ranked seeds are expanded along typed edges to produce a local evidence subgraph, which is serialized and injected into the reasoning context.

**Failure-Driven Lifelong Self-Evolution**: After each evaluation split $D_t$, the system collects traces $T_t$ and runs:

$$T_t, G_t = \text{Run}(D_t; G_{t-1}, A_{<t})$$
$$\Delta A_t = \text{Gate}(\text{Compile}(\text{Propose}(\text{Diagnose}(T_t))))$$
$$A_{\leq t} = A_{<t} \cup \Delta A_t$$

An asset $a$ is accepted only if:

$$\text{Accept}(a) = \mathbb{I}[\Delta S_{\text{target}}(a) \geq \tau_{\text{gain}} \land \Delta S_{\text{reg}}(a) \geq -\tau_{\text{reg}}]$$

This ensures no ground-truth leakage from the current split.

### Training Pipeline (Deployable Small Model)

A text-based semantic sandbox environment is used to generate teacher trajectories (via strong models). These trajectories are filtered and used for SFT. The student policy is then improved via online RL with an LLM-as-a-Judge reward engine. The reward engine provides turn-level and episode-level rewards:

$$r_{\text{episode}}(\tau) = \lambda_{\text{eff}} R_{\text{eff}}(\tau) + \lambda_{\text{cons}} R_{\text{cons}}(\tau) + \lambda_{\text{comp}} R_{\text{comp}}(\tau)$$

$$R(\tau) = \sum_{t=1}^{T} \hat{r}_t + r_{\text{episode}}(\tau)$$

A **Meta-Judge** validates reward quality using five dimensions and a quality score:

$$Q(x_t) = \sum_{k=1}^{5} w_k q_k$$

Low-quality cases are clustered and used to refine the reward prompt via a multi-agent self-evolution loop.

## Empirical Validation / Results

### Agent Evaluation on EmbodiedWorldBench Subset

| Agent Model | TSR | GCR |
|-------------|-----|-----|
| ReAct (Qwen3.6-Plus) | 49.97% | 57.95% |
| ABot-AgentOS (Qwen3.6-Plus) | 61.96% | 68.79% |
| ABot-AgentOS (DeepSeek-V4-Pro) | 68.18% | 74.62% |

ABot-AgentOS improves over the single-controller ReAct baseline by 11.99% in TSR and 10.84% in GCR under the same LLM. Using a stronger main LLM (DeepSeek-V4-Pro) further improves results.

### Memory Evaluation

**LoCoMo (Long-term conversational memory)** – Table 2:

| Method | Single-hop | Temporal | Multi-hop | Open-domain | Adversarial | Overall |
|--------|-----------|----------|-----------|-------------|-------------|---------|
| Mem0 | 94.1 | 96.6 | 90.8 | 68.8 | 62.3 | 85.6 |
| **ABot-AgentOS Static** | **92.9** | **87.5** | **90.9** | **70.8** | **80.9** | **87.5** |
| Human | 95.1 | 92.6 | 85.8 | 75.4 | 89.4 | 87.9 |

**OpenEQA (Embodied QA)** – Table 3 (EM-EQA, 24 frames):

| Method | ScanNet | HM3D | Overall |
|--------|---------|------|---------|
| GPT-5.4 Direct VQA (24) | 75.9 | 70.5 | 74.1 |
| **ABot-AgentOS Static** | **61.9** | **55.7** | **59.9** |
| Human | 87.7 | 85.1 | 86.8 |

**Mem-Gallery (Multi-modal memory)** – Table 4 (Overall):

| Method | Overall |
|--------|---------|
| GPT-5.4 Full context | 92.6 |
| **ABot-AgentOS Static** | **88.6** |
| MemGPT | 87.6 |

**NExT-QA (Video QA)** – Table 5 (Acc@All):

| Method | Acc@All |
|--------|---------|
| Qwen3.6-Plus Direct QA | 81.9 |
| GraphVideoAgent | 73.3 |
| **ABot-AgentOS Static** | **76.5** |

**EgoLife (Egocentric daily-life QA)** – Table 6 (Avg. accuracy):

| Method | Frames | Avg. |
|--------|--------|------|
| EGAgent-Gemini2.5 Pro | 1FPS→50 | 57.5 |
| WorldMM-Qwen3.5 Flash | Full | 56.0 |
| **ABot-AgentOS-Qwen3.5 Flash** | **1FPS→1** | **65.4** |

**Self-Evolution Results** (Figure 8): Self-evolution improves primary scores across all benchmarks: LoCoMo +1.2, OpenEQA +1.2, Mem-Gallery +0.4, NExT-QA +4.1, EgoLife +0.8. Gains are most notable in categories requiring temporal normalization, scene disambiguation, and causal-temporal reasoning.

## Theoretical and Practical Implications

- **Theoretical**: The work demonstrates that a structured, source-grounded graph memory is superior to flat text retrieval or raw visual buffers for long-horizon embodied reasoning. The separation of roles (main LLM, Skill Runner, Verifier) provides a principled way to handle the lack of explicit completion signals in embodied tasks. The self-evolution protocol formalizes a safe, cumulative improvement process without evaluation leakage.

- **Practical**: ABot-AgentOS offers a deployable OS that can work across heterogeneous robot embodiments (humanoids, quadrupeds, mobile manipulators) by using a plugin-based skill integration. The edge-cloud collaboration reduces latency and cost. The training pipeline enables transferring capabilities to smaller models for deployment. The benchmark (EmbodiedWorldBench) provides a standardized, reproducible evaluation for embodied agents.

- **Privacy**: The edge-cloud memory system with privacy-aware gating (99% accuracy in identifying non-shareable memories) allows sharing of public environmental knowledge while protecting sensitive personal information.

## Conclusion

ABot-AgentOS provides a unified foundation for physically grounded, memory-augmented, and continuously improving embodied agents. Key contributions include:

- A modular Agent OS with scene-conditioned planning, context-isolated skill execution, multi-stage verification, and edge-cloud collaboration.
- EmbodiedWorldBench, an executable benchmark covering diverse scenes and tasks with deterministic, trace-grounded scoring.
- A universal multi-modal graph memory with failure-driven self-evolution, enabling persistent, auditable, and improvable memory.

Results show that the Agent OS layer improves long-horizon execution, and the memory system achieves strong performance across multiple benchmarks. Self-evolution further improves later splits without leaking ground truth.

**Limitations** include: need for large-scale real-world validation under noisy perception and heterogeneous embodiments; limited scene diversity and social interaction depth in the current benchmark; dependence on structured traces and post-hoc feedback; and the need for stronger auditing and user control in privacy-aware sharing.

**Future directions** include: complete benchmark evaluation release, extending the training pipeline to visual observations and multi-modal feedback, and improving fine-grained observation policies and spatial reasoning.

---

_Markdown view of https://picx.dev/p/mmBNvc, served by PicX — AI-generated visual whiteboard summaries of research papers._