JarvisHub: An Open Harness for Canvas-Native Multimodal Creative Agents

Summary (Overview)

• Creative AI paradigm shift: JarvisHub addresses the gap between single-step asset generation and long-horizon multimodal production by treating the canvas as the shared workspace, external memory, and action space for both humans and agents.
• Three-layer architecture: The harness combines a canvas state layer (storing typed artifacts, dependencies, versions, and feedback), a protocol bridge (validating and logging all canvas mutations), and an agent runtime (planning, tool invocation, and state synchronization).
• Formalized project graph: Creative projects are represented as a typed artifact graph $G_t = (V_t, E_t)$ with addressable nodes, provenance metadata, user interaction records, and spatial layouts—enabling reuse, inspection, and traceability across turns.
• Demonstrated on high-value tasks: Qualitative results on narrative media generation, interactive web development, and presentation deck generation show that JarvisHub supports inspectable, feedback-driven, and recoverable creative workflows.
• Open harness for research: The system provides structured trajectories $\tau = \{(q_t, C_t, \Gamma_t, \Omega_t, a_t, o_t, f_t, r_t, C_{t+1})\}$ that enable process-level evaluation, benchmark construction, and training data for future creative agents.

Introduction and Theoretical Foundation

Background: Recent advances in multimodal generation (images, videos, audio, UI elements, etc.) have made high-quality asset production substantially easier. However, real-world creative work involves references, drafts, alternatives, edits, failed attempts, version relations, tool actions, evaluation signals, and human feedback—forming an evolving project state that existing systems fail to capture.

Limitations of existing approaches:

• Prompt-to-output tools (e.g., standalone generation models): Effective for individual assets but discard intermediate decisions, failed trials, and revision history.
• Chat-based creative agents (e.g., Claude, GPT-based assistants): Follow multi-step instructions but rely on linear conversations, making it difficult to represent spatial layouts, asset relationships, version branches, and local editing targets.
• Node-based workflow tools (e.g., ComfyUI, storyboard tools): Make execution steps visible but are organized around manually specified pipelines rather than continuously editable project states.

Theoretical insight: The key problem is not how to call stronger models, but how to give agents a workspace they can read, update, and keep consistent across an extended workflow. The canvas must serve as both a visual interface for users and the shared project state that agents can observe and modify.

Formal definition: At turn $t$, a creative project is represented as:

$$C_t = (G_t, X_t, M_t, U_t, L_t), \quad G_t = (V_t, E_t)$$

where $G_t$ is a typed artifact graph, $X_t$ stores editable contents, $M_t$ stores provenance and metadata, $U_t$ records user selections and feedback, and $L_t$ stores spatial layouts.

Each canvas node is defined as:

$$v_i = (\text{id}_i, k_i, p_i, x_i, y_i, m_i, s_i), \quad k_i \in \mathcal{K}_{\text{node}}$$

with stable identifier, node kind, position, editable inputs, generated outputs, provenance, and runtime status.

Methodology

Canvas-Native Project State: The canvas stores multimodal artifacts, dependencies, versions, and user choices as typed, addressable nodes and directed edges $E_t \subseteq V_t \times \mathcal{R} \times V_t$ where $r \in \mathcal{R}$ indicates relations such as reference use, version lineage, generation dependency, grouping, or workflow continuation.

Protocol-Constrained Canvas Interaction: A protocol bridge enforces explicit, valid, and recoverable canvas updates. At each turn, the agent receives:

• User query $q_t$
• Current canvas state $C_t$
• Capability manifest $\Gamma_t$ (available node types, operations, tools)
• Execution grant $\Omega_t$ (permitted actions for the current request)

The canvas evolves as:

$$C_{t+1} = \mathcal{F}(C_t, a_t, o_t, f_t, r_t)$$

where $\mathcal{F}$ is the state-transition operator that writes accepted actions and returned evidence to the canvas.

Agent Runtime: The runtime decides what the agent can do in the current turn and converts it into a valid canvas update. The granted action space is:

$$\mathcal{A}_t = \mathcal{A}(\Omega_t, \Gamma_t, C_t, q_t), \quad a_t \in \mathcal{A}_t$$

Actions must be grounded in the observed canvas, exposed by the manifest, allowed by the grant, and committed through the bridge.

Tool Families: Five core tool families are supported:

Higher-level supports: Skills (reusable creative procedures), Memory (user preferences across turns), and Subagents (parallel subtask execution) help sequence capabilities across long tasks.

Trajectory Recording: Each turn's complete execution is recorded as:

$$\tau = \{(q_t, C_t, \Gamma_t, \Omega_t, a_t, o_t, f_t, r_t, C_{t+1})\}_{t=1}^T$$

Empirical Validation / Results

Experimental Setup:

• Main agent backend: GPT-5.5
• Image generation: GPT Image 2
• Video generation: Seedance 2.0
• Multimodal evaluation: Gemini 3.1 Pro
• All tasks run in the same JarvisHub environment

Three Long-Horizon Creative Tasks:

Qualitative Results (key findings):

1. Narrative media generation: The workspace trace externalized story planning, visual references, dependency links, and agent progress. The final output (a short drama about a "cowboy robot zombie scavenger") showed recurring characters, setting cues, and action continuity across shots.

2. Interactive web development: The canvas tracked design references, implementation progress, previews, and revision state. The final photography website demonstrated consistent typography, image placement, page structure, and visual direction.

3. Presentation deck generation: The canvas recorded lecture content, generated diagrams, slide drafts, and revision state. The final deck on decision trees showed consistent layout, diagram style, emphasis color, and slide-level organization.

Important: The experiments are qualitative demonstrations rather than a completed benchmark or leaderboard. Final artifact quality depends on the external models and tools used.

Theoretical and Practical Implications

Theoretical Contributions:

• Formalization of long-horizon creation as an agent process over an editable project graph, providing a structured representation for creative workflows
• Canvas-native perspective that unifies workspace, memory, and action space—moving beyond isolated tool use toward sustained creative automation
• Process-level analysis via trajectories, enabling evaluation of context preservation, tool-use appropriateness, dependency correctness, feedback adherence, and repair success

Practical Implications:

• Open research platform: JarvisHub enables project-state benchmarks where tasks specify initial canvas, references, tools, constraints, feedback events, and expected checkpoints
• Data flywheel for model improvement: Each run produces structured examples of canvas states, agent actions, tool observations, feedback signals, and repair decisions—usable for training future creative agents
• Inspectable and steerable workflows: Users can inspect, guide, and intervene throughout the process, not just at the final output
• Applicable across domains: Narrative media, web development, and presentation generation demonstrate broad applicability

Conclusion

Main takeaways:

• JarvisHub presents a canvas-native agent harness that treats the editable canvas as the shared project state, enabling agents to progressively plan, generate, revise, and organize multimodal projects while users remain able to inspect, guide, and intervene.
• The three-layer architecture (canvas state, protocol bridge, agent runtime) provides explicit, valid, and recoverable canvas updates, with comprehensive trajectory recording for analysis.
• Qualitative demonstrations on three high-value creative tasks show that JarvisHub supports more inspectable and reusable agentic creative workflows.

Limitations:

1. Experiments are qualitative demonstrations, not a completed benchmark or leaderboard
2. Final artifact quality depends on external models and tools, not JarvisHub itself
3. The protocol bridge ensures operation validity but not semantic correctness of creative decisions
4. Raw trajectories require quality filtering, consent, anonymization, and copyright review before use as research data

Future directions:

• Developing project-state benchmarks with structured initial canvases, reference materials, and expected checkpoints
• Establishing evaluation protocols combining final artifact quality with process-level measures
• Building training datasets from curated trajectories for planning, tool selection, multimodal state tracking, local repair, and feedback-guided revision
• Extending the harness to additional creative domains and tool ecosystems