# AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design

> AutoDesign recursively optimizes the harness surrounding a fixed model, achieving state-of-the-art paper-to-poster generation (78.32 on PosterBench) and improving performance by up to 19.6 points across model configurations.

- **Source:** [arXiv](https://arxiv.org/abs/2608.13560)
- **Published:** 2026-08-15
- **Permalink:** https://picx.dev/p/KnJfBf
- **Whiteboard:** https://picx.dev/p/KnJfBf/image

## Summary

## Summary (Overview)

- **AutoDesign** is a meta-harness optimization framework that recursively improves a *design harness* (the system surrounding a fixed model) rather than individual artifacts, enabling long-horizon, human-aligned multimodal design (e.g., paper-to-poster generation).
- The framework operates via two nested loops: an **inner loop** (design harness) that iteratively generates and revises artifacts under critic feedback, and an **outer loop** (meta-harness) that aggregates rollouts, evaluates outputs, proposes bounded updates to one of five harness components, and gates acceptance using a train/dev split to prevent overfitting.
- The optimized system, **DesignHarness**, achieves the highest score (**78.32**) on the new **PosterBench** benchmark (100 papers across five disciplines), surpassing the closed-source commercial system Claude Design by **7.45 points** under matched configurations.
- Across seven controlled code-agent–model configurations, attaching DesignHarness consistently improves performance, raising the average PosterBench Score from **54.99 to 67.39 (+12.4%)**.
- A system-blind human study (936 responses from 11 reviewers) confirms AutoDesign's superiority, with a Bradley–Terry preference estimate of **64.0%** against a randomly sampled alternative, and demonstrates that PosterBench scores align with human preference, especially for large score gaps (74.4% agreement for gaps ≥20 points).

## Introduction and Theoretical Foundation

Human communication often requires transforming multimodal sources into human-facing artifacts (webpages, slides, posters, videos). This is naturally a **long-horizon agentic task** requiring evidence extraction, reasoning over heterogeneous information, planning, and iterative refinement based on feedback.

**Key limitation of existing systems:** Current multimodal design systems treat human-aligned feedback (critiques, revisions) as *transient signals* rather than *reusable design knowledge*. Unlike human creators who accumulate experience from successes and failures, these systems do not convert evidence, structural constraints, and feedback into persistent design-aligned capabilities.

**AutoDesign's core insight:** Frame human-aligned design generation as a **meta-harness optimization problem** — an agentic system that recursively optimizes the *design harness itself* (the scaffolding around a fixed model) based on evaluation grounded in human preferences (annotated reference artifacts, natural language guidance).

**Formal definition:** A design harness $H$ transforms a multimodal source $x$ into an artifact $y$ given context $c$:

$$y \sim H(\pi_{\theta}, x, c) \tag{1}$$

The harness is decomposed into five components: (1) Context and Memory, (2) Tools and Specifications, (3) Execution Runtime, (4) Orchestration, and (5) Evaluation and Feedback.

The meta-harness optimizes expected quality:

$$J(H) = \mathbb{E}_{(x,c) \sim p_{\text{task}}, y \sim H(\pi_{\theta}, x, c)}[R_{\text{meta}}(y, x, c)] \tag{2}$$

$$H^{*} = \arg\max_{H} J(H) \tag{3}$$

The model parameters $\theta$ remain **fixed** throughout; optimization acts on the system surrounding the model.

## Methodology

### Inner Loop (Design Harness)

The inner loop consists of a designer $M_{\text{design}}$ and a critic $M_{\text{critic}}$:

$$\begin{array}{l} y_k = M_{\text{design}}(y_{k-1}, f_{k-1}; x, c), \\ f_k = M_{\text{critic}}(y_k; x, c), \end{array} \tag{4}$$

where $y_k$ is the artifact at step $k$ and $f_k$ is the feedback. The designer generates and revises; the critic evaluates and provides localized feedback.

### Outer Loop (Meta-Harness)

Each outer-loop iteration has four stages:

1. **Rollout:** Execute current harness $H_t$ on training set $\mathcal{D}_{\text{train}}$, collecting trajectories $\tau_t$ and scores $s_t$.
2. **Evaluation:** A fixed evaluator $R_{\text{meta}}$ (constructed from human-annotated reference artifacts across seven quality dimensions) scores each artifact.
3. **Update Proposal:** A coding agent $P$ (acting as planner and code editor) analyzes trajectories, scores, and the optimization record $\mathcal{L}$ to propose a candidate harness:

$$H'_{t+1} = P(H_t, \tau_t, s_t, \mathcal{L}) \tag{5}$$

Each iteration modifies **exactly one** of the five harness components to maintain interpretable credit assignment.

4. **Acceptance Gate:** A candidate is accepted only if it improves on training *and* does not degrade on the development set:

$$\text{Accept}(H'_{t+1}) \iff J_{\text{train}}(H'_{t+1}) > J_{\text{train}}(H_t) \land J_{\text{dev}}(H'_{t+1}) \geq J_{\text{dev}}(H_t) \tag{6}$$

The dev set is never exposed to the optimizer, serving as a guard against overfitting.

**Human-in-the-Loop:** Optional natural language guidance $g_t$ can redirect the search when autonomous optimization stagnates, or revise the evaluator when systematic bias is detected.

### DesignHarness Architecture

The optimized system has four stages:
1. **Paper Ingestion:** Extracts metadata, outlines, key passages, and figures with provenance tracking.
2. **Artifact Generation/Revision:** A coding agent generates/revises editable HTML with localized edits.
3. **Validation:** Rule-based blocking checks (missing assets, broken provenance, overflow) plus a VLM-based visual critic.
4. **Finalization:** Post-processing (rendering, math typesetting, asset inlining) with fallback mechanisms if the budget ($K=12$ attempts) is exhausted.

## Empirical Validation / Results

### PosterBench Benchmark

PosterBench comprises a 100-paper Main Track and a 10-paper mini subset across five disciplines (AI/ML, biomedicine, climate, economics, physics). The seven-dimensional rubric score:

$$R_{\text{rubric}}(p_i, A_i) = \sum_{j=1}^{7} \alpha_j q_{i,j}/10, \quad \boldsymbol{\alpha} = (10, 10, 15, 10, 20, 25, 10) \tag{7}$$

with final score capped by record-level ceilings:

$$R_{\text{poster}}(p_i, A_i) = \min\left(R_{\text{rubric}}(p_i, A_i), C_i^{\text{layout}}, C_i^{\text{viability}}, C_i^{\text{failure}}, C_i^{\text{gate}}\right) \tag{8}$$

### Main Results (PosterBench Main Track)

| System | Design Harness | Coding Agent | Model | Score | Faith. | Cover. | Density | Vis. Ev. | Layout | Read. | Aesth. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| **AutoDesign** | DesignHarness | Claude Code | Claude 4.8 | **78.32** | 9.35 | 9.40 | 8.41 | 5.97 | 8.55 | 8.17 | 5.59 |
| AutoDesign | DesignHarness | Codex | GPT 5.5 | 77.97 | 9.57 | 9.35 | 8.63 | 5.47 | 7.75 | 8.08 | 7.16 |
| Claude Design | Claude Design | Claude Code | Claude 4.8 | 70.87 | 9.22 | 9.90 | 6.48 | 7.62 | 8.08 | 5.96 | 7.36 |
| OpenDesign | OpenDesign | Claude Code | Claude 4.8 | 69.45 | 9.17 | 9.37 | 7.51 | 6.12 | 7.08 | 6.19 | 7.01 |
| Codex | — | Codex | GPT 5.5 | 73.37 | 9.64 | 9.71 | 7.69 | 7.43 | 8.03 | 6.08 | 6.60 |
| Claude Code | — | Claude Code | Claude 4.8 | 70.01 | 9.25 | 9.88 | 5.71 | 7.01 | 9.46 | 6.68 | 6.53 |

### Ablation: Effect of DesignHarness Attachment

| Configuration | Original | AutoDesign | Gain |
|---|---|---|---|
| GPT-5.5 + Codex | 75.87 | 81.46 | +5.59 |
| Claude 4.8 + Claude Code | 69.55 | 74.56 | +5.01 |
| Seed 2.1 Pro + Claude Code | 54.01 | 71.83 | +17.82 |
| Kimi K2.7 + Claude Code | 57.20 | 70.12 | +12.92 |
| GLM 5.2 + Claude Code | 50.32 | 64.33 | +14.01 |
| LongCat 2.0 + Claude Code | 43.26 | 55.13 | +11.87 |
| DeepSeek V4 Pro + Claude Code | 34.73 | 54.29 | +19.56 |

### Cost–Performance Trade-off

On PosterBench-mini, the Pareto frontier spans from LongCat-2.0 (55.13 at **$0.27/poster**) through Doubao Seed 2.1 Pro (71.83 at $2.75) and Claude 4.8 (74.56 at $7.63) to GPT-5.5 (81.46 at $10.02). Doubao achieves 88% of GPT-5.5's score at 27% of its cost.

### Human Evaluation

Using a Bradley–Terry model:

$$\Pr(i \succ j) = \frac{\exp(\beta_i)}{\exp(\beta_i) + \exp(\beta_j)}$$

AutoDesign achieves the highest preference estimate at **64.0%** (95% interval: 55.2–77.8%). Tie-adjusted preferences: 61.3% vs. Claude Code, 63.1% vs. OpenDesign, 67.6% vs. Claude Design.

**Benchmark–human alignment:** PosterBench Score correlates with human preference ($r = 0.34$), and agreement rises from 51.9% (0–3 point gaps) to **74.4%** (≥20 point gaps), validating the benchmark's discriminative power.

## Theoretical and Practical Implications

- **Meta-harness optimization as a paradigm:** AutoDesign demonstrates that persistent, recursive improvement of the *system surrounding a model* (rather than the model itself) is both feasible and effective for complex multimodal generation tasks. This aligns with recent taxonomies distinguishing model improvement from scaffold improvement.
- **Credit assignment via component isolation:** Restricting each update to one of five harness components makes gains/losses attributable to single coherent interventions, enabling interpretable, reproducible optimization trajectories.
- **Practical accessibility:** DesignHarness makes high-quality paper-to-poster generation affordable (under $3 per poster in fully autonomous mode, ~40 minutes, 253 tool calls, 11 editing turns), democratizing conference-quality poster production.
- **Benchmark design:** PosterBench's hybrid evaluation (rule-based + VLM) with record-level ceilings provides a robust, human-aligned protocol that could generalize to other design tasks (slides, webpages, videos).
- **Generalization potential:** The framework extends beyond posters to slides, webpages, and videos (pilot results shown), suggesting a path toward general multimodal-in/out agentic design systems.

## Conclusion

AutoDesign transforms recurring design failures into persistent improvements to the system that generates future outputs. Through 7 days of evolving traces, it invokes 224 subagents, records at least 123 recursive iterations, and accumulates 54 harness updates — converting human-designed reference artifacts, rollout traces, rendering diagnostics, and evaluator feedback into durable design priors.

**Key takeaways:**
1. Meta-harness optimization with acceptance gating effectively converts transient feedback into reusable design knowledge.
2. DesignHarness achieves state-of-the-art paper-to-poster generation (78.32 on PosterBench) with broad model/coding-agent compatibility (+5.0 to +19.6 points across configurations).
3. PosterBench provides a comprehensive, human-aligned evaluation protocol for academic poster generation.

**Future directions** include: (a) extending to additional media with medium-specific evaluators and validation gates, (b) better component selection via failure attribution and uncertainty estimation, (c) versioned adaptive evaluators anchored by frozen reference tasks, and (d) model–harness co-evolution where long-horizon trajectories provide execution-time supervision for model post-training.

---

_Markdown view of https://picx.dev/p/KnJfBf, served by PicX — AI-generated visual whiteboard summaries of research papers._