Summary (Overview)

• ReDesign is an agentic framework that reconstructs editable design files (e.g., Figma, Illustrator) from raster images by progressively growing a layer hierarchy through tool composition.
• The system uses a Vision-Language Model (VLM) controller to select actions (text extraction, multi-layer decomposition, detection/segmentation, vectorization) and applies graceful verification at every node expansion to accept, prune, or retry—preventing error accumulation.
• A new benchmark, the Figma Edit Replay Benchmark, is introduced with 909 real Figma files and 14,796 controlled edit instructions to evaluate editability.
• ReDesign achieves state-of-the-art editability across layout, color, and text edits, while also attaining strong visual fidelity (L1, PSNR, LPIPS, PQ, F1) on both the Figma and Crello datasets.
• Tree-structured reconstruction enables parallel execution (up to 7.1× speedup) and local repair, outperforming serial tool-using agents and layered decomposition baselines.

Introduction and Theoretical Foundation

Designers frequently need to modify raster images (screenshots, exports) that lack editable structure. Rasterization is a many-to-one mapping from editable layer structures to pixels; recovering the original structure is an ill-posed inverse problem. An editable design file contains typography, vector geometry, colors, grouping, and layer ordering as separate objects with parameters. Previous work solved subproblems (text recognition, layer decomposition, shape extraction), but straightforward composition fails due to cascading errors and the diversity of real designs. ReDesign casts reconstruction as recovering a layer hierarchy from a raster image, treating it as a structured tree expansion problem.

Methodology

ReDesign operates as a VLM agent that maintains a partial reconstruction tree (a hierarchy of nodes with incomplete metadata). At each step, the controller selects an action from a fixed set:

• Text extraction: OCR + segmentation + inpainting to produce a text layer and remainder.
• Multi-layer decomposition: Uses a layered image generation model (e.g., Qwen-Image-Layered) to separate the image into RGBA layers.
• Connected Component Labeling: Splits spatially disjoint elements within a layer.
• Detection and segmentation: VLM identifies the foremost object, open-set detector localizes it, segmentation model extracts it, inpainting fills the hole.
• Vectorization: Leaf nodes are vectorized (SVG) for shapes or stored as raster with placement metadata; text leaves get font recognition and typographic properties.

Graceful verification is applied after each expansion. The verifier checks two criteria: (1) Do the child nodes jointly cover the parent content? (2) Do any children expand beyond the parent? Outcomes are accept, prune (remove invalid/redundant children), or retry (request a different tool or parameters). Only the offending branch is repaired, avoiding large-scale reruns.

Memory management stores lineal history per node (tool choice, outputs, verification outcomes) to guide future decisions and enable parallel expansion across leaf nodes.

Empirical Validation / Results

Editability (Figma Edit Replay Benchmark)

The benchmark consists of 909 raw Figma files and 14,796 edit instructions (reposition, rotation, recolor, text edit, delete, reorder, opacity). Edits are applied to ground-truth and reconstructed outputs; SSIM and text recall are measured. ReDesign achieves the highest SSIM across all edit types and the highest text recall after text edits, outperforming:

• Qwen-Image-Layered [48]
• LayerD [34]
• Tool Agent (ReAct-style linear tool chain) [45]

Baselines often fail because edits bleed into background or nearby layers, or error cascades corrupt text early.

Reconstruction Accuracy

Results on the Figma dataset (Table 1) and Crello dataset (Table 2) show ReDesign is best or competitive on all metrics: L1 (lower better), PSNR, LPIPS (lower better), PQ (panoptic quality), and detection F1.

Table 1: Reconstruction accuracy on the Figma dataset.

Table 2: Reconstruction accuracy on the Crello dataset [43].

Inference Cost and Parallelism

• Graceful verification is faster and more accurate than terminal verification; it avoids expensive restarts and reduces variance in tool call count.
• Parallel tree expansion yields up to 7.1× speedup over serial tool execution because independent leaf nodes can be expanded concurrently.

Theoretical and Practical Implications

• Structural constraints on agentic progress (tree hierarchy) improve controllability, accuracy, and efficiency by making failures local and repairable.
• The system is not tied to a fixed training distribution; users can adjust granularity (finer or coarser decomposition) via prompting, supporting user-in-the-loop control.
• Compared to raster-based image editing models (e.g., Nano Banana 2), ReDesign provides deterministic, predictable edits through explicit objects and parameters, avoiding unintended global warping.
• Practical direction: step-level verification and tree-structured execution are more reliable than terminal checking and serial tool chains.

Conclusion

ReDesign recovers editable design structures from raster images by agentic tree expansion with graceful verification. On the new Figma Edit Replay Benchmark (14,796 edits), it achieves strong visual fidelity and the highest editability across layout, color, and text edits. The work demonstrates that imposing structural constraints on an agent’s progress (a layer hierarchy) improves controllability, accuracy, and efficiency. Future work could extend the tool set, explore user-in-the-loop refinements, or apply the paradigm to other domains requiring structured reconstruction.