Summary (Overview)

• Meshy T2 is a fast, native mesh generation framework built on flow matching that generates compact, artist-style polygonal meshes directly from a single reference image, completing end-to-end generation in a median of 6 seconds — over an order of magnitude faster than autoregressive baselines.
• The core innovation is a nearly lossless vertex-set mesh VAE (extending SpaceMesh) that encodes a mesh into one continuous latent token per vertex, decoding vertices, edge connectivity, and face winding order jointly in a single pass — without vertex quantization or welding.
• Generation proceeds via a coarse-to-fine cascade of two flow-matching models: an image-conditioned voxel flow sketches a coarse 64³ occupancy scaffold, and a mesh flow populates it with per-vertex latent tokens, conditioned on the image, scaffold, and a requested vertex budget.
• The system provides effective face-count control (via the vertex budget, leveraging Euler's formula $F = 2V - 4 \approx 2V$) and native multi-part asset support, where connected components emerge directly from the generated connectivity.
• Meshy T2 achieves state-of-the-art geometric fidelity on both retopology and image-to-mesh tasks, with a non-manifold edge count of just 0.14 and 70.1% tri-to-quad mergeability.

Introduction and Theoretical Foundation

Background and Motivation

Polygonal meshes are the standard surface representation for modern 3D pipelines (game engines, AR/VR, robotics, digital content). Production-ready assets must capture shape with minimal primitives while preserving sharp edges, thin structures, and semantically meaningful part boundaries. However, such meshes are predominantly hand-authored — a slow, costly workflow.

Limitations of Prior Approaches

1. Implicit/volumetric methods (3DShape2VecSet, TRELLIS, LATTICE): Generate neural fields and extract meshes via Marching Cubes. These produce hundreds of thousands of near-uniform triangles that poorly respect sharp features and part structure.
2. Autoregressive sequence models (MeshGPT, EdgeRunner, TreeMeshGPT, BPT, DeepMesh): Serialize meshes into token sequences. Inference is inherently sequential and expensive, with error accumulation causing incomplete surfaces, inconsistent winding, or cracks.
3. Existing diffusion/flow methods: Face-level tokens (MeshCraft, MeshFlow) duplicate shared vertices; sparse-voxel latents (TriFlow, LATO) re-derive primitives at decode time; per-vertex tokens (Nexus) still decouple geometry and topology. Gaps include vertex quantization artifacts, heuristic face reconstruction, and lack of native image-to-mesh conditioning.

Theoretical Foundation: Flow Matching

The framework uses flow matching with the linear interpolation schedule of Rectified Flow. Given a clean target $x_0$ and Gaussian noise $x_1$, the interpolated state at time $t$ is:

$$x_t = (1-t)x_0 + t x_1$$

The model learns a velocity field $v_\theta(x_t, t, c)$ to map noise to data, enabling parallel generation of all primitives rather than sequential token-by-token decoding.

Methodology

2.1 Vertex-Set Mesh VAE

Representation: A mesh $M = (V, F)$ with vertices $V = \{x_i\}_{i=1}^{V}$, $x_i \in \mathbb{R}^3$ is encoded into a latent set $Z = \{z_i\}_{i=1}^{V}$ with exactly one token $z_i \in \mathbb{R}^C$ per vertex.

Encoder:

• A local PointNet pools surface samples (positions and normals, drawn preferentially along mesh edges) into features on occupied cells of a 256³ sparse voxel grid.
• Each vertex query is initialized from Fourier features of its continuous position; all attention uses 3D RoPE.
• Queries are refined by cross-attention into voxel context, followed by interleaved graph-attention (restricted to ground-truth edges) and self-attention layers.

Decoder: A pure set decoder consuming latent tokens without positional encoding, predicting:

• Continuous position $\hat{x}_i$ (vertex branch)
• Edge embedding $e_i = (e^{\text{space}}_i, e^{\text{time}}_i) \in \mathbb{R}^{d_e} \times \mathbb{R}^{d_e}$ (topology branch)
• Face embedding $f_i = (f^{\text{root}}_i, f^{\text{prev}}_i, f^{\text{next}}_i)$ (topology branch)

Edge prediction uses the SpaceMesh spacetime logit:

$$A_{ij} = \|e^{\text{time}}_i - e^{\text{time}}_j\|^2_2 - \|e^{\text{space}}_i - e^{\text{space}}_j\|^2_2$$

An edge is predicted when temporal separation exceeds spatial separation. The edge loss is class-balanced binary cross-entropy:

$$\mathcal{L}_{\text{edge}} = \frac{1}{Z}\left[\sum_{\{i,j\} \in E} \text{softplus}(-A_{ij}) + \lambda \sum_{\substack{i<j \\ \{i,j\} \notin E}} \text{softplus}(A_{ij})\right]$$

Face prediction extends SpaceMesh with a NULL element $\emptyset$ to handle open (non-watertight) boundaries. Each vertex's fan defines a successor mapping $\pi_i$ on neighbors $N(i)$:

$$\pi_i(p) = n \text{ for every triangle } (p, i, n) \in F$$

The face head predicts a soft permutation via Sinkhorn iterations:

$$\Phi_i[p, q] = \mathbf{1}^\top \left(f^{\text{root}}_i \odot f^{\text{prev}}_p \odot f^{\text{next}}_q\right)$$

trained with negative log-likelihood:

$$\mathcal{L}_{\text{face}} = -\frac{1}{\sum_i (D_i + 1)} \sum_i \sum_{p \in N(i) \cup \{\emptyset\}} \log P_i\left[p, \pi_i(p)\right]$$

Training objective:

$$\mathcal{L}_{\text{VAE}} = w_v \mathcal{L}_{\text{vertex}} + w_e \mathcal{L}_{\text{edge}} + w_f \mathcal{L}_{\text{face}}$$

with weights $w_v = 100$, $w_e = w_f = 10$, latent channels $C = 32$, and embedding sizes $d_e = d_f = 16$.

2.2 Stage I: Image-Conditioned Voxel Flow

• A Voxel VAE compresses 64³ binary occupancy grids into an 8-channel Gaussian posterior at 16³ resolution via two stride-2 3D convolutional stages, decoded with 3D pixel-shuffle upsampling.
• The flow model is a Transformer velocity field over the flattened 4096 latent tokens, with:
  • 3D positional self-attention
  • Cross-attention to frozen DINOv3 image features
  • Timestep injection via AdaLN modulation
• Hidden dimension 1536, 28 blocks, 12 heads, SwiGLU feed-forward.

2.3 Stage II: Image-and-Voxel-Conditioned Mesh Flow

Conditions:

• Image: Frozen DINOv3 at 768×768, patch size 16 → 2304 image tokens
• Voxel: 64³ occupancy grid → Voxel VAE encoder → 4096 voxel tokens (fine-tuned at 0.25× learning rate)
• Vertex count: Fourier embedding of $N/N_{\max}$, added to time embedding

Count control via padding: Zero-valued pad tokens are randomly appended up to ratio $p$, with an extra existence channel (+1 real, −1 pad). The flow operates on $C+1 = 33$ channels. At inference, tokens with positive existence are kept as real vertices.

Latent PE via Optimal Transport: Latent tokens are an unordered set, so they receive positions from a deterministic Sobol point set. During training, real vertex coordinates are matched to Sobol candidates via optimal transport (minimum squared Euclidean cost). At inference, the Sobol set provides positions directly.

Training: Velocity-prediction flow matching with logit-normal timestep sampling. Conditions are dropped with probabilities 0.2 (image), 0.3 (voxel), 0.2 (count), and 0.05 (all) for classifier-free guidance.

Empirical Validation / Results

Position Encoding / Optimal Transport Ablation

Sobol OT reduces Chamfer distance by 45% vs. no OT, Hausdorff by over 3×, and non-manifold edge ratio by 58% vs. Sobol + Morton.

High-Poly Mesh Retopology (Table 2)

Meshy T2 achieves the best geometric fidelity (CD 0.020, HD 0.044, NC 0.860) and runs in median 3 seconds — more than an order of magnitude faster than all baselines.

Image-to-Mesh Generation (Table 3)

Meshy T2 leads on DINOv2 FD (2312.01), indicating strongest semantic and structural alignment, with 100% success in median 6 seconds.

Theoretical and Practical Implications

Theoretical Contributions

• Nearly lossless mesh representation: By avoiding vertex quantization and welding, and recovering vertices, edges, and face winding jointly, the VAE preserves high-precision geometry and artist-authored topology exactly — addressing a fundamental gap in prior latent mesh representations.
• Optimal transport for unordered latent sets: The Sobol OT position assignment resolves the permutation ambiguity inherent to set-based generation, providing a principled solution applicable to any set-based generative model.
• Unified flow-matching cascade: The coarse-to-fine split (voxel scaffold → mesh latent) decomposes the ill-conditioned image-to-mesh problem into tractable sub-problems.

Practical Implications

• Interactive speed: Median 6-second end-to-end generation makes Meshy T2 practical for interactive asset creation workflows.
• Face-count control: Users can directly specify vertex budgets (with $F \approx 2V$), enabling level-of-detail pipelines without post-hoc simplification.
• Multi-part assets: Connected components emerge naturally from generated connectivity, eliminating separate part-wise generation or stitching stages.
• Production-ready topology: Low non-manifold edge counts (0.14) and high tri-to-quad mergeability (70.1%) indicate outputs are closer to artist-quality than prior automated methods.

Conclusion

Meshy T2 demonstrates that flow-based parallel generation over a carefully designed vertex-set latent representation can match or exceed the geometric fidelity of autoregressive methods while being over an order of magnitude faster. The key insights are:

1. Vertex-set latent representation with continuous coordinates and explicit connectivity prediction enables nearly lossless mesh encoding.
2. Coarse-to-fine flow cascade (voxel scaffold → mesh flow) effectively handles the ambiguity of image-to-mesh generation.
3. Optimal transport position assignment resolves the permutation ambiguity of unordered latent sets.

Future directions identified by the authors include:

• Improving topological robustness on highly irregular source meshes
• Extending to richer material and part-level controls
• Scene-level generation with multiple compact meshes under a shared global budget

The code and weights will be released at https://github.com/meshy-dev/meshy-t2.