# InsightTok: Improving Text and Face Fidelity in Discrete Tokenization for Autoregressive Image Generation > InsightTok improves text and face fidelity in autoregressive image generation by augmenting standard tokenizer training with localized, content-aware perceptual losses on detected text and face regions. - **Source:** [arXiv](https://arxiv.org/abs/2605.14333) - **Published:** 2026-05-19 - **Permalink:** https://picx.dev/p/pKNI6t - **Whiteboard:** https://picx.dev/p/pKNI6t/image ## Summary # InsightTok: Improving Text and Face Fidelity in Discrete Tokenization for Autoregressive Image Generation ## Summary (Overview) * **Core Problem**: Standard discrete visual tokenizers, when trained with generic reconstruction losses (e.g., LPIPS), struggle to preserve fine-grained details crucial for **text legibility** and **facial identity fidelity**, as these perceptually critical regions often occupy small image areas. * **Proposed Solution**: **InsightTok**, a novel tokenizer training framework that augments standard objectives with **localized, content-aware perceptual losses** (`L_text` and `L_face`) computed on detected text and face regions using domain-specific recognition models. * **Key Innovation**: Introduces an **area-based weighting scheme** for the specialized losses (`w = Area(region)/Area(image)`) to balance improvements on critical content with maintaining general reconstruction quality, preventing small, difficult regions from dominating optimization. * **Superior Performance**: With a compact 16k codebook and 16× downsampling, InsightTok achieves **state-of-the-art** text and face reconstruction on the TokBench benchmark, significantly outperforming prior tokenizers including those with much larger codebooks (e.g., 262k entries). * **Effective Transfer**: The improved tokenizer directly benefits downstream **autoregressive image generation (InsightAR)**, producing images with clearer text and more faithful facial details without compromising general text-to-image capability. ## Introduction and Theoretical Foundation Discrete tokenization is foundational for autoregressive image generation and unified multimodal modeling. However, the aggressive spatial compression (e.g., 16× downsampling) and quantization inherent in visual tokenizers often discard the fine-grained structures necessary for **readable text** and **distinctive facial features**. This is increasingly problematic as generative models are used in text- and face-centric applications like graphic design and portrait synthesis. The authors identify the root cause as **insufficiently targeted supervision** in standard tokenizer training. Objectives like pixel reconstruction ($L_{rec}$) and general perceptual loss ($L_{perc}$, e.g., LPIPS) treat all image content uniformly and are poorly aligned with domain-specific metrics like text readability (OCR accuracy) or identity preservation (face similarity). Consequently, the training signal for small but critical text/face regions is diluted by the surrounding scene. Previous approaches often address this by **reducing compression**—increasing codebook size or token count—which incurs computational overhead without explicitly prioritizing fidelity-critical structures. This paper proposes a different direction: enhancing tokenizer training with **specialized, content-aware supervision** to learn a discrete representation that inherently preserves these details. ## Methodology ### Preliminary: Standard Discrete Tokenizer A tokenizer consists of an encoder $E$, a quantizer $Q$ with a codebook $\mathcal{C} = \{e_k \in \mathbb{R}^d\}_{k=1}^K$, and a decoder $D$. Given an image $x$, it produces a latent $z = E(x)$, quantized tokens $\hat{z} = Q(z)$, and a reconstruction $\hat{x} = D(\hat{z})$. The standard training objective combines several losses: $$ L_{image} = L_{rec} + \beta \cdot L_{codebook} + \gamma \cdot L_{perc} + \eta \cdot L_{GAN} $$ where $L_{rec}$ is an $\ell_1$ or $\ell_2$ reconstruction loss, $L_{codebook}$ is the commitment loss ($\|z - \text{sg}(\hat{z})\|_2^2$), $L_{perc}$ is a general perceptual loss (e.g., LPIPS), and $L_{GAN}$ is an adversarial loss. ### InsightTok Framework InsightTok augments the standard objective with two new, localized perceptual losses: $$ L_{\text{InsightTok}} = L_{image} + \alpha_1 \cdot L_{text} + \alpha_2 \cdot L_{face} $$ where $\alpha_1$ and $\alpha_2$ are scalar weights. #### 1. Text Perceptual Loss ($L_{text}$) 1. **Detection**: For each training image $x$, a text detector identifies $N$ bounding boxes $\{b_n^{text}\}_{n=1}^N$. 2. **Extraction**: Corresponding patches $\{r_n^{text}\}$ and $\{\hat{r}_n^{text}\}$ are cropped from the original and reconstructed images. 3. **Supervision**: Each patch pair is compared in the feature space of a pretrained text recognition network $F_{text}(\cdot)$. The region-level loss is: $$ L_n^{text} = \frac{1}{L} \sum_{l=1}^L \frac{1}{H_l W_l} \| F_{text}^{(l)}(r_n^{text}) - F_{text}^{(l)}(\hat{r}_n^{text}) \|^2 $$ 4. **Aggregation**: Region losses are aggregated with area-based weighting to prevent small text from dominating: $$ L_{text} = \sum_{n=1}^N w_n^{text} \cdot L_n^{text}, \quad \text{where } w_n^{text} = \frac{\text{Area}(b_n^{text})}{\text{Area}(x)} $$ #### 2. Face Perceptual Loss ($L_{face}$) 1. **Detection & Alignment**: A face detector provides bounding boxes $\{b_m^{face}\}$ and five facial landmarks $\{p_k^m\}$ for $M$ faces. Each face is aligned to a canonical template via a similarity transform $T(u) = sRu + t$ minimizing landmark error: $$ \min_{s, R, t} \sum_{k=1}^5 \| sRp_k + t - p_k^* \|^2 $$ 2. **Extraction**: Aligned patches $r_{face}$ and $\hat{r}_{face}$ are extracted via inverse warping: $r_{face}[c] = x[T^{-1}(c)]$. 3. **Supervision**: Similarity is measured in the feature space of a pretrained face recognition network $F_{face}(\cdot)$: $$ L_m^{face} = \frac{1}{L} \sum_{l=1}^L \frac{1}{H_l W_l} \| F_{face}^{(l)}(r_m^{face}) - F_{face}^{(l)}(\hat{r}_m^{face}) \|^2 $$ 4. **Aggregation**: Again, using area-based weighting: $$ L_{face} = \sum_{m=1}^M w_m^{face} \cdot L_m^{face}, \quad w_m^{face} = \frac{\text{Area}(b_m^{face})}{\text{Area}(x)} $$ ### InsightAR: Autoregressive Image Generator The improved tokens from InsightTok are used to train a standard autoregressive Transformer, **InsightAR**, for text-to-image generation. It models the token sequence conditioned on a text prompt $T$: $$ p(t | T) = \prod_{i=1}^n p(t_i | t_{m | T-NEDm | F-Simm | rFID ↓ | PSNR ↑ | | VQGAN | 16,384 | 0.0547 | 6.12 | 17.32 | 0.19 | 1.27 | 22.24 | | LlamaGen | 16,384 | 0.0547 | 15.01 | 30.44 | 0.25 | 0.69 | 22.80 | | O-MAGVIT2-16k | 16,384 | 0.0547 | 20.62 | 39.96 | 0.26 | 1.20 | 23.31 | | **IBQ-16k** | 16,384 | 0.0547 | **24.16** | **43.66** | **0.27** | **0.64** | **23.39** | | **InsightTok** | 16,384 | 0.0547 | **53.05** | **71.40** | **0.36** | 0.69 | 23.64 | | O-MAGVIT2-262k | 262,144 | 0.0703 | 27.33 | 47.28 | 0.31 | 0.65 | 23.98 | | Emu3.5-IBQ | 131,072 | 0.0664 | 41.52 | 65.39 | 0.30 | 0.46 | 23.82 | * **Key Findings**: InsightTok **doubles** the text accuracy (T-ACC) and significantly improves face similarity (F-Sim) over the best comparable baseline (IBQ-16k), while maintaining competitive general reconstruction metrics (PSNR, rFID). It even outperforms models with **8-16x larger codebooks** (O-MAGVIT2-262k, Emu3.5-IBQ) on its target domains. ### 2. Autoregressive Text-to-Image Generation (InsightAR) The tokenizer improvements consistently transfer to the generative model InsightAR. **Table 2: Image generation performance of InsightAR and existing autoregressive models.** | Model | #Params | #Tokens | **Face** | **Text** | **General** | | :--- | :--- | :--- | :--- | :--- | :--- | | | | | MagFace-Score ↑ | NED (%) ↑ | GenEval ↑ | DPG-Bench ↑ | | LlamaGen | 0.8B | 1,024 | 22.37 | 18.78 | 0.32 | 65.16 | | Janus-Pro | 7B | 576 | 22.09 | 32.29 | 0.80 | 84.19 | | LlamaGenTok-AR | 7B | 1,024 | 22.29 | 79.86 | 0.81 | 83.78 | | **InsightAR** | 7B | 1,024 | **23.33** | **95.83** | **0.82** | **84.11** | * **Key Findings**: InsightAR achieves the **highest face quality score** and **lowest text error rate (NED)** among models with the same token count, demonstrating clear benefits from the improved tokenizer. It maintains strong performance on general text-to-image benchmarks (GenEval, DPG-Bench). ### 3. Analytical Experiments & Ablations **Table 3: Effect of specialized perceptual losses and area-based loss weighting.** | $L_{text}$ & $L_{face}$ | Area-based Weighting | T-ACCm ↑ | Face-Simm ↑ | rFID ↓ | IN-PSNR ↑ | | :--- | :--- | :--- | :--- | :--- | :--- | | ✗ | - | 30.89 | 0.29 | **0.60** | **23.65** | | ✓ | ✗ | 55.18 | 0.42 | 1.11 | 22.41 | | ✓ | ✓ | **53.05** | **0.36** | 0.69 | 23.64 | * **Finding**: Adding specialized losses without weighting improves text/face metrics but **degrades general reconstruction** (worse rFID, PSNR). The proposed area-based weighting scheme **preserves the gains while maintaining overall quality**. **Table an: Additional key ablation results (from Appendix).** | **Ablation** | **Key Result** | | :--- | :--- | | **Decoder-only fine-tuning** (Table 4) | Applying losses only to the decoder yields minimal gains, confirming that improvements come from a **refined latent representation**, not just a stronger decoder. | | **Comparison with OCR-VQGAN** (Table 5) | InsightTok's localized loss (`L_text`) is more effective than OCR-VQGAN's global OCR loss. | | **Scaling codebook size** (Table 6) | InsightTok's benefits hold for both 16k and 65k codebooks. | | **Isolating losses** (Table 8) | `L_text` and `L_face` specifically improve their respective targets; combining them works well. | | **Detector coverage** (Table 9) | Better region identification (higher recall) leads to stronger reconstruction quality. | ## Theoretical and Practical Implications * **Tokenizer Training Paradigm**: The work shifts the focus from **increasing bottleneck capacity** (larger codebooks) to **improving supervision alignment**. It demonstrates that incorporating richer, **content-aware objectives** is a highly effective and computationally lightweight way to advance discrete representation learning. * **Specialization vs. Generality**: The paper successfully navigates the trade-off between specializing for critical content (text/faces) and maintaining general-purpose utility. The **area-based weighting** is a simple yet crucial mechanism for achieving this balance. * **Downstream Impact**: Improvements in the tokenizer **directly and consistently transfer** to autoregressive generation, enabling higher-quality text and face synthesis without changes to the core AR modeling architecture. This validates the tokenizer as a critical component for improving specific failure modes in generative models. * **Practical Deployability**: The method adds only ~**2%** overhead to training (FLOPs and wall-clock time) as detection is performed offline and recognition models are applied only to small crops. This makes it a practical enhancement to existing tokenizer training pipelines. ## Conclusion The paper identifies **lack of targeted supervision** as a key bottleneck for text and face fidelity in discrete visual tokenizers. The proposed **InsightTok** framework addresses this by augmenting standard training with **localized, domain-specific perceptual losses** for text and face regions, balanced via an area-based weighting scheme. With a compact 16k codebook, InsightTok achieves **state-of-the-art text and face reconstruction**, significantly outperforming prior tokenizers including those with much larger capacities. These gains effectively transfer to the **InsightAR** autoregressive generator, producing images with clearer text and more faithful faces. The findings highlight that **aligning tokenizer supervision with perceptually critical content** is a promising and practical direction for advancing discrete image generation. --- _Markdown view of https://picx.dev/p/pKNI6t, served by PicX — AI-generated visual whiteboard summaries of research papers._