Summary (Overview)

• Penguin-VL is a compact vision-language model family (2B and 8B parameters) designed for efficiency on resource-constrained devices while delivering strong multimodal performance on both image and video tasks.
• The core innovation is Penguin-Encoder, a vision encoder initialized directly from a text-only LLM (Qwen3-0.6B) rather than a contrastively pretrained encoder like CLIP/SigLIP. This avoids the "objective mismatch" of contrastive learning, which suppresses fine-grained visual cues needed for dense reasoning.
• A set of three reconstruction losses—amplitude, direction, and relation —are used during encoder pretraining to align the LLM-initialized encoder with visual features, enabling data-efficient learning through both labeled and unlabeled structured data (e.g., charts).
• The paper presents a complete training recipe: two-stage encoder pretraining (low/high resolution), joint VLM pretraining, a two-stage SFT strategy, and a Temporal Redundancy-Aware (TRA) token compression mechanism for efficient video processing.
• Empirical results show that Penguin-VL matches or surpasses leading baselines (e.g., Qwen3-VL, InternVL3.5) on document understanding, visual knowledge, mathematical reasoning, and video temporal grounding, with much smaller model sizes and far less pretraining data dependency.

---

Introduction and Theoretical Foundation

The paper challenges the prevailing assumption that a state‑of‑the‑art VLM must rely on vision encoders pretrained with massive contrastive learning (CLIP/SigLIP). The authors identify a fundamental objective mismatch: contrastive learning is optimized for discrimination and applies supervision only to a global summarization token (e.g., [CLS]). This encourages coarse, category‑level invariances that suppress fine-grained spatial and temporal details essential for dense captioning, document understanding, and complex reasoning.

The paper argues that transformers trained with a sequence‑token‑prediction objective—as in modern LLMs—are better suited for vision encoders in VLMs. They propose initializing the vision encoder from a text‑only LLM rather than from scratch or from contrastive checkpoints. Their reasoning:

• LLMs contain rich semantic priors and world knowledge.
• LLM architectures include modern design elements (e.g., QK normalization, sophisticated attention) that improve feature stability and expressiveness.
• A vision encoder initialized from the same LLM family as the downstream decoder is in a "native" representation space, reducing the modality gap and simplifying alignment.
• This approach scales more predictably than contrastive training, as it leverages established LLM design principles.

This perspective is inspired by advances in speech modeling, where text‑only LLMs have been successfully adapted to process continuous signals. Figure 2 contrasts three encoder training paradigms: (a) contrastive training, (b) direct LLM supervision with contrastively pretrained encoder, and (c) Penguin-Encoder (ours), which starts directly from LLM weights and uses generative supervision.

---

Methodology

Model Architecture

Penguin-VL consists of three modules:

1. Penguin-Encoder: a vision encoder initialized from Qwen3-0.6B (approximately 400M visual parameters). The causal self‑attention is modified to bidirectional full attention, and the model is equipped with 2D‑RoPE (rotary positional embeddings) to handle variable‑resolution inputs.
2. MLP‑based vison–language projector: a lightweight 2‑layer MLP with GELU activation, mapping visual features to the LLM hidden size without spatial compression.
3. LLM decoder: Qwen3‑1.7B or Qwen3-8B, depending on the model variant.

Penguin-Encoder Pretraining with Mixed Supervision

The encoder is trained in two stages:

• Low‑resolution pretraining (~100M samples, up to 2048 visual tokens, ~600×600 pixels): trained with standard image–text caption loss plus a reconstruction (distillation) loss that uses a teacher encoder (VL3‑SigLIP‑NaViT) to inject visual knowledge into the LLM‑initialized encoder.
• High‑resolution fine‑tuning (~47M samples, up to 10240 visual tokens): removes the reconstruction branch and focuses on fine‑grained alignment using high‑quality recaptioned data.

The reconstruction loss comprises three terms, applied to the vision encoder features $F_s$ and teacher features $F_t$:

Amplitude Loss (supervises absolute magnitudes):

$$L_A = \frac{1}{N} \sum_{N} |F_s - F_t|$$

Direction Loss (cosine similarity to align feature direction):

$$L_D = \frac{1}{N} \sum_{N} \left( \mathrm{tr} \left( \frac{F_s F_t^{\top}}{\|F_s\|_2 \|F_t\|_2} \right) \right)$$

Relation Loss (supervises inter‑patch self‑correlations):

$$L_R = \frac{1}{N} \sum_{N} \left( \left| \frac{F_s F_s^{\top}}{\|F_s\|_2^2} - \frac{F_t F_t^{\top}}{\|F_t\|_2^2} \right| \right)$$

These losses enable the encoder to use large‑scale unlabeled structured data (e.g., charts) and to preserve fine‑grained visual relations crucial for later reasoning.

Temporal Redundancy‑Aware (TRA) Token Compression

For video inputs, frames are first sampled (e.g., 1 FPS capped at 180 frames). TRA then allocates a global visual‑token budget $T_{\max}$, distinguishing key frames (fast changes) from intermediate frames (stable context). The compression proceeds in three stages:

• Stage 1 (Resolution Preservation): if the full‑resolution token counts satisfy

$$\sum_{k \in \mathcal{K}} T_k + \sum_{i \in \mathcal{I}} T_i \leq T_{\max},\tag{1}$$

no spatial compression is applied.

• Stage 2 (Synchronous Downscaling): if the budget is exceeded, all frames are downscaled by a continuous factor $\alpha$ while preserving the relative ratio (intermediate frames are 4× spatially downsampled relative to key frames, so $T_k \approx 16 T_i$):

$$T_k \leftarrow \alpha T_k, \qquad T_i \leftarrow \alpha T_i.\tag{2}$$

• Stage 3 (Saturation‑Aware Scaling): when intermediate frames reach a minimum token floor $T_{\min}$, they are clamped and further compression is applied only to key frames, ensuring essential context is preserved.

This design allows smooth, dynamic resolution adaptation since the model is pretrained over a wide range of resolutions.

Data Curation

• Image data (Penguin-Recap‑I): 57.2M image–text pairs from COYO‑700M and DataComp‑1B, filtered by quality (resolution, corruption) and balanced via hierarchical k‑means semantic clustering. Images are re‑annotated with detailed long captions covering global semantics, subjects, actions, spatial relationships, OCR text, scene attributes, and knowledge‑oriented descriptions.
• Video data (Penguin-Recap‑V): 3.7M video–text pairs from 29 public datasets, filtered for duplicates and static content (optical‑flow motion scores), balanced by duration. Multi‑granularity annotations are generated at three levels: event‑level atomic descriptions, chapter‑level narratives, and holistic summaries. Temporal QA pairs are constructed for ordering and grounding tasks.
• Training data mixtures: The final pretraining mixture (~121M samples) includes 64% general captions, with the rest covering document, fine‑grained, OCR, code, math, science, and pure text data. For region‑level perception, 7.7M grounding QA and 1.5M region‑caption QA samples are included.

Training Stages

1. Stage 1 (Encoder Pretraining): Optimize only the encoder and projector. Low‑resolution phase uses a learning rate of $1.0 \times 10^{-3}$; high‑resolution phase removes reconstruction loss and lowers the encoder LR to $5.0 \times 10^{-4}$.
2. Stage 2 (Pre‑training): All parameters trainable; the vision encoder LR drops to $1.0 \times 10^{-4}$. The model is trained on ~121M mixed multimodal samples.
3. Stage 3 (Supervised Fine‑tuning): All parameters jointly optimized with LR $1.0 \times 10^{-5}$. Image SFT uses 39M samples (32.6% general/caption, 20.9% document/chart/table, 16.6% OCR/text QA, 10.1% grounding/counting, 8.9% math, 3.7% multi‑image/science). Video SFT contains four pillars: general understanding (77.6%), action recognition (12.7%), temporal grounding (6.9%), and ego‑video understanding (2.8%).

---

Empirical Validation / Results

Implementation Details

• Vision encoder initialized from Qwen3‑0.6B; LLM backbones from Qwen3‑1.7B and Qwen3‑8B.
• Cosine LR schedule with 3% warm‑up; max sequence length 16,384 tokens (up to 10,240 visual tokens).
• For video, frames are sampled at 1 FPS (max 180 frames during training); at inference, up to 300 frames and 3 FPS are considered, with two frame‑sampling strategies: TRA and TRA‑codec (using I‑frames from the video codec).

Image Benchmarks

2B variant results (Table 1, best in bold, second underlined):

Penguin‑VL 2B achieves the best results on most benchmark categories—especially document understanding (DocVQA, ChartQA), general knowledge (AI2D, RealWorldQA, V‑star), math (MathVista), and all video tasks except VideoMME. It notably surpasses Qwen3‑VL on LongVideoBench (+7.4) and Charades‑STA (+1.7, with InternVL3.5 more than 34 points behind).

8B variant results (Table 2):

Penguin‑VL 8B is the strongest overall in document/chart understanding (DocVQA, ChartQA, InfoVQA), knowledge tasks (AI2D, RealWorldQA, V‑star), and video temporal reasoning (LongVideoBench, NextQA, Charades‑STA, Perception Test). Qwen3‑VL retains advantages on OCRBench, CharXiv, MMMU‑Pro, BLINK, MathVerse, and VideoMME, indicating room for improvement in certain advanced reasoning and OCR‑heavy settings.

Ablation Study Results (Table 3)

The ablation uses a lightweight three‑stage pipeline (encoder pretraining on 24M samples, modality alignment on DenseFusion‑1M, SFT on LLaVA‑665k) with Qwen3‑1.7B as the LLM backbone.

Key findings:

• LLM initialization provides a substantial +3.3 average improvement over random initialization (31.3 → 34.6), confirming that reusing LLM weights gives a well‑conditioned starting point that eases later alignment and stabilizes training.
• Relation loss is critical: adding it boosts the average score from 33.3 (without it) to 34.6, because attention‑based models mainly rely on inter‑token relationships rather than absolute feature magnitudes/directions.
• In a LMM integration comparison (Table 3, full rows) with SigLIP2 (original/any resolution), VL3‑SigLIP‑NaViT, and Qwen3‑VL ViT, the Penguin‑encoder achieves an average of 49.3 on the same five benchmarks—higher than Qwen3‑VL‑32B ViT (47.3) and significantly higher than SigLIP2 variants (32.4–45.3), even when SigLIP2 is given the same training data (matched‑data setting). This demonstrates that generation‑aligned, LLM‑initialized encoders are more data‑efficient and architecturally suited for VLM reasoning than contrastively pretrained encoders.

Case Studies

The paper provides qualitative examples (Figures 9–12) showing that Penguin‑VL can:

• Solve complex competitive programming problems from images, generating correct dynamic‑programming code.
• Perform high‑fidelity OCR on historical documents with severe visual degradation, preserving reading order.
• Extract quantitative data from multivariate charts and reason about global trends (e.g., identifying minima and volatility over 50 years).
• Generate evocative, multi‑stanza poems from traditional paintings, capturing implicit artistic semantics.

---

Theoretical and Practical Implications

• Rethinking vision encoder pretraining: The results suggest that contrastive pretraining (e.g., CLIP/SigLIP) is not an essential prerequisite for high‑performing VLMs. An LLM‑initialized vision encoder, trained with generative/reconstruction losses, can provide better fine‑grained visual representations for reasoning‑centric multimodal models, while using orders of magnitude less pretraining data (∼240M vs. >40B samples).
• Data efficiency and scalability: The LLM‑to‑vision transfer demonstrates that pretrained semantic knowledge and language‑model architecture inductive biases can be reused for vision, making the approach particularly appealing for compute‑limited environments. The authors note that the framework can be extended to larger vision encoders (e.g., 1.5B parameters) more predictably than contrastive training.
• Practical impact for edge deployment: Penguin‑VL achieves leading performance at 2B and 8B scales, with lightweight architecture and an efficient video token compression mechanism (TRA). This makes it suitable for smartphones, robots, and other latency‑sensitive applications where full‑scale VLMs are not deployable.
• Guiding future VLM design: The work provides a concrete alternative to the "scale‑up everything" trend, showing that architectural choices and representation learning can be more impactful than raw parameter count. It also advocates for aligning vision encoders with the generative nature of LLMs, which may inspire further research into unified modality encoders.

---

Conclusion

Penguin-VL demonstrates that compact, efficient VLMs can achieve state‑of‑the‑art performance by rethinking the vision encoder. The central contribution is Penguin‑Encoder, a vision encoder initialized from a text‑only LLM and trained with mixed supervision (caption loss plus amplitude, direction, and relation reconstruction losses). This encoder is natively aligned with the LLM decoder, preserves fine‑grained spatial and temporal details, and reduces the need for massive contrastive pretraining.

The complete training recipe—two‑stage encoder pretraining, joint multimodal pretraining, a two‑stage SFT, and TRA video token compression—yields consistently strong results on image and video benchmarks at 2B and 8B scales, often surpassing much larger or more data‑hungry baselines. Ablations confirm that the LLM initialization and the relation loss are key drivers of the performance gains.

Future directions include scaling the vision encoder to larger capacities (e.g., 1.5B), expanding to a wider range of modalities (e.g., audio), and further improving data curation pipelines. The code and models are publicly released (GitHub and Hugging Face links provided), enabling reproducibility and further research into efficiency‑focused multimodal learning.

---

Code: https://github.com/tencent-ailab/Penguin-VL | 2B Model: https://huggingface.co/tencent/Penguin-VL-2B | 8B Model: https://huggingface.co/tencent/Penguin-VL-8B