WeVisOne-VL: Unified Visual Perception from Objects to Pixels

Summary (Overview)

• WeVisOne-VL (WeChat Vision All-in-One) is a unified vision-language model (VLM) that integrates object-level perception (detection, referring localization, counting, pointing, OCR) and pixel-level perception (segmentation, matting) through a single structured next-token prediction interface.
• The model introduces a lightweight pixel decoder (~9.6M parameters) connected to the VLM via a special <OBJ> token, enabling dense predictions (masks and alpha mattes) without relying on a separate large segmentation model like SAM.
• Training uses a three-stage curriculum SFT (task alignment → resolution/density scaling → token–pixel joint optimization) followed by mixed-task reinforcement learning (GRPO) with uncertainty-aware sample selection.
• The WeVisOne-VL Data Engine (WDE) generates over 100M images with ~3.6B bounding-box annotations, integrating knowledge graphs, multi-expert localization, visual-language verification, hard-example construction, and a dedicated matting pipeline.
• Across 30+ benchmarks, WeVisOne-VL achieves state-of-the-art results on object detection (e.g., 74.0 F1 Mean on COCO), referring segmentation (e.g., 81.4 cIoU on RefCOCO val), counting, OCR, and matting, while demonstrating strong rejection of absent-target queries.

Introduction and Theoretical Foundation

Background

The paper addresses the challenge of building reliable visual perception systems that can:

• Recognize what objects are present in an image
• Identify which object a user refers to
• Determine where objects exist at multiple granularities (points, boxes, masks, alpha mattes)

Motivation

Existing VLMs remain unreliable for fine-grained perception due to:

• Hallucination: Predicting non-existent objects
• Misinterpretation: Failing on unfamiliar object descriptions
• Incomplete recall: Overlooking non-salient instances in crowded scenes

Theoretical Basis

The work builds on:

• Qwen3-VL as the backbone VLM
• Rex-Omni's approach of quantized coordinate tokens
• Deformable DETR-style query decoding
• SAM2/SAM3 for annotation generation
• GRPO (Group Relative Policy Optimization) for reinforcement learning

The core hypothesis is that unifying object-level and pixel-level perception through structured next-token prediction with a lightweight decoder can achieve specialized performance without the overhead of separate large models.

Methodology

1. Data Engine (WDE)

Knowledge Graph Construction:

• Organizes category concepts and referring expressions in a shared knowledge graph
• Initialized from Wikipedia and WordNet
• Supports taxonomic relations and whole–part relations

Detection Annotation:

• Proposals generated in parallel from SAM3 and an in-house DINO-style open-vocabulary detector
• Qwen3-VL verifies whether each proposed region matches the category query

Referring Annotation:

• SAM3 Agent framework + Rex-Omni as localization experts
• Qwen3-VL consolidates proposals by checking attributes, states, actions, and relations

Mask Annotation:

• SAM2 generates binary masks for each localized instance
• Manual spot-checks for quality control

Hard-Example Construction:

• Derived from confusable concepts, visually similar categories, and model failures
• Negative queries verified against the whole image before inclusion

Matting Pipeline:

• Expert-assisted annotation combining BiRefNet and MatAnyone2 predictions with spatial consistency checks
• Dedicated matting annotator trained with frozen SAM2.1-Base+ backbone and trainable alpha refinement head
• Corpus re-annotation for all instances

2. Model Architecture

Task Formulation:

• All tasks cast as structured next-token prediction
• Dedicated tokens <0> to <1000> encode normalized coordinates
• Output formats:
  • Detection/Referring: [x₁, y₁, x₂, y₂]
  • Pointing: [x, y]
  • Segmentation/Matting: adds <OBJ> token after box coordinates

Visual Feature Adapter:

• Extracts $L$ pre-merging features from intermediate ViT layers
• Feature transformation:

$$X_{l}^{(0)} = W_{l} F_{l}, \qquad E_{l} = P_{\mathrm{2D}} + e_{l} \tag{1}$$

• Adapter blocks with depthwise/pointwise convolutions and FFN:

$$\bar{X}_{l}^{(t)} = \text{LayerNorm}\left(X_{l}^{(t)} + \text{PWConv}_{t}\left(\text{GELU}\left(\text{DWConv}_{t}(X_{l}^{(t)} + E_{l})\right)\right)\right) \tag{2}$$

Box-Guided Query Decoder:

• Queries initialized from VLM outputs: content query $q_i = W_h h_i$ and positional query $p_i = \text{MLP}(\phi(b_i))$
• Six deformable blocks with multi-scale deformable cross-attention

Instance-Conditioned Dense Prediction:

• Multi-level pixel fusion with top-down feature integration:

$$G_{l} = \begin{cases} \text{Conv}_{1\times1}(X_{L}^{(T)}), & l = L \\ \text{GELU}(\text{GroupNorm}(\text{Conv}_{3\times3}(G_{l+1} + \text{Conv}_{1\times1}(X_{l}^{(T)})))), & l = L-1, \ldots, 1 \end{cases} \tag{3}$$

• Dynamic mask head with instance-aware kernels:

$$[k_i, \beta_i] = \text{MLP}(\tilde{q}_i), \qquad M_i = k_i^{\top} P + \beta_i \tag{5}$$

• Coarse-to-fine refinement across three upsampling stages

3. Training Pipeline

Stage 1: Task Alignment

• 90M examples, 13K steps
• Vision encoder and projector frozen; only language model trained
• 16–512 visual tokens per image; max 50 boxes/points per category

Stage 2: Resolution and Density Scaling

• 40M examples, 9K steps
• Max 1,960 visual tokens; max 500 boxes/points per response

Stage 3: Alternating Token–Pixel Optimization

• Segmentation warm-up: 250M samples, 100K steps with loss:

$$\mathcal{L}_{\mathrm{seg}} = 20\mathcal{L}_{\mathrm{focal}} + \mathcal{L}_{\mathrm{dice}} \tag{9}$$

• Joint training with 1:1 NTP:pixel ratio:

$$\mathcal{L}_{\mathrm{joint}} = \mathcal{L}_{\mathrm{NTP}} + 10\mathcal{L}_{\mathrm{seg}} \tag{10}$$

• Matting fine-tuning (VLM frozen): 130M samples, 100K steps:

$$\mathcal{L}_{\mathrm{mat}} = 200\mathcal{L}_{l1} + 5\mathcal{L}_{\mathrm{grad}} + 50\mathcal{L}_{\mathrm{lap}} + \mathcal{L}_{\mathrm{hard}} \tag{11}$$

Mixed-Task Reinforcement Learning:

• Uncertainty-aware sample selection:

$$U(x) = 4\mu_x(1 - \mu_x)\sigma_x \tag{12}$$

• GRPO with grounding reward:

$$r_i = \frac{2\sum_{(\hat{b}, b^*) \in \mathcal{M}_i} \text{IoU}(\hat{b}, b^*)^3}{N_{\mathrm{gt}} + N_{\mathrm{pred}}} \tag{13}$$

Empirical Validation / Results

Object Detection (Table 2)

Referring Object Detection (Table 3)

• RefCOCOg test: 81.8 F1 Mean (WeVisOne-VL-4B) vs. 77.6 (LocateAnything-3B)
• WeRef-Bench: 78.7 F1 Mean with 96.4% rejection rate (WeVisOne-VL-4B)
• PR-Bench Rej: 52.6% vs. 0.2% for LocateAnything

Object Counting (Table 4)

• PixMo-Count test: 89.6% exact match (WeVisOne-VL-4B) vs. 82.8% (LocateAnything)
• MixCount test: 40.4% vs. 15.2% (LocateAnything)

Referring Segmentation (Table 7)

Image Matting (Table 8)

• P3M-500-NP (Box): MAD 6.6 vs. 15.4 (SAM2-Matting SAM3)
• WeRefMat (Referring): MAD 9.8 vs. 22.4 (Qwen3.5-9B + SAM2-Matting)

Key Findings

1. Superior localization precision at strict IoU=0.95 thresholds
2. Strong rejection capability for absent-target queries (up to 96.4% rejection rate)
3. Competitive pixel-level performance without a separate large segmentation model
4. Efficiency: 2B and 4B models outperform larger 7B–13B baselines

Theoretical and Practical Implications

Theoretical Contributions

1. Unified perception paradigm: Demonstrates that object-level and pixel-level tasks can share a single interface via structured next-token prediction
2. Token-to-pixel bridging: The <OBJ> token hidden state effectively serves as an instance representation for dense decoding
3. Curriculum learning: Three-stage SFT establishes that progressive task complexity (alignment → density → pixels) is effective for multi-task perception training
4. RL for grounding: GRPO with uncertainty filtering improves localization quality beyond token-level likelihood optimization

Practical Implications

1. Deployment-ready: Small models (2B/4B) with competitive performance are suitable for real-world deployment
2. Unified system: Replaces multiple specialized models (detector, segmenter, matting model) with one system
3. Reliable rejection: Critical for practical applications where query targets may be absent
4. Data engine value: The WDE demonstrates a scalable approach to generating high-quality, multi-task supervision

Conclusion

Main Takeaways

• WeVisOne-VL unifies object-level and pixel-level perception through structured next-token prediction
• A lightweight pixel decoder (~9.6M params) connected via <OBJ> tokens achieves competitive segmentation and matting
• Three-stage curriculum SFT + mixed-task RL yields strong performance across 30+ benchmarks
• The model excels at both precise localization and reliable rejection of absent targets

Future Directions

1. General capabilities: Incorporate more general-purpose training data to preserve base model abilities
2. Chain-of-thought reasoning: Enable CoT for complex reasoning tasks
3. Multi-turn interaction: Integrate agent frameworks for progressive visual understanding
4. Intent reasoning: Handle vague, real-world user instructions more effectively
5. Open-source release: Code and weights are publicly available to accelerate community development

The paper concludes that by unifying these capabilities within a single language-guided model, WeVisOne-VL provides a foundation for practical, reliable visual systems suitable for real-world deployment.