# WeVisOne-VL: Unified Visual Perception from Objects to Pixels

> WeVisOne-VL unifies object detection, segmentation, and matting in one VLM via structured next-token prediction, beating larger specialized models across 30+ benchmarks.

- **Source:** [arXiv](https://arxiv.org/abs/2610.31329)
- **Published:** 2026-10-24
- **Permalink:** https://picx.dev/p/Ws3hGS
- **Whiteboard:** https://picx.dev/p/Ws3hGS/image

## Summary

# WeVisOne-VL: Unified Visual Perception from Objects to Pixels

## Summary (Overview)

- **WeVisOne-VL** (WeChat Vision All-in-One) is a unified vision-language model (VLM) that integrates object-level perception (detection, referring localization, counting, pointing, OCR) and pixel-level perception (segmentation, matting) through a single structured next-token prediction interface.
- The model introduces a **lightweight pixel decoder (~9.6M parameters)** connected to the VLM via a special `<OBJ>` token, enabling dense predictions (masks and alpha mattes) without relying on a separate large segmentation model like SAM.
- Training uses a **three-stage curriculum SFT** (task alignment → resolution/density scaling → token–pixel joint optimization) followed by **mixed-task reinforcement learning (GRPO)** with uncertainty-aware sample selection.
- The **WeVisOne-VL Data Engine (WDE)** generates over 100M images with ~3.6B bounding-box annotations, integrating knowledge graphs, multi-expert localization, visual-language verification, hard-example construction, and a dedicated matting pipeline.
- Across 30+ benchmarks, WeVisOne-VL achieves state-of-the-art results on object detection (e.g., 74.0 F1 Mean on COCO), referring segmentation (e.g., 81.4 cIoU on RefCOCO val), counting, OCR, and matting, while demonstrating strong rejection of absent-target queries.

## Introduction and Theoretical Foundation

### Background
The paper addresses the challenge of building reliable visual perception systems that can:
- Recognize what objects are present in an image
- Identify which object a user refers to
- Determine where objects exist at multiple granularities (points, boxes, masks, alpha mattes)

### Motivation
Existing VLMs remain unreliable for fine-grained perception due to:
- **Hallucination**: Predicting non-existent objects
- **Misinterpretation**: Failing on unfamiliar object descriptions
- **Incomplete recall**: Overlooking non-salient instances in crowded scenes

### Theoretical Basis
The work builds on:
- **Qwen3-VL** as the backbone VLM
- **Rex-Omni**'s approach of quantized coordinate tokens
- **Deformable DETR**-style query decoding
- **SAM2/SAM3** for annotation generation
- **GRPO** (Group Relative Policy Optimization) for reinforcement learning

The core hypothesis is that **unifying object-level and pixel-level perception through structured next-token prediction** with a lightweight decoder can achieve specialized performance without the overhead of separate large models.

## Methodology

### 1. Data Engine (WDE)

**Knowledge Graph Construction:**
- Organizes category concepts and referring expressions in a shared knowledge graph
- Initialized from Wikipedia and WordNet
- Supports taxonomic relations and whole–part relations

**Detection Annotation:**
- Proposals generated in parallel from SAM3 and an in-house DINO-style open-vocabulary detector
- Qwen3-VL verifies whether each proposed region matches the category query

**Referring Annotation:**
- SAM3 Agent framework + Rex-Omni as localization experts
- Qwen3-VL consolidates proposals by checking attributes, states, actions, and relations

**Mask Annotation:**
- SAM2 generates binary masks for each localized instance
- Manual spot-checks for quality control

**Hard-Example Construction:**
- Derived from confusable concepts, visually similar categories, and model failures
- Negative queries verified against the whole image before inclusion

**Matting Pipeline:**
- Expert-assisted annotation combining BiRefNet and MatAnyone2 predictions with spatial consistency checks
- Dedicated matting annotator trained with frozen SAM2.1-Base+ backbone and trainable alpha refinement head
- Corpus re-annotation for all instances

### 2. Model Architecture

**Task Formulation:**
- All tasks cast as structured next-token prediction
- Dedicated tokens `<0>` to `<1000>` encode normalized coordinates
- Output formats:
  - Detection/Referring: `[x₁, y₁, x₂, y₂]`
  - Pointing: `[x, y]`
  - Segmentation/Matting: adds `<OBJ>` token after box coordinates

**Visual Feature Adapter:**
- Extracts $L$ pre-merging features from intermediate ViT layers
- Feature transformation:

$$X_{l}^{(0)} = W_{l} F_{l}, \qquad E_{l} = P_{\mathrm{2D}} + e_{l} \tag{1}$$

- Adapter blocks with depthwise/pointwise convolutions and FFN:

$$\bar{X}_{l}^{(t)} = \text{LayerNorm}\left(X_{l}^{(t)} + \text{PWConv}_{t}\left(\text{GELU}\left(\text{DWConv}_{t}(X_{l}^{(t)} + E_{l})\right)\right)\right) \tag{2}$$

**Box-Guided Query Decoder:**
- Queries initialized from VLM outputs: content query $q_i = W_h h_i$ and positional query $p_i = \text{MLP}(\phi(b_i))$
- Six deformable blocks with multi-scale deformable cross-attention

**Instance-Conditioned Dense Prediction:**
- Multi-level pixel fusion with top-down feature integration:

$$G_{l} = \begin{cases} \text{Conv}_{1\times1}(X_{L}^{(T)}), & l = L \\ \text{GELU}(\text{GroupNorm}(\text{Conv}_{3\times3}(G_{l+1} + \text{Conv}_{1\times1}(X_{l}^{(T)})))), & l = L-1, \ldots, 1 \end{cases} \tag{3}$$

- Dynamic mask head with instance-aware kernels:

$$[k_i, \beta_i] = \text{MLP}(\tilde{q}_i), \qquad M_i = k_i^{\top} P + \beta_i \tag{5}$$

- Coarse-to-fine refinement across three upsampling stages

### 3. Training Pipeline

**Stage 1: Task Alignment**
- 90M examples, 13K steps
- Vision encoder and projector frozen; only language model trained
- 16–512 visual tokens per image; max 50 boxes/points per category

**Stage 2: Resolution and Density Scaling**
- 40M examples, 9K steps
- Max 1,960 visual tokens; max 500 boxes/points per response

**Stage 3: Alternating Token–Pixel Optimization**
- Segmentation warm-up: 250M samples, 100K steps with loss:

$$\mathcal{L}_{\mathrm{seg}} = 20\mathcal{L}_{\mathrm{focal}} + \mathcal{L}_{\mathrm{dice}} \tag{9}$$

- Joint training with 1:1 NTP:pixel ratio:

$$\mathcal{L}_{\mathrm{joint}} = \mathcal{L}_{\mathrm{NTP}} + 10\mathcal{L}_{\mathrm{seg}} \tag{10}$$

- Matting fine-tuning (VLM frozen): 130M samples, 100K steps:

$$\mathcal{L}_{\mathrm{mat}} = 200\mathcal{L}_{l1} + 5\mathcal{L}_{\mathrm{grad}} + 50\mathcal{L}_{\mathrm{lap}} + \mathcal{L}_{\mathrm{hard}} \tag{11}$$

**Mixed-Task Reinforcement Learning:**
- Uncertainty-aware sample selection:

$$U(x) = 4\mu_x(1 - \mu_x)\sigma_x \tag{12}$$

- GRPO with grounding reward:

$$r_i = \frac{2\sum_{(\hat{b}, b^*) \in \mathcal{M}_i} \text{IoU}(\hat{b}, b^*)^3}{N_{\mathrm{gt}} + N_{\mathrm{pred}}} \tag{13}$$

## Empirical Validation / Results

### Object Detection (Table 2)

| Method | COCO Mean | LVIS Mean | Dense200 Mean | VisDrone Mean |
|--------|-----------|-----------|---------------|---------------|
| Qwen3-VL-4B | 48.7 | 44.0 | 15.2 | 31.0 |
| LocateAnything-3B | 54.7 | 50.7 | 58.7 | 39.9 |
| **WeVisOne-VL-2B** | **55.2** | **53.3** | **61.4** | **40.4** |
| **WeVisOne-VL-4B** | **57.1** | **55.3** | **71.3** | **44.7** |

### Referring Object Detection (Table 3)

- **RefCOCOg test**: 81.8 F1 Mean (WeVisOne-VL-4B) vs. 77.6 (LocateAnything-3B)
- **WeRef-Bench**: 78.7 F1 Mean with 96.4% rejection rate (WeVisOne-VL-4B)
- **PR-Bench Rej**: 52.6% vs. 0.2% for LocateAnything

### Object Counting (Table 4)

- **PixMo-Count test**: 89.6% exact match (WeVisOne-VL-4B) vs. 82.8% (LocateAnything)
- **MixCount test**: 40.4% vs. 15.2% (LocateAnything)

### Referring Segmentation (Table 7)

| Method | RefCOCO val | RefCOCO+ val | RefCOCOg val | ReasonSeg test |
|--------|-------------|--------------|--------------|----------------|
| UniPixel-7B | 80.8 | 75.3 | 76.4 | - |
| **WeVisOne-VL-4B** | **81.4** | **76.5** | **77.7** | **66.8** |

### Image Matting (Table 8)

- **P3M-500-NP (Box)**: MAD 6.6 vs. 15.4 (SAM2-Matting SAM3)
- **WeRefMat (Referring)**: MAD 9.8 vs. 22.4 (Qwen3.5-9B + SAM2-Matting)

### Key Findings
1. **Superior localization precision** at strict IoU=0.95 thresholds
2. **Strong rejection capability** for absent-target queries (up to 96.4% rejection rate)
3. **Competitive pixel-level performance** without a separate large segmentation model
4. **Efficiency**: 2B and 4B models outperform larger 7B–13B baselines

## Theoretical and Practical Implications

### Theoretical Contributions
1. **Unified perception paradigm**: Demonstrates that object-level and pixel-level tasks can share a single interface via structured next-token prediction
2. **Token-to-pixel bridging**: The `<OBJ>` token hidden state effectively serves as an instance representation for dense decoding
3. **Curriculum learning**: Three-stage SFT establishes that progressive task complexity (alignment → density → pixels) is effective for multi-task perception training
4. **RL for grounding**: GRPO with uncertainty filtering improves localization quality beyond token-level likelihood optimization

### Practical Implications
1. **Deployment-ready**: Small models (2B/4B) with competitive performance are suitable for real-world deployment
2. **Unified system**: Replaces multiple specialized models (detector, segmenter, matting model) with one system
3. **Reliable rejection**: Critical for practical applications where query targets may be absent
4. **Data engine value**: The WDE demonstrates a scalable approach to generating high-quality, multi-task supervision

## Conclusion

### Main Takeaways
- WeVisOne-VL unifies object-level and pixel-level perception through structured next-token prediction
- A lightweight pixel decoder (~9.6M params) connected via `<OBJ>` tokens achieves competitive segmentation and matting
- Three-stage curriculum SFT + mixed-task RL yields strong performance across 30+ benchmarks
- The model excels at both precise localization and reliable rejection of absent targets

### Future Directions
1. **General capabilities**: Incorporate more general-purpose training data to preserve base model abilities
2. **Chain-of-thought reasoning**: Enable CoT for complex reasoning tasks
3. **Multi-turn interaction**: Integrate agent frameworks for progressive visual understanding
4. **Intent reasoning**: Handle vague, real-world user instructions more effectively
5. **Open-source release**: Code and weights are publicly available to accelerate community development

The paper concludes that by unifying these capabilities within a single language-guided model, WeVisOne-VL provides a foundation for practical, reliable visual systems suitable for real-world deployment.

---

_Markdown view of https://picx.dev/p/Ws3hGS, served by PicX — AI-generated visual whiteboard summaries of research papers._