# ZGCM-1: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search

> ZGCM-1-7B, a fully open-source 7.39B model, matches frontier reasoning and agentic search performance by pairing deliberate chain-of-thought thinking with external tool use.

- **Source:** [arXiv](https://arxiv.org/abs/2609.13356)
- **Published:** 2026-09-16
- **Permalink:** https://picx.dev/p/ZjyTcS
- **Whiteboard:** https://picx.dev/p/ZjyTcS/image

## Summary

## Summary (Overview)

- **ZGCM-1** is a fully open-source 7.39B dense foundation model trained from scratch, designed for mathematical reasoning and agentic search across a 256K context, challenging the assumption that frontier intelligence requires hundreds of billions of parameters.
- The core thesis: compact models are bounded by static parametric capacity but can transcend this through a dual engine of **deliberate internal thinking** (chain-of-thought reasoning) and **active external tool use** (web search, terminal interaction, binary analysis).
- Key technical contributions include: (1) hybrid gated sliding-window/global attention (5:1 ratio) achieving a 3.94× throughput speedup and 6.4× KV-cache reduction at 256K; (2) FP8 pre-training with the Muon optimizer and TWEO outlier regularization yielding ~4.2× time-to-loss speedup; (3) progressive curriculum mid-training (16K→64K→256K) with MDP-formulated interaction traces; (4) AI-native R&D workflow with agent swarms.
- ZGCM-1-7B achieves state-of-the-art results among sub-10B models on reasoning benchmarks (75.0% AIME 2026, 97.1% MATH-500, 70.4% HMMT 2025) and remains competitive with frontier models orders of magnitude larger on agentic tasks (63.1% WebWalkerQA, 62.0% Binary Function Search).
- The project releases model weights from all training stages, intermediate checkpoints, training code, data recipes, and W&B logs for full reproducibility.

## Introduction and Theoretical Foundation

The paper addresses two critical bottlenecks in foundation model research:

- **The Scale Barrier**: Frontier reasoning and agentic search are often seen as exclusive to hundred-billion-parameter systems, locking compute-constrained researchers out.
- **The Opacity Barrier**: Most competitive models are released as open-weight rather than fully open-source, keeping training recipes, curricula, and agent traces proprietary.

The central thesis is:

> Compact models are inherently bounded by static parametric capacity, but they can transcend this limitation through a dual engine of deliberate internal thinking and active external seeking.

Rather than relying on passive memorization of the open web, ZGCM-1 bridges knowledge gaps by coupling long-horizon chain-of-thought reasoning with autonomous tool use—actively gathering web evidence, interacting with system terminals, and analyzing stripped binary programs.

## Methodology

### Architecture

ZGCM-1 uses a decoder-only Transformer with:
- **Grouped-Query Attention (GQA)**: 32 query heads, 8 KV heads (head dimension 128)
- **Hybrid attention**: 27 gated sliding-window attention (SWA) layers with 128-token window + 5 global attention layers at positions 6, 12, 18, 24, 30
- **RMSNorm, SwiGLU activation, Partial RoPE** (rotary fraction 0.33), QK normalization

The gated SWA module computes:

$$
\operatorname{GatedSWA} (h) = o _ {\text { proj }} \left(A _ {\text { SWA }} (q, k, v) \odot \sigma \left(g _ {\text { proj}} (h)\right)\right).
$$

### Pre-Training

- **General Pre-Training**: Two stages (~0.99T + 3.20T tokens) with curriculum pretraining (lexical complexity ordering for general text, separate interleaving for code/math)
- **Mid-Training**: 600.51B tokens across three context stages (16K, 64K, 256K) with MDP-formulated agentic data
- **Optimization**: Muon optimizer (momentum 0.9, 5 Newton-Schulz steps, LR $2 \times 10^{-4}$), hybrid FP8 precision (E4M3 forward, E5M2 backward), TWEO activation regularization

The estimated pre-training efficiency gain:

$$
1.4 \times 1.5 \times 1.8 \times 1.1 \approx 4.2
$$

### Post-Training

- **SFT**: 4,921,933 examples (96.46% general, 3.54% agentic), with tiered quality filtering, 8-gram decontamination, and mixed think/no-think supervision
- **RL**: GRPO with dynamic sampling, KL regularization, and length penalties

## Empirical Validation / Results

### 7B Scale Comparison (Selected Results)

| Benchmark | ZGCM-1-7B | DeepSeek-R1-0528-Qwen3-8B | MiniCPM4.1-8B | Qwen3-8B |
|---|---|---|---|---|
| MATH-500 | **97.13** | 96.32 | 95.60 | 96.20 |
| AIME 2026 | **75.00** | 69.17 | 71.67 | 66.67 |
| HMMT 2025 | **70.42** | 61.50 | 52.50 | 43.33 |
| AGIEval SAT Math | **99.09** | 91.14 | 98.98 | 99.09 |
| HumanEval+ | **90.24** | 88.87 | 89.63 | 80.20 |

### Agentic Search Results

| Model / System | WebWalkerQA | BrowseComp | GAIA text-only |
|---|---|---|---|
| **ZGCM-1-7B** | **63.09** | **19.43** | 42.52 |
| Qwen3-235B-A22B | 59.60 | 2.30 | 45.60 |
| Kimi-K2 | 63.00 | 14.10 | 57.30 |
| Claude 4 Sonnet | 61.70 | 12.20 | 68.30 |
| GPT-4o | 33.80 | 1.90 | 34.60 |

### Binary Function Search (50 tasks)

| Model | Accuracy (%) |
|---|---|
| DeepSeek-V4 Flash | 76.00 |
| Qwen3.5-397B-A17B | 76.00 |
| GLM-5.1 | 66.00 |
| **ZGCM-1-7B** | **62.00** |
| Kimi-K2 | 62.00 |
| Qwen3-8B | 12.00 |

## Theoretical and Practical Implications

### Eight Empirical Findings

1. **Curriculum Pre-training**: Lexical complexity is a cheap ordering signal for general text but poor for code/math; sorting general documents while interleaving technical domains improves BPB on technical domains.
2. **TWEO enables FP8 pretraining**: Suppressing activation outliers is crucial for FP8 stability, sustaining 60% BF16-equivalent MFU.
3. **Pretraining Efficiency Gain**: ~4.2× speedup from co-design of SWA (1.4×), FP8 (1.5×), Muon (1.8×), and Pre-LN (1.1×).
4. **Data Quality Trumps Volume**: Pruning ~50% of SFT candidates improved six-benchmark mean from 67.78 to 68.83.
5. **Long-CoT Trade-offs**: Oversaturating with extended reasoning degrades instruction following; dynamic calibration is needed.
6. **General Data for Agent Training**: Agent-only fine-tuning degrades interaction fidelity; co-training with general data is essential.
7. **Long Agentic Mid-Training Bypasses Ultra-Long SFT**: Moderate-length SFT suffices to activate 256K capabilities established in mid-training.
8. **AI Autonomy Is Task-Dependent**: Operational tasks reach L4 autonomy, while architecture/algorithm design remains at L2.

### AI-Native R&D

The team developed an AI-native workflow where agents autonomously manage cluster operations, data curation, and rapid diagnostic evaluation. The Atomic Capability Evaluation (ACE) suite (2,503 probes, 183 capabilities, 18 categories) provides ~2-3 minute diagnostic feedback loops.

## Conclusion

ZGCM-1 demonstrates that compact models can achieve frontier-competitive reasoning and agentic performance through the coupling of deliberate internal thinking with active external tool use. The fully open-source release (weights, data, code, logs) enables reproducible research on efficient foundation model development.

**Future directions** include:
1. Extension to sparse MoE architectures
2. End-to-end interactive agentic RL in real-world sandboxes
3. Autonomous dynamic knowledge retrieval
4. Self-evolving AI4AI R&D ecosystems

**Key limitations**: parametric knowledge bounds in closed-book tasks, instruction adherence vs. reasoning verbosity trade-offs, nascent general software/terminal agency, and environment/protocol brittleness.

---

_Markdown view of https://picx.dev/p/ZjyTcS, served by PicX — AI-generated visual whiteboard summaries of research papers._