Summary (Overview)

• ZGCM-1 is a fully open-source 7.39B dense foundation model trained from scratch, designed for mathematical reasoning and agentic search across a 256K context, challenging the assumption that frontier intelligence requires hundreds of billions of parameters.
• The core thesis: compact models are bounded by static parametric capacity but can transcend this through a dual engine of deliberate internal thinking (chain-of-thought reasoning) and active external tool use (web search, terminal interaction, binary analysis).
• Key technical contributions include: (1) hybrid gated sliding-window/global attention (5:1 ratio) achieving a 3.94× throughput speedup and 6.4× KV-cache reduction at 256K; (2) FP8 pre-training with the Muon optimizer and TWEO outlier regularization yielding ~4.2× time-to-loss speedup; (3) progressive curriculum mid-training (16K→64K→256K) with MDP-formulated interaction traces; (4) AI-native R&D workflow with agent swarms.
• ZGCM-1-7B achieves state-of-the-art results among sub-10B models on reasoning benchmarks (75.0% AIME 2026, 97.1% MATH-500, 70.4% HMMT 2025) and remains competitive with frontier models orders of magnitude larger on agentic tasks (63.1% WebWalkerQA, 62.0% Binary Function Search).
• The project releases model weights from all training stages, intermediate checkpoints, training code, data recipes, and W&B logs for full reproducibility.

Introduction and Theoretical Foundation

The paper addresses two critical bottlenecks in foundation model research:

• The Scale Barrier: Frontier reasoning and agentic search are often seen as exclusive to hundred-billion-parameter systems, locking compute-constrained researchers out.
• The Opacity Barrier: Most competitive models are released as open-weight rather than fully open-source, keeping training recipes, curricula, and agent traces proprietary.

The central thesis is:

Compact models are inherently bounded by static parametric capacity, but they can transcend this limitation through a dual engine of deliberate internal thinking and active external seeking.

Rather than relying on passive memorization of the open web, ZGCM-1 bridges knowledge gaps by coupling long-horizon chain-of-thought reasoning with autonomous tool use—actively gathering web evidence, interacting with system terminals, and analyzing stripped binary programs.

Methodology

Architecture

ZGCM-1 uses a decoder-only Transformer with:

• Grouped-Query Attention (GQA): 32 query heads, 8 KV heads (head dimension 128)
• Hybrid attention: 27 gated sliding-window attention (SWA) layers with 128-token window + 5 global attention layers at positions 6, 12, 18, 24, 30
• RMSNorm, SwiGLU activation, Partial RoPE (rotary fraction 0.33), QK normalization

The gated SWA module computes:

$$\operatorname{GatedSWA} (h) = o _ {\text { proj }} \left(A _ {\text { SWA }} (q, k, v) \odot \sigma \left(g _ {\text { proj}} (h)\right)\right).$$

Pre-Training

• General Pre-Training: Two stages (~0.99T + 3.20T tokens) with curriculum pretraining (lexical complexity ordering for general text, separate interleaving for code/math)
• Mid-Training: 600.51B tokens across three context stages (16K, 64K, 256K) with MDP-formulated agentic data
• Optimization: Muon optimizer (momentum 0.9, 5 Newton-Schulz steps, LR $2 \times 10^{-4}$), hybrid FP8 precision (E4M3 forward, E5M2 backward), TWEO activation regularization

The estimated pre-training efficiency gain:

$$1.4 \times 1.5 \times 1.8 \times 1.1 \approx 4.2$$

Post-Training

• SFT: 4,921,933 examples (96.46% general, 3.54% agentic), with tiered quality filtering, 8-gram decontamination, and mixed think/no-think supervision
• RL: GRPO with dynamic sampling, KL regularization, and length penalties

Empirical Validation / Results

7B Scale Comparison (Selected Results)

Agentic Search Results

Binary Function Search (50 tasks)

Theoretical and Practical Implications

Eight Empirical Findings

1. Curriculum Pre-training: Lexical complexity is a cheap ordering signal for general text but poor for code/math; sorting general documents while interleaving technical domains improves BPB on technical domains.
2. TWEO enables FP8 pretraining: Suppressing activation outliers is crucial for FP8 stability, sustaining 60% BF16-equivalent MFU.
3. Pretraining Efficiency Gain: ~4.2× speedup from co-design of SWA (1.4×), FP8 (1.5×), Muon (1.8×), and Pre-LN (1.1×).
4. Data Quality Trumps Volume: Pruning ~50% of SFT candidates improved six-benchmark mean from 67.78 to 68.83.
5. Long-CoT Trade-offs: Oversaturating with extended reasoning degrades instruction following; dynamic calibration is needed.
6. General Data for Agent Training: Agent-only fine-tuning degrades interaction fidelity; co-training with general data is essential.
7. Long Agentic Mid-Training Bypasses Ultra-Long SFT: Moderate-length SFT suffices to activate 256K capabilities established in mid-training.
8. AI Autonomy Is Task-Dependent: Operational tasks reach L4 autonomy, while architecture/algorithm design remains at L2.

AI-Native R&D

The team developed an AI-native workflow where agents autonomously manage cluster operations, data curation, and rapid diagnostic evaluation. The Atomic Capability Evaluation (ACE) suite (2,503 probes, 183 capabilities, 18 categories) provides ~2-3 minute diagnostic feedback loops.

Conclusion

ZGCM-1 demonstrates that compact models can achieve frontier-competitive reasoning and agentic performance through the coupling of deliberate internal thinking with active external tool use. The fully open-source release (weights, data, code, logs) enables reproducible research on efficient foundation model development.

Future directions include:

1. Extension to sparse MoE architectures
2. End-to-end interactive agentic RL in real-world sandboxes
3. Autonomous dynamic knowledge retrieval
4. Self-evolving AI4AI R&D ecosystems

Key limitations: parametric knowledge bounds in closed-book tasks, instruction adherence vs. reasoning verbosity trade-offs, nascent general software/terminal agency, and environment/protocol brittleness.