Summary of Intern-S2-Preview: Scientific Agentic Foundation Model

Summary (Overview)

• Intern-S2-Preview-397B is a scientific agentic foundation model designed for multimodal scientific understanding, reasoning, generation, and long-horizon agentic tasks, integrating both static question-answering and iterative tool-grounded problem solving.
• The training pipeline combines scientific multimodal pre-training (including Visual Pre-training on rendered documents and interleaved image-text data) with a unified post-training pipeline comprising supervised fine-tuning, scalable multi-task RL, black- and white-box agentic RL, and on-policy distillation.
• Key technical innovations include partial rollout with off-policy correction, adaptive length regularization, online speculative decoding, Group-level Entropy-Controlled Policy Optimization (GEPO), and a harness × task abstraction for unified agentic RL across heterogeneous environments.
• The architecture extends time series modelling to numerical forecasting via a dedicated forecasting branch, and introduces Memory Decoder as a separate memory-augmented path for rapid scientific specialization without modifying the frozen 397B backbone.
• Evaluations show competitive or leading results across scientific, multimodal, agentic, general-purpose, and time-series benchmarks; the Memory Decoder extension improves Biology-Instructions average score from 56.92 to 60.32 without backbone modification.

Introduction and Theoretical Foundation

Scientific discovery increasingly requires AI systems that can reason over heterogeneous evidence modalities, interact with tools and environments, and sustain progress across long task horizons. The authors identify a gap in existing model families:

• General-purpose LLMs (e.g., GPT, Claude) provide broad instruction following but lack specialization for heterogeneous scientific modalities, domain protocols, or verifiable tool interaction.
• Scientific multimodal models improve perception over specialized inputs but are often evaluated as static QA systems rather than long-horizon agents.

The theoretical foundation rests on two complementary architectural requirements:

1. Long numerical signal understanding + forecasting: Scientific workflows require models to both understand long numerical signals and predict future system states, motivating the dedicated forecasting branch.
2. Fast domain adaptation without catastrophic forgetting: Directly fine-tuning a backbone for each new domain risks perturbing general capabilities, motivating the Memory Decoder approach where external parametric memories are attached to a frozen backbone.

The post-training framework is grounded in recent advances in reinforcement learning for reasoning (e.g., GRPO, DAPO, REINFORCE variants), agentic RL systems, and distillation techniques, building on prior work including Intern-S1-Pro, Agent-FLAN, Lagent, MindSearch, and SciExplore.

Methodology

Architecture

Memory Decoder: A separate extension model that attaches domain-specific knowledge via external parametric memories while keeping the backbone frozen. At each decoding step, a lightweight token-level router predicts a fusion weight $\lambda_t \in [0,1]$ combining the backbone and memory distributions:

$$p_{\mathrm{final}}(\cdot \mid c_t) = (1-\lambda_t) p_{\mathrm{S2}}(\cdot \mid c_t) + \lambda_t p_{\mathrm{mem}}(\cdot \mid c_t)$$

Training uses retrieval distillation from a token-level datastore with a soft next-token teacher distribution:

$$p_{\text{ret}}(y \mid c_t) \propto \sum_{(k_j, v_j) \in \mathcal{N}(k_t)} \mathbb{I}_{y = v_j} \exp(-d(k_t, k_j)/\tau)$$

Time Series Modules: The upgraded encoder partitions input into temporal chunks, applies compressive patching (normalization, CNN feature extraction, Q-Former compression), and introduces a channel-wise Transformer encoder for inter-channel dependencies. Maximum input length increased from ~240,000 to 300,000 time steps with 5–6× faster inference and ~20% GPU memory. A dedicated numerical forecasting branch preserves numerical fidelity via a causal Transformer forecaster conditioned on fused LLM and encoder representations.

Pre-training

• Visual Pre-training (VP): Learns from rendered scientific pages by predicting visual latents with a contrastive next-latent prediction objective, preserving document structure lost in text extraction.
• Interleaved image-text data: Uses MinerU2.5-Pro for OCR/layout parsing, crops visual units (images, equations, tables), and applies visual-gain filtering based on perplexity reduction when visual content is included.
• Image retrieval enhancement: Large-scale vector database (Milvus) with text-to-image and image-to-image retrieval using an 8B embedding model, plus reranking for quality.

Post-Training

Supervised Fine-Tuning: Large-scale multimodal dataset covering general conversation, code, tool use, scientific tasks, and agentic trajectories.

Scalable Multi-Task RL with four key techniques:

1. Partial rollout with off-policy correction: Pause-and-resume mechanism with importance-sampling ratios:

$$\rho_{i,t}(\theta) = \frac{\pi_\theta(y_{i,t} \mid s_{i,t})}{\pi_{\text{beh}(i,t)}(y_{i,t} \mid s_{i,t})}$$

Clipped as $\bar{\rho}_{i,t}(\theta) = \mathrm{clip}(\rho_{i,t}(\theta), 1-\epsilon_{\text{low}}^{\text{IS}}, 1+\epsilon_{\text{high}}^{\text{IS}})$, plus Rollout Routing Replay (R3) and bidirectional binary KL divergence masking for numerical consistency.

2. Adaptive length regularization: Reweights advantages of positive responses only when pass rate exceeds threshold $\tau$:

$$\widetilde{A}_i = \frac{\sum_{j \in \mathcal{P}_q} \hat{A}_j}{\sum_{j \in \mathcal{P}_q} w_j \hat{A}_j + \epsilon} w_i \hat{A}_i$$

with length-dependent weight $w_i = \alpha + (1-\alpha)(1 - \frac{L_i - L_{\min}^+}{L_{\max}^+ - L_{\min}^+ + \epsilon})^\gamma$.

3. Online speculative decoding: Draft model trained with hybrid LK loss combining forward KL and total variation distance, with adaptive mixing coefficient $\lambda_k = \exp(-\eta \operatorname{sg}[\bar{\alpha}_k])$ based on acceptance rate. Achieves ~2× rollout speedup and 1.7× end-to-end speedup.

4. GEPO (Group-level Entropy-Controlled Policy Optimization): Scales advantages based on group-level entropy to balance exploration across heterogeneous tasks:

$$\hat{A}_i = \begin{cases} \alpha_{\text{low}} \cdot A_i & \text{if } A_i > 0 \text{ and } \mathcal{H}_g(x) < \mathcal{H}_{\text{low}}^{(t)}, \\ \alpha_{\text{high}} \cdot A_i & \text{if } A_i < 0 \text{ and } \mathcal{H}_g(x) > \mathcal{H}_{\text{high}}^{(t)}, \\ A_i & \text{otherwise.} \end{cases}$$

Unified RL objective (leave-one-out REINFORCE with all stabilizations):

$$\mathcal{L}_{\text{RL}}(\theta) = -\mathbb{E}_{(q, \{y_i\}_{i=1}^G) \sim \mathcal{B}} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} m_{i,t}^{\text{BKL}} \mathrm{sg}[\bar{\rho}_{i,t}(\theta)] \widetilde{A}_i \log \pi_\theta(y_{i,t} \mid s_{i,t}) \right]$$

Agentic RL Framework: Uses a harness × task abstraction decoupling agent runtimes from executable task distributions. Supports white-box (orchestrated directly) and black-box (OpenClaw, Claude Code, OpenCode, OpenHands, Mini-SWE) harnesses. Tasks come from public coding/terminal sources (Table 1) and a self-evolving task-synthesis system with skill-state graphs and progressive validation. A token-in-token-out (TITO) interface with PrefixTree trace storage enables trace-aware experience assembly. Process-aware advantage control applies process weights only to positive advantages.

On-Policy Distillation: Separately trained reasoning and agentic experts are consolidated into a unified model using reverse KL divergence objective:

$$\mathcal{J}_{\text{OPD}}(\theta) = \sum_{d \in \{\text{rea,agt}\}} \lambda_d \mathbb{E}_{q \sim \mathcal{D}_d, y \sim \pi_\theta(\cdot|q)} \left[ \sum_{t=1}^H \left(\log \pi_{T_d}(y_t \mid s_t) - \log \pi_\theta(y_t \mid s_t)\right) \right]$$

with sampled-token distillation advantage $\widehat{A}_{i,t}^{\text{OPD}} = \operatorname{sg}[\log \pi_{T_d}(y_{i,t} \mid s_{i,t}) - \log \pi_{\text{prox}}(y_{i,t} \mid s_{i,t})]$, reducing teacher payload from $O(HV)$ or $O(Hk)$ to $O(H)$.

Empirical Validation / Results

Scientific Benchmarks (Table 2)

General Benchmarks (Table 3)

Intern-S2-Preview-397B achieves best open-source results on MMLU-Pro (89.75), SimpleQA-Verified (69.90), MMMU-Pro (80.46), and ChartQAPro (69.65).

Time Series Understanding (Table 4)

Outperforms general-purpose Text/VL LLMs across all tasks. Surpasses trillion-parameter Intern-S1-Pro on 7/9 shared tasks (e.g., PHU01 F1: 36.8 → 66.9). Extends to radar signal analysis tasks (RAU01: 88.4, RAU02: 60.2).

Time Series Forecasting (Table 5)

With dedicated numerical forecasting branch, achieves best MAPE on all 7 SciTS forecasting tasks (e.g., ENG02: 60.2, URG05: 60.6) while maintaining 100% success rate. Horizon predictor achieves 99% accuracy. Zero-shot MASE of 0.785 on GIFT-Eval.

Memory Decoder Study

Intern-MemDec-4B improves Biology-Instructions from 56.92 to 60.32 (Table 6 per-task results show gains on DNA-cpd: 63.11→72.57, RNA-CRISPROnTarget: 6.61→17.18) while remaining close to the frozen backbone on cross-domain benchmarks (MMLU Pro, Mol-Instructions, MMMU Pro, MicroVQA, SFE).

Theoretical and Practical Implications

• Scientific capability assessment: The work argues that scientific AI should be evaluated not only on isolated benchmark-answer accuracy but on whether models can connect reasoning to executable, verifiable, iterative workflows—a shift toward workflow-oriented evaluation.
• Modular specialization: Memory Decoder demonstrates that domain extension can be achieved without backbone rewriting, avoiding the trade-off between domain performance and general capability preservation. This has practical implications for deploying general models across continuously evolving scientific subfields.
• Scalable RL infrastructure: The partial rollout system, online speculative decoding, and GEPO provide a blueprint for stable, efficient RL training over heterogeneous task mixtures with long rollouts, addressing practical bottlenecks in large-scale agentic training.
• Unified agentic training: The harness × task abstraction enables heterogeneous agent runtimes and task distributions to share a common rollout, verification, and training protocol, reducing the cost of adding new environments or harnesses.
• Time series as first-class modality: The dedicated forecasting branch demonstrates that numerical forecasting should be treated as a native output modality rather than text-token generation, preserving numerical fidelity and enabling flexible prediction horizons.

Conclusion

Intern-S2-Preview-397B demonstrates broad capability coverage across scientific, multimodal, agentic, general-purpose, and time-series evaluations. The main design choices in architecture (time series modules, Memory Decoder), pre-training (Visual Pre-training, interleaved data, image retrieval), and post-training (multi-task RL, agentic RL, on-policy distillation) are validated by empirical results. The Memory Decoder study shows that targeted scientific specialization can be added to the frozen backbone while preserving its role as a general model.

Future directions identified by the authors include:

• Improving reliability over longer scientific workflows
• Expanding domain-specific memories and task environments
• Strengthening verifiers against reward hacking and leakage
• Deepening integration with specialized scientific tools

Key limitations: Intern-S2-Preview remains a preview system; some benchmarks (MP20, ProteinBinder-9) are internal evaluation sets; agentic performance varies across harnesses and task families; the model still trails closed-source models on several general-purpose and agentic benchmarks (e.g., TerminalBench 2.1, WildClawBench).