Summary (Overview)

• InCoder-32B is the first 32B-parameter code foundation model purpose-built for industrial code intelligence, unifying code intelligence across chip design (Verilog/RTL), GPU kernel optimization (CUDA/Triton), embedded systems, compiler optimization, and 3D modeling (CAD).
• The model is trained from scratch using a three-stage Code-Flow pipeline: (1) pre-training with curated industrial code data, (2) mid-training with progressive context extension from 8K to 128K tokens using synthetic industrial reasoning data and agentic trajectories, and (3) post-training with execution-grounded verification.
• InCoder-32B achieves competitive general code performance (74.8% on SWE-bench Verified, 49.14% on LiveCodeBench, 60.99% on BFCL) while establishing strong open-source baselines across all industrial domains, even surpassing the proprietary Claude-Sonnet-4.6 on CAD-Coder IoU and KernelBench L1/L2/L3.
• The paper introduces two new industrial benchmarks: VeriScope (568 Verilog generation problems across 5 difficulty levels) and VeriRepair (~22,000 training + 300 test Verilog bug repair samples), alongside existing benchmarks like RealBench, ArchXBench, KernelBench, TritonBench, EmbedCGen, SuperCoder, and CAD-Coder.
• Key ablations reveal that repository transition data outperforms static snapshots for planning, mid-training reasoning trajectories improve robustness under distribution shift, and thinking paths unlock emergent capabilities absent in standard instruction tuning.

---

Introduction and Theoretical Foundation

Background and Motivation

Code intelligence has advanced rapidly with LLMs like Qwen3.5, DeepSeek-V3.2, and Claude-4.6 achieving strong performance on general programming tasks. However, a critical gap persists between general code intelligence and the demands of industrial software development:

"Scenarios such as CUDA kernel optimization, Verilog hardware description, embedded firmware programming, and compiler optimization impose requirements that fundamentally differ from conventional software engineering with specialized language semantics, strict timing and resource constraints, reasoning about hardware behavior, and rigorous verification methodologies."

Existing models struggle on industrial tasks, with the best models achieving only 28.80% call success rate on Triton operator generation and 33.3% accuracy on Verilog code that passes formal equivalence checking.

Key Insight: Simulation-Grounded Training

The central theoretical premise is that industrial code correctness can only be established by running code in the same environment where it will ultimately be deployed. This motivates reconstructing four classes of industrial environments in software:

Architectural Choice

InCoder-32B adopts an efficient recurrent architecture with 32B parameters, designed to balance capability with deployability in industrial settings. It is trained with both autoregressive language modeling and fill-in-the-middle (FIM) completion objectives using a standard decoder-only Transformer architecture on 4,096 GPUs.

---

Methodology

Three-Stage Training Pipeline

Stage 1: Pre-Training

Data Collection: Industrial codes are collected from public repositories, technical literature, and domain-specific web data using a three-step recall strategy. OCR is used to extract code snippets and structured content from technical literature.

Data Cleaning and Refinement:

• License filtering, PII removal, file-level validation
• Deduplication at multiple levels: exact hash matching, token-level near-duplicate detection, repository-level fork consolidation, cross-source deduplication
• Domain-specific checks, surface-level formatting normalization, structured annotation
• AST comparison and re-compilation verification for all refined samples

Stage 2: Mid-Training

Context Extension: Two sub-stages progressively extend context from 8K → 32K → 128K tokens. The first sub-stage focuses on file-level tasks (e.g., completing RTL modules); the second unlocks long-context capabilities (e.g., extended debugging sessions).

Industrial Data Synthesis includes three components:

1. Synthetic Industrial Code QA: Three-step pipeline — (i) scenario specification with practicing engineers, (ii) seed code generation reflecting realistic hardware patterns, (iii) QA pair synthesis with automated verification
2. Agent Trajectories: Multi-step debugging/repair following the Thought-Action-Observation cycle with tool feedback from simulators, synthesis tools, compilers, and formal verification engines
3. Industrial Code Artifacts: Testbenches (SystemVerilog/UVM), timing constraints (SDC), synthesis scripts, GPU profiling traces, memory sanitizer logs

Stage 3: Post-Training

Data Construction: 2.5M samples constructed from real-life industrial coding tasks grounded in execution, spanning hardware design, GPU kernel development, systems programming, and embedded firmware.

Task Construction → Candidate Generation → Verification → Feedback-Driven Repair → Quality Filtering:

• Tasks decomposed into structured instructions with interface constraints, platform/toolchain specifications, and verification scripts
• Candidate diversity via template-based perturbation and cross-language migration
• Execution-grounded verification in real production environments
• Failed solutions repaired using full feedback context (compiler errors, runtime logs, waveform differences, profiling bottlenecks)
• Final filtering by executability, stability, and information density, categorized into: direct solutions, defect repairs, and performance/structural optimization samples

Evaluation Setup

General benchmarks (14): EvalPlus (HumanEval, MBPP), BigCodeBench, FullStackBench, CRUXEval, LiveCodeBench, Mercury, Spider, BIRD, Terminal-Bench v1.0/v2.0, SWE-bench Verified, Mind2Web, BFCL V3, τ²-bench

Industrial benchmarks (9):

---

Empirical Validation / Results

General Code Performance (Key Results)

Table 1 (Code Generation): InCoder-32B achieves competitive results:

Agentic Coding (Table 3): InCoder-32B achieves 74.8% on SWE-bench Verified, ranking first among all open-weight baselines and competitive with much larger models (DeepSeek-V3.2: 73.1%, GLM-4.7: 73.8%).

Industrial Code Performance (Key Results)

Table 4 (Chip Design): InCoder-32B achieves the best open-weight results on RealBench module-level tasks by a wide margin:

Table 5 (GPU, Code Optimization, 3D Modeling): InCoder-32B leads all open-weight baselines and surpasses Claude-Sonnet-4.6 on several tasks:

Error Analysis

Analysis of 1,882 failure cases across 9 industrial benchmarks reveals five recurring error themes:

1. Compilation and syntax errors dominate Verilog tasks: 71% of RealBench failures (malformed literals, incorrect port declarations, bit-width mismatches) and 51% of ArchXBench failures
2. Incomplete industrial API knowledge: 47% of EmbedCGen failures are linker errors from undefined HAL/CMSIS functions; 33% NameErrors and 24% TypeErrors on TritonBench
3. Functional correctness gaps: 79% of VeriRepair failures compile but fail test cases; 93% of CAD-Coder failures are geometric (mostly Euler angle convention misinterpretation)
4. Optimization insufficiency: 33% of KernelBench failures are functionally correct but insufficiently fast; 83% of SuperCoder failures are the model simply copying input assembly
5. Format errors: 46% of VeriScope failures are unparseable outputs ignoring required structured format

Scaling Analysis

Training on 83M → 167M → 250M SFT tokens shows consistent improvements across most benchmarks, with only minor regressions on a few RealBench and TritonBench sub-metrics at the 250M stage, suggesting verification-related understanding may saturate early.

---

Theoretical and Practical Implications

Theoretical Implications

1. Industrial code intelligence requires environment-grounded training: The paper demonstrates that reconstructing real industrial toolchains (simulators, compilers, hardware emulators) for training signal generation is critical for transferring capabilities to real deployment.

2. General code competence does not automatically transfer to industrial domains: Despite strong general performance from baseline models, all show significant degradation on industrial benchmarks, confirming these tasks are fundamentally out-of-distribution for general code LLMs.

3. Thinking paths unlock emergent capabilities: The paper's ablations show that thinking/reasoning paths enable capabilities not present in standard instruction tuning, suggesting that reasoning about hardware constraints is a distinct skill requiring explicit training.

4. Repository transition data > static snapshots: For planning abilities, data capturing the process of code evolution (transitions) is more valuable than static repository states.

Practical Implications

1. Unified industrial foundation model: InCoder-32B demonstrates that a single model can serve multiple fragmented industrial domains (chip design, GPU optimization, embedded systems, compiler optimization, 3D modeling) rather than requiring domain-specific models.

2. Hardware constraint awareness: The paper's motivating example (Figure 2) shows InCoder-32B correctly flattening CUDA grid dimensions to avoid the 65,535 hardware limit on gridDim.y, demonstrating learned hardware constraint reasoning.

3. Benchmark infrastructure: The paper provides the most comprehensive industrial code evaluation to date (14 general + 9 industrial benchmarks), including two new benchmarks (VeriScope, VeriRepair) that will serve as resources for future research.

4. Practical deployment: As a 32B dense model, InCoder-32B is more deployable than the 100B+ MoE models it competes with, making it practical for industrial settings with resource constraints.

---

Conclusion

InCoder-32B bridges the long-standing gap between general code intelligence and industrial software development through:

1. A systematic three-stage Code-Flow training pipeline that combines general code pre-training, curated industrial annealing, context extension with synthetic industrial reasoning data, and execution-grounded post-training.

2. Comprehensive evaluation demonstrating competitive general code performance (74.8% SWE-bench Verified, 49.14% LiveCodeBench) alongside state-of-the-art industrial capabilities (best open-weight results on RealBench module tasks, CAD-Coder, and KernelBench across all levels).

3. Key insights from ablations: repository transition data improves planning, mid-training reasoning trajectories enhance robustness under distribution shift, and thinking paths unlock emergent capabilities.

Future directions identified by the error analysis include:

• Data curation targeting rare APIs and hardware semantics
• Training with verification signals in the loop
• Explicit reasoning about low-level performance
• Addressing systematic misinterpretations (e.g., Euler angle conventions in CAD)
• Improving optimization capabilities (only 33% of KernelBench failures are correctness issues; performance optimization remains a major bottleneck)

The authors acknowledge that "there is still a long journey to apply LLMs in realistic industrial scenarios," but InCoder-32B represents a significant step toward unified industrial code intelligence.