# SemaPLC: A Project-Grounded, Verification-Gated Agent Harness for PLC Code Generation

> SemaPLC, a verification-gated agent harness using project grounding and three external check layers, improves PLC code generation reliability across all models, achieving up to 52.2% dynamic behavior accuracy versus 31.4% for baselines.

- **Source:** [arXiv](https://arxiv.org/abs/2608.18565)
- **Published:** 2026-08-21
- **Permalink:** https://picx.dev/p/xxNgxc
- **Whiteboard:** https://picx.dev/p/xxNgxc/image

## Summary

# SemaPLC: A Project-Grounded, Verification-Gated Agent Harness for PLC Code Generation

## Summary (Overview)

- **SemaPLC** is a verification-gated agent harness for PLC (Programmable Logic Controller) code generation that grounds generation in the target project and withholds completion until external checks confirm the result—rather than relying on the model's own self-assessment.
- The harness enforces **three verification layers**: specification audits, compilation checks, and live runtime validation, with a strict completion rule requiring logged external evidence for every claimed pass.
- On the **function track** (117 independent-POU tasks), SemaPLC achieves the highest strict verified pass rate on all seven models (72.6% mean vs. 63.9% for the strongest baseline), improving every backbone by 8.5–33.3 points.
- On the **project-context track** (65 tasks across ten industrial plants), SemaPLC leads in integrated compilation (89.4%), static behavior (81.6%), and dynamic behavior (52.2% vs. at most 31.4% for baselines)—demonstrating that methods looking similar under static evaluation differ sharply at runtime.
- The harness is open-sourced at https://github.com/midea-ai/SemaPLC.

## Introduction and Theoretical Foundation

### Background
Programmable logic controllers (PLCs) run industrial plants—factory lines, power plants, and water-treatment facilities—and are programmed primarily in IEC 61131-3 languages, of which **Structured Text (ST)** is the textual member. Prior work has established that LLMs can generate independent PLC program organization units (POUs), benefiting from:
- Compiler-in-the-loop repair
- Formal or property checking
- Multi-agent iteration
- Vendor IDEs
- Physical or simulated testbeds

### The Gap
In production, control logic is rarely an isolated POU. Deployment imposes two further requirements:

1. **Project grounding**: Generated logic must integrate into an established project, reuse its modules and function blocks, respect its variables/types/interfaces, and honor build, reset, initialization, and safety conventions.
2. **Correct runtime behavior**: Even when integrated programs compile and pass static checks, they can still misconfigure timers, take wrong state transitions, miss resets, break interlocks, or drive outputs with wrong timing.

**Key insight**: Prior systems execute generated code to *demonstrate* it can run, not to *measure* how reliably it runs. No unified assessment establishes, across methods and models, how reliably generated logic meets these two requirements.

### Three Design Principles of SemaPLC
1. **Project-grounded generation**: Anchors code generation and editing in available task or project context.
2. **Multi-source verification**: Treats specification checks, compilation, and live runtime validation as first-class external evidence.
3. **Verification-gated iteration**: Forbids the agent from terminating on its own judgment; failures, absences, or indeterminate outcomes require repair or incompletion reporting.

## Methodology

### Problem Formulation

**Function track**: Given requirement $R_f$ and local interface $I_f$, the system produces a POU $L_f = G(R_f, I_f)$. The primary metric is:

$$
\operatorname{VerifiedPass}(L_f) = \mathbb{1}[V_f \geq 0.80], \tag{1}
$$

where $V_f$ is the fraction of model-checked properties verified satisfied, with the 0.80 threshold following Agents4PLC.

**Project-context track**: Given requirement $R_p$ and existing project $P$, the system produces logic $L_p = G(R_p, P)$ and integrated program $P' = P \oplus L_p$, scored by three independent metrics:
- Integrated compilation $C(P') \in \{0, 1\}$
- Static behavior $S(P', R_p) \in [0, 100]$ (assertion-based, on program text)
- Dynamic behavior $D(P', \mathcal{T}) \in [0, 100]$ (golden-trace differential on live runtime)

### Algorithm: Verification-Gated Generation

The harness runs on a generic event-driven tool-use core with five components:
1. **Agent core**: Plans, edits, interprets verification results
2. **Project/task grounding**: Retrieves project structure, reuses existing variables and function blocks
3. **PLC skill library**: Domain knowledge in documents (rules file, curated wiki, procedural skills)
4. **Verification processes**: Specification audits, compilation, live runtime validation
5. **Verification gate**: Decides completion per Algorithm 1

**Three gate invariants**:
- **Bounded retries**: Each check allows at most $r = 2$ repair rounds
- **Edit invalidation**: Any modification voids all prior verdicts; every check re-runs
- **Earned claims**: Each outcome is a machine-readable sentinel cross-validated against the tool-call log

### Live Runtime Validation
Builds and deploys the implementation, initializes the runtime, injects scenario inputs, and samples external variables. Failure stages prescribe repair targets:
- Flat trace → wiring problem
- Changing but wrong trace → block logic
- Late transition → timer or edge detector

## Empirical Validation / Results

### RQ1: Function-Level Reliability

**Table 1: Function track strict verified pass rate (%, denominator 117)**

| Model | LLM4PLC | AutoPLC | Agents4PLC | SemaPLC bare | SemaPLC full |
|-------|---------|---------|------------|--------------|--------------|
| MiniMax-M2.7 | 22.2 | 49.6 | 53.8 | 39.3 | **69.2** |
| MiniMax-M3 | 15.4 | 65.0 | 55.6 | 60.7 | **69.2** |
| Qwen3.5-Plus | 13.7 | 67.5 | 67.5 | 62.4 | **75.2** |
| DS-V4-Flash | 41.0 | 54.7 | 54.7 | 34.2 | **67.5** |
| DS-V4-Pro | 43.6 | 61.5 | 62.4 | 55.6 | **69.2** |
| GLM-5.2 | 30.8 | 59.0 | 74.4 | 63.2 | **76.1** |
| GPT-5.5 | 44.4 | 79.5 | 78.6 | 71.8 | **82.1** |
| **Mean** | 30.2 | 62.4 | 63.9 | 55.3 | **72.6** |
| **Worst** | 13.7 | 49.6 | 53.8 | 34.2 | **67.5** |

Key findings:
- SemaPLC attains the highest rate on all seven models (+8.8 points mean over strongest baseline)
- Cross-model spread shrinks from 37.6 points (bare) to 14.6 points (full)
- Weakest models gain most (MiniMax-M2.7 +29.9, DeepSeek-V4-Flash +33.3)

### RQ2: Project-Grounded Reliability

**Table 2: Project-context track (0–100, best per column in bold)**

| Method | MiniMax M2.7 | MiniMax M3 | Qwen3.5 Plus | DS V4-Flash | DS V4-Pro | GLM-5.2 | GPT-5.5 | Worst | Best | Mean |
|--------|-------------|------------|--------------|-------------|-----------|---------|---------|-------|------|------|
| **Integrated compilation** |
| LLM4PLC | 47.7 | 53.8 | 16.9 | 60.0 | 52.3 | 80.0 | 100.0 | 16.9 | 100.0 | 58.7 |
| AutoPLC | 69.2 | 95.4 | 58.4 | 58.5 | 95.4 | 95.4 | 98.5 | 58.4 | 98.5 | 81.5 |
| Agents4PLC | 47.7 | 69.2 | 40.0 | 75.4 | 78.5 | 89.2 | 98.5 | 40.0 | 98.5 | 71.2 |
| **SemaPLC** | **81.5** | **95.4** | **80.0** | **84.6** | **89.2** | **95.4** | **100.0** | **80.0** | **100.0** | **89.4** |
| **Static behavior** |
| LLM4PLC | 76.1 | 76.7 | 74.5 | 70.3 | 69.2 | 77.1 | 86.3 | 69.2 | 86.3 | 75.7 |
| AutoPLC | 68.9 | 73.5 | 68.9 | 76.2 | 66.5 | 75.5 | 88.8 | 66.5 | 88.8 | 74.0 |
| Agents4PLC | 63.4 | 73.5 | 77.6 | 71.4 | 64.5 | 62.8 | 88.6 | 62.8 | 88.6 | 71.7 |
| **SemaPLC** | **74.9** | **84.9** | **79.9** | **78.0** | **81.2** | **88.0** | 84.1 | **74.9** | **88.0** | **81.6** |
| **Dynamic behavior** |
| LLM4PLC | 3.0 | 26.1 | 6.6 | 18.4 | 13.9 | 34.6 | 54.5 | 3.0 | 54.5 | 22.4 |
| AutoPLC | 4.0 | 43.5 | 19.8 | 23.9 | 45.7 | 21.9 | 61.1 | 4.0 | 61.1 | 31.4 |
| Agents4PLC | 4.5 | 30.8 | 11.4 | 28.3 | 28.8 | 44.6 | 63.6 | 4.5 | 63.6 | 30.3 |
| **SemaPLC** | **31.3** | **52.1** | **43.1** | **54.1** | **57.4** | **61.9** | **65.4** | **31.3** | **65.4** | **52.2** |

### RQ3: Verification Across Layers

**Key finding**: All methods fall within 10 static points of one another, whereas dynamic scores separate them sharply (22.4 to 31.4 for baselines vs. 52.2 for SemaPLC).

**Layer ablation (Table 3, DeepSeek-V4-Flash, 65 tasks)**:

| Verification layers | Comp. | Static | Dyn. | Tok. | Reqs. |
|--------------------|-------|--------|------|------|-------|
| None (generate only) | 64.6 | 71.5 | 23.1 | 34k | 8.9 |
| + Spec | 70.8 | 74.0 | 30.3 | 60k | 14.6 |
| + Compile | 83.1 | 77.8 | 43.7 | 74k | 25.5 |
| + Runtime (full) | 84.6 | 78.0 | 54.1 | 129k | 47.8 |

**Formal verification coverage (Table 5)**: Timers are the boundary—none of the 174 properties across 32 timer-bearing programs obtained a conclusive verdict, motivating runtime validation for stateful timing constructs.

### RQ4: Interaction Cost

| Method | Requests/task | Time/task (s) |
|--------|--------------|---------------|
| **Function track** |
| Agents4PLC | 6.3 (4.4–7.2) | 454 (241–688) |
| SemaPLC | 6.5 (5.5–7.6) | 71 (41–156) |
| **Project track** |
| Agents4PLC | 6.9 (6.8–7.0) | 344 (47–917) |
| SemaPLC | 34.1 (16.4–60.4) | 347 (25–1380) |

## Theoretical and Practical Implications

1. **Execution is the faithful test**: Static scoring compresses and even reorders methods that runtime validation separates sharply. Benchmarks must execute generated logic to distinguish reliable methods from unreliable ones.

2. **Verification-gated iteration is a model-agnostic reliability layer**: The harness improves every backbone model (8.5–33.3 points), with the weakest models gaining most—the external checks compensate for model limitations.

3. **Delivery-integrity guarantee**: The three invariants (bounded retries, edit invalidation, earned claims) ensure the delivered program is identical to the candidate that earned every reported pass, with no unlogged self-reported pass surviving.

4. **Formal verification has coverage boundaries**: Timers (TON) fall outside conclusive model-checking coverage in practice, making runtime validation a necessary complement rather than an alternative.

5. **Cost is architectural**: SemaPLC's open-ended loop issues more requests (each tool call is a model decision), but achieves comparable wall-clock time on the project track and lower time on the function track.

## Conclusion

SemaPLC demonstrates that a verification-gated agent harness—assembled from conventional tools but governed by strict external-check completion discipline—substantially improves PLC code generation reliability across models and evaluation layers. Key takeaways:

- **Best results across all layers**: 72.6% mean verified pass rate (function track), 89.4% integrated compilation, 81.6% static behavior, and 52.2% dynamic behavior (project track).
- **Runtime validation is essential**: Similar static scores mask sharply different runtime behavior; execution, not static scoring, is the faithful test.
- **Two limitations remain**: (1) dynamic scoring exercises a bounded scenario set, so behavior under unseen conditions remains unmeasured; (2) the advantage narrows on the strongest model (GPT-5.5).

**Future directions**: An end-to-end PLC development environment on this harness architecture, with editing, generation, and deployment governed by the same verification gate, and validation extended to process simulation.

---

_Markdown view of https://picx.dev/p/xxNgxc, served by PicX — AI-generated visual whiteboard summaries of research papers._