Summary (Overview)

• ReqClarifyBench is introduced as the first interactive requirement clarification benchmark for repository-level software development, containing 341 tasks across game, miniapp, and web development domains.
• The benchmark includes three core components per task: a high-level vague user intent, a human-validated detailed requirement document, and code evaluation artifacts from source benchmarks (GameCraft-Bench, MiniAppBench, WebGen-Bench).
• Three representative user personas (Expert Nitpicker, Vague Requester, Mind Changer) are derived from the SWE-chat dataset to power LLM-based user simulators, validated with 95.8% classification accuracy and Cohen's κ of 0.892.
• Key finding: clarification effectiveness does not necessarily translate into requirement document quality — GPT-5.6-Sol achieves the best ReqAlign (0.727) but the lowest clarification effectiveness (EQR 0.196), while Gemini-3.1-Pro shows the opposite pattern.
• Interactive clarification consistently improves requirement document quality (ReqAlign improves by 133.1% on average) and further boosts code generation performance across all three source benchmarks.

Introduction and Theoretical Foundation

The paper addresses a critical bottleneck in LLM-based software development: requirement clarification. As LLMs' implementation capabilities improve, the ability to understand what users actually want becomes increasingly important. Real-world users often begin with vague intents, leaving many requirements to be clarified through interaction. In repository-level development, an incorrect guess about user intent can propagate through subsequent phases, causing costly rework.

The authors identify a gap in existing research:

• Repository-level code generation benchmarks (e.g., SWE-bench, FeatureBench, WebGen-Bench) evaluate code generation from either complete specifications or underspecified requests, but overlook multi-turn requirement clarification.
• Prior requirement clarification studies (e.g., ClarifyGPT, ClarifyCodeBench, ReqElicitGym) focus on function-level tasks and use simulated users with fixed interaction patterns, failing to capture real-world complexity and diverse user behaviors.

The theoretical foundation draws from Requirements Engineering (RE), where requirement clarification involves iterative interaction with stakeholders to clarify goals, constraints, and unstated requirements. The paper argues that this process is essential for LLM-based software development but remains underexplored.

Methodology

Benchmark Construction Pipeline

ReqClarifyBench is built through a semi-automated human-in-the-loop pipeline with three stages:

1. Source Benchmark Curation: Selects GameCraft-Bench (games), MiniAppBench (miniapps), and WebGen-Bench (web) based on criteria of real-world requirements, repository-level complexity, and behavioral evaluation protocols.

2. Requirement Reconstruction and Intent Abstraction: An LLM reconstructs structured requirement documents from diverse artifacts, using a requirement–artifact traceability mechanism to prevent hallucinated details. A second LLM pass performs intent abstraction, removing implementation details while preserving core software objectives.

3. Human Validation: Two authors validate requirements against ISO/IEC/IEEE 29148:2018 criteria (Correctness, Unambiguity, Completeness, Consistency), using traceability evidence. Only samples passing both validation stages are included.

LLM-based User Simulator

Three personas are derived from SWE-chat (covering ~80% of its users):

• Expert Nitpicker: Provides accurate, detailed responses with stable goals
• Vague Requester: Describes needs in broad terms, leaving details unspecified
• Mind Changer: Revises requirements as interaction progresses

The simulator uses GPT-5.6-Sol with temperature 0.7, limited to 20 interaction rounds maximum.

Evaluation Metrics

Clarification Effectiveness:

$$\mathrm{EQR} = \frac{1}{T} \sum_{t=1}^{T} \mathbb{I}(c_t > 0) \tag{1}$$

where $T$ is the total number of clarification questions, $\mathbb{I}(\cdot)$ is the indicator function, and $c_t$ denotes the number of atomic requirements clarified in the $t$-th question-response pair.

$$\text{AvgClarified} = \frac{1}{T} \sum_{t=1}^{T} c_t \tag{2}$$

Requirement Document Quality:

• BERTScore F1: Semantic similarity between generated and ground-truth documents
• ReqAlign: Atomic requirement-level alignment, where each requirement $r_i$ is scored:

$$s_i = \begin{cases} 1, & y_i = \text{Covered}, \\ 0, & y_i = \text{Omitted}, \\ -1, & y_i = \text{Violated}, \end{cases} \tag{3}$$ $$\mathrm{ReqAlign} = \frac{1}{N} \sum_{i=1}^{N} s_i \tag{4}$$

where $N$ is the total number of atomic requirements.

Studied Models

Six LLMs are evaluated: GPT-5.6-Sol, Claude-Opus-5, Gemini-3.1-Pro (proprietary); GLM-5.3, DeepSeek-V4-Pro, Qwen3-235B-A22B (open-weight). All evaluated LLMs use temperature 0.0 for fair comparison.

Empirical Validation / Results

RQ1: Overall Performance

Table 4: Overall performance of studied LLMs on ReqClarifyBench

EN., VR., and MC. denote Expert Nitpicker, Vague Requester, and Mind Changer personas respectively.

Key findings:

• GPT-5.6-Sol achieves best document quality (ReqAlign 0.727) but lowest clarification effectiveness
• Gemini-3.1-Pro performs best in clarification but poorly in document quality (ReqAlign 0.357)
• All models perform best under Expert Nitpicker; ReqAlign drops from 0.609 (EN) to 0.433/0.439 (VR/MC) on average
• GPT-5.6-Sol shows strongest robustness across personas (retains ~81%/78% of ReqAlign), while Qwen3 is most sensitive (retains only ~46%/53%)

RQ2: Clarification Behavior

• Stopping behavior: Interactions end earliest with Expert Nitpicker, latest with Vague Requester. GPT-5.6-Sol shows extreme variation (2-3 rounds for EN, 14-15 for VR), while Gemini-3.1-Pro stays at 2-3 rounds across all personas.
• Clarification dynamics: Expert Nitpicker interactions are short but information-intensive; Mind Changer interactions reveal new information later; Vague Requester interactions are low-yield but sustained.

RQ3: Impact of Clarification

All models benefit from clarification:

• Average BERTScore F1: 0.664 → 0.712 (+7.2% relative)
• Average ReqAlign: 0.212 → 0.494 (+133.1% relative)
• ReqAlign improvements range from 83.8% (GLM-5.3) to 241.3% (Qwen3)

RQ4: Effect of Coding-Agent Harness

Claude Code and Codex (agent harnesses) achieve higher EQR and AvgClarified than base models, but produce lower-quality requirement documents. ReqAlign decreases by 15.2% (Claude) and 24.0% (GPT) under Expert Nitpicker when using agent harnesses.

RQ5: Impact on Code Generation

Table 5: Code-generation performance across three source benchmarks

Requirement documentation alone improves code generation, and interactive clarification provides further gains across all benchmarks.

Theoretical and Practical Implications

1. Proactive clarification is essential: LLMs should identify unclear requirements and ask developers for clarification rather than guessing, as this improves downstream code generation.

2. Requirement documents as intermediate representation: Even without user feedback, generating an explicit requirement document before coding outperforms direct code generation from vague intents.

3. Decoupled architecture recommendation: The paper suggests a staged workflow where base models handle requirement analysis and documentation, while coding-agent harnesses (Claude Code, Codex) handle repository-level implementation — leveraging each system's strengths.

4. Persona-aware design: The substantial performance variation across user personas suggests that LLM-based development systems should adapt their clarification strategies to different user interaction styles.

Conclusion

The paper introduces ReqClarifyBench as the first interactive requirement clarification benchmark for repository-level software development, demonstrating that:

• Clarification effectiveness and document quality are distinct capabilities
• User interaction styles substantially affect model performance
• Interactive clarification consistently improves both requirement quality and code generation
• Coding-agent harnesses trade off document quality for clarification effectiveness

The work highlights that software development agents' autonomy should encompass understanding user intent through proactive interaction, offering a user-centered perspective for designing future autonomous development systems. The code and data are open-sourced for community use and further research.