Summary (Overview)

• This paper introduces a response-centric framework for measuring "no-thinking" behavior in LLMs, decomposing outputs into pre-answer text $T$ and final answer $A$, evaluated at three levels: Empty-Thinking Rate (ETR), Question–Pre-answer Relevance (QRel.), and Explicit Inference Rate (EIR).
• The authors identify "Thinking Inertia": visible explicit inference persists even under strict no-thinking controls and becomes progressively more prevalent as the answer space opens (Boolean < multiple-choice < open-ended).
• Across six LLMs (Qwen3 family, DeepSeek-V4-Flash, GPT-5.4, Gemini-3-Flash) and six benchmarks, explicit no-think controls are unreliable—even DeepSeek-V4-Flash retains visible explicit inference in 99.9% of open-ended responses under its disabled-thinking setting.
• A no-thinking–performance trade-off emerges on open-ended tasks: strict answer-only prompting raises mean ETR from near zero to ~40% while reducing mean accuracy by ~15 percentage points.
• Answer space matters: rewriting the same questions across answer spaces shows candidate-visible formats make answer-only responses substantially easier to produce than open-generation formats.

---

Introduction and Theoretical Foundation

Background and Motivation

Large Language Models (LLMs) benefit substantially from extended reasoning, from early chain-of-thought prompting [42, 19] to test-time scaling [31, 3]. However, extended reasoning raises inference costs in latency and token consumption, motivating work on shortening or compressing reasoning traces [1, 18, 11, 45]. Beyond efficiency, humans often respond from intuition and deliberate only when needed, motivating adaptive reasoning capabilities.

The Proxy Problem

Current practice identifies "no-thinking" through three surface proxies, each with failure cases:

1. Prompt-side control: Running the model in a no-think setting does not guarantee the absence of question-conditioned content.
2. Visible-format control: The absence of special thinking tokens does not prevent reasoning from appearing as ordinary language.
3. Length-based control: Short outputs can still encode compact, task-specific reasoning steps.

"These proxies conflate instruction control with the suppression of visible reasoning. For example, a generic lead-in such as 'Let me think about this. The answer should be <ans>' carries no task-specific content, yet would be counted as reasoning by surface proxies."

Theoretical Foundation

The paper formalizes no-thinking as an observable response behavior. The response is decomposed as:

$$\pi \big(S(Q)\big) = (T, A)\tag{1}$$

where $A$ is the last parseable final answer, $T$ is all readable visible text preceding it, and $S(\cdot)$ is an intervention applied to the question $Q$. Each task sample is written as $Q = (q, c, \mathcal{A})$, where $q$ is the question, $c$ is optional context, and $\mathcal{A}$ is the admissible answer space.

---

Methodology

Three-Level Response Evaluation

Level 1: Empty-Thinking Rate (ETR) — Strict answer-only compliance:

$$e_{i} = \mathbf{1}\{|T_{i}| = 0\}, \qquad \mathrm{ETR} = 100\% \times \frac{1}{N} \sum_{i=1}^{N} e_{i}\tag{2}$$

Level 2: Question–Pre-answer Relevance (QRel.) — Continuous semantic similarity using instruction-aware Qwen3-Embedding-4B:

$$\operatorname{QRel.}(Q_{i}, T_{i}) = \begin{cases} 0, & |T_{i}| = 0, \\ \cos\bigl(\phi(Q_{i}), \phi(T_{i})\bigr), & |T_{i}| > 0. \end{cases}\tag{3}$$

Level 3: Explicit Inference Rate (EIR) — Blinded LLM judge (GPT-5.5) classifies non-empty $T$ into five mutually exclusive labels (Table 1): Generic/off-topic, Paraphrase-only, Relevant non-inferential, Explicit reasoning, and Unclear:

$$\mathrm{EIR} = 100\% \times \frac{1}{N} \sum_{i=1}^{N} \operatorname{Judge}(Q_{i}, T_{i})\tag{4}$$

Intervention Spectrum

Six modes organized along a no-think intensity axis:

Experimental Setup

• Models: Qwen3-4B, Qwen3-32B, Qwen3-235B-A22B, DeepSeek-V4-Flash (284B), GPT-5.4, Gemini-3-Flash
• Benchmarks: BoolQ, StrategyQA (Boolean); MMLU, MMLU-Pro (multiple-choice); GSM8K, MATH (open-ended)
• Human validation: 600 outputs sampled, 5 annotators, blinded to model identity and intervention

---

Empirical Validation / Results

Q1: Native Think-on vs. Think-off

• Boolean tasks: Most models reach ~80–100% ETR under THINK-OFF, with sharply reduced QRel. and EIR.
• MCQ tasks: Intermediate but heterogeneous—ETR ranges from near zero to 95% across models.
• Open-ended tasks: ETR collapses to near zero everywhere; QRel. remains close to THINK-ON levels and EIR remains high.

"Visible explicit inference is organized more by the task's answer space than by the switch itself."

Q2: No-Think Variants Performance

• Prefix forcing (M6) is not a stronger STRICT: Mean ETR drops from M5 to M6 on MCQ and Open-ended tasks, while QRel. and EIR rebound.
• Open-ended trade-off: Moving from M2 to M5 raises mean ETR from near zero to ~40% and reduces mean accuracy by ~15 percentage points. EIR decreases from 99.69% to 62.00% across the complete corpus.
• Bool and MCQ show little accuracy trade-off: EIR decreases from 5.15% to 0.12% (Bool) and 45.57% to 25.39% (MCQ), while accuracy remains largely flat.

Q3: Domain and Answer-Space Variation

Domain analysis (Table 4) — Under fixed four-choice MCQ interface:

STEM combines the highest accuracy with the lowest ETR (0.3–36.4%), arguing against a simple positive relationship between task success and answer-only compliance.

Answer-space rewrites (Figure 6) — Same MMLU questions rewritten for Qwen3-32B under M2:

• Boolean rewrite: ETR rises from 10.5% → 62.5%, EIR drops from 86.95% → 36.90%, accuracy nearly unchanged (77.8% → 78.9%)
• Open-ended rewrite: ETR drops to 0%, EIR rises to 90.14%, accuracy drops to 52.6%

Human Validation

• ETR boundary audit: 94.2% agreement (565/600 outputs)
• QRel. correlation: Mean Spearman ρ = 0.829 with human relevance judgments
• EIR agreement: 98.8% mean binary agreement with GPT-5.5 judge; 96.7% with Claude Opus 4.8
• Human annotations reproduce both the answer-space staircase and persistent visible inference on open-ended tasks under M5

---

Theoretical and Practical Implications

Theoretical Contributions

1. No-thinking as a measurable behavior: The paper establishes no-thinking as a distinct, testable capability requiring systematic evaluation alongside reasoning ability itself.
2. Thinking inertia: Visible inference persistence is organized by answer space structure, not by model settings or instructions alone.
3. Answer-space dependence: The structural support of the question (candidate visibility) governs compressibility, not question difficulty or model capability alone.

Practical Implications

• Task-aware routing: Models should answer directly when the interface provides sufficient support (e.g., Boolean verification) and preserve explicit computation when suppression would sacrifice accuracy (e.g., open-ended generation).
• Caution on cost assumptions: Visible compliance with no-thinking instructions should not be equated with lower inference cost when hidden reasoning may remain active.
• High-stakes settings: Concise outputs may reduce cost and disclosure, but explanations remain important for verification and error detection.

---

Conclusion

The paper establishes that "no-thinking" in LLMs should be evaluated as a response-level property rather than inferred from surface proxies. The key findings are:

1. Explicit no-think controls are not reliable: Even frontier models retain visible explicit inference under disabled-thinking settings.
2. Thinking inertia: Visible inference persists and scales with answer-space openness.
3. No-thinking–performance trade-off: Stronger suppression improves compliance but can reduce accuracy on open-ended tasks.
4. Answer space matters: Candidate-visible formats make answer-only responses easier to produce.

Future Directions

• Calibrated stopping: Train models to jointly optimize accuracy, answer-only compliance, visible relevance/inference, latency, and actual reasoning-token cost.
• Cross-domain extension: Test generality across languages, code, tool-use, multi-turn, and stochastic settings.
• Internal-state analysis: Examine whether computation disappears, moves to a hidden channel, or is only omitted from view.
• Auditable decisions: Make models' decisions about when and how to expose intermediate computation controllable and auditable, rather than suppressing computation universally.

"Building LLMs that can stop thinking on demand requires more than hiding reasoning traces or enforcing shorter outputs: it requires evaluating when direct answering is actually supported by the task."