# Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation

> Local serving stacks silently confound tool-use benchmarks: Ollama rejects some models' tool requests before inference, making capable models score 0% without ever running.

- **Source:** [arXiv](https://arxiv.org/abs/2609.26693)
- **Published:** 2026-09-26
- **Permalink:** https://picx.dev/p/IwI86D
- **Whiteboard:** https://picx.dev/p/IwI86D/image

## Summary

# Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation

## Summary (Overview)

- **Core finding**: Local serving stacks (Ollama, llama.cpp, vLLM, SGLang) can systematically confound tool-use evaluation results, causing the measurement instrument to reflect serving-layer behavior rather than model capability.
- **Silent contamination mechanism**: In Ollama, the `tools=` request is gated per model by a static template flag—some models (Phi-3, Gemma-3) are rejected with HTTP 400 before inference ever runs, yet the harness records these rejections as ordinary model non-calls, producing misleading 0% fidelity scores.
- **Prompt vs. channel separation**: A control arm shows that for accepted models, adding a text tool hint while keeping the native channel recovers most measured fidelity (e.g., Qwen-0.5B: 0% → 35%), while a uniform text protocol can *hurt* models with real native tool-call support (Llama-3.2: 82% → 44%).
- **Cross-stack variability**: The identical request with identical weights yields different outcomes across four serving stacks—Ollama rejects some models, llama.cpp runs them, vLLM refuses by default regardless of model, and SGLang accepts but returns calls as text.
- **Fragile metrics**: Turn-pooled and per-seed fidelity estimates differ by up to ~55 percentage points due to looping episodes dominating thin denominators (8–16 turns over 8 seeds for weaker models).

## Introduction and Theoretical Foundation

### Background and Motivation

A coding agent must emit a *valid tool call*—a parseable invocation of a tool matching the provided schema—before the harness can execute its chosen action. This is a **protocol step**, distinct from the semantic step of choosing the right tool. The authors sought to measure, on small local models, how often this step succeeds—and discovered that the measurement itself was the problem.

Every local agent stack interposes a serving layer between the harness and the model weights. This layer decides how a tool-call request is handled **before the model is ever consulted**. The paper's contribution is not another tool-use benchmark but a **measurement study** demonstrating that serving-layer behavior can systematically confound evaluation of local coding agents.

### Theoretical Foundation

The OpenAI-compatible protocol that local servers expose includes a `tools=` parameter carrying a `tools` array of function specifications (name, description, JSON-Schema parameter object). The model never receives this array directly—the serving layer must:

1. **Render** the specifications into the prompt using the model's chat template
2. **Parse** the generated text back into a structured `tool_call`

Both steps happen **outside the weights**, so whether a server performs them is determined by the model's template and server launch configuration, not by model capability. Ollama exposes this per-model through a capabilities list via `/api/show`; vLLM and SGLang require an explicit `--tool-call-parser` flag.

### Related Work

The paper situates itself among:
- **Function-calling evaluation**: BFCL (Patil et al., 2025) distinguishes native FC mode from prompting mode but doesn't treat the serving interface as an experimental variable
- **Agent benchmarks**: AgentBench, τ-bench, Tool-Sandbox target hosted/fixed endpoints and don't isolate the local serving stack
- **Serving backends as measurement variables**: Pape et al. (2026) quantify how inference backend choice changes LLM outputs; Harness-Bench measures how the harness shifts outcomes
- **Constrained decoding**: Standard fix for unparseable tool calls (Outlines, GBNF grammars) but removes the model's ability to stop

## Methodology

### Harness and Per-Turn Taxonomy

The authors use an off-the-shelf ReAct coding-agent harness (an extension of LOCA-bench) on a fixed aggregation task requiring several tool calls. Every assistant turn is labeled into one of five categories:

- **Valid in-schema call**: parseable, tool exists
- **Hallucinated call**: parseable, tool not found (schema misalignment)
- **Unparseable text**: cannot be parsed
- **No-call prose**: text without any tool call
- **Non-response**: harness wrote its retry-exhaustion error

**Protocol fidelity** is defined as the valid-in-schema-call rate over turns the model actually produced; non-responses are excluded from the denominator and reported separately.

### Experimental Design

**Three serving conditions**:
1. **native**: harness default, sends `tools=` (gated per model on Ollama)
2. **native+hint**: `tools=` still sent, plus plain-text tool list and explicit JSON call format with allowed-name list injected into the prompt
3. **texttools**: `tools=` dropped, same text guidance in prompt, calls parsed from text

**Models**: Qwen2.5-Coder 0.5B/1.5B/3B/7B/14B, Llama-3.2-3B, Phi-3-mini, Gemma-3-4B, Gemma-3-270m (local, via Ollama), plus deepseek-v4-flash (cloud anchor).

**Statistical protocol**: 8 seeds per model, reporting both turn-pooled rate and per-seed mean with seed-level bootstrap 95% CI (10,000 resamples). Proportions on 4–8 seeds are non-normal, so bootstrap is preferred over normal ±SE intervals.

**Cross-stack probes**: llama.cpp (same GGUF weights), vLLM and SGLang (upstream Hugging Face checkpoints for Qwen-0.5B and Phi-3).

## Empirical Validation / Results

### Native FC Mismeasures in Two Ways

Under native FC, a naive per-turn analysis reports **Phi-3 and Gemma-3 at 0%**—but 100% of their turns are rejected requests; the model never ran, so the rate is properly *undefined* (marked "rej"). The second mechanism is **non-response**: Gemma-3-4B emits one valid call per seed then fails to respond.

### The Prompt, Not the Channel, Drives the Accepted-Model Gap

For every model the server accepts, adding the text hint while keeping the native channel raises per-seed fidelity substantially:

| model | native seed (pool) | native+hint seed (pool) | text-tools seed (pool) |
|---|---|---|---|
| Qwen-0.5B | 0 (0) | 35 (29) | 34 (85) |
| Qwen-1.5B | 38 (24) | 58 (91) | 74 (95) |
| Qwen-3B | 23 (79) | 82 (96) | 84 (90) |
| Qwen-7B | 59 (79) | 89 (80) | 91 (89) |
| Qwen-14B | 60 (74) | 80 (71) | 92 (88) |
| Llama-3.2-3B | 60 (62) | 82 (89) | 44 (47) |
| Phi-3-mini | rej | — | 38 (30) |
| Gemma-3-4B | rej | — | 100 (100) |
| Gemma-3-270m | rej | — | 38 (38) |
| deepseek (cloud) | 100 (100) | — | 100 (100) |

*Table 1: Protocol fidelity as per-seed mean with turn-pooled rate in parentheses, in percent. "rej" = native request rejected on all 8 seeds, so native fidelity is undefined, not 0%.*

**Llama-3.2 is the informative exception**: native+hint reaches 82% but uniform text-tools drops to 44% because Llama-3.2 has real native `tool_calls` support that the text protocol discards. The best-performing interface is **model-dependent**.

### The Rejection Is Stack Policy, Not the Model

Cross-stack handling of the same `tools=` request:

| model | Ollama | llama.cpp | vLLM | SGLang |
|---|---|---|---|---|
| Qwen-0.5B | 200 txt | 200 txt | 400† | 200 txt |
| Llama-3.2 | 200 nat | 200 nat | — | — |
| Phi-3 | 400 rej | 200 txt | 400† | 200 txt |
| Gemma-3-270m | 400 rej | 200 txt | — | — |

*Table 2: Cross-stack handling of the same `tools=` request. txt = call returned as text, nat = native tool_call, rej = request rejected. †vLLM refuses by default, independent of model, until launched with `--enable-auto-tool-choice` and `--tool-call-parser`.*

### The Fidelity Number Is Not Robust

Turn-pooled and per-seed rates diverge sharply when a model produces one long looping episode. Qwen-0.5B under text-tools is 85% pooled but 34% per-seed: seven of eight episodes fail in one or two turns while a single 41-turn episode of repeated valid calls dominates the pool. Bootstrap 95% CIs are correspondingly wide: Qwen-0.5B text-tools is 34% [9, 59], Phi-3 38% [12, 75].

### Constrained Decoding: Removes Protocol Failures at a Cost

Constraining decoding to a JSON schema with a name field as an enum of available tools makes **all nine local models** emit valid in-schema calls on 8/8 trials. However, forced to emit a tool call on every turn, weaker models never terminate—Qwen-0.5B reached 869 assistant turns in one seed before cutoff. The failure is *moved*, not removed.

### Replication

- **Second task** (dependency-chain, 4 seeds): Native gating replicates exactly (Phi-3/Gemma rejected, Llama native, Qwen text); magnitudes are task-dependent (Llama-3.2 falls to 12% per-seed vs. 60% on aggregation)
- **HumanEval** (single-turn): Native gating replicates; text-tools valid-call rate rises with Qwen scale (4/6, 5/6, 6/6); Gemma-3-270m 0/6 (all unparseable prose)

### The Harness Conflates Serving Failures with Model Non-Calls

The contamination is silent because the inference wrapper collapses the HTTP 400 rejection into an assistant message with a generic error string (e.g., `"Failed to get response after multiple retries."`), and the **error type is not persisted** into the saved trajectory. Downstream per-turn analysis reads an ordinary non-call assistant turn and attributes it to the model unless it string-matches the harness's specific error text.

## Theoretical and Practical Implications

### The Structural Nature of the Confound

The confound is not a one-off quirk but a **structural consequence** of how local agent stacks are assembled. A modern agent request passes through:
1. A harness
2. A tool-protocol adapter
3. A serving engine (Ollama, vLLM, llama.cpp)
4. A per-model chat template

Each layer can accept, rewrite, or reject a tool call independently of the model's ability. The trend is toward *more* layers, not fewer—the Model Context Protocol (MCP) and similar standards plausibly add another translation step.

### Two Distinct Evaluation Targets

The paper distinguishes two questions that should not be conflated:
- **Standardized model comparison**: The serving interface should be held fixed as part of the controlled measurement instrument
- **Deployable model–server system evaluation**: The full model–server configuration should be reported as part of the system being evaluated

### The Remedy Checklist

> **A protocol for measuring local-agent tool use.**
> **Measure:**
> 1. For standardized model comparisons, hold the serving interface fixed across models and pin and report the stack and its version.
> 2. Log a transport- or serving-level failure (rejected request, timeout, empty response) as a distinct outcome, never as a model non-call.
> 3. Report per-seed rates with intervals, not a single turn-pooled number.
> **Diagnose a 0% tool-call rate, in order:**
> 1. Did the serving layer refuse or empty the request (HTTP 4xx, retry exhaustion)? If so, the model never ran.
> 2. Is a tool-call parser configured for this stack and model? An accepted request can still surface the call as text.
> 3. Only after (1)–(2) are ruled out, attribute the failure to the model.

## Conclusion

Native function calling on a local server gates the tool-call request per model and records a refused or empty request as an ordinary non-call turn, so a capable model can be measured at 0% without ever running. The paper's key takeaways:

1. **The default native path under-measures fidelity** for accepted models due to lack of prompt guidance, not model inability
2. **Forcing a uniform text protocol can hurt** models with real native support
3. **The rejection is stack policy**, confirmed across four serving stacks
4. **Pooled vs. per-seed estimates diverge** by up to ~55 points due to looping episodes on thin denominators

The lesson is a measurement checklist for small or local agents: hold the serving interface fixed, separate refused/empty requests from model non-calls, and report per-seed rates with intervals. Otherwise the instrument measures the serving stack, not the model.

### Limitations

- Per-seed fidelity magnitudes are task-dependent and not generalized across tasks or stacks
- vLLM and SGLang probes cover only Qwen-0.5B and Phi-3
- Constrained-decoding baseline is single-turn only (agentic version didn't terminate on weak models)
- The hint includes an allowed-name list; a weaker hint might recover less
- An earlier 3-seed pilot showed a clean curve that 8 seeds dissolved—small model rates are high-variance

---

_Markdown view of https://picx.dev/p/IwI86D, served by PicX — AI-generated visual whiteboard summaries of research papers._