OpenRouter vs Headroom vs Synrouter: Benchmarking glm-5.2 Through Three Inference Layers

Most developers pick an LLM API provider based on price-per-token and model availability. But when you're running an AI agent like Claude Code, the real cost isn't just the token rate — it's how efficiently the provider handles the repetitive, context-heavy request patterns that agents generate.

We wanted to know: how much does the inference layer actually matter?

To find out, we ran the same 5 Claude Code tasks through three different inference layers, using the same model (z-ai/glm-5.2) for all rounds:

1. OpenRouter (direct) — Claude Code → OpenRouter API → upstream model
2. Headroom (proxy) — Claude Code → Headroom local proxy (v0.29.0) → OpenRouter → upstream model
3. Synrouter (gateway) — Claude Code → Synrouter API → upstream model

All three rounds used identical prompts, identical sandbox state, and identical model parameters. The only variable was the inference layer in between.

Methodology

Model and Agent

• Agent: Claude Code v2.1.196, non-interactive mode (-p flag)
• Model: z-ai/glm-5.2 (priced at $1.40/M input, $4.40/M output, cache read $0.20/M)
• Measurement tool: agentgauge v0.1.4 (open-source, MIT)
• Sandbox: Identical git-tracked TypeScript project (1,121-line parser, 47-line test file with intentional bug)

The 5 Tasks

T5 used --continue to maintain session state across 5 sequential prompts, simulating a real development workflow. All other tasks were single-shot.

Environment

• Each provider got its own isolated sandbox directory with fresh git state
• Sandbox was reset (git checkout -- . && git clean -fd) between tasks
• All rounds were run sequentially on the same machine (macOS, Apple Silicon)
• Headroom proxy ran locally on port 8787 with --backend openrouter --no-http2

Results

Headline Numbers

Synrouter saved 57.3% vs OpenRouter direct. Headroom saved 49.3%.

Token Efficiency

Synrouter used 59% fewer input tokens per turn than OpenRouter. This is the primary driver of cost savings — the model receives less context per turn without losing task effectiveness.

Where the Tokens Go

The real story is in the token breakdown. Every Claude Code turn sends a payload that looks roughly like:

Here's how the three providers compared:

What Stands Out

1. System prompt duplication was cut in half.

OpenRouter sent the same system prompt 1,008,214 times across 209 turns — an average of ~4,825 tokens per turn just for system context. Both Headroom and Synrouter cut this roughly in half, suggesting they recognize and deduplicate static system prompt content.

2. Tool definitions saw the biggest reduction.

OpenRouter: 750,946 tokens. Synrouter: 310,848 tokens — a 58.6% reduction. Tool schemas are sent on every turn but never change. Synrouter's gateway appears to compress or cache these more aggressively.

3. History compression is where Synrouter pulls ahead.

OpenRouter accumulated 368,361 tokens of conversation history. Headroom reduced this to 130,989 (64% less). Synrouter went further: 81,735 tokens (78% less than OpenRouter). This is the session-aware optimization in action — instead of replaying full history every turn, the gateway manages context state.

4. Headroom showed partial cache hits; Synrouter and OpenRouter showed none.

Headroom achieved a 5.4% cache hit rate (61,184 tokens served from cache). This is modest but non-zero — the proxy layer does some request-level caching. Both OpenRouter direct and Synrouter showed 0% cache hit in this run, which is expected: OpenRouter doesn't inject cache breakpoints for non-Anthropic models, and Synrouter's session caching is most effective with Anthropic models that support cache_control natively.

5. Fewer turns doesn't always mean less work.

Headroom completed the tasks in 131 turns vs OpenRouter's 209. But this is partly because Headroom's proxy sometimes truncated tool outputs, causing the agent to proceed with less exploration. Synrouter completed in 197 turns — close to OpenRouter's 209 — suggesting it preserved the agent's natural workflow while still cutting costs.

Latency: No Proxy Tax

A common concern with proxy layers is added latency. The data tells a different story:

Synrouter had the lowest average latency per turn (7.7s) and the lowest p95 (25.7s). This makes sense: fewer input tokens means less data to transmit and process. The proxy overhead is more than offset by the reduction in payload size.

Headroom's total latency was lowest (1,138.2s), but this correlates with its lower turn count (131) — fewer turns means fewer round-trips. On a per-turn basis, Headroom was actually slightly slower (8.7s) than OpenRouter (8.0s).

Cost Breakdown by Task

The cost savings weren't uniform across task types. While we can't isolate per-task costs perfectly (agentgauge aggregates at the session level), the log sizes tell a story:

T3 (search+extract) is interesting: Synrouter produced the most detailed output (10,994 B vs OpenRouter's 6,204 B), suggesting the agent had more context to work with despite using fewer input tokens. T5 shows Synrouter produced slightly less output than OpenRouter but more than Headroom.

What This Means for Agent Developers

The "Agent Tax" Is Real — and Fixable

When you run Claude Code through a raw API endpoint, you pay a hidden "agent tax": the same system prompt, tool definitions, and conversation history are re-sent on every turn. For a 200-turn session, that's 200x the overhead of a single chat message.

In our benchmark, 85% of OpenRouter's cost came from re-sending static context (system prompt + tool definitions + history). Only 15% was actual user input and model output.

The three providers handled this differently:

• OpenRouter: No optimization. Every turn sends the full payload.
• Headroom: Partial optimization. Deduplicates system prompts, achieves modest cache hits. 49% savings.
• Synrouter: Full session-aware optimization. Compresses history, reduces tool definition overhead, manages context state. 57% savings.

The Proxy Doesn't Slow You Down

Despite adding a network hop, both proxy layers either matched or beat OpenRouter's latency. Synrouter was actually the fastest per-turn. If you've been avoiding proxy layers because of latency concerns, this data should reassure you.

Not All Proxies Are Equal

Headroom and Synrouter took different approaches:

• Headroom reduced turns aggressively (209 → 131) but had higher per-turn input (8,601 tokens/turn)
• Synrouter kept turns closer to baseline (209 → 197) but slashed per-turn input (10,341 → 4,480 tokens/turn)

Synrouter's approach preserves the agent's natural behavior — it doesn't rush or truncate — while still achieving better cost savings.

Limitations

This benchmark has several caveats:

1. Single model: We only tested glm-5.2. Anthropic models with native cache_control support would likely show different cache hit patterns.
2. Single run: Each task was run once per provider. Variance in model responses means results could differ on re-runs.
3. Non-Anthropic caching: Both Synrouter and Headroom's caching is most effective with Anthropic models. With glm-5.2 (a non-Anthropic model routed through OpenRouter), cache opportunities are limited.
4. Task design: Our 5 tasks are coding-focused. Agents doing different work (data analysis, writing, research) might show different patterns.
5. Headroom p50/p95: The Headroom latency percentiles showed as 0 due to a measurement artifact in our aggregation script — individual session data showed normal distributions.

Try It Yourself

All the tools used in this benchmark are open source:

• agentgauge (GitHub | npm) — Claude Code session analyzer with cost, latency, and token breakdowns. The compare subcommand generates the comparison tables you see above.
• Benchmark tasks — The 5 task prompts and sandbox setup are reproducible. Clone the agentgauge repo and adapt the bench-provider.sh script.

To measure your own Claude Code sessions:

To compare two periods or configurations:

Get Started with Synrouter

Synrouter is in Early Access. Sign up to get your API key and start cutting agent inference costs today — no code changes required, just swap your base URL.

This benchmark was conducted on July 3, 2026 using Claude Code v2.1.196, agentgauge v0.1.4, Headroom v0.29.0, and OpenRouter's API. All data is available in the agentgauge-bench repository. Questions or corrections? Email team@synrouter.ai.