One vLLM flag, −68.6% cost per token: what a counterbalanced benchmark looks like

On a rented A100, one vLLM flag took the cost of serving a small model from $0.746 to $0.234 per million output tokens, a 68.6% drop. Before anything else, the honest part: the baseline was deliberately bad. max_num_seqs=1 makes vLLM process one sequence at a time, and nobody should run production that way.

So this post isn't "use this flag". It's about the method: how to measure a before/after change so the answer holds up, with every raw number from the run. The protocol is the one Throttle's golden command implements, and the artifacts are in the repo.

The setup

Chunked prefill was on in both variants, so it can't take credit for the difference. The client always offered 8 concurrent requests, enough to exercise the candidate's limit, so the only thing that changed was how many sequences the server was allowed to run together.

Why not just run before, then after?

Because the machine drifts. In a separate test on my laptop, four back-to-back checks of an unchanged server swung +27% and then −11%. If the GPU host gets slower over the session (a noisy neighbour, thermal state, anything), "baseline first, candidate second" hands the whole drift to the candidate.

The protocol: six positions, interleaved

The run has six positions on the same physical GPU, in this order:

B1  C1  B2  C2  B3  C3

Before each position, Throttle stops and tells the operator which config it needs. The operator restarts vLLM with that flag in another terminal, verifies the effective setting, and confirms before any traffic is sent. Throttle never touches the server itself. Each position runs three warm-up requests, then three measured blocks of 67 requests.

The analysis uses two phase contrasts, each bracketing one variant with the other:

• Phase one: C1 against the average of B1 and B2, which run on either side of it.
• Phase two: the average of C2 and C3 against B3, which runs between them.

If the machine drifts linearly over the session, each bracket cancels it: the bracketing pair averages out to the same moment in time as the thing it's compared with. One phase is B‑C‑B, the other C‑B‑C, so a drift that favours one variant in one phase favours the other in the next.

The raw numbers

Every position produced exactly 25,728 completion tokens (201 valid responses, zero errors), so both variants did the same amount of work.

The three baseline positions agree within 0.4%, even though they were spread across the whole session. That's what you want to see: little drift, and the protocol would have caught it if there were.

The dollar figures follow directly from throughput:

$/M = 1.39 ÷ (3,600 × 516.55) × 1,000,000 = $0.7475

The result

The two phase contrasts, computed on throughput:

phase one:  C1 vs mean(B1, B2)
            1,632.41 vs 517.22   → +215.6%
phase two:  mean(C2, C3) vs B3
            1,659.78 vs 518.55   → +220.1%

The order-balanced estimate is their mean: +217.8% throughput, 95% interval +189.5% to +246.2%. With only two contrasts, the interval uses t with 1 degree of freedom (12.71), which is why it's that wide even though the two phases agree within 5 points. The whole interval is far from zero, and Throttle marked the result decision_eligible: true.

In cost terms, the mean of the three baseline positions is $0.746 per million output tokens and the mean of the three candidates is $0.234: −68.6%.

One detail if you check the files yourself: each position also records a whole-run cost summary that charges the GPU for the full elapsed time, including warm-up and gaps between blocks. On that basis the numbers are $0.778 → $0.261, or −66.5%. Both are real. They answer slightly different questions, and I quote the measured-block figure because it matches how the throughput was measured.

What this does and doesn't show

It shows that on this GPU, engine version, model and workload, letting vLLM batch up to 8 sequences instead of 1 roughly tripled output throughput, and so cut the cost per token by about two-thirds, with drift controlled for.

It does not show:

• A saving you should expect. Your baseline almost certainly isn't max_num_seqs=1. Going from a sane default to a tuned value moves much less.
• Anything about bigger models. A 0.5B model leaves a lot of an A100 unused. Larger models change where the bottleneck is.
• Your traffic. This is closed-loop load at concurrency 8 with 128-token completions. Open-loop traffic, long prompts or long outputs can behave differently.
• Quality. Same model, same revision, temperature 0, identical token counts: nothing here trades quality for speed. With a quantization or a smaller model, that would be a separate question you'd have to test.

What it cost to find out

The six measurement windows used $0.08 of GPU time. Including startup, model download, restarts, validation and a first GPU that never came up, the whole session was at most about $0.74 (an elapsed-time estimate, not an invoice). Measuring a config change properly is cheap. Running the wrong config for a month isn't.

Run it on your own server

Preview the full six-position plan first. This sends no traffic and needs no key:

throttle golden --dry-run \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --url https://inference.example/v1 \
  --api-key-env VLLM_API_KEY \
  --baseline-config max_num_seqs=1 \
  --candidate-config max_num_seqs=8 \
  --concurrency 8 \
  --cost-model dedicated-hourly --gpus 1 --total-hourly-price 1.39 \
  --cache-policy disabled

The plan prints the sequence, the request and token ceilings, and a spend cap. It also lists what a decision-grade run still needs from you, such as a pinned model revision and runtime-verified engine flags. golden currently supports max_num_seqs as the treatment. For quick day-to-day re-checks after any config change, use throttle check, which calibrates against run-to-run noise instead of interleaving (how that works).

pipx install throttle-pro

The code is MIT-licensed: github.com/KushagraKanaujia/throttle. If you run a counterbalanced comparison on your own hardware, I'd love to see it: throttle check --share prints a sanitized summary for a results issue.