The Second User Buys No Batching At All. The Fourth Buys a Third of What Theory Promises.

Every serving stack sells concurrency as free efficiency. Batch more requests through the same weights and you amortise the weight reads that dominate generation — that is the entire economic argument for a shared inference endpoint over a per-user one, and it is the reason a GPU hour is worth renting.

The argument rests on a premise this site has already measured on this machine. Generation here is memory-bandwidth-bound, and halving the bytes per weight buys 58% more tokens per second. Both say the bottleneck is moving weights, not doing arithmetic. If that is true, batching should be close to free: one pass over the weights, N sequences served.

An earlier post named parallel slots as a suspect for a factor of two it was chasing, pinned OLLAMA_NUM_PARALLEL:1, and showed the server log to rule them out. That established what concurrency was not doing. Nothing here had measured what it does.

So I measured it. The headline is roughly what a sceptic would guess. The decomposition underneath it is not.

The design

Three arms. Every one sends the same four requests; they differ only in how many are in flight at once.

Holding the request count constant is what makes serial-1 more than a baseline: it is the queueing control, the same four requests taken a user at a time. The obvious alternative — sweep concurrency and fire that many requests — would have each arm doing a different amount of work, and on a fanless M1 the heavier arms heat the machine more. This one loses 8.2% between burst and plateau with macOS reporting thermal pressure in zero of 424 samples, which is the same order as the effect being looked for.

Two details decide whether the numbers mean anything.

The four prompts in a batch differ. Four concurrent requests carrying a byte-identical prompt can share a prefix cache, and the parallel arms would book that saving as a concurrency win. Each request carries its own nonce, so the four are four different users asking a same-length question. Prompt length is held equal; prompt bytes are not.

The arm order rotates by rep. An arm that always ran last would be charged the thermal decay. The analyser re-checks this on every run by comparing each arm's median across the positions it occupied. Across the ten reps here the widest spread is 3.9%, well inside the 10% at which the analyser stops reporting a headline and starts reporting a refusal.

What it returns

Ten reps, two runs, pooled. Medians across reps; the speedups are within-rep ratios against serial-1, which is the statistic the rotated design supports.

Four-way concurrency buys the operator 39% more throughput. It costs each user 65% of their generation rate and nearly triples the wait before the first token appears.

Both of those are true at once, and which one you quote is a choice. A dashboard reporting tokens per second across the server would show this change as a clear win. Every individual user would experience it as the machine getting substantially worse. Neither party is wrong.

The part I did not expect

The efficiency column falls from 55% to 35%, which reads like a smooth story about diminishing returns. It is not one story. Splitting the aggregate into the stage that produces the tokens shows the two arms winning for entirely different reasons.

Multiply each arm's per-user generation rate by the number of streams and you get what the generation stage delivered in total:

At four-way, the generation-side gain is 1.39x and the aggregate gain is 1.38x. The batching is the whole story — every token of the server's improvement comes from the generation stage doing more work per pass.

At two-way it inverts. The generation stage delivers 0.94x — two streams together produce 6% fewer tokens per second than one stream alone. Yet the server still reports a 9% aggregate gain.

That gain is real, and it is not batching. It is pipelining: one request's prefill overlapping another's generation, two stages of a pipeline kept busy at once. Nothing is being amortised. The weights are being read just as often, slightly less efficiently, and the win comes entirely from the machine having something to do during a phase it would otherwise have spent waiting.

So the practical shape is not a smooth curve of diminishing returns. Adding a second user buys you no batching at all — only overlap. Batching does not begin paying until the fourth, and when it does it returns about a third of what the arithmetic promises.

The trap in the method

num_predict looks like it sets the output length. It does not — it is a ceiling, and EOS almost always wins first. At num_predict: 200 this summarisation task ends around 45 tokens, so a four-request batch generates about 180 tokens, not 800. Ollama does not honour llama.cpp's ignore_eos; passing it returns a three-token reply with done_reason: "stop". Output length cannot be pinned from the client.

The first version of this harness asserted in its own header that every arm generated exactly num_predict tokens. The first real run disproved it. Fourteen of fifteen cells generated exactly 180 tokens and one generated 233 — and that cell was simultaneously 13% slower by the wall clock and 14% faster by throughput.

A wall-over-wall speedup would have recorded that as a concurrency effect. It is an artefact of one request that happened not to stop. So batch_wall_ms is still recorded but is never compared; every number above is tokens actually produced over the wall clock that produced them, which is correct whether or not two cells generated the same count. The analyser names any cell deviating more than 5% from its run's modal output, and it names that one.

What it means past this laptop

The obvious caution first: this is a 1.2B model at Q4_K_M on a fanless 8GB M1, at concurrency up to four. A served 70B model on an accelerator has different arithmetic, more memory bandwidth, and a batch scheduler doing considerably more than this one.

What travels is the shape of the question. Two numbers move in opposite directions under concurrency, and an inference platform gets to choose which one it publishes. "Throughput up 39%" and "every user 65% slower" describe the same configuration change. If you are sizing a shared endpoint, the aggregate is the number that determines your bill and the per-user rate is the number that determines whether anyone wants to use it — and the gap between them widens with every slot you add.

The narrower lesson is that the second user is the worst deal on the menu. It is the point where you pay the full latency cost of sharing and receive none of the benefit that sharing is supposed to provide.