Doubling the Prompt Costs 2.2x the Wait, Not 2x

Two measurements on this site already bound the context-window decision from either side. A prompt is read in full up to num_ctx, and cut to half the window the moment it exceeds it. Every declared token of that window costs 33.0 KiB of KV cache, whether a prompt fills it or not.

Neither says what a long prompt costs in seconds. That is the number a person actually waits through, and it is the one I had not measured.

The measurement

Seven prompt lengths, from 329 to 19,167 tokens. The window pinned at 32,768 for every cell, so a growing prompt is never confounded with a growing window. One output token per call, because generation is bandwidth-bound and has nothing to do with prompt length. Three repetitions of the whole sweep.

Fitting time against tokens on the logs gives an exponent of 1.133.

An exponent of 1 would be linear — twice the prompt, twice the wait. At 1.133, doubling the prompt multiplies the wait by 2.2x. Fifty-eight times the tokens costs about a hundred times the time.

The rate column is the same fact read the other way: throughput falls from 523 tokens per second to 303 as the prompt grows. The machine is not getting slower. The work per token is getting larger.

The control that could have killed it

There is a real problem with the design of a single pass, and it has to be answered rather than glossed.

Within each repetition the lengths run shortest-first. So the longest prompt always runs last, and always on the warmest machine. This laptop is fanless, and this site has already measured it losing 8.2% between burst and plateau under sustained load, with macOS reporting thermal pressure in 0 of 424 samples. Heating alone would bend a straight line upward and hand me an exponent above 1 for free.

So the whole sweep repeats three times, and the exponent is fitted separately for each repetition — the first on a cool machine, the last on a thoroughly warm one.

Spread of 0.037, against a tolerance of 0.10.

The analyser refits this on every run and exits non-zero if the spread exceeds tolerance, so a future run on a hotter machine produces a refusal rather than a bigger number.

Cold, every time

One more way this measurement could have been wrong, and it is the trap that already caught an earlier post on this site: ollama reuses the KV cache for a matching prefix. Send the same prompt twice and the second call reports a prefill rate no hardware could produce.

So every prompt here leads with a distinct nonce — as its first tokens, not its last — and no two prompts in the run share a prefix. Every prefill measured is cold.

The harness also records whether any cell landed on num_ctx/2 + 2, the signature of the silent truncation that happens when a prompt exceeds its window. A truncated prompt would look wonderfully fast. None were.

Does a bigger window cost time too?

The obvious follow-up, given that a big window demonstrably costs memory: hold the prompt fixed at ~1,825 tokens and vary num_ctx across five values.

The answer is that I do not know, and the analyser says so on my behalf.

range across windows: 1003 ms
worst spread within a single window: 1227 ms
INCONCLUSIVE: the noise within one window is as large as the difference between windows.
This design cannot detect a prefill cost of declaring a bigger window. Do not claim one.

The medians across 4,096 to 65,536 tokens of window are 3,388, 4,391, 4,357, 4,078 and 3,891 ms — not monotonic, and each cell's own three runs vary by up to 1,227 ms. There is no signal here to report.

That is a useful non-result rather than a wasted arm. Taken with the memory measurement, the current position is: a large declared window costs memory, and no prefill time that this design can see. If you have the RAM, headroom in num_ctx appears to be cheap in time. The cost is the cache.

What it adds up to

Three measurements now bound the same decision, and they do not point the same way.

• Do not undershoot the window. Exceed num_ctx and you lose half the prompt, from the head, silently.
• Do not over-declare it either — every declared token is 33.0 KiB of cache, held whether used or not, and at the full 131,072-token window this machine spills 1.243 GB out of its GPU budget.
• And keep the prompt itself short, because prompt length is the one that compounds. Memory is linear in the window. Time is not linear in the prompt.

The practical version: pick the smallest window that comfortably clears your largest prompt, assert on prompt_eval_count, and treat the prompt itself — not the window — as the thing to economise. A retrieval step that trims 19,000 tokens to 5,000 cuts the token count by 74% and the wait by about 78% — the superlinearity works in your favour on the way down.

Limits

One model, one machine, one serving stack. Three repetitions establish that the curvature is not thermal; they do not establish that a 1.2B model on a fanless M1 predicts a 70B model on anything.

No mechanism is proven. An exponent of 1.13 is consistent with attention's quadratic term becoming visible against a linear baseline — at these lengths the quadratic part is still small — but this is black-box timing and cannot name the kernel responsible.

The harness, the analyser, the per-repetition control and all 36 cells are in the repository. Eight minutes and no credentials will fit the exponent for your own hardware.