A Context Token Costs 33 Kilobytes Whether You Use It or Not

Last week's measurement ended with a piece of advice: this server reads only num_ctx/2 + 2 prompt tokens, so set the window to twice the largest prompt you plan to send. Advice is cheap. This is what it costs.

Every token of context window is KV cache, KV cache is memory, and this machine has 8 GB of it shared between CPU and GPU. So I swept nine windows, from 2,048 tokens to the model's full trained 131,072, and weighed each one.

The number

The weights do not change with num_ctx, so the difference between two adjacent rows is the cache and nothing else. Five of the eight intervals give exactly 33.0 KiB per token. The two that do not — 42.0 then 28.5 — are adjacent and cancel, which is what a single slightly-low reading at one window looks like.

Why 33 is the right answer

A measurement that agrees only with itself is a number, not a finding. This one has an independent prediction to check against.

Llama 3.2 1B has 16 layers and 8 key-value heads of 64 dimensions. The cache stores a key and a value per layer per token, at two bytes each in f16:

16 layers × 8 heads × 64 dim × 2 (K and V) × 2 bytes = 32,768 bytes = 32 KiB

And the server prints the same figure without being asked:

llama_kv_cache: size = 128.00 MiB (4096 cells, 16 layers, 1/1 seqs),
                K (f16): 64.00 MiB, V (f16): 64.00 MiB

128 MiB for 4,096 cells is 32 KiB per token exactly.

So: 32 KiB predicted, 33.0 KiB measured. The 1.6 KiB of daylight is compute buffers, which also scale with the window. Three routes — arithmetic, the server's own log, and a footprint sweep that knows nothing about either — landing on the same constant is worth more than any one of them.

The control that had to hold

The obvious way to run this sweep is smallest window first. On an 8 GB laptop with a browser open, that design hands every later arm a more contended machine than the one before it, and quietly books accumulated pressure as cache cost. This site has already paid for that mistake in the time domain: throughput falls 8.2% from burst to plateau on this machine with macOS reporting thermal pressure in 0 of 424 samples.

So the sweep runs ascending and descending, and the analyser recomputes every window's footprint from each direction separately.

Two more guards, both fatal to the run. An arm whose granted context_length differs from the window it asked for is measuring a different allocation than its label claims. An arm that starts with the model still resident is measuring the previous arm. Neither fired, but both are checked every time rather than assumed.

Where it breaks

At 131,072 tokens — the full window the model was trained for, and the one you would naively reach for — the footprint is 5.313 GB and only 4.070 GB of it is in VRAM. 1.243 GB has spilled out of the GPU budget.

That final interval also costs more per token: 45.4 KiB against the 33.0 that held all the way up to it. The allocation stops being uniform once it stops fitting.

The model loads. It answers. size_vram is the only place the machine admits that a fifth of the allocation is somewhere slower.

What I could not establish

The obvious next question is whether the spill makes generation slower. I have the data and it does not support a claim.

Spilled arms averaged 32.7 tokens per second against 36.4 for unspilled ones — a difference of 3.6. But the worst spread between the two sweep directions at a single window was 16.4 tokens per second. The noise is four times the effect.

INCONCLUSIVE: the difference is smaller than the run-to-run spread at a single window.
This design cannot detect a speed cost of the spill. Do not claim one.

That is the analyser refusing on my behalf, and it is in the repository so it refuses every time rather than only when I remember. A 48-token generation is too short to rate reliably on a machine this noisy; measuring the speed of the spill needs a design built for it, which this is not.

The part that actually costs you

Now put the two measurements together, because separately they are trivia and together they are a budget.

Every arm above was measured with the same tiny prompt — "Count from one to twenty" — and the footprint still tracked num_ctx exactly. That is the first thing to be clear about: the cache is charged on the window you declare, not the prompt you send. Configure 32,768 tokens and use 500 of them, and you have still bought 1.05 GB of cache.

So the window should be as small as it can be. Except that it cannot be too small, because of what the companion measurement found: a prompt is read in full right up to num_ctx, and the moment it exceeds num_ctx the server does not trim the overflow — it cuts the prompt to num_ctx/2 + 2 and says nothing. At an 8,192-token window that is 8,102 tokens accepted, and then 4,098.

That is the tension, and it is a genuinely awkward one:

• Undershoot the window and the penalty is not proportional. Forty words over the line cost 4,004 tokens, and the half that goes is the head, where the instructions live.
• Overshoot it and you pay 33.0 KiB for every token of headroom, used or not.

A gigabyte of an eight-gigabyte machine to hold a 32,000-token window open, on a model whose weights are 0.8 GB. The cache is not a rounding error beside the model. Past roughly 25,000 tokens of window it is the model, and then some — whether or not a single prompt ever fills it.

The practical shape: size the window to your realistic maximum prompt plus a margin you have actually measured, assert on prompt_eval_count, and treat num_ctx/2 + 2 in a response as an alarm rather than a number.

This also reframes the memory ceiling this site measured in August, which found that disk, not GPU memory, is what stops an 8 GB machine. That post held context constant and varied parameters. Vary the context instead and there is a second road to the same wall — one you can walk down without changing models at all, just by declaring a window you never fill.

Limits

One model, one serving stack, one machine. The 33 KiB constant belongs to this architecture's layer and head counts; a model with more layers or more KV heads has a different number, and the arithmetic in this post is how you compute yours rather than a figure to copy.

The footprints are what ollama reports through /api/ps, not an independent count of physical pages. Swap is recorded per arm but never attributed — vm.swapusage is machine-global and every other process moves it.

The harness, the analyser, the three guards and the raw eighteen arms are in the repository. Eight minutes and no credentials will price your own window.