Forty Words Over the Line, and the Server Threw Away Half My Prompt

At a context window of 8,192 tokens, this server accepted a 2,700-word prompt whole: 8,102 tokens, every one of them read.

I made the prompt forty words longer. It came back 4,098.

Not 8,192. Not a truncation of the overflow. Not an error — error: null, HTTP 200. Forty words past the line, and slightly more than half the prompt was gone.

The rule

Two things are true, and keeping them apart is the whole point of this post.

A prompt is accepted in full right up to num_ctx. Whatever the window is set to, the server will read a prompt of that size.

A prompt that exceeds num_ctx is cut to num_ctx/2 + 2. Not to num_ctx. Not to the last N tokens that fit. To half the window, plus two.

That is the shape of the cliff. You do not pay for the overflow. You pay half the window.

The target reproduces everywhere

The truncation target itself is not model-specific. One oversized prompt, held byte-identical, sent at four window sizes across three models:

Twelve cells, one offset. Two model families, two quantisations — so this is the serving layer, not a tokeniser. It also holds at 32,768 and 65,536, which accept 16,386 and 32,770.

Is anything actually lost?

A token count is not yet harm. A server that reported conservatively while reading the whole prompt would produce the same number.

The obvious test does not work. A 1.2B model failing to recall a nonce from 4,000 tokens of high-entropy filler may simply be a weak model, and comparing a truncated prompt against a short one confounds truncation with length.

So the control arm is length-matched: a prompt sized to land just under the truncation target, so both arms put roughly four thousand tokens in the window and differ only in whether the prompt was cut to get there.

Under 1% apart in occupancy. Same model, same seed. Unclipped it returns HEAD CODE: alpha101, TAIL CODE: omega101 three times from three. Truncated it returns mangled fragments and finds neither.

Length is excluded. Capability is excluded. What is left is the truncation.

Which half goes

Knowing something was lost does not say what. For that I stopped asking the model to remember and started asking it to obey: one instruction at the very head demanding the single word RED, one at the very tail demanding BLUE.

Obeying a one-word instruction is something a 1.2B model does reliably, which is the point — here, obeyed neither is evidence of absence rather than of weakness. Head-only and tail-only run separately, because with both present the model prefers the more recent, and that recency preference is exactly what would masquerade as truncation.

Unclipped, the head instruction is obeyed every time it is the only one present. Truncated, the identical instruction is obeyed never — while the tail instruction, in the same run, still works.

Long prompts are assembled the same way nearly everywhere: system prompt and constraints first, then retrieved documents or history, then the user's actual question last. Cross the line and what you lose is the framing — the persona, the output format, the do not do X — while the raw material and the question survive to be answered without any of it.

The request succeeds. The model answers, in a plausible register, because the question is still there. It has simply stopped being told how.

Not parallel slots

A factor of exactly two has an obvious suspect: the KV cache split across concurrent request slots. It is a good hypothesis and it is wrong here, and the server says so itself.

OLLAMA_NUM_PARALLEL:1
llama-server ... -c 8192 -np 1 --context-shift --keep 4
llama_context: n_ctx      = 8192
llama_context: n_ctx_seq  = 8192

One slot. The whole window allocated to one sequence. And an oversized prompt still lands on half of it. What remains visible from outside is --context-shift --keep 4 on the same launch line; I did not read the source, so I will not tell you which line does it.

What to do about it

The practical rule is simpler than the one I published in the first draft of this post, and stricter than it looks.

Set num_ctx at least as large as your largest prompt. Not twice as large — that was my error, and it costs real memory for no reason. Just large enough.

Then verify, because the failure is silent. prompt_eval_count comes back on every response. Compare it against your own token estimate, and treat num_ctx/2 + 2 as a fire alarm: it means you crossed the line and the top of your prompt is gone.

if (res.prompt_eval_count === numCtx / 2 + 2) {
  throw new Error('prompt exceeded num_ctx; the head was discarded')
}

And if you cannot raise the window far enough on the memory you have — on 8 GB, the window itself costs 33 KiB per token, and disk is the real ceiling long before the model is — then put the instructions at the end. It is the opposite of how nearly every prompt template is written, and on this stack it is the half that survives.

Limits

One serving stack, one version, one machine. Three models is enough to show the target is not model-specific and nowhere near enough to show it is universal across serving stacks — I did not test llama.cpp directly, or vLLM, or LM Studio.

Everything here is black-box. The run establishes what the server does and rules out one explanation for it; it does not identify the code responsible.

One thing is genuinely unmeasurable from outside: the true token length of a prompt that got truncated. Once the server cuts it, it reports only what survived. That is why the overflow is quantified in words — forty of them — rather than in tokens.

The harness, the analyser, the four arms and the raw runs are in the repository. Six minutes and no credentials will tell you whether your build still does this.