Four-Bit Quantisation Made This Laptop 58% Faster. The Eight-Bit Model Refused the Question.

"Use a quantised model" is the standard advice for running an LLM on hardware like mine, and it is almost always given without a number. This site has already measured what this particular machine does at one precision: where it becomes bandwidth-bound, and how much it loses between burst and plateau with no thermal pressure reported at all. Neither says what the quantisation choice itself is worth.

So I measured it. The answer to the question I asked is clean and slightly boring. The answer to a question I had not asked is the reason this post is longer than a table.

Two arms, one model

llama3.2:1b and llama3.2:1b-instruct-q4_K_M are the same 1.2B weights at two precisions, Q8_0 and Q4_K_M. That claim is the entire foundation of the comparison, so the harness re-checks it on every run rather than trusting an inspection I did once: architecture, parameter count, context length, embedding length, block count, and a SHA-1 of the chat template must all match across arms, and the run aborts if they do not.

The template check is the one that earns its place. Two models with different templates are being asked different questions however identical their weights, and nothing else in the output would reveal it.

Why the runs interleave

The obvious design is to measure one model for ten minutes and then the other. On this machine that design is wrong, and this site has already published why: throughput falls 8.2% from burst to plateau on a fanless M1, with macOS reporting thermal pressure in 0 of 424 samples. A sequential run hands the second model a hotter machine and books the difference as quantisation. Whichever arm ran second would lose.

So the pair is the unit, not the block. Within each repetition both arms answer a byte-identical prompt back to back, at the same thermal moment, and the within-pair order alternates by repetition index so that went first is not itself confounded with a model. Five pairs per cell, three prompt sizes, cold and warm cache states kept as separate conditions — 30 pairs, 60 generations.

The result, and where it lives

Generation runs 1.58x faster at four bits, averaged over all 30 pairs, and the sign test is unanimous: 30 of 30 pairs favour the smaller model. In absolute terms, across all 30 reps per arm, the median is 28.1 tokens per second at eight bits against 44.0 at four.

Prefill runs at 0.99x — no difference, and the sign test splits 6 of 15. Reading the prompt costs the same at either precision.

That split is not a surprise so much as a confirmation, and it is the reason the two phases are never averaged into a single "tokens per second" here. Generation is memory-bound: every token drags the whole weight set across the bus, so halving the bytes per weight is worth something close to proportional. Prefill is compute-bound: it processes the prompt in parallel and saturates arithmetic rather than bandwidth, so there is no bus pressure to relieve. An independent measurement on this site already found this machine bandwidth-bound at generation. This run reproduces that finding from a completely different direction, which is worth more than either result alone.

The memory side is the half that throughput hides: ollama holds 1.14 GB resident for Q4_K_M against 1.65 GB for Q8_0, a 31% reduction. On an 8 GB machine that is frequently the difference between a model running and not running at all.

The check that could have killed it

An interleaved design is a claim that interleaving worked. That claim is testable, so the analyser tests it every time: it recomputes the generation speedup separately for the pairs where Q8_0 ran first and the pairs where Q4_K_M ran first.

Q8_0 ran first     n=18   generation 1.59x
Q4_K_M ran first   n=12   generation 1.58x
gap 0.01x — order did not move the result; the pairing held.

Had those disagreed, the analyser would have printed a refusal to publish the headline rather than leaving me to notice. They agree to 0.01x, so the 1.58x is a quantisation effect and not a thermal one.

I would rather report the check that passed than let a reader assume I ran it.

The bug that discarded the entire dataset

The first version of the analyser reported 0 clean (30 excluded for a mid-pair model load) and a null speedup. A run that excludes every one of its own observations looks exactly like a run that failed, which is the only reason I caught it.

The filter tested load_duration_ns > 0, on the assumption that a resident model reports zero. It does not: ollama reports a non-zero load duration on every request as readiness bookkeeping. All 60 reps reported between 177 ms and 520 ms, so the filter excluded all of them.

The fix was not to pick a bigger number. It was to measure where the boundary actually is — force an unload with keep_alive: 0, then time the next generation:

llama3.2:1b                  cold load 1552 ms
llama3.2:1b-instruct-q4_K_M  cold load 1291 ms
paired run, all 60 reps      177–520 ms

Two populations that do not touch. The threshold now sits at 900 ms, in the empty band between them, and the comment in the analyser says where those numbers came from so the next person does not have to guess whether 900 was principled or convenient.

The thing I was not looking for

At temperature 0 with a fixed seed both models are deterministic, so I sent each pair's byte-identical prompt to both arms and diffed the replies. This measures how often the two disagree. It is not a quality measurement, and the distinction matters more than usual here.

All 30 prompts produced different text. Not most — all of them.

Then the reply heads showed why, and it was not what I expected:

On the 16-clause prompt, Q8_0 answered "There is no passage provided. The text you've shared appears to be a random string of characters" in all ten pairs. Q4_K_M summarised the passage in all ten. At 64 clauses the split was 7 and 0. Across the run: 17 refusals from the eight-bit model, zero from the four-bit one.

The reason for the care is that the tempting sentence — four-bit was more helpful — is the same class of error as the cache-state bug that once inverted a result on this site: a real number answering a question nobody asked. The divergence rate is real. The interpretation that four bits is better is not in the data.

Neither arm hit the 200-token cap, incidentally. Both stopped naturally between 34 and 83 tokens, so the rate comparison is not an artefact of one model being cut off mid-sentence.

What this is worth, and to whom

If you are running a small model on a memory-constrained machine, the four-bit weights are worth taking: roughly 1.6x the generation speed and a third less resident memory, for no measurable prefill penalty. If your workload is prompt-heavy and generation-light — classification, extraction, anything that reads a lot and writes a little — the headline number does not apply to you, because the phase it lives in is not the phase you are paying for.

And whatever the benchmark says, run your own prompts through both. Two builds of one model, at temperature 0, disagreed on every single prompt I sent them, and on more than half of them they disagreed about whether there was a question to answer.

Reproducing this

node scripts/bench/quantisation.mjs --reps 5 \
  --out data/quantisation/run-$(date -u +%Y%m%dT%H%M%S).json
node scripts/bench/analyse-quantisation.mjs data/quantisation/run-*.json

Needs pmset (macOS), a local ollama, and both model tags pulled. No API keys, no accounts, nothing paid. A full run takes about twelve minutes and the machine should be idle and cool before it starts — two earlier benchmark runs on this site were wasted by ignoring that.

Every number in this post recomputes from the committed run file with the committed scripts, including the order-effect check that would have withdrawn the headline.