A 2.5× Bigger Model Costs Only 1.55× the Speed — I Measured Why on an 8GB M1

Two earlier posts on this site measured local LLMs on this same 8GB MacBook Air: one benchmarked tokens per second, and one found that macOS swap — not GPU memory — was what actually capped the model size the machine could hold. Both answered what. Neither answered why, and I've been unsatisfied with that for a couple of weeks.

So this time I designed the run as an experiment rather than a benchmark. Three models, five repetitions each, fifteen measured generations — but arranged so that each comparison moves exactly one variable. Qwen2.5 0.5B and Llama 3.2 1B, both at Q4_K_M, vary parameter count with quantization held still. Llama 3.2 1B at Q4_K_M and at Q8_0 vary quantization with parameter count held exactly still — same architecture, same 1,235,814,432 parameters, same weights, different precision.

The scripts, the raw run log and the per-sample telemetry are all committed in data/energy-per-token/ and scripts/bench/, along with a data dictionary and a list of the things this measurement cannot tell you. Everything below comes out of that dataset.

The methodology, plainly

Every generation used the same prompt, seed 42, temperature 0, and num_predict: 200 — so each model was asked for exactly the same amount of work, and produced exactly 200 tokens on all fifteen runs. Token counts are not estimated from output length: ollama reports eval_count and eval_duration directly, so throughput is an exact token count over a nanosecond-resolution generation time.

Each model got one warm-up generation that was discarded, because the first call after a model loads reads high. Between every repetition the machine sat idle for twelve seconds. Swap usage and free memory were sampled at every phase boundary, since on an 8 GB machine those are not incidental.

One thing I want to state before the numbers rather than after: this run has no energy figures in it. I built the harness to measure joules per token, which is the number I actually wanted. On Apple Silicon that requires powermetrics, which is root-only, or the battery gauge, which reports system draw only while discharging. The laptop was plugged in, and on AC the battery's InstantAmperage is the charging current flowing into the pack — not what the machine is consuming. The harness records the AC state alongside every sample and refuses to compute energy from a non-discharging window, so those fields came out null. That is the correct behaviour and I'd rather publish a null than a number that is silently measuring the wrong thing. The energy run is a re-run on battery, and it's the next post.

What I got instead turned out to be more interesting.

The throughput ladder

Qwen2.5 0.5B at Q4_K_M runs at a median 84.4 tokens/sec. Llama 3.2 1B at the same quantization runs at 54.6. The same Llama at Q8_0 runs at 37.9.

The middle comparison is the one worth pausing on. Going from 494 million parameters to 1.24 billion — 2.5× the parameters — cost only 1.55× the throughput. If inference were compute-bound in the way most people intuitively assume, 2.5× the arithmetic should have cost something much closer to 2.5× the time. It didn't, and that gap is the whole finding.

Why bigger models are cheaper than they look

A decoder-only transformer generating one token has to read essentially its entire weight set out of memory to do it. Not once per sentence — once per token. That makes the arithmetic almost incidental and the memory traffic dominant, which means the number that actually matters is bytes moved per second.

I can compute that directly, because I know both halves: the exact on-disk weight size and the exact measured throughput.

• Qwen2.5 0.5B Q4_K_M: 398 MB × 84.4 tok/s = 33.6 GB/s — 49% of peak
• Llama 3.2 1B Q4_K_M: 808 MB × 54.6 tok/s = 44.1 GB/s — 65% of peak
• Llama 3.2 1B Q8_0: 1,321 MB × 37.9 tok/s = 50.1 GB/s — 73% of peak

There's the explanation. The small model isn't fast because it's efficient — it's fast while leaving half the machine's memory bandwidth unused. At 494M parameters, per-token overhead that doesn't scale with model size (sampling, framework dispatch, kernel launch, Python-side bookkeeping) is a large enough share of each token's cost that the memory system never gets saturated. The larger model has more bytes to amortise that fixed overhead across, so it runs closer to the hardware's actual limit. Scaling looks sub-linear because the baseline was inefficient, not because the big model is doing something clever.

This also predicts where it stops. As models grow, saturation climbs toward 100% and the sub-linear discount disappears — throughput then falls roughly in proportion to weight bytes. On this machine that ceiling is close: the Q8_0 model is already at 73%.

I should be careful about the precision I'm claiming. That 68.25 GB/s is Apple's published specification for the M1, not something I measured on this laptop, so every percentage inherits whatever error the spec carries. And the traffic figure counts weights only — the KV cache and activations add real traffic I haven't accounted for, which means the true saturation is higher than these numbers and the headroom I'm describing is, if anything, smaller.

The quantization tax, with parameter count nailed down

This is the cleanest comparison in the dataset, because nothing varies except the number of bits per weight. Q4_K_M generates 1.44× faster than Q8_0 — 54.6 versus 37.9 tokens/sec — while its weights occupy 770 MB against 1,260 MB.

Note that the speedup (1.44×) is smaller than the size ratio (1.64×). That's the same story again from the other direction: the Q4 model, being smaller, sits further from the bandwidth ceiling and wastes more of the machine, so it doesn't convert its full byte-count advantage into speed.

The swap column is the practical one for anyone on 8 GB. During the Q8_0 runs the machine held 2,139 MB of swap against 1,795 MB for Q4_K_M — 344 MB more pressure on exactly the resource the earlier post on this site identified as the real ceiling on this hardware. Choosing Q4 over Q8 on an 8 GB machine buys throughput and headroom simultaneously.

What this actually means, practically

If you're running models locally on Apple Silicon, quantization is the highest-leverage knob you have — it moves throughput, disk, and memory pressure together, in the same direction, and Q4_K_M's quality cost on these small models is far smaller than a 1.44× speed difference suggests.

Don't assume a bigger model costs proportionally more. On bandwidth-limited hardware, stepping up from a very small model is often cheaper than the parameter ratio implies, because the small model wasn't using the machine anyway. The corollary matters more, though: that discount is borrowed from idle capacity, and it runs out. Once you're near saturation — which on an M1 happens around the 1–2B mark at 8-bit — scaling goes back to being linear and unforgiving.

And the bottleneck is not the one most people name. Local inference on this class of hardware is not compute-starved. It is memory-bandwidth-starved, which is why Apple's unified memory architecture punches above its weight for this workload, and why the M1's successors improved on it primarily by widening the memory bus rather than by adding arithmetic units.

What I still owe this measurement

The energy numbers. Joules per token is the figure that connects a laptop generating text to the argument this site actually cares about — the physical footprint of AI compute — and I don't have it yet, because I ran this plugged in. The harness is built, the null-handling is correct, and the re-run on battery gives whole-system draw including DRAM, SSD and display, which is closer to a wall-plug reading than the package-power figure powermetrics would have given me anyway.

Until then, this post claims exactly what it measured: three models, fifteen runs, one laptop, and a bottleneck that turned out to be made of memory bandwidth rather than arithmetic.

Reproducing this

Everything is in the repository. scripts/bench/energy-per-token.mjs drives the run, scripts/bench/sample-battery.mjs samples power, scripts/bench/summarise.mjs produces the derived table, and data/energy-per-token/README.md documents every field and every limitation. The raw phase log and the 577 power samples from this specific run are committed alongside them, so the numbers above can be recomputed rather than taken on trust.