Loading
AI Infrastructure2026-07-27· 11 min read
Running a local LLM,
measured on real hardware.
I ran these models myself, on the 8GB Apple M1 I actually work on. Here's what a small model costs you in speed, and exactly where an 8GB machine stops being enough.
Qwen2.5 0.5B
0 tok/s
Measured on this M1, 8GB
Llama 3.2 1B
0 tok/s
Measured on this M1, 8GB
Llama 3 8B
0 tok/s
Same 8GB tier, cited (~9.7)
RTX 4090
0 tok/s
Same 8B model, cited
← Back to Global Tech SearchNo cloud GPU, no rented box.
I didn't run an 8B model myself.
Match the model to the machine,
What I Actually Ran, And On What
No cloud GPU, no rented box.
Just the laptop I write on.
Every “run a local LLM” post I'd read before writing this one assumed a 24GB+ GPU or at least a 32GB unified-memory Mac. I run this site's tooling from an 8GB Apple M1 — a genuinely small, ordinary machine — so I benchmarked on exactly that instead of a borrowed high-end rig. I installed Ollama locally, pulled two small quantized models, and hit Ollama's own generation API directly, which returns precise eval token counts and eval durations in nanoseconds — no manual stopwatch, no guessing.
Methodology: same prompt (“Explain in detail how a hash table resolves collisions, with an example”), num_predict: 200, temperature 0 for determinism, three runs per model, generation throughput computed as eval_count / eval_duration. Both models reported “100% GPU” in ollama ps, meaning inference ran on the M1's integrated Metal GPU, not the CPU cores.
Measured Generation Throughput on This Machine
Apple M1, 8GB unified memory — Ollama, 3 runs/model, num_predict=200, temp=0
Qwen2.5's 0.5B model held a steady ~77.5 tokens/second across all three runs — fast enough to feel conversational, not just usable. Llama 3.2's 1B model, roughly 2.6x the parameter count, ran at ~38 tokens/second— almost exactly half. That's the first honest data point: on this hardware, throughput fell roughly in proportion to model size, not worse — small models are not a compromise, they're simply fast.
Where This Hardware Tier Hits A Wall
I didn't run an 8B model myself.
Here's why, honestly.
I stopped at 1B parameters on this machine deliberately — an 8B model at Q4 quantization needs roughly 5-6 GB just to load the weights, leaving almost nothing for the OS, the browser, and everything else an 8GB machine is also doing. Rather than force a swap-thrashing run and report a misleading number, I'm citing a community benchmark instead: XiongjieDai's GPU-Benchmarks-on-LLM-Inference, a widely-cited open community dataset, measured an M1 with a 7-core GPU and 8GB of memory — the same tier as this machine — at just 9.72 tokens/second on Llama 3 8B (Q4_K_M). That's not a small drop; it's an 8x fall for roughly 8x more parameters, and it lines up with what unified-memory bandwidth limits predict once a model stops comfortably fitting in fast memory.
Llama 3 8B (Q4_K_M), Same Model, Different Hardware
Community-benchmarked, cited — GPU-Benchmarks-on-LLM-Inference (GitHub) & Puget Systems
The pattern across that chart is straightforward: more unified memory bandwidth (M1 Max's 34.49 tok/s vs. the base M1's 9.72) buys real headroom on Apple Silicon, but a dedicated GPU with its own high-bandwidth VRAM — an RTX 4090 or A100 — still leads by roughly 3-15x at the same model size. If your workload genuinely needs an 8B-class model at conversational speed, this machine tier isn't it; if it needs something in the 0.5-1B range, this machine tier is already fast.
What This Means For Picking A Model
Match the model to the machine,
not the machine to the model.
The instinct when trying local AI is to reach for the biggest model your disk can hold. The numbers above argue for the opposite: on an 8GB machine, a well-chosen sub-1B or 1B model at Q4 quantization stays genuinely responsive, while an 8B model on the same machine is technically possible but not pleasant to use. If your task is classification, structured extraction, or drafting short text — the kind of thing I cover in my Next.js/Prisma agentic extraction pipeline post— a fast small model beats a slow large one almost every time in practice, because most of these tasks don't need frontier reasoning, they need a response before the person waiting has moved on. For anything that genuinely needs 8B+ class reasoning at real speed, the honest answer on this hardware tier is a hosted API — which is its own cost question I cover in Cloud 3.0: The AI Inference Economics Reckoning.
I'm publishing the exact prompt, parameters, and raw eval numbers above rather than a polished-looking single “tokens per second” headline, because the point of running this myself was to show real, reproducible numbers on ordinary hardware — not to make local AI look either better or worse than it is. Run the same prompt on your own machine and the numbers should land close to these, memory bandwidth being the dominant variable either way.
Written by Abhishek Kushwaha
Founder and writer at Global Tech Search, based in Kathmandu, Nepal. Covers AI, infrastructure, markets, and climate with sourced data and original analysis. More about the author →
Recommended Tools
Affiliate disclosure — we may earn a commission at no cost to you.
Sources
- Measured directly by the author, July 2026: Ollama v0.32.4, Qwen2.5 0.5B and Llama 3.2 1B (both Q4), Apple M1 / 8GB unified memory, 3 runs each via the Ollama generate API
- XiongjieDai, GPU-Benchmarks-on-LLM-Inference (GitHub) — community-maintained Apple Silicon vs. NVIDIA GPU inference benchmarks, Llama 3 8B Q4_K_M figures
- Puget Systems, LLM Inference — Consumer GPU Performance — RTX 4090 / A100 reference figures
The 0.5B and 1B figures in this post are first-hand measurements taken on the author's own machine. The 8B-and-above figures are clearly cited third-party community benchmarks, not re-measured here, because this machine does not have the memory headroom to run them responsibly.
Global Tech Search
Real numbers, on real hardware.
More independent, measured analysis of the infrastructure decisions that actually matter.
Back to Dashboard →