The Network Was Never the Problem — Measuring Local Inference Against the Cloud From Kathmandu

This site has published two measurements that never spoke to each other. One walked the network distance from Kathmandu to the major cloud regions. The other three ran a local model on a fanless MacBook Air until it stopped speeding up. Separately they say the cloud is far and the M1 is slow, and neither answers the question anyone here actually has, which is whether the far-but-fast thing is worth the trip.

I assumed the answer was about distance. It is not. It is about whether the model has read your prompt before.

Measuring the toll without paying it

The cloud side needs a number for what a request costs before inference begins. I have no API budget for benchmarking, and inventing a figure for how fast someone else's model generates is exactly the thing this site exists not to do.

So I measured the rejection. An HTTPS request to a real inference endpoint with no credentials attached comes back 401 or 403. The timing of that response contains DNS, TCP, TLS and one round trip, and provably no inference whatsoever — there is nothing for the model to do. Seven reps per target, each a separate curl process so no connection is reused.

Read the split, not the totals. The lower band is the handshake — the part that completes at whatever CDN edge is nearest. It is fast: 22 ms to Anthropic. The upper band is everything after it.

That upper band is the point. The earlier latency post measured handshakes and flagged, correctly, that a quick handshake to a CDN-fronted API tells you the edge is near and says nothing about where the model is. It could flag the problem but not size it. This sizes it: 294 of Anthropic's 315 ms happens after the handshake completes. The near edge terminated my TLS in 22 ms and then the answer came from somewhere thirteen times further away.

The AWS Mumbai control inverts exactly. It is not CDN-fronted, so the handshake is the expensive part (100 ms of a 152 ms total) and the response comes straight back in 43 ms. That is what a request looks like when the thing answering is the thing you connected to.

The bug, which had it backwards

The local side is where I got it wrong, and the way I got it wrong is worth more than the result.

The first harness sent one fixed prompt three times per size and took the median — the same shape as every other benchmark on this site. It produced a clean, plausible table: a 7,201-token prompt answered in 1.8 seconds, prefill costing 31 ms. I nearly wrote the post from it. The conclusion would have been that a five-year-old laptop beats a round trip to the cloud on any prompt size, which is a good headline and is false.

What tripped it was a derived quantity that had no business existing. The harness records prefill throughput as a sanity field, and 7,201 tokens in 31 ms is 232,000 tokens per second. No consumer silicon does that. Nothing was reading those tokens at all.

ollama reuses the KV cache when a prompt shares a prefix with the last one. Rep 1 paid real prefill; reps 2 and 3 read the cache and paid nothing. The median across the three therefore reported the cached case:

prompt_eval_count=7201  rep0 prefill=9458.6ms  wall=11303ms   <- real
prompt_eval_count=7201  rep1 prefill=  31.2ms  wall= 1838ms   <- cache hit
prompt_eval_count=7201  rep2 prefill=  30.1ms  wall= 1841ms   <- cache hit
                        median  =    31.2ms                   <- what got reported

A nine-and-a-half-second error, in the direction that confirmed what I expected. The run completed, the reps were tight, the trend sloped the right way. Nothing looked wrong.

The fix is not to discard the cached reps. Both numbers are real, and they describe two situations a person is actually in. So cache state became an experimental condition rather than an accident of rep order:

• Cold — every rep gets a distinct nonce prefixed to the prompt, so no rep can reuse another's prefix. This is a one-off question: paste a document, ask about it.
• Warm — one identical prompt, sent once to populate the cache and then measured. The populating send is dropped. This is a conversation continuing over context already sent.

The nonce goes at the front because prefix matching runs from the start. A suffix nonce leaves the expensive part of the prefix cached and would have reproduced the bug while looking like a fix — I tried that first.

And the labels now check themselves. Cold rows prefill at 590–780 tok/s, which is this machine's real rate. Warm rows report 2,600 to 234,000 tok/s, which is not a speed, it is a cache being read. The analyser refuses to print anything if a row labelled cold ever reports above 2,000 tok/s.

What it costs the laptop

The warm series is nearly flat: 1.06 s at 61 prompt tokens, 1.84 s at 7,201. The cold series is not a series so much as a cliff — 1.27 s, 1.72 s, 3.68 s, 13.83 s.

The dashed line is the largest transport floor I measured, 476 ms, drawn to the same scale. Against the cold bars it is almost invisible. That is the finding, and it is the opposite of what I set out to test: from a city that is genuinely far from every cloud region, the network is not what makes local inference attractive or unattractive. It is a rounding error next to what the laptop does with a prompt it has not seen.

Generation barely moves: 970 ms at 72 prompt tokens, 1,641 ms at 7,212. Output length was held fixed, so it should not move much, and the rise is the cost of attending over a longer context. Prefill goes from 103 ms to 12,202 ms — 88% of the cold wall time at the largest prompt is the machine reading, before it writes anything.

The bar the cloud has to clear

I cannot tell you whether the cloud wins, because I did not measure cloud inference and will not guess at it. What the two measured quantities support is a threshold. A provider starting F ms behind must deliver the same 200 tokens in under local − F to be worth the trip:

Against a warm laptop the bar is high everywhere — 250 to 270 tok/s across most of the range, which is demanding for a frontier model streaming to Kathmandu. Against a cold one at 7,200 tokens the bar is 15 tokens per second, which is not a bar at all.

So the crossover is not a prompt size and it is not a distance. It is a state:

• Long context you are still working in — the laptop is hard to beat, and 315 ms of network is the least of the cloud's problems.
• Long context the model has never seen — the laptop spends twelve seconds reading before it starts, and any real provider clears 15 tok/s without noticing.

Which means the practical advice from a slow-network city is the opposite of the intuitive one. Do not choose local because the network is far. Choose local when you are staying in one context long enough to amortise the read.

What this does not say

It does not say a 1B model is a substitute for a frontier one. llama3.2:1b-q4_K_M is 807 MB and the comparison here is quality-blind — 1.1 seconds of a small model and 1.1 seconds of a large one are not the same product, and no threshold above should be read as a recommendation to use the small one for work that needs the large one. Every number is one machine, one residential ISP in Kathmandu, one morning. A 7B model would move every local figure and none of the network ones, which would push the crossover further toward the cloud in both states.

And the cloud column is a floor, not a cost. Nothing here measures a successful API call.

The three runs taken before the cold/warm split are not in that directory. They were the ones the cache had quietly contaminated, and publishing them beside the corrected ones would have put two incompatible definitions of "local wall time" in the same folder under the same field name. They are described in the README and nowhere else, which is the right amount of visibility for a measurement that was wrong.