I Tried to Measure What One LLM Query Costs in Energy. The Instrument Lied Twice.

I wanted a number I could stand behind: the energy cost of a single LLM query, measured on hardware I own, from Kathmandu, with the script committed so anyone could recompute it. The throughput side of this machine I already have — 424 generations, a settled sustained rate, the whole dataset in the repo. Energy looked like the same exercise with a different sensor.

It isn't. I ran three sessions across two days and have no joules-per-token figure to show for any of them. What I have instead is a precise account of why the obvious method cannot work, which turns out to be the more useful thing to publish — partly because the method is the one most people reach for, and partly because it failed in a way that produced a confident-looking wrong answer rather than an error.

The only instrument available without root

macOS will tell you the machine's power draw through powermetrics, and powermetrics requires root. That is a real obstacle for a reproducible measurement: "run this as root" is a much bigger ask than "run this," and it reports SoC package power rather than what the whole machine pulls.

The unprivileged alternative is the battery itself. AppleSmartBattery exposes InstantAmperage and Voltage through ioreg, needs no privileges, and their product is the draw of the entire laptop — DRAM, SSD, display, regulator losses — which is closer to what a wall meter would read than package power is. It only works while actually discharging, because on AC the amperage is charge current flowing into the pack. So: unplug the laptop, sample the gauge, run generations, integrate.

That is the method. It has one fatal property that nothing in the API hints at.

Failure one: the gauge refreshes once a minute

The sampler polls at 2 Hz. Over a 21.3-minute discharging window it wrote 626 rows. Those 626 rows contain 21 distinct values, each held for 51 to 59 seconds — plus one tread lasting 119 seconds where a refresh simply never arrived.

The register updates about once a minute. Polling it faster does not produce more information; it produces more copies of the same information. A mean over 626 samples looks like it has 626 samples behind it, and reports a reassuringly tight spread, and is really one number repeated.

Now put a generation next to that. Each rep in the energy benchmark is a fixed 200 tokens at seed 42, temperature 0 — between 2.8 and 5.4 seconds depending on the model. The instrument is roughly fifteen times slower than the event it is supposed to measure.

That chart is the whole problem in one frame. Fifteen generations, six power readings. The five reps between minutes 1.99 and 2.99 all ran while the gauge read 2.79 W — against a measured idle floor of 5.79 W. Taken at face value, generating tokens on this machine costs less than leaving it alone.

Pooled across all inference windows the reading is 6.91 W against that 5.79 W floor: a 1.12 W effect, resting on five distinct gauge values, in a window where the same instrument also claims generation is cheaper than idling. And notice which model gets credited with the single 14.88 W tread — whichever one happened to be running when the register latched. That is alignment luck. It is not a measurement of anything.

Failure two: even a long window doesn't reconcile

The obvious fix is a longer window. Run for half an hour instead of five minutes, span twenty-odd refreshes instead of six, divide total energy by exact token count. The refresh rate stops mattering if the window is long enough.

So I ran that, and then checked it — because the same pack offers a second, independent measurement of the same energy. InstantAmperage × Voltage integrated over time is one answer. The pack's own charge counter, AppleRawCurrentCapacity, is another: it accumulates continuously in hardware, and the charge it lost across the window is the energy it delivered. Same battery, same minutes, two derivations. They should agree.

They do not. The integral says 5.22 Wh. The charge counter fell 209 mAh which at a mean pack voltage of 12.19 V is 2.55 Wh. A factor of 2.05×.

The pack cannot have delivered both. And the direction of the error matters: the integral, built from those 21 instantaneous readings, is the one with a mechanism for being wrong. If the gauge's once-a-minute refreshes land preferentially during load rather than in the gaps, the integral weights the peaks and overstates the mean. Coulomb counting has no such bias.

One trap worth naming, because it hides part of the gap. The check needs AppleRawMaxCapacity — the pack's present full-charge capacity, 3484 mAh on this battery at 514 cycles — and not DesignCapacity, which reads 4382 mAh. Using the design figure understates the charge drop by about 20% and pulls the two numbers closer together, which is precisely the direction that would have let me convince myself the check had passed.

I can show these two numbers cannot both be right. I cannot yet prove which one is wrong. Until that is settled, no joules-per-token figure from this instrument is publishable, however long the window.

Failure three: the run itself was two runs

The 30-minute run I did all of this on has a separate problem, and it is the one I am least comfortable about, because it was mine and not the hardware's.

The laptop was plugged into AC at minute 21.2. The harness recorded the power source with every one of its 403 samples and never once compared two consecutive values, so nothing flagged it. The file's power_at_start field says Battery Power, which is true, and which is exactly why it is dangerous: it is a true statement that certifies the wrong thing.

Read as a single run, it reports a 13.6% decline from first fifth to last, a trend of −0.379 tok/s per minute, and a t-statistic of −16.3. That is a decisive-looking thermal result. It is two experiments averaged together.

Nor can the segment means rescue it. Battery: 294 generations, mean 49.96 tok/s. AC: 109 generations, mean 44.22 tok/s. Tempting — until you notice the AC segment is also the later segment, so power source and time-under-load are perfectly confounded. The run establishes neither a thermal curve nor an AC-versus-battery difference. It establishes nothing at all.

The two bugs in my own analysis

The instrument failing is a fact about Apple's hardware. My code publishing a number anyway is a fact about me, and it is the part I would most want someone else to learn from.

I had already written a guard. summarise.mjs refuses to compute energy for a window shorter than five gauge refreshes, with a long comment explaining why. It worked — joules_per_token came back null for every rep, correctly.

Directly beneath it, joules_per_token_marginal was computed like this:

joules_per_token_marginal:
  watts != null && idleWatts != null
    ? (Math.max(0, watts - idleWatts) * seconds) / r.eval_count
    : null,

No window check. It tested only that the two inputs existed. So the marginal figure sailed straight past the guard sitting two lines above it and published values from 2.8-second windows that the same file had just declared unmeasurable.

And Math.max(0, …) is the worse half. When an inference window reads below the idle floor — the exact diagnostic tell that exposed the whole problem — the clamp converts it to 0 and emits it as a measurement. Two of three models came back at 0.00 J/token marginal. That does not read as a broken sensor. It reads as a clean result meaning inference is free.

Both are fixed. The guard is now a single measurable predicate applied to both figures, gated on the count of distinct gauge readings in the window rather than on its duration — because duration was always a proxy for evidence, and a poor one. A sub-idle reading is now a refusal carrying a stated reason, not a clamped zero. Every rep in the regenerated summaries says which condition it failed.

Two more guards exist now for the third failure: the sustained harness watches the power source per sample, records every transition, and warns as it happens, with --require-battery to abort outright; and the analyser recovers transitions from the per-sample field so that runs recorded before the fix — including the one above — get checked too.

What I would tell someone attempting this

Consumer battery telemetry cannot resolve a single inference. Not on this hardware. The gauge is an order of magnitude slower than the event, and no amount of oversampling changes that. If you need per-query energy on a Mac, you need powermetrics and therefore root, or an external meter on the charger.

Whatever you measure, reconcile it against something independent. The coulomb-counter check took an afternoon to write and invalidated two days of runs. Most published local-inference energy numbers I can find do not appear to have run any equivalent check — which does not make them wrong, but does mean nobody has shown they are right.

Watch for the zero that isn't a zero. A clamped negative and a genuine null effect are indistinguishable downstream. If your pipeline can emit "no cost detected," make sure it can also emit "did not resolve," and make sure they are different values.

The question I started with — what does one query actually cost? — is still open. The throughput and memory-bandwidth work on this machine stands, because exact token counts over nanosecond durations are a sound measurement and always were. The energy question needs a different instrument, and I would rather say that than publish 0.0522 J/token because an unguarded line of arithmetic was willing to produce it.