Cost per million tokens is not one number

Every conversation I have about AI infrastructure runs into the same wall. One side describes the workload in tokens, context length, concurrency and a latency target. The other side pays for it in dollars, watts, rack units and a depreciation schedule. There is no checkable translation between those two languages, so the decision often comes down to whoever has the more confident slide.

That is a poor way to spend capital, and it is the gap I am trying to close. This note is the argument. The model that would test it does not exist yet.

The claim

Cost per unit of useful inference work is decided almost entirely by two things: how much accelerator memory the deployment needs, and how much utilization it can actually hold at its latency target. Peak throughput, the figure most comparisons lead with, is a poor predictor of either.

If that is right, the correct way to size an inference deployment is backwards. Start from the memory footprint and the latency target, derive the concurrency you can realistically sustain, and only then talk about hardware. Most cost arguments settle themselves once the binding constraint is named.

Why a single number cannot survive

Serving a request has two distinct phases with different bottlenecks. Processing the prompt is largely a compute problem; generating each subsequent token is largely a memory-bandwidth problem, because the model’s weights have to be read again for every token produced. A workload that is mostly long prompts and short answers has a different cost shape from one that is mostly short prompts and long answers, on identical hardware.

On top of that, at least six things move independently:

Model size and numeric precision. Weights have to fit in accelerator memory. Running at lower precision shrinks that footprint, at some cost in output quality that is workload-specific and rarely quantified in the same document as the speed claim.

Context length. The cache holding attention state for a request grows with the length of that request. Long contexts consume memory that would otherwise hold more concurrent requests.

Concurrency. Larger batches raise total throughput and raise the latency of any individual request. There is no setting that is best on both axes; there is only the setting that meets your latency target.

Which latency target. Time to the first token and time between subsequent tokens are separate promises with separate constraints. A system tuned to look fast on one can be poor at the other.

Duty cycle. A deployment sized for peak concurrency spends most of a normal week below it. Cost per token is total cost divided by tokens actually served, not by tokens the machine could theoretically have served.

What you are buying. On-premises cost is mostly capital, power and utilization. Cloud cost is mostly rate and commitment. They do not have the same shape, so a single comparison number is almost always dishonest without its assumptions written beside it.

Why memory tends to bind first

Accelerator memory holds two things at once: the model weights, which are fixed for a given model and precision, and the per-request attention cache, which is not. Whatever is left after the weights determines how many concurrent requests can be in flight, and concurrency is what turns an expensive accelerator into an affordable cost per request.

This is why memory is usually the first thing that binds. It sets the ceiling on concurrency, concurrency sets achievable utilization, and utilization is the denominator of the cost. A faster accelerator with less memory can easily be the more expensive machine per unit of work, which is not what a throughput comparison would tell you.

Why utilization, not peak throughput

Published throughput figures describe a machine running flat out at a batch size chosen to make the number large. Nobody serves production traffic that way, because the latency at that batch size is unacceptable.

The number that decides the bill is much less flattering: the utilization you can sustain while still meeting the latency you promised, averaged over a real week including the quiet hours. My working assumption is that the gap between the two is large enough to reorder most hardware comparisons, and that it is a systems and scheduling question at least as much as a hardware one.

What a model would have to capture

For this to be more than an opinion, it has to produce numbers somebody can argue with. That means four layers, each with its assumptions exposed as editable inputs rather than buried as constants:

A memory model, covering weights at a stated precision plus attention cache as a function of context length and concurrency, which decides what fits.

A utilization model, for what concurrency is achievable at the stated latency target, across a stated traffic pattern rather than at peak.

A power and footprint model, for watts per serving unit and what that implies for rack density and cooling.

A cost layer that keeps the on-premises and cloud shapes separate, and reports a range rather than a point.

If a reader disagrees with the utilization assumption, they should be able to change it and watch the answer move. That is the entire reason to build a model rather than publish a conclusion.

What would show this is wrong

A thesis that cannot fail is marketing. Three things would falsify this one, and I would rather name them now than defend the claim later:

If deployments that are memory-constrained and deployments that are compute-constrained turn out to have similar cost per token at the same latency target, then memory is not the variable I think it is.

If achievable utilization under a realistic traffic pattern turns out to sit close to the published peak, say within a modest margin, then peak throughput is a reasonable proxy after all, and the extra machinery buys nothing.

If the range this produces is so wide that it contains every candidate configuration, the model is not useful even if the reasoning is sound. A model that cannot distinguish between options has not helped anyone decide anything.

Where this actually is

This note is the whole of it. There is no model, no calculator, no benchmark and no result. I would rather publish the argument first and be corrected on it than publish a number that looks authoritative because it has a decimal point in it.

The next step is a fixed schema for describing a workload, then the memory and utilization layers, then back-testing against published deployment configurations where the real cost is known. If you size inference for a living and you disagree with a specific assumption above, that is the most useful thing that could happen to this piece.