InferSpend
Inference gets specified in tokens and latency, and paid for in dollars, watts and rack space. This is an attempt to put something honest between the two.
In development
Why it exists
One side of every AI infrastructure conversation describes the workload in tokens, context length, concurrency and a latency target. The other side pays for it in dollars, watts, rack units and a depreciation schedule. There is no checkable translation between them, so the decision often goes to whoever sounds more certain.
The bet I am testing
That cost per unit of useful work is decided by accelerator memory and by the utilization a deployment can actually hold at its latency target, rather than by peak throughput, which is what almost every comparison leads with.
If that holds, sizing should start from the memory footprint and the latency target and work backwards. The full argument, including what would show it to be wrong, is published as a written thesis.
What it would have to do
Take a workload shape and answer three questions with the working shown: what infrastructure envelope does this need, what does a million tokens cost on it, and how does that compare with the equivalent cloud position.
Every assumption stays an editable input rather than a constant buried in a formula. If a reader disagrees with the utilization assumption, they should be able to change it and watch the answer move.
How it would be judged
Not on whether it produces a number. On whether the number survives contact with someone who has actually deployed the workload. So: back-testing against published, reproducible deployment configurations, a stated range rather than a point estimate, and assumptions visible enough to be argued with.
Where it is
Three phases- Phase 1 · in progress
Thesis and schema
A written thesis, and a fixed input schema for describing a workload.
The thesis is published. The schema is not final.
Done when: Somebody who sizes inference for a living disagrees with a specific assumption rather than with the whole framing.
- Phase 2 · not started
The model
The memory, utilization, power and cost layers, with every assumption exposed as an editable input.
Not started, and deliberately gated behind phase 1.
Done when: A reader can change the utilization assumption and watch the answer move, without reading any code.
- Phase 3 · not started
Back-test and publish
Back-testing against published deployment configurations, then the method, the failure modes and the code.
Not started.
Done when: The model produces a range that contains the real cost for configurations it has never seen, and the write-up says plainly where it does not.
Next: Finish the input schema, then build the memory and utilization layers.
kushagragoyal.com/lab/inferspend