AI workloads
Training, inference and retrieval use many of the same infrastructure components, but they stress them differently. I am building deeper expertise in how those differences affect compute, memory, storage, networking and cost.
What I work on
I currently work on the infrastructure side: how compute, storage and networking need to be sized for these workloads. The retrieval and agentic projects shown in Lab are things I am still building.
Accelerator count, and how they are connected to each other.
Read at high sustained throughput. Data has to arrive faster than the accelerators consume it.
Traffic between accelerators. Often where distributed training stops scaling.
Power and cooling density, frequently the first real ceiling.
Utilization. An idle accelerator is the most expensive thing in the room.
Three workloads
Switch between them to see where each one loads the system.
Training
Teaching a model from a large data set.
The accelerators are the expensive part, so most of the design question is keeping them fed with data and connected to each other.
The limit is usually outside the accelerator: whether data arrives fast enough, whether the network carries traffic between accelerators, and whether the rack can power and cool the density the design assumes. So the design has to be validated as a whole.
Inference
What happens when a trained model answers a request.
The questions change: memory, utilization, latency and cost per request, rather than raw scale.
Size backwards from the latency target and the memory footprint. Most of the rest of the configuration follows from those two. Peak throughput figures describe a machine nobody is actually running.
Retrieval
Looking something up in an external data set before the model answers.
It moves part of the problem off the accelerator and onto storage and data access, so retrieval latency becomes part of what the user experiences.
It binds on the data layer and the round trip rather than the accelerator. A system that generates quickly and looks up slowly is a slow system. This is the workload that most changes what a storage platform is being asked to do.
kushagragoyal.com/stack/ai-workloads