Planning capacity around a training run
Training runs have a shape: a ramp, a long plateau and a tail. Capacity that fits that shape costs less than capacity that doesn’t.
Most teams we talk to don’t need a cluster forever. They need a specific amount of compute, for a specific run, starting on a specific date. The run has a shape, and the best capacity plan follows it.
The shape of a run
A run starts with experiments on a fraction of the cluster, ramps to the full machine for the main training, then tails off into evaluation and fine-tuning. Checkpoints set the rhythm: how often you save decides how much work you lose when a node fails.
Plan for failure
At scale, hardware failure isn’t an exception; it’s a rate. Plan spare nodes into the cluster, keep checkpoints fast, and agree in advance how failed parts are swapped. The cheapest hour of a run is the one you don’t have to repeat.