How to cut model training cost in half
Where training compute actually goes, why the data decision is the one lever that halves it, and what we measured: up to 63% less compute for the same held-out loss, with the receipts.
A training run is a fixed bill: GPUs, times hours, times the price per GPU-hour. Nothing you do after the run starts gives that money back. The only decisions that halve the bill are the ones made before the first step, and the largest of them is which data the model trains on.
Where the cost goes
Take a 90-day run at $2 per GPU-hour. On 1,000 GPUs that is $4.3M. On 10,000 it is $43M. On 100,000 it is $432M. Every 1% of compute on the largest run is worth about $4M, and every week is a week your competitors do not have to wait.
Why data is the lever
Most efficiency work happens inside the run: kernels, precision, parallelism, schedulers. Each is worth a few percent, and each is already in every serious stack. Data is different. A model trained on the right quarter of a pool reaches the quality of a model trained on a random quarter in a fraction of the steps. That is not a few percent. Measured on real corpora, it is between a third and two thirds of the run.
Compute saved is the training compute the curated run needed to reach the best held-out loss the random-data baseline ever reached, averaged over four seeds per arm, same model and budget on both sides. Every curve is on the benchmarks page.
The catch, and why it matters more than the gain
Curating toward a target does not always help. On one target in our battery the curated model never caught up with the random baseline at all. A team that curates blind on such a target pays full price for a worse model. So the decision is not "curate or not"; it is "know, before the run, whether curating toward this target will pay". That forecast is what the Echo Program issues: a verdict, timestamped before the first training step, and checked against what the run then does.
What to do
- Decide the data before the compute. Anything after the first step is sunk cost.
- Forecast the decision on the target and the pool you actually have, not on a rule of thumb from another corpus.
- When the forecast says curate, keep a quarter of the pool by a plain published selector and train on that.
- When it says do not curate, train on a random sample of the same size and bank the compute.
- Train both arms once, on a small model, and keep the receipt. That is how every number on this site was made.
The numbers above are measured on models up to 41.7M parameters and one 256M vision-language model. The projection to large clusters is arithmetic on those measurements, labelled as such, until a pilot confirms it on your own job. That pilot is free.
- How much does it cost to train a model?
- GPUs times hours times the price per GPU-hour. A 90-day run at $2 per GPU-hour costs about $4.3M on 1,000 GPUs, $43M on 10,000, and $432M on 100,000.
- What reduces training cost the most?
- Choosing the training data before the run. On real corpora, a model trained on the right quarter of a pool reached the random-data baseline's best quality with 35% to 63% less compute.
- Does data selection always reduce training cost?
- No. On some targets the curated model never catches up with random data. The Echo Program forecasts, before training, whether curating toward a given target will pay.
- How is compute saved measured?
- As the training compute the curated run needed to reach the best held-out loss the random-data baseline ever reached, averaged over four seeds per arm, same model and budget on both sides.