Benchmarks

saved per training run on a 100,000-GPU cluster

and about six weeks back on the calendar for every run

Every 1% of compute on a run this size is worth about $4M. Half of it is worth the race.

Projection: $2 per GPU-hour, a 90-day run, and our measured reduction of about half carried to this scale. Measured so far on models up to 41.7M parameters; a pilot on your own job verifies it on yours.

19 verdicts. Every one finished on the side we named. 15 were sealed and timestamped before their first training step.

More than 300 full training runs sit behind these lines, four seeds per arm. The loss column is the curated run's best held-out loss minus the random baseline's; below zero, curating helped.

Sealed: the call was fixed and timestamped before the first run, then checked against the outcome. Measured: the first four cells, where the rule was found and its boundary located; no prior call existed, so they are listed and not counted as sealed. Receipt: a single-seed run through the public service, kept as a check on the service, not as a battery result.

Same quality, about half the training

Training compute the curated run needed to reach the best quality the random-data baseline ever reached.

Training steps the curated run needs to reach the best held-out quality the random-data baseline ever achieved, averaged over four runs per domain. On the DON'T job the curated model never reaches the baseline at any amount of training, as the verdict said. A near-control domain (9 to 22% less) is left out.

At equal compute, the right call is worth up to a third

Quality gained by following the call, against training on a random sample of the same size.

Quality is perplexity on held-out target text, the standard accuracy measure for language models. On the DON'T job, curating would have cost 8%; the verdict kept the baseline. One near-control domain (2% better, GO) is left out. Every call was made before training.

Against the frontier

Two targets where our call was GO. The most advanced published curation methods, run at full strength, against our call followed by a plain published selector. Same model, same training budget, four seeds each.

Quality is perplexity on held-out target text against the random-data baseline at the same training budget. MATES re-probes the model during training; its runs took about 1,900 GPU-seconds against about 480 for every other arm on the same hardware, so its compute is roughly three times the others. Its influence predictor was verified as working in every round, so this is the method at its best, not a weakened copy. Forty runs, preregistered, every comparison 4 of 4 seeds with the interval excluding zero.

Eight domains of real text, from scratch

Every domain: four seeds per arm, six thousand steps, the same model and budget on both sides. The only difference is which quarter of the pool the model trained on.

Loss is cross-entropy on held-out target text, evaluated every 200 steps. Compute saved is where the curated run's mean curve first reaches the best loss the baseline's mean curve ever reached, against the step where the baseline reached it. The 95% interval is a 20,000-draw bootstrap over the four paired seeds. Every call was fixed and timestamped before the first run.

The same calls, on a model eight times larger

Three of the domains rerun on a model with eight times the non-embedding capacity. Same pool, same selections, same frozen calls. Nothing retuned.

41.7M parameters, four seeds per arm, six thousand steps. The medical gain grew, the near-control gain came out identical to the small model, and the DON'T stayed a DON'T.

Image and text together

A 256M-parameter vision-language model, continued on curated image-text pairs. The first completed target of a 48-run battery, and the same target run end to end through the public service.

Single seed so far; these are signs, not intervals. Through the service, the verdict was issued and sealed before training, the selection list came from the API, and the run happened on a different GPU stack and framework; its gap matched our own run of the same target to three decimals. The remaining 42 runs of the battery follow.