Data selection for pretraining, done so you can check it
The procedure behind every number on this site: samples, a verdict, a plain importance-weighting selector, a keep list computed where the data lives, and a two-arm test that produces a receipt.
Targeted data selection means training on the part of a large pool that most resembles a target domain. The idea is old and the selectors are published. What has been missing is the decision before it, and a way to check the result that a skeptic would accept. This is the procedure we use.
1. Samples, not corpora
Draw a seeded random sample of whole documents from the target and from the pool: about two million target words and five million pool words by default. Record the seed and the full corpus size. The corpus never leaves your machines; the sample is enough to decide.
2. The verdict
A sealed engine reads the two samples and answers in seconds, on CPU, without touching your model: Select, Do not select, or No forecast. It carries a confidence tier and a scope card that says whether your model size, keep fraction and language are inside the validated range. The verdict is timestamped before your first training step.
3. The keep list, computed where the pool lives
On Select, the response carries a scoring table: a plain published importance-weighting selector fitted on your samples, one weight per word-bigram bucket. Score every pool document by the mean weight over its bigrams and keep the top fraction. A terabyte pool is one pass on your own hardware and is never uploaded. The hashing is versioned and published with test vectors, because a scorer that drifts produces a keep list that is silently wrong.
4. The two-arm test
Train two models with the same architecture and budget: one on the curated keep list, one on a random keep list of the same size. Evaluate both on held-out target text every few hundred steps. Two numbers fall out: the best held-out loss each arm reached, and the step at which the curated arm first reached the baseline's best. The second is compute saved.
saved = 1 - (step at which the curated mean curve first reaches the baseline's best) / (step at which the baseline reached it) mean over seeds of held-out cross-entropy, evaluated every 200 steps, linear interpolation of the crossing
5. The receipt
Report the outcome under the forecast id: which arms you trained, the metric, both values. The service records whether the forecast agreed with the outcome. Over a program, that ledger is the calibration, and it is the only thing that lets anyone trust the next verdict.
What we have measured with this procedure
- Eight real text domains from scratch, four seeds per arm: compute saved from 35% to 63% where the call was Select, and a curated model that never caught up where the call was Do not select.
- Three domains rerun on a model eight times larger, same frozen calls, nothing retuned: the calls held.
- One image-and-text cell on a 256M-parameter model in continued training: 67% less compute, reproduced end to end through the public service at 65%.
- Nineteen of nineteen verdicts on the side we named. Fifteen sealed before their first training step.
- What is targeted data selection for pretraining?
- Training on the part of a large pool that most resembles a target domain, chosen by a selector fitted on samples of the target and the pool.
- How do you verify that data selection worked?
- Train two arms with the same model and budget, one on the curated data and one on a random sample of the same size, evaluate both on held-out target text, and compare the best loss and the step at which the curated arm reached the baseline's best.
- Does the pool have to be uploaded?
- No. The scoring table comes back with a Select verdict and the keep list is computed on your own hardware; only samples travel.