Can data curation make a model worse?

Yes. On a small encyclopedia target the curated model finished 0.076 nats worse in held-out loss than the random-data model and never caught up, at two model sizes.

Do advanced data selection methods always beat random data?

No. In a preregistered head-to-head, MATES at full strength used about three times the compute and delivered models 28% to 37% worse than random data on two targets.

How do I know whether to curate toward a target?

Forecast it on your own target and pool before training. The Echo Program issues a sealed verdict from samples, in seconds, and checks it against the run's outcome.

Guide

2026-09-27

When curating training data backfires

The results nobody publishes: targets where selecting data toward the goal made the model worse, a frontier method that used three times the compute to lose to random data, and how to know before you train.

Data selection papers report the wins. The losses are the useful part. Here are ours, with the same measurement as the wins.

A target where curating never catches up

On a small encyclopedia target, the model trained on the curated quarter of the pool finished with a held-out loss 0.076 nats worse than the model trained on a random quarter, and at eight times the model size it was still 0.062 worse. Its training curve never crossed the baseline's best. Our call on that cell, sealed before training, was Do not select. A team curating by habit would have spent the full budget for a worse model.

A frontier method that lost to random data

In a preregistered head-to-head on two targets where our call was Select, we ran the most advanced published curation methods at full strength against our call followed by a plain published selector. Same model, same training budget, four seeds each.

Arm

Target A vs random

Target B vs random

Compute

MATES, full

Target A vs random 36.7% worse

Target B vs random 28.2% worse

Compute about 3×

BETR classifier

Target A vs random 4.1% better

Target B vs random 5.1% better

Compute 1×

Our call + plain selector

Target A vs random 8.5% better

Target B vs random 11.3% better

Compute 1×

MATES re-probes the model during training, which is where the compute goes; its influence predictor was verified as working in every round. It still delivered a model worse than random data. Every comparison held on four of four seeds with the interval excluding zero.

What makes a target unsafe to curate

The pattern across our battery is simple to state and expensive to guess: some targets carry a usable signal that selection can follow, and some do not, or carry so little that selection amplifies noise. Which one you have is a property of your target and your pool together. It is not visible from the domain name, and it does not transfer from someone else's corpus.

How to know before you train

Take a seeded random sample of whole documents from the target and from the pool. A few million words in all.
Get a verdict on that pair: Select, Do not select, or No forecast when the evidence is too close to call.
On Do not select, train on a random sample of the same size. That is the recommendation, not a fallback.
Record the forecast before the run and the outcome after. Nineteen of our nineteen verdicts finished on the side we named; fifteen were sealed and timestamped before their first training step.

Questions this answers

Can data curation make a model worse?: Yes. On a small encyclopedia target the curated model finished 0.076 nats worse in held-out loss than the random-data model and never caught up, at two model sizes.
Do advanced data selection methods always beat random data?: No. In a preregistered head-to-head, MATES at full strength used about three times the compute and delivered models 28% to 37% worse than random data on two targets.
How do I know whether to curate toward a target?: Forecast it on your own target and pool before training. The Echo Program issues a sealed verdict from samples, in seconds, and checks it against the run's outcome.