Soft Contamination Means Benchmarks Test Shallow Generalization

We call training on such items soft contamination; when the resulting gains extend to held-out items of the same benchmark (but not to related benchmarks), we call the result shallow generalization. It comes from a paper posted to arXiv on 10 November 2026. Training on benchmark test items is a classic way to overestimate generalization to unseen problems. Standard decontamination methods miss semantic duplicates (SemDs): training items equivalent in content to a test item, but distant in string space. We first embed Olmo 3's entire post-training dataset and search it for SemDs of benchmark items; we then finetune Olmo 3 and Qwen models on natural and synthetic semantic duplicates. We find (1) that soft contamination is widespread: the Olmo corpus contains SemDs of 76% of the CodeForces test set and 100% of MBPP's; (2) that finetuning on SemDs of a benchmark raises accuracy on its held-out items by up to 22pp in controlled experiments and 10-11pp at realistic contamination rates, though the effect varies across benchmarks; (3) that the gains from training on SemDs mostly do not transfer to conceptually similar benchmarks, i.e. we find shallow generalization; (4) that finetuning on items with similar embeddings that are not SemDs produces no such gain, indicating that the mechanism is not mere topical similarity; and (5) that at realistic contamination rates, soft contamination can raise held-out performance about as much as exact duplicates (or more). Benchmark gains here are therefore confounded - in a way that standard decontamination does not address.

Why it counts

We call training on such items soft contamination; when the resulting gains extend to held-out items of the same benchmark (but not to related benchmarks), we call the result shallow generalization. We find (1) that soft contamination is widespread: the Olmo corpus contains SemDs of 76% of the CodeForces test set and 100% of MBPP's; (2) that finetuning on SemDs of a benchmark raises accuracy on its held-out items by up to 22pp in controlled experiments and 10-11pp at realistic contamination rates, though the effect varies across benchmarks; (3) that the gains from training on SemDs mostly do not transfer to conceptually similar benchmarks, i.e. we find shallow generalization; (4) that Benchmark gains here are therefore confounded - in a way that standard decontamination does not address.

Sources

Primary source: primary source

What is not known

This brief does not claim independent replication. Claims that appear only on X and not in the primary source stay unknown.