When Bigger Becomes Brittle: Why Compression Hurts More as Language Models Scale

Quantization and pruning reduce the cost of deploying language models, but their effects are usually reported as benchmark scores. It comes from a paper posted to arXiv on 6 November 2026. Such scores record only the loss that remains at the output. Inside the model, compression damages some components more than others, and the rest of the network can partly compensate. The observed loss therefore reflects both the damage and this compensation, which benchmark scores cannot separate. How the two combine, and whether their balance changes with model size, remains unclear. We study this compensation for in-context retrieval, a capability carried by a small, identifiable set of attention heads. To separate the two, the expected loss from ablating these heads is decomposed exactly into the part restored by the rest of the network, attributed to the components responsible, and the part that reaches the output. This decomposition is then compared across model sizes. For damage of comparable size, smaller models restore most of the loss, whereas larger models retain most of it, because the compensating responses offset less of it as models grow. Compressing the heads produces the same pattern. Compression that is nearly harmless in small models can therefore be costly in large ones.

Why it counts

Quantization and pruning reduce the cost of deploying language models, but their effects are usually reported as benchmark scores. The observed loss therefore reflects both the damage and this compensation, which benchmark scores cannot separate.

Sources

Primary source: primary source

What is not known

This brief does not claim independent replication. Claims that appear only on X and not in the primary source stay unknown.