Where Cross-Question Structure in LLM Errors Comes From: Shared Seeds, Shared Prompts, and

Within one model, independently drawn answers show no detectable structure; it appears when all questions in one pass over the benchmark share a random seed or few-shot prompt. It comes from a paper posted to arXiv on 26 October 2026. Disagreement between language models is often read as missing knowledge. Questions that need the same missing knowledge should then be right or wrong together. We test for this structure with the eigenvalues of the questions' correlation matrix. Across 24 models, the largest structure beyond overall skill is a preference among answer letters: when the options are rotated, it follows the letter, not the question, and it depends on one model, Llama-3.2-1B; without it only a smaller excess remains. Counting a question correct only if right in every rotation removes the part that acts on guesses. What remains is a marginal excess at the 5% line, tied post hoc to two near-duplicate model pairs and to Llama-3.2-1B.

Why it counts

Within one model, independently drawn answers show no detectable structure; it appears when all questions in one pass over the benchmark share a random seed or few-shot prompt.

Sources

Primary source: primary source

What is not known

This brief does not claim independent replication. Claims that appear only on X and not in the primary source stay unknown.