Sanity Checking Causal Representation Learning on a Simple Real-World System

Our efforts highlight the contrast between the theoretical promise of the state of the art and the challenges in its application. It comes from a paper posted to arXiv on 16 September 2026. We evaluate methods for causal representation learning (CRL) on a simple, real-world system that satisfies the basic problem setup of CRL. The system consists of a controlled optical experiment producing a variety of measurements where the underlying causal factors---the control inputs to the experiment---are known, providing a ground truth. We select methods representative of different approaches to CRL and find that they all fail to consistently recover the underlying causal factors. To understand the failure modes of the evaluated algorithms, we perform an ablation on the data by substituting the real data-generating process with a simpler synthetic equivalent. The results reveal a reproducibility problem, as most methods already fail on this synthetic ablation despite its simple data-generating process. Additionally, we observe that common assumptions on the mixing function are crucial for the performance of some of the methods but do not hold in the real data. We hope the benchmark serves as a simple, real-world sanity check to further develop and validate methodol- ogy, bridging the gap towards CRL methods that work in practice.

Why it counts

Our efforts highlight the contrast between the theoretical promise of the state of the art and the challenges in its application.

Sources

Primary source: primary source

What is not known

This brief does not claim independent replication. Claims that appear only on X and not in the primary source stay unknown.