LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving

A prevailing assumption is that successive VLM generations will continually improve driving performance and eventually outperform state-of-the-art methods. It comes from a paper posted to arXiv on 4 September 2026. Rapid advances in vision-language models (VLMs) have generated growing interest in their application to autonomous driving. To systematically examine this assumption, we introduce LightEMMA, a longitudinal framework for evaluating the autonomous driving performance of VLMs. LightEMMA uses a lightweight, unified evaluation protocol that assesses each model's intrinsic driving capability without model-specific fine-tuning, architectural changes, or prompt engineering. Using this protocol, we evaluate 15 models from five major families on the challenging nuScenes prediction benchmark. Empirical findings show that, despite increased model scale and enhanced general reasoning capabilities, successive VLM generations do not consistently achieve better driving performance. Further analysis of driving scenarios reveals recurring failure modes, including overreliance on historical actions and difficulty reconciling conflicting visual cues. These findings highlight the need for domain-specific adaptation to improve the safety of VLM-based autonomous driving systems. The source code is available at https://github.com/michigan-traffic-lab/LightEMMA.

Why it counts

A prevailing assumption is that successive VLM generations will continually improve driving performance and eventually outperform state-of-the-art methods. These findings highlight the need for domain-specific adaptation to improve the safety of VLM-based autonomous driving systems.

Sources

Primary source: primary source

What is not known

This brief does not claim independent replication. Claims that appear only on X and not in the primary source stay unknown.