SWE-Shift: Evaluating Coding Agents in Evolving Workspaces

We introduce SWE-Shift, a benchmark for evaluating coding agents under such mid-execution changes. It comes from a paper posted to arXiv on 26 October 2026. Repository-level coding benchmarks largely assume that tasks and environments remain unchanged throughout execution. In software development, code, dependencies, runtime state, and requirements can change while an agent is still working. From the same saved history and workspace, a Clean continuation proceeds without intervention, while a matched Shifted continuation receives a task-relevant update. Success requires completing the current task, preserving still-valid work, and satisfying the applicable state and evidence checks. The benchmark contains 120 scenarios across 12 change families, each evaluated at nine scheduled intervention positions. We evaluate 17 coding-agent systems, all of which have lower Shifted than Clean success. GPT-6 Astra drops from 67.35% to 24.79%, with 20.34% paired success. Tool/session changes and stale evidence yield zero paired success across all systems. Late-position paired success is lower for some systems and higher for others. These results show that success on the unchanged task can coexist with failures to preserve work, recover state, or refresh evidence after an update.

Why it counts

We introduce SWE-Shift, a benchmark for evaluating coding agents under such mid-execution changes. The benchmark contains 120 scenarios across 12 change families, each evaluated at nine scheduled intervention positions.

Sources

Primary source: primary source

What is not known

This brief does not claim independent replication. Claims that appear only on X and not in the primary source stay unknown.