WorkflowPerturb: Calibrated Stress Tests for Evaluating Multi-Agent Workflow Metrics

We introduce WorkflowPerturb, a controlled benchmark that applies realistic, graded perturbations to golden workflows. It comes from a paper posted to arXiv on 14 September 2026. Multi-agent LLM systems that generate structured workflows from natural-language requests are now deployed in production across cloud automation, DevOps, and enterprise orchestration. Operating them exposes a recurring change-management problem. Routine updates, such as re-running an input, swapping the LLM, or refactoring an agent's prompt or orchestration code, often produce workflows that differ substantially from validated references. Engineers then lack a principled way to decide whether a change is safe to ship. Automatic workflow evaluation is the natural tool, but in practice metric scores are poorly calibrated, and a numeric change rarely communicates the severity of the underlying degradation. It contains 4,973 golden workflows and 44,757 perturbed variants across three perturbation types (Missing Steps, Compressed Steps, Description Changes) at severities of 10%, 30%, and 50%. We benchmark multiple metric families, analyzing their sensitivity and calibration using expected score trajectories and instance-level alert rates. Our results characterize systematic differences across families and support severity-aware interpretation of workflow evaluation scores in change-management settings. WorkflowPerturb is publicly available at https://huggingface.co/datasets/microsoft/WorkflowPerturb .

Why it counts

We introduce WorkflowPerturb, a controlled benchmark that applies realistic, graded perturbations to golden workflows. We benchmark multiple metric families, analyzing their sensitivity and calibration using expected score trajectories and instance-level alert rates.

Sources

Primary source: primary source

What is not known

This brief does not claim independent replication. Claims that appear only on X and not in the primary source stay unknown.